Research log Small Model Experimentation
GitHub

Qwen Structural Compiler Attribution Ablation

Finished2026-06-24imported · line YBenchmark GeneralizationInterpretability and DiagnosticsStructured Execution and CompilersGitHub ↗
Graded practice, not clever expansion, builds long programs

The one idea you need

The model turns each problem into a 24-step recipe of clock-style arithmetic — numbers wrapping like hours on a dial — which a built-in calculator then runs. Like a pianist handed the whole score but drilled on easy passages first, it needs graded practice, not special wiring.

The question

What makes a small model write long, 24-step arithmetic programs: practicing easy-to-hard, or a trick that grows short programs by copying earlier steps into new ones?

What we found

It's the practice schedule. Give the model its full size from the start, then feed examples easy-to-hard — 8 steps, then 16, then 24 — and it solves nearly every standard 24-step program (about 97%). The popular guess, growing short programs by copying earlier steps into the new ones, managed under 2%. And training only up to 16 steps, then testing at 24, got exactly zero.

Why it matters

To teach a small model a hard multi-step task, invest in an easy-to-hard training schedule at full capacity, not tricks that stitch long programs from copied pieces — and never assume it handles lengths beyond what you trained on.

Winning recipe on the hardest test96.9%solved on standard 24-step programs, using easy-to-hard practice at full size
Copy-and-grow recipe on the same test1.6%the popular explanation, refuted here
Trained only to 16 steps, tested at 240%no ability to stretch past the trained length
Getting matched twin problems both right78.1% vs 0%winning recipe versus every other recipe at 24 steps
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Question
    2. Method
    3. Runs
    4. Results
    5. Figures
    6. Interpretation
    7. Artifacts
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 7

How each training approach does on the hardest 24-step programs

How to read

Each cluster is one training approach; the three bars show 24-step accuracy for standard wording, unfamiliar wording, and matched twin-prompt pairs. Taller bars are better.

0%25%50%75%100%max-24 + curriculummax-24 + curriculum96.9%46.9%89.1%max-24 no curriculummax-24 no curriculum4.7%18.8%9.4%expand (copy slots)expand (copy slots)1.6%51.6%29.7%expand (random slots)expand (random slots)1.6%6.2%14.1%train to 16, eval 24train to 16, eval 240%1.6%1.6%

Takeaway → The full-size easy-to-hard approach towers over all others on standard wording and twin-prompts; copy-and-grow only shows up on unfamiliar wording.

Data table
training armstandard wordingheld-out wordingpaired consistency
max-24 + curriculum96.9%46.9%89.1%
max-24 no curriculum4.7%18.8%9.4%
expand (copy slots)1.6%51.6%29.7%
expand (random slots)1.6%6.2%14.1%
train to 16, eval 240%1.6%1.6%

Numbers from experiments/qwen_structural_compiler_attribution_ablation/reports/structural_compiler_attribution_ablation_report.md (attribution summary table)

Technical framing

Final length-24 accuracy by training arm — The max-24 curriculum compiler dominates on standard and paired splits; copied structural expansion only helps on held-out wording.

Accuracy stays high until programs reach full length

How to read

The horizontal axis is program length from 8 to 24 steps; each line is one training approach; higher is better.

0%50%100%150%101520

Takeaway → Every approach stays high (81% or better) at 8 and 16 steps, then all but the full-size easy-to-hard approach collapse toward zero at 24 steps.

Data table
program lengthmax-24 + curriculummax-24 no curriculumexpand (copy slots)expand (random slots)train to 16, eval 24
8100%100%100%98.4%100%
16100%100%100%93.8%81.2%
2496.9%4.7%1.6%1.6%0%

Numbers from experiments/qwen_structural_compiler_attribution_ablation/reports/structural_compiler_attribution_ablation_report.md (final-stage standard_L8/L16/L24 rows) and reports/aggregate_metrics.csv (train_to16_eval24 L8/L16 rows)

Technical framing

Accuracy vs program length at the final checkpoint (standard split) — Every arm scores 81%+ at lengths 8 and 16; only the curriculum-trained max-24 compiler carries accuracy to length 24.

The winner's 24-step skill appears late in training

How to read

The horizontal axis is training progress; the lines track the winning approach's 24-step accuracy for standard wording, unfamiliar wording, and twin-prompt pairs; higher is better.

0%50%100%150%400600standard wordingpaired consistencyheld-out wording

Takeaway → All three lines stay low, then jump sharply in the final stretch — the long-program skill emerges suddenly, not gradually.

Data table
training stepstandard wordingheld-out wordingpaired consistency
3004.7%0%1.6%
45034.4%1.6%18.8%
75096.9%46.9%89.1%

Numbers from experiments/qwen_structural_compiler_attribution_ablation/reports/structural_compiler_attribution_ablation_report.md (per-stage table, max24_curriculum L24 rows at steps 300/450/750)

Technical framing

Winning arm's length-24 accuracy across curriculum stages — Length-24 skill in the max-24 curriculum arm emerges late, jumping sharply in the final curriculum stage.

Reworded prompts flip which approaches look good

How to read

Each cluster is a training approach; two bars compare 24-step accuracy on standard wording versus reworded-but-familiar prompts; higher is better.

0%25%50%75%100%max24_curriculummax24_curriculum96.9%81.2%max24_no_curriculummax24_no_curriculum4.7%15.6%expand_copyexpand_copy1.6%75%expand_random_new_slotsexpand_random_new_slots1.6%29.7%train_to16_eval24train_to16_eval240%0%

Takeaway → The copy-and-grow approach scores 75% on reworded prompts but under 2% on standard ones — wording, not just the approach, drives failures.

Data table
Training armStandard templates (L24)Paraphrase templates (L24)
max24_curriculum96.9%81.2%
max24_no_curriculum4.7%15.6%
expand_copy1.6%75%
expand_random_new_slots1.6%29.7%
train_to16_eval240%0%

Numbers from experiments/qwen_structural_compiler_attribution_ablation/reports/structural_compiler_attribution_ablation_report.md (final length-24 splits table)

Technical framing

Length-24 accuracy: standard wording vs seen-family paraphrases — expand_copy inverts: 75% on paraphrases but 1.6% on standard wording, so template wording, not just arm, drives length-24 failures.

Only the winning approach answers twin prompts consistently

How to read

Each cluster is a training approach; the bars show how often it gets both halves of a matched prompt pair right at 24 steps, for familiar and unfamiliar wording; higher is better.

0%20%40%60%80%max24_curriculummax24_curriculum78.1%56.2%max24_no_curriculummax24_no_curriculum0%0%expand_copyexpand_copy0%0%expand_random_new_slotsexpand_random_new_slots0%3.1%train_to16_eval24train_to16_eval240%0%

Takeaway → Every approach except the winner drops to essentially zero — none of the others can reliably nail both twin prompts.

Data table
Training armPaired split (L24)Paired held-out split (L24)
max24_curriculum78.1%56.2%
max24_no_curriculum0%0%
expand_copy0%0%
expand_random_new_slots0%3.1%
train_to16_eval240%0%

Numbers from experiments/qwen_structural_compiler_attribution_ablation/reports/structural_compiler_attribution_ablation_report.md (final length-24 splits table, executor_pair_both_correct)

Technical framing

Paired-prompt consistency at length 24: only the curriculum arm holds up — Pairwise both-correct rates collapse to ~0% for every arm except the max-24 curriculum winner (78.1% / 56.2%).

Failing approaches trace most steps but botch the final answer

How to read

Each cluster is a training approach; one bar is final-answer accuracy, the other is the share of intermediate steps traced correctly, on standard 24-step programs; higher is better.

0%25%50%75%100%max24_curriculummax24_curriculum96.9%99.7%max24_no_curriculummax24_no_curriculum4.7%82.2%expand_copyexpand_copy1.6%85.9%expand_random_new_slotsexpand_random_new_slots1.6%67.9%train_to16_eval24train_to16_eval240%62%

Takeaway → Losing approaches keep 62-86% of intermediate steps right while final answers sink near zero — errors cluster in the last few steps.

Data table
Training armFinal-answer executor accuracyCorrect state-prefix fraction
max24_curriculum96.9%99.7%
max24_no_curriculum4.7%82.2%
expand_copy1.6%85.9%
expand_random_new_slots1.6%67.9%
train_to16_eval240%62%

Numbers from experiments/qwen_structural_compiler_attribution_ablation/reports/structural_compiler_attribution_ablation_report.md (final length-24 splits table)

Technical framing

Failing arms still trace most intermediate states at length 24 — Losing arms keep 62-86% of the intermediate state trace correct despite near-zero final answers, so errors concentrate in late steps.

Short-program skill dips mid-training as longer programs arrive

How to read

The horizontal axis is training progress; the lines show accuracy on short, unfamiliar-wording programs for three approaches as longer programs enter training; higher is better.

20%40%60%80%100%120%400600max24_curriculumexpand_copyexpand_random_new_slotsexpand_random_new…

Takeaway → The winning approach sags mid-training as longer programs arrive, then rebounds to about 91%, while one control ends lower near 58%.

Data table
Training stepmax24_curriculumexpand_copyexpand_random_new_slots
30081.2%75%81.2%
45034.4%90.6%96.9%
75090.6%79.7%57.8%

Numbers from experiments/qwen_structural_compiler_attribution_ablation/reports/structural_compiler_attribution_ablation_report.md (per-stage splits table, heldout_L8 rows)

Technical framing

Held-out length-8 accuracy dips mid-training as longer programs arrive — Short held-out programs are transiently forgotten: max24_curriculum drops to 34.4% mid-run before recovering; expand_random ends down at 57.8%.

In the author’s words from the Report · “Results”

The strongest arm is the max-24 compiler trained from the start with the staged length curriculum. Copied structural expansion is not the winning explanation in this run.

Overview

This standalone experiment tests why a tuned executable latent compiler learns length-24 modular programs.

The arms isolate:

  • Copy-based structural expansion.
  • Same curriculum with a max-24 compiler from the start.
  • Expansion with random initialization for newly introduced slots.
  • No-curriculum max-24 training.
  • Training only through length 16, then evaluating length 24.

Large checkpoints are stored outside this directory:

/workspace/large_artifacts/qwen_structural_compiler_attribution_ablation/checkpoints

Report

Rendered from reports/structural_compiler_attribution_ablation_report.md

Question

Which factor causes a one-shot executable latent compiler to learn length-24 modular programs: copied structural expansion, curriculum alone, random-slot expansion, no-curriculum training, or length extrapolation?

Method

  • Every arm uses a Qwen causal LM plus a direct executable compiler head.
  • The compiler predicts one initial value plus typed operation and argument slots.
  • A differentiable modular executor supervises final answer probability and intermediate state traces.
  • The main controls separate copied structural expansion from curriculum, random new slots, no-curriculum training, and length extrapolation.
  • Evaluation includes standard templates, seen-family paraphrases, held-out wording templates, and paired consistency splits.

Runs

runarmelapsed_secmodelstage_max_stepsstage_stepstrain_examplesgpu
main_expand_copy_s750expand_copy1914Qwen/Qwen3-4B8,16,24300,150,300512NVIDIA RTX 6000 Ada Generation
main_expand_random_new_slots_s750expand_random_new_slots1915Qwen/Qwen3-4B8,16,24300,150,300512NVIDIA RTX 6000 Ada Generation
main_max24_curriculum_s750max24_curriculum2726Qwen/Qwen3-4B24,24,24300,150,300512NVIDIA RTX 6000 Ada Generation
main_max24_no_curriculum_s750max24_no_curriculum2697Qwen/Qwen3-4B24750512NVIDIA RTX 6000 Ada Generation
main_train_to16_eval24_s750train_to16_eval242442Qwen/Qwen3-4B24750512NVIDIA RTX 6000 Ada Generation
pilot_expand_copyexpand_copy105.5Qwen/Qwen3-4B8,16,2430,15,3096NVIDIA RTX 6000 Ada Generation
pilot_expand_random_new_slotsexpand_random_new_slots106.3Qwen/Qwen3-4B8,16,2430,15,3096NVIDIA RTX 6000 Ada Generation
pilot_max24_curriculummax24_curriculum157Qwen/Qwen3-4B24,24,2430,15,3096NVIDIA RTX 6000 Ada Generation
pilot_max24_no_curriculummax24_no_curriculum134.7Qwen/Qwen3-4B247596NVIDIA RTX 6000 Ada Generation
pilot_train_to16_eval24train_to16_eval24125.3Qwen/Qwen3-4B247596NVIDIA RTX 6000 Ada Generation
smoke_random_new_slotssmoke_random_new_slots10.16Qwen/Qwen3-4B8,16,241,1,14NVIDIA RTX 6000 Ada Generation

Results

Attribution summary, final length-24 executable accuracy:

armrunstandard_L24heldout_L24paired_L24paired_heldout_L24
max24_curriculummain_max24_curriculum_s75096.9%46.9%89.1%78.1%
max24_no_curriculummain_max24_no_curriculum_s7504.7%18.8%9.4%7.8%
expand_copymain_expand_copy_s7501.6%51.6%29.7%25.0%
expand_random_new_slotsmain_expand_random_new_slots_s7501.6%6.2%14.1%1.6%
train_to16_eval24main_train_to16_eval24_s7500.0%1.6%1.6%0.0%

The strongest arm is the max-24 compiler trained from the start with the staged length curriculum. Copied structural expansion is not the winning explanation in this run.

Final length-24 splits by arm:

armrunsplitexecutor_accuracyprogram_exactstate_prefix_fractionexecutor_pair_both_correctcompiler_pair_state_consistency
expand_copymain_expand_copy_s750standard_L241.6%0.0%85.9%
expand_copymain_expand_copy_s750paraphrase_L2475.0%75.0%97.3%
expand_copymain_expand_copy_s750heldout_L2451.6%51.6%68.6%
expand_copymain_expand_copy_s750paired_L2429.7%29.7%91.0%0.0%9.4%
expand_copymain_expand_copy_s750paired_heldout_L2425.0%23.4%74.1%3.1%0.0%
expand_random_new_slotsmain_expand_random_new_slots_s750standard_L241.6%0.0%67.9%
expand_random_new_slotsmain_expand_random_new_slots_s750paraphrase_L2429.7%29.7%85.6%
expand_random_new_slotsmain_expand_random_new_slots_s750heldout_L246.2%4.7%46.5%
expand_random_new_slotsmain_expand_random_new_slots_s750paired_L2414.1%14.1%77.5%0.0%0.0%
expand_random_new_slotsmain_expand_random_new_slots_s750paired_heldout_L241.6%0.0%56.6%0.0%0.0%
max24_curriculummain_max24_curriculum_s750standard_L2496.9%96.9%99.7%
max24_curriculummain_max24_curriculum_s750paraphrase_L2481.2%81.2%98.8%
max24_curriculummain_max24_curriculum_s750heldout_L2446.9%45.3%85.3%
max24_curriculummain_max24_curriculum_s750paired_L2489.1%89.1%99.3%78.1%78.1%
max24_curriculummain_max24_curriculum_s750paired_heldout_L2478.1%75.0%91.1%56.2%50.0%
max24_no_curriculummain_max24_no_curriculum_s750standard_L244.7%0.0%82.2%
max24_no_curriculummain_max24_no_curriculum_s750paraphrase_L2415.6%14.1%89.1%
max24_no_curriculummain_max24_no_curriculum_s750heldout_L2418.8%18.8%81.6%
max24_no_curriculummain_max24_no_curriculum_s750paired_L249.4%7.8%84.9%0.0%0.0%
max24_no_curriculummain_max24_no_curriculum_s750paired_heldout_L247.8%6.2%82.4%0.0%0.0%
train_to16_eval24main_train_to16_eval24_s750standard_L240.0%0.0%62.0%
train_to16_eval24main_train_to16_eval24_s750paraphrase_L240.0%0.0%69.6%
train_to16_eval24main_train_to16_eval24_s750heldout_L241.6%0.0%61.9%
train_to16_eval24main_train_to16_eval24_s750paired_L241.6%0.0%65.3%0.0%0.0%
train_to16_eval24main_train_to16_eval24_s750paired_heldout_L240.0%0.0%60.7%0.0%0.0%
armrunstagesplitglobal_stepmax_stepsexecutor_accuracyprogram_exactstate_prefix_fractionstate_all_exactexecutor_pair_both_correctcompiler_pair_state_consistencylength
expand_copymain_expand_copy_s750stage1_max8heldout_L8300875.0%73.4%84.2%0.73448
expand_copymain_expand_copy_s750stage1_max8paired_L83008100.0%100.0%100.0%1100.0%100.0%8
expand_copymain_expand_copy_s750stage1_max8paired_heldout_L8300882.8%82.8%91.8%0.828165.6%65.6%8
expand_copymain_expand_copy_s750stage1_max8paraphrase_L8300898.4%98.4%98.4%0.98448
expand_copymain_expand_copy_s750stage1_max8standard_L83008100.0%100.0%100.0%18
expand_copymain_expand_copy_s750stage2_max16heldout_L84501690.6%90.6%93.8%0.90628
expand_copymain_expand_copy_s750stage2_max16paired_L845016100.0%100.0%100.0%1100.0%100.0%8
expand_copymain_expand_copy_s750stage2_max16paired_heldout_L84501695.3%95.3%97.9%0.953190.6%90.6%8
expand_copymain_expand_copy_s750stage2_max16paraphrase_L845016100.0%100.0%100.0%18
expand_copymain_expand_copy_s750stage2_max16standard_L845016100.0%100.0%100.0%18
expand_copymain_expand_copy_s750stage2_max16heldout_L164501671.9%71.9%92.2%0.718816
expand_copymain_expand_copy_s750stage2_max16paired_L164501690.6%90.6%98.3%0.906281.2%81.2%16
expand_copymain_expand_copy_s750stage2_max16paired_heldout_L164501670.3%68.8%92.8%0.687546.9%43.8%16
expand_copymain_expand_copy_s750stage2_max16paraphrase_L164501693.8%93.8%99.5%0.937516
expand_copymain_expand_copy_s750stage2_max16standard_L164501695.3%95.3%98.5%0.953116
expand_copymain_expand_copy_s750stage3_max24heldout_L87502479.7%79.7%82.8%0.79698
expand_copymain_expand_copy_s750stage3_max24paired_L875024100.0%100.0%100.0%1100.0%100.0%8
expand_copymain_expand_copy_s750stage3_max24paired_heldout_L87502487.5%87.5%91.4%0.87575.0%75.0%8
expand_copymain_expand_copy_s750stage3_max24paraphrase_L875024100.0%100.0%100.0%18
expand_copymain_expand_copy_s750stage3_max24standard_L875024100.0%100.0%100.0%18
expand_copymain_expand_copy_s750stage3_max24heldout_L167502473.4%73.4%77.9%0.734416
expand_copymain_expand_copy_s750stage3_max24paired_L1675024100.0%100.0%100.0%1100.0%100.0%16
expand_copymain_expand_copy_s750stage3_max24paired_heldout_L167502484.4%84.4%89.0%0.843868.8%68.8%16
expand_copymain_expand_copy_s750stage3_max24paraphrase_L1675024100.0%100.0%100.0%116
expand_copymain_expand_copy_s750stage3_max24standard_L1675024100.0%100.0%100.0%116
expand_copymain_expand_copy_s750stage3_max24heldout_L247502451.6%51.6%68.6%0.515624
expand_copymain_expand_copy_s750stage3_max24paired_L247502429.7%29.7%91.0%0.29690.0%9.4%24
expand_copymain_expand_copy_s750stage3_max24paired_heldout_L247502425.0%23.4%74.1%0.23443.1%0.0%24
expand_copymain_expand_copy_s750stage3_max24paraphrase_L247502475.0%75.0%97.3%0.7524
expand_copymain_expand_copy_s750stage3_max24standard_L24750241.6%0.0%85.9%024
expand_random_new_slotsmain_expand_random_new_slots_s750stage1_max8heldout_L8300881.2%79.7%87.9%0.81258
expand_random_new_slotsmain_expand_random_new_slots_s750stage1_max8paired_L83008100.0%100.0%100.0%1100.0%100.0%8
expand_random_new_slotsmain_expand_random_new_slots_s750stage1_max8paired_heldout_L8300895.3%95.3%97.7%0.953190.6%90.6%8
expand_random_new_slotsmain_expand_random_new_slots_s750stage1_max8paraphrase_L8300898.4%98.4%98.4%0.98448
expand_random_new_slotsmain_expand_random_new_slots_s750stage1_max8standard_L83008100.0%100.0%100.0%18
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16heldout_L84501696.9%96.9%97.9%0.96888
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16paired_L845016100.0%100.0%100.0%1100.0%100.0%8
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16paired_heldout_L84501693.8%93.8%96.1%0.937587.5%87.5%8
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16paraphrase_L845016100.0%100.0%100.0%18
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16standard_L845016100.0%100.0%100.0%18
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16heldout_L164501679.7%79.7%90.4%0.796916
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16paired_L164501698.4%98.4%98.7%0.984496.9%96.9%16
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16paired_heldout_L164501687.5%87.5%93.4%0.87575.0%75.0%16
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16paraphrase_L164501695.3%95.3%96.9%0.953116
expand_random_new_slotsmain_expand_random_new_slots_s750stage2_max16standard_L1645016100.0%100.0%100.0%116
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24heldout_L87502457.8%57.8%71.9%0.57818
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24paired_L875024100.0%100.0%100.0%1100.0%100.0%8
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24paired_heldout_L87502482.8%82.8%88.5%0.828165.6%65.6%8
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24paraphrase_L875024100.0%100.0%100.0%18
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24standard_L87502498.4%98.4%98.8%0.98448
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24heldout_L167502440.6%39.1%59.7%0.406216
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24paired_L167502495.3%95.3%99.3%0.953190.6%90.6%16
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24paired_heldout_L167502473.4%70.3%84.3%0.718850.0%46.9%16
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24paraphrase_L1675024100.0%100.0%100.0%116
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24standard_L167502493.8%93.8%99.1%0.937516
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24heldout_L24750246.2%4.7%46.5%0.0468824
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24paired_L247502414.1%14.1%77.5%0.14060.0%0.0%24
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24paired_heldout_L24750241.6%0.0%56.6%00.0%0.0%24
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24paraphrase_L247502429.7%29.7%85.6%0.296924
expand_random_new_slotsmain_expand_random_new_slots_s750stage3_max24standard_L24750241.6%0.0%67.9%024
max24_curriculummain_max24_curriculum_s750stage1_max24heldout_L83002481.2%81.2%87.7%0.81258
max24_curriculummain_max24_curriculum_s750stage1_max24paired_L830024100.0%100.0%100.0%1100.0%100.0%8
max24_curriculummain_max24_curriculum_s750stage1_max24paired_heldout_L83002492.2%92.2%97.1%0.921984.4%84.4%8
max24_curriculummain_max24_curriculum_s750stage1_max24paraphrase_L83002498.4%98.4%98.4%0.98448
max24_curriculummain_max24_curriculum_s750stage1_max24standard_L830024100.0%100.0%100.0%18
max24_curriculummain_max24_curriculum_s750stage1_max24heldout_L16300244.7%3.1%55.4%0.0312516
max24_curriculummain_max24_curriculum_s750stage1_max24paired_L163002450.0%50.0%89.2%0.531.2%31.2%16
max24_curriculummain_max24_curriculum_s750stage1_max24paired_heldout_L163002428.1%28.1%76.9%0.28120.0%0.0%16
max24_curriculummain_max24_curriculum_s750stage1_max24paraphrase_L163002426.6%25.0%81.8%0.2516
max24_curriculummain_max24_curriculum_s750stage1_max24standard_L163002450.0%48.4%89.9%0.484416
max24_curriculummain_max24_curriculum_s750stage1_max24heldout_L24300240.0%0.0%37.2%024
max24_curriculummain_max24_curriculum_s750stage1_max24paired_L24300241.6%0.0%59.8%00.0%0.0%24
max24_curriculummain_max24_curriculum_s750stage1_max24paired_heldout_L24300241.6%0.0%46.0%00.0%0.0%24
max24_curriculummain_max24_curriculum_s750stage1_max24paraphrase_L24300241.6%0.0%57.8%024
max24_curriculummain_max24_curriculum_s750stage1_max24standard_L24300244.7%1.6%61.3%0.0156224
max24_curriculummain_max24_curriculum_s750stage2_max24heldout_L84502434.4%34.4%60.0%0.34388
max24_curriculummain_max24_curriculum_s750stage2_max24paired_L845024100.0%100.0%100.0%1100.0%100.0%8
max24_curriculummain_max24_curriculum_s750stage2_max24paired_heldout_L84502460.9%59.4%76.0%0.609421.9%21.9%8
max24_curriculummain_max24_curriculum_s750stage2_max24paraphrase_L84502496.9%96.9%97.9%0.96888
max24_curriculummain_max24_curriculum_s750stage2_max24standard_L845024100.0%100.0%100.0%18
max24_curriculummain_max24_curriculum_s750stage2_max24heldout_L164502410.9%10.9%42.1%0.109416
max24_curriculummain_max24_curriculum_s750stage2_max24paired_L1645024100.0%100.0%100.0%1100.0%100.0%16
max24_curriculummain_max24_curriculum_s750stage2_max24paired_heldout_L164502459.4%59.4%72.1%0.593818.8%18.8%16
max24_curriculummain_max24_curriculum_s750stage2_max24paraphrase_L164502496.9%96.9%98.6%0.968816
max24_curriculummain_max24_curriculum_s750stage2_max24standard_L1645024100.0%100.0%100.0%116
max24_curriculummain_max24_curriculum_s750stage2_max24heldout_L24450241.6%0.0%32.5%024
max24_curriculummain_max24_curriculum_s750stage2_max24paired_L244502418.8%17.2%79.6%0.17190.0%0.0%24
max24_curriculummain_max24_curriculum_s750stage2_max24paired_heldout_L244502415.6%14.1%60.0%0.14060.0%0.0%24
max24_curriculummain_max24_curriculum_s750stage2_max24paraphrase_L24450240.0%0.0%69.5%024
max24_curriculummain_max24_curriculum_s750stage2_max24standard_L244502434.4%34.4%88.6%0.343824
max24_curriculummain_max24_curriculum_s750stage3_max24heldout_L87502490.6%90.6%93.4%0.90628
max24_curriculummain_max24_curriculum_s750stage3_max24paired_L875024100.0%100.0%100.0%1100.0%100.0%8
max24_curriculummain_max24_curriculum_s750stage3_max24paired_heldout_L87502493.8%93.8%97.3%0.937587.5%87.5%8
max24_curriculummain_max24_curriculum_s750stage3_max24paraphrase_L875024100.0%100.0%100.0%18
max24_curriculummain_max24_curriculum_s750stage3_max24standard_L875024100.0%100.0%100.0%18
max24_curriculummain_max24_curriculum_s750stage3_max24heldout_L167502485.9%85.9%89.4%0.859416
max24_curriculummain_max24_curriculum_s750stage3_max24paired_L1675024100.0%100.0%100.0%1100.0%100.0%16
max24_curriculummain_max24_curriculum_s750stage3_max24paired_heldout_L167502490.6%90.6%94.1%0.906281.2%81.2%16
max24_curriculummain_max24_curriculum_s750stage3_max24paraphrase_L1675024100.0%100.0%100.0%116
max24_curriculummain_max24_curriculum_s750stage3_max24standard_L1675024100.0%100.0%100.0%116
max24_curriculummain_max24_curriculum_s750stage3_max24heldout_L247502446.9%45.3%85.3%0.453124
max24_curriculummain_max24_curriculum_s750stage3_max24paired_L247502489.1%89.1%99.3%0.890678.1%78.1%24
max24_curriculummain_max24_curriculum_s750stage3_max24paired_heldout_L247502478.1%75.0%91.1%0.7556.2%50.0%24
max24_curriculummain_max24_curriculum_s750stage3_max24paraphrase_L247502481.2%81.2%98.8%0.812524
max24_curriculummain_max24_curriculum_s750stage3_max24standard_L247502496.9%96.9%99.7%0.968824
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24heldout_L87502493.8%93.8%96.5%0.93758
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24paired_L875024100.0%100.0%100.0%1100.0%100.0%8
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24paired_heldout_L87502498.4%98.4%99.2%0.984496.9%96.9%8
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24paraphrase_L875024100.0%100.0%100.0%18
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24standard_L875024100.0%100.0%100.0%18
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24heldout_L167502496.9%96.9%98.5%0.968816
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24paired_L1675024100.0%100.0%100.0%1100.0%100.0%16
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24paired_heldout_L1675024100.0%100.0%100.0%1100.0%100.0%16
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24paraphrase_L1675024100.0%100.0%100.0%116
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24standard_L1675024100.0%100.0%100.0%116
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24heldout_L247502418.8%18.8%81.6%0.187524
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24paired_L24750249.4%7.8%84.9%0.078120.0%0.0%24
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24paired_heldout_L24750247.8%6.2%82.4%0.06250.0%0.0%24
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24paraphrase_L247502415.6%14.1%89.1%0.140624
max24_no_curriculummain_max24_no_curriculum_s750stage1_max24standard_L24750244.7%0.0%82.2%024

Figures

executor_accuracy_by_length.png

executor_accuracy_by_step.png

program_exact_by_step.png

state_prefix_by_step.png

paired_state_consistency_by_step.png

training_loss.png

training_state_accuracy.png

Interpretation

This report is intentionally standalone. The key readout is whether the copied-expansion arm beats same-budget max-24 curriculum and random-new-slot controls on length-24 standard, held-out wording, and paired consistency splits.

Artifacts

Experiment log 5

Show the running log (5 entries, 2026-06-24)

2026-06-24

  • Created a fresh standalone attribution-ablation experiment directory.
  • Core question: does the long-program compiler result come from structural slot expansion, from curriculum/objective tuning, or from general training budget?

  • Required controls:

    • expand_copy: staged 8 -> 16 -> 24 compiler with copied learned slots.
    • max24_curriculum: max-24 compiler from the start, same staged training lengths and budget.
    • expand_random_new_slots: staged compiler, but newly introduced slots are randomly initialized instead of copied from the last learned slot.
    • max24_no_curriculum: max-24 compiler from the start, trained directly on lengths 1..24.
    • train_to16_eval24: max-24 compiler trained only on lengths up to 16, evaluated on length 24.
  • Added held-out wording templates that are not sampled by mixed training mode.

  • Planned process:

    • Smoke: verify all new split names and expansion modes.
    • Pilot: short controlled arms to verify the suite runs and exposes expected floor/learning behavior.
    • Main: full ablation arms with Markdown/HTML report and charts.

Smoke

  • Run: smoke_random_new_slots
  • Configuration: Qwen/Qwen3-4B QLoRA, one update per stage, random-new-slot expansion, tiny eval.
  • Outcome: completed successfully.

  • Verified:

    • random_new expansion mode.
    • Held-out wording splits: heldout_L*.
    • Standard-vs-heldout paired splits: paired_heldout_L*.
    • Markdown/HTML report generation.

Pilot

  • Runs:

    • pilot_expand_copy
    • pilot_max24_curriculum
    • pilot_expand_random_new_slots
    • pilot_max24_no_curriculum
    • pilot_train_to16_eval24
  • Configuration: reduced-width compiler, small datasets, short budgets, no checkpoints.
  • Outcome: all five controls completed and produced comparable metrics. As expected at this tiny budget, all length-24 final program-exact metrics stayed at 0%.
  • Interpretation: the pilot was a harness validation, not a result-bearing run. Proceeding to full ablation with tuned width, loss weights, batch size, evaluation size, and checkpoints.

Main Ablation

  • Runs:

    • main_expand_copy_s750
    • main_max24_curriculum_s750
    • main_expand_random_new_slots_s750
    • main_max24_no_curriculum_s750
    • main_train_to16_eval24_s750
  • Configuration: Qwen/Qwen3-4B QLoRA, 512-wide compiler, LoRA rank 8, batch 8, strong init/argument trace supervision, full state loss, 64 examples per single-template split, 32 paired examples per paired split.

  • Main length-24 executable accuracy:

    • max24_curriculum: standard 96.9%, heldout 46.9%, paired 89.1%, paired-heldout 78.1%.
    • expand_copy: standard 1.6%, heldout 51.6%, paired 29.7%, paired-heldout 25.0%.
    • expand_random_new_slots: standard 1.6%, heldout 6.2%, paired 14.1%, paired-heldout 1.6%.
    • max24_no_curriculum: standard 4.7%, heldout 18.8%, paired 9.4%, paired-heldout 7.8%.
    • train_to16_eval24: standard 0.0%, heldout 1.6%, paired 1.6%, paired-heldout 0.0%.

  • Interpretation:

    • Copied structural expansion is not the winning explanation in this run.
    • The best explanation is a max-24 compiler trained from the start with a staged length curriculum.
    • Randomly initialized new expansion slots perform poorly at length 24.
    • Removing curriculum fails at length 24 despite strong length-8 and length-16 performance.
    • Training only through length 16 does not extrapolate to length 24.
    • Held-out wording is much harder than seen-family paraphrase, even for the best arm.

Reports

  • Markdown: reports/structural_compiler_attribution_ablation_report.md
  • HTML: reports/structural_compiler_attribution_ablation_report.html
  • Figures: reports/figures/

Figures 7

executor accuracy by length
executor accuracy by length · reports/figures/
executor accuracy by step
executor accuracy by step · reports/figures/
paired state consistency by step
paired state consistency by step · reports/figures/
program exact by step
program exact by step · reports/figures/
state prefix by step
state prefix by step · reports/figures/
training loss
training loss · reports/figures/
training state accuracy
training state accuracy · reports/figures/

Data files 23

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗