Research log Small Model Experimentation
GitHub

Qwen Compiler Multi-Seed Reattribution

Finished2026-06-24 → 25imported · line YBenchmark GeneralizationCollective Experimentation InfrastructureStructured Execution and CompilersGitHub ↗
Random seed, not recipe, decides whether it works

The one idea you need

The model writes a fixed 24-step recipe of typed number operations, and a built-in calculator runs it — grading not just the final answer but every intermediate value. Re-running the same training from three different random starting points reveals whether it learns reliably.

The question

Can a small model be trained to reliably write correct 24-step number programs, and does the training schedule you pick actually decide whether it succeeds?

What we found

No. The best schedule averaged 43% correct on plainly worded problems, yet the identical training swung from total failure to 81% just by changing the random starting number — so no schedule earns credit for the wins. Both growing programs from short to long and jumping straight to full length collapsed unpredictably. The seed, not the recipe, decided the outcome.

Why it matters

Before crediting a training trick for a small-model gain, re-run it across several random starting points. One lucky start can look like a breakthrough while the identical recipe flatlines on the next, faking a real effect.

Best recipe's swing across three random starts0% → 81%identical training, plainly worded 24-step problems
Best average accuracy of any recipe43%top schedule, hiding a 41-point spread across starts
Jumping straight to full lengthnear 0%weakest schedule on plainly worded problems
Long opening run of steps recovered70% to 87%across all recipes, even when the full program fails
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Question
    2. Method
    3. Arms
    4. Runs
    5. Results
    6. Key Findings
    7. Figures
    8. Interpretation
    9. Artifacts
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 7

Which training recipe runs 24-step programs best, on average

How to read

Grouped bars along four kinds of test question (plain wording, unfamiliar wording, matched question pairs, unfamiliar pairs); each recipe is a colored bar; height is the share of 24-step programs run correctly. Taller is better.

0%20%40%60%80%standardstandard43.2%14.1%0.5%held-out wordingheld-out wording34.4%38.5%16.1%pairedpaired60.4%40.1%24.5%paired held-outpaired held-out40.6%22.9%9.9%

Takeaway → The grow-from-short recipe is tallest on most groups, yet even its best plain-wording bar sits below half, while training everything at once barely clears the floor.

Data table
evaluation splitexpand-copymax-24 curriculummax-24 no curriculum
standard43.2%14.1%0.5%
held-out wording34.4%38.5%16.1%
paired60.4%40.1%24.5%
paired held-out40.6%22.9%9.9%

Numbers from report table (reports/qwen_compiler_multiseed_reattribution_report.md); also reports/aggregate_final_l24_accuracy.csv

Technical framing

Length-24 program execution accuracy by training arm (mean of 3 seeds) — Copied structural expansion has the best mean on most splits; same-budget no-curriculum training nearly fails on standard wording.

The same recipe's score swings wildly by random starting point

How to read

Bars grouped by random starting point (three of them), each recipe a colored bar; height is share of plainly worded 24-step programs run correctly. Taller is better; equal heights within one color would mean a dependable recipe.

0%25%50%75%100%0%23.4%0%seed 12348.4%1.6%0%seed 45681.2%17.2%1.6%seed 789

Takeaway → The best recipe goes from zero at one starting point to 81% at another, so a single run tells you almost nothing about the recipe.

Data table
random seedexpand-copymax-24 curriculummax-24 no curriculum
seed 1230%23.4%0%
seed 45648.4%1.6%0%
seed 78981.2%17.2%1.6%

Numbers from report table (reports/qwen_compiler_multiseed_reattribution_report.md); also reports/per_seed_final_l24_accuracy.csv

Technical framing

Standard length-24 accuracy per seed: means hide huge seed variance — The best arm ranges from total failure to 81% across seeds, so no single-seed run supports a stable attribution claim.

The early steps are right; the whole program is where it breaks

How to read

For each recipe, two bars: how often the opening run of steps is correct versus how often the entire 24-step program is correct. Height is accuracy; higher is better.

0%25%50%75%100%expand-copyexpand-copy87.3%43.2%max-24 curriculummax-24 curriculum81.4%14.1%max-24 no curriculummax-24 no curriculum70%0.5%

Takeaway → Every recipe gets a long opening run of steps right far more often than the full program, so the failure is finishing consistently, not knowing individual steps.

Data table
training armstate-prefix recoveryexact execution
expand-copy87.3%43.2%
max-24 curriculum81.4%14.1%
max-24 no curriculum70%0.5%

Numbers from report table (reports/qwen_compiler_multiseed_reattribution_report.md)

Technical framing

Partial execution is not the bottleneck: state-prefix recovery vs exact execution (standard L24, mean of 3 seeds) — Every arm recovers long state prefixes far more often than exact programs; the failure is late-step global consistency, not local operators.

Programs work at 16 steps, then fall off a cliff at 24

How to read

Each run (recipe and starting point) has two bars: accuracy on 16-step programs versus 24-step programs. Height is share run correctly; higher is better.

0%25%50%75%100%expand s123expand s123100%0%expand s456expand s456100%48.4%expand s789expand s789100%81.2%curric s123curric s123100%23.4%curric s456curric s45690.6%1.6%curric s789curric s789100%17.2%no-curr s123no-curr s12348.4%0%no-curr s456no-curr s456100%0%no-curr s789no-curr s78996.9%1.6%

Takeaway → Almost every run nails the 16-step programs but crashes on the 24-step ones, so the trouble is tied to that extra length, not a general inability.

Data table
Run (arm / seed)Length 16Length 24
expand s123100%0%
expand s456100%48.4%
expand s789100%81.2%
curric s123100%23.4%
curric s45690.6%1.6%
curric s789100%17.2%
no-curr s12348.4%0%
no-curr s456100%0%
no-curr s78996.9%1.6%

Numbers from experiments/qwen_compiler_multiseed_reattribution/reports/aggregate_metrics.csv (final step 750, standard_L16/standard_L24 executor_accuracy)

Technical framing

Capability cliff between length 16 and length 24 — Nearly every run solves length-16 programs, then falls off a cliff at length 24 -- the failure is length-specific, not general.

Training looks like a clean success even when the final test collapses

How to read

Lines track, step by step during training, how often the model's intermediate work is right; one line per random starting point. Rising toward the top means training is going well.

0%50%100%150%200400600Seed 123Seed 789Seed 456

Takeaway → All lines climb to near-perfect and look interchangeable, yet their final test scores differ hugely, so a smooth training curve gives no warning of a bad outcome.

Data table
Training stepSeed 123Seed 456Seed 789
10%0%3.1%
500%4.9%0%
10032.4%0%88.9%
150100%85.4%84.8%
200100%100%100%
250100%100%100%
300100%93.2%100%
301100%76.1%81.8%
350100%100%100%
400100%100%100%
450100%100%100%
45189.6%96.1%96.4%
50096.7%98.6%100%
550100%100%100%
600100%100%100%
650100%93.5%100%
700100%100%100%
750100%85.2%100%

Numbers from experiments/qwen_compiler_multiseed_reattribution/reports/aggregate_train_log.csv (state_train_accuracy by global_step, max24_curriculum runs)

Technical framing

Training looks identical across seeds even when eval collapses — All three curriculum seeds train to near-perfect state accuracy, yet seed 456 scores 1.6% at eval -- the collapse is invisible in training.

Reworded questions are far easier than the plain ones

How to read

Each run has two bars: accuracy on the plain question wording versus on reworded versions of the same programs. Height is share run correctly; higher is better.

0%25%50%75%100%expand s123expand s1230%78.1%expand s456expand s45648.4%73.4%expand s789expand s78981.2%98.4%curric s123curric s12323.4%92.2%curric s456curric s4561.6%1.6%curric s789curric s78917.2%98.4%no-curr s123no-curr s1230%46.9%no-curr s456no-curr s4560%100%no-curr s789no-curr s7891.6%0%

Takeaway → Most runs handle reworded questions far better than plain ones — some near 100% reworded while plain wording is near zero — revealing sensitivity to phrasing.

Data table
Run (arm / seed)Standard templatesParaphrase templates
expand s1230%78.1%
expand s45648.4%73.4%
expand s78981.2%98.4%
curric s12323.4%92.2%
curric s4561.6%1.6%
curric s78917.2%98.4%
no-curr s1230%46.9%
no-curr s4560%100%
no-curr s7891.6%0%

Numbers from experiments/qwen_compiler_multiseed_reattribution/reports/qwen_compiler_multiseed_reattribution_report.md (Final split rows table, executor_accuracy)

Technical framing

Standard wording is the hardest split: paraphrase templates score far higher — Most runs execute paraphrased length-24 programs far better than standard wording -- some hit 92-100% paraphrase with near-zero standard.

Getting one wording right rarely means getting both

How to read

For each run, two bars: how often a single question is answered correctly versus how often both rewordings of the same program are answered. Height is the rate; higher is better.

0%25%50%75%100%expand s123expand s12332.8%0%expand s456expand s45654.7%21.9%expand s789expand s78993.8%90.6%curric s123curric s12359.4%25%curric s456curric s4560%0%curric s789curric s78960.9%21.9%no-curr s123no-curr s12323.4%3.1%no-curr s456no-curr s45650%3.1%no-curr s789no-curr s7890%0%

Takeaway → The both-versions bar sits well below the single-question bar in nearly every run, so success on a program usually doesn't survive a change of wording.

Data table
Run (arm / seed)Single-item accuracy (paired split)Both paraphrases correct
expand s12332.8%0%
expand s45654.7%21.9%
expand s78993.8%90.6%
curric s12359.4%25%
curric s4560%0%
curric s78960.9%21.9%
no-curr s12323.4%3.1%
no-curr s45650%3.1%
no-curr s7890%0%

Numbers from experiments/qwen_compiler_multiseed_reattribution/reports/qwen_compiler_multiseed_reattribution_report.md (Final split rows table, paired_L24 executor_accuracy and executor_pair_both_correct)

Technical framing

Answering both paraphrases of the same program is much rarer than answering one — Pair-level consistency collapses relative to per-item accuracy in every run except the strongest seed (expand_copy 789).

In the author’s words from the Report · “Key Findings”

standard_L24: best mean is expand_copy at 43.2% with 40.9 percentage points seed standard deviation. paired_L24: best mean is expand_copy at 60.4% with 30.9 percentage points seed standard deviation. heldout_L24: best mean is max24_curriculum at 38.5% with 47.9 percentage points seed standard deviation. State-prefix recovery is high even when exact execution fails: standard_L24 ranges from 70.0% mean prefix recovery for max24_no_curriculum to 87.3% for expand_copy. Same-budget no-curriculum training is the weakest standard-L24 arm at 0.5% mean accuracy.

Overview

This standalone experiment tests whether a one-shot executable latent compiler reliably learns length-24 modular programs across random seeds.

The result-bearing arms are:

  • max24_curriculum: a max-24 compiler trained from the start with staged train lengths.
  • expand_copy: an 8 -> 16 -> 24 expanding compiler where new slots copy the last learned slot.
  • max24_no_curriculum: a max-24 compiler trained on the full length range immediately.

All result-bearing runs use the same seed set across arms. The main report aggregates final length-24 executable accuracy with mean, standard deviation, min, and max across seeds.

Large checkpoints are stored outside this directory:

/workspace/large_artifacts/qwen_compiler_multiseed_reattribution/checkpoints

Primary reports:

  • reports/qwen_compiler_multiseed_reattribution_report.md
  • reports/qwen_compiler_multiseed_reattribution_report.html

Report

Rendered from reports/qwen_compiler_multiseed_reattribution_report.md

Question

Does a one-shot executable latent compiler reliably learn length-24 modular programs across random seeds, and which training factor best explains the result: staged length curriculum, copied structural expansion, or same-budget no-curriculum training?

Method

  • Each arm uses Qwen/Qwen3-4B with QLoRA and a direct executable compiler head.
  • The compiler predicts one initial value and a sequence of typed operation and argument slots.
  • A differentiable modular executor supervises final answers and intermediate state traces.
  • The same seed set is used for each arm, so the main readout is mean and spread across matched random seeds.
  • Evaluation includes standard templates, held-out wording templates, seen-family paired consistency, and held-out paired consistency.

Arms

  • max24_curriculum: max-24 compiler from the start, with staged train lengths.
  • expand_copy: compiler capacity expands in stages and newly introduced slots copy the last learned slot.
  • max24_no_curriculum: max-24 compiler from the start, trained on the full length range immediately.

Runs

runarmseedelapsed_secstage_max_stepsstage_stepstrain_exampleseval_examplesgpu
main_expand_copy_seed123expand_copy12319128,16,24300,150,30051264NVIDIA RTX 6000 Ada Generation
main_expand_copy_seed456expand_copy45619148,16,24300,150,30051264NVIDIA RTX 6000 Ada Generation
main_expand_copy_seed789expand_copy78919088,16,24300,150,30051264NVIDIA RTX 6000 Ada Generation
main_max24_curriculum_seed123max24_curriculum123272224,24,24300,150,30051264NVIDIA RTX 6000 Ada Generation
main_max24_curriculum_seed456max24_curriculum456272124,24,24300,150,30051264NVIDIA RTX 6000 Ada Generation
main_max24_curriculum_seed789max24_curriculum789271724,24,24300,150,30051264NVIDIA RTX 6000 Ada Generation
main_max24_no_curriculum_seed123max24_no_curriculum12326912475051264NVIDIA RTX 6000 Ada Generation
main_max24_no_curriculum_seed456max24_no_curriculum45626932475051264NVIDIA RTX 6000 Ada Generation
main_max24_no_curriculum_seed789max24_no_curriculum78926952475051264NVIDIA RTX 6000 Ada Generation
pilot_expand_copy_seed123expand_copy12330.618,16,244,2,4328NVIDIA RTX 6000 Ada Generation
pilot_expand_copy_seed456expand_copy45630.938,16,244,2,4328NVIDIA RTX 6000 Ada Generation
pilot_max24_curriculum_seed123max24_curriculum12352.4524,24,244,2,4328NVIDIA RTX 6000 Ada Generation
pilot_max24_curriculum_seed456max24_curriculum45653.3424,24,244,2,4328NVIDIA RTX 6000 Ada Generation
pilot_max24_no_curriculum_seed123max24_no_curriculum12323.182410328NVIDIA RTX 6000 Ada Generation
pilot_max24_no_curriculum_seed456max24_no_curriculum45623.112410328NVIDIA RTX 6000 Ada Generation
smoke_max24_curriculum_seed123max24_curriculum12324.0124,24,241,1,1124NVIDIA RTX 6000 Ada Generation

Results

Final length-24 executable accuracy, mean +/- standard deviation across seeds:

Key Findings

  • standard_L24: best mean is expand_copy at 43.2% with 40.9 percentage points seed standard deviation.
  • paired_L24: best mean is expand_copy at 60.4% with 30.9 percentage points seed standard deviation.
  • heldout_L24: best mean is max24_curriculum at 38.5% with 47.9 percentage points seed standard deviation.
  • State-prefix recovery is high even when exact execution fails: standard_L24 ranges from 70.0% mean prefix recovery for max24_no_curriculum to 87.3% for expand_copy.
  • Same-budget no-curriculum training is the weakest standard-L24 arm at 0.5% mean accuracy.
armstandard_L24heldout_L24paired_L24paired_heldout_L24
expand_copy43.2% +/- 40.9 (0.0-81.2, n=3)34.4% +/- 36.9 (7.8-76.6, n=3)60.4% +/- 30.9 (32.8-93.8, n=3)40.6% +/- 35.8 (9.4-79.7, n=3)
max24_curriculum14.1% +/- 11.3 (1.6-23.4, n=3)38.5% +/- 47.9 (0.0-92.2, n=3)40.1% +/- 34.7 (0.0-60.9, n=3)22.9% +/- 23.5 (0.0-46.9, n=3)
max24_no_curriculum0.5% +/- 0.9 (0.0-1.6, n=3)16.1% +/- 19.3 (0.0-37.5, n=3)24.5% +/- 25.0 (0.0-50.0, n=3)9.9% +/- 13.3 (0.0-25.0, n=3)

Per-seed final length-24 executable accuracy:

armseedrunstandard_L24heldout_L24paired_L24paired_heldout_L24
expand_copy123main_expand_copy_seed1230.0%18.8%32.8%9.4%
expand_copy456main_expand_copy_seed45648.4%7.8%54.7%32.8%
expand_copy789main_expand_copy_seed78981.2%76.6%93.8%79.7%
max24_curriculum123main_max24_curriculum_seed12323.4%23.4%59.4%21.9%
max24_curriculum456main_max24_curriculum_seed4561.6%0.0%0.0%0.0%
max24_curriculum789main_max24_curriculum_seed78917.2%92.2%60.9%46.9%
max24_no_curriculum123main_max24_no_curriculum_seed1230.0%10.9%23.4%4.7%
max24_no_curriculum456main_max24_no_curriculum_seed4560.0%37.5%50.0%25.0%
max24_no_curriculum789main_max24_no_curriculum_seed7891.6%0.0%0.0%0.0%

Exact program recovery, aggregated across seeds:

armstandard_L24heldout_L24paired_L24paired_heldout_L24
expand_copy42.2% +/- 40.7 (0.0-81.2, n=3)32.3% +/- 38.4 (7.8-76.6, n=3)59.9% +/- 31.6 (31.2-93.8, n=3)39.1% +/- 37.3 (6.2-79.7, n=3)
max24_curriculum10.4% +/- 9.2 (0.0-17.2, n=3)38.0% +/- 48.2 (0.0-92.2, n=3)38.5% +/- 33.4 (0.0-59.4, n=3)22.4% +/- 22.7 (0.0-45.3, n=3)
max24_no_curriculum0.0% +/- 0.0 (0.0-0.0, n=3)16.1% +/- 19.3 (0.0-37.5, n=3)23.4% +/- 25.1 (0.0-50.0, n=3)9.9% +/- 13.3 (0.0-25.0, n=3)

State-prefix recovery, aggregated across seeds:

armstandard_L24heldout_L24paired_L24paired_heldout_L24
expand_copy87.3% +/- 13.4 (71.7-95.1, n=3)70.5% +/- 24.9 (43.7-92.8, n=3)92.2% +/- 6.6 (84.6-96.6, n=3)78.4% +/- 21.0 (54.9-95.6, n=3)
max24_curriculum81.4% +/- 8.5 (71.8-87.9, n=3)82.7% +/- 13.9 (74.6-98.7, n=3)86.8% +/- 11.1 (74.1-93.7, n=3)81.6% +/- 8.5 (73.0-89.9, n=3)
max24_no_curriculum70.0% +/- 8.5 (62.2-79.0, n=3)72.0% +/- 6.6 (66.1-79.2, n=3)78.0% +/- 9.8 (71.4-89.2, n=3)72.2% +/- 9.6 (66.0-83.3, n=3)

Final split rows:

armseedrunsplitnexecutor_accuracyprogram_exactstate_prefix_fractionexecutor_pair_both_correctcompiler_pair_state_consistency
expand_copy123main_expand_copy_seed123standard_L24640.0%0.0%71.7%
expand_copy123main_expand_copy_seed123paraphrase_L246478.1%71.9%97.3%
expand_copy123main_expand_copy_seed123heldout_L246418.8%12.5%43.7%
expand_copy123main_expand_copy_seed123paired_L246432.8%31.2%84.6%0.0%0.0%
expand_copy123main_expand_copy_seed123paired_heldout_L24649.4%6.2%54.9%0.0%0.0%
expand_copy456main_expand_copy_seed456standard_L246448.4%45.3%95.1%
expand_copy456main_expand_copy_seed456paraphrase_L246473.4%71.9%96.8%
expand_copy456main_expand_copy_seed456heldout_L24647.8%7.8%74.9%
expand_copy456main_expand_copy_seed456paired_L246454.7%54.7%95.3%21.9%21.9%
expand_copy456main_expand_copy_seed456paired_heldout_L246432.8%31.2%84.5%3.1%3.1%
expand_copy789main_expand_copy_seed789standard_L246481.2%81.2%94.9%
expand_copy789main_expand_copy_seed789paraphrase_L246498.4%98.4%98.4%
expand_copy789main_expand_copy_seed789heldout_L246476.6%76.6%92.8%
expand_copy789main_expand_copy_seed789paired_L246493.8%93.8%96.6%90.6%93.8%
expand_copy789main_expand_copy_seed789paired_heldout_L246479.7%79.7%95.6%59.4%59.4%
max24_curriculum123main_max24_curriculum_seed123standard_L246423.4%17.2%87.9%
max24_curriculum123main_max24_curriculum_seed123paraphrase_L246492.2%90.6%99.3%
max24_curriculum123main_max24_curriculum_seed123heldout_L246423.4%21.9%74.7%
max24_curriculum123main_max24_curriculum_seed123paired_L246459.4%56.2%93.7%25.0%18.8%
max24_curriculum123main_max24_curriculum_seed123paired_heldout_L246421.9%21.9%81.8%3.1%3.1%
max24_curriculum456main_max24_curriculum_seed456standard_L24641.6%0.0%71.8%
max24_curriculum456main_max24_curriculum_seed456paraphrase_L24641.6%0.0%73.0%
max24_curriculum456main_max24_curriculum_seed456heldout_L24640.0%0.0%74.6%
max24_curriculum456main_max24_curriculum_seed456paired_L24640.0%0.0%74.1%0.0%0.0%
max24_curriculum456main_max24_curriculum_seed456paired_heldout_L24640.0%0.0%73.0%0.0%0.0%
max24_curriculum789main_max24_curriculum_seed789standard_L246417.2%14.1%84.6%
max24_curriculum789main_max24_curriculum_seed789paraphrase_L246498.4%98.4%99.9%
max24_curriculum789main_max24_curriculum_seed789heldout_L246492.2%92.2%98.7%
max24_curriculum789main_max24_curriculum_seed789paired_L246460.9%59.4%92.8%21.9%18.8%
max24_curriculum789main_max24_curriculum_seed789paired_heldout_L246446.9%45.3%89.9%21.9%21.9%
max24_no_curriculum123main_max24_no_curriculum_seed123standard_L24640.0%0.0%62.2%
max24_no_curriculum123main_max24_no_curriculum_seed123paraphrase_L246446.9%46.9%78.8%
max24_no_curriculum123main_max24_no_curriculum_seed123heldout_L246410.9%10.9%66.1%
max24_no_curriculum123main_max24_no_curriculum_seed123paired_L246423.4%20.3%71.4%3.1%0.0%
max24_no_curriculum123main_max24_no_curriculum_seed123paired_heldout_L24644.7%4.7%67.2%0.0%0.0%
max24_no_curriculum456main_max24_no_curriculum_seed456standard_L24640.0%0.0%79.0%
max24_no_curriculum456main_max24_no_curriculum_seed456paraphrase_L2464100.0%100.0%100.0%
max24_no_curriculum456main_max24_no_curriculum_seed456heldout_L246437.5%37.5%79.2%
max24_no_curriculum456main_max24_no_curriculum_seed456paired_L246450.0%50.0%89.2%3.1%3.1%
max24_no_curriculum456main_max24_no_curriculum_seed456paired_heldout_L246425.0%25.0%83.3%0.0%0.0%
max24_no_curriculum789main_max24_no_curriculum_seed789standard_L24641.6%0.0%68.8%
max24_no_curriculum789main_max24_no_curriculum_seed789paraphrase_L24640.0%0.0%77.5%
max24_no_curriculum789main_max24_no_curriculum_seed789heldout_L24640.0%0.0%70.8%
max24_no_curriculum789main_max24_no_curriculum_seed789paired_L24640.0%0.0%73.3%0.0%0.0%
max24_no_curriculum789main_max24_no_curriculum_seed789paired_heldout_L24640.0%0.0%66.0%0.0%0.0%

Figures

mean_accuracy_by_split.png

standard_accuracy_by_seed.png

per_seed_accuracy_heatmap.png

final_accuracy_by_length.png

training_loss_by_seed.png

training_state_accuracy_by_seed.png

Interpretation

On the standard length-24 split, the strongest mean arm is expand_copy at 43.2% with 40.9 percentage points of seed standard deviation. The decisive criterion is whether the arm ranking remains stable across seeds and whether any arm's seed spread is large enough to make a single-seed conclusion unreliable. The observed spread is large enough that no single seed supports a stable attribution claim. Copied expansion has the best mean in this seed set, but it ranges from complete standard-L24 failure to strong performance. Full-width curriculum is also unstable, and no-curriculum training is especially weak on standard-L24 despite sometimes doing well on other wording splits. A second conclusion is that partial execution is not the bottleneck: every arm recovers long state prefixes far more often than it recovers exact length-24 programs. The remaining failure is late-step/global program consistency, not the absence of local operator knowledge.

Artifacts

Experiment log 4

Show the running log (4 entries, 2026-06-24)

2026-06-24

  • Created a fresh standalone multi-seed reattribution experiment directory.
  • Core question: does the executable latent compiler's length-24 performance remain stable across random seeds, and which controlled arm has the best mean behavior?

  • Result-bearing arms:

    • max24_curriculum
    • expand_copy
    • max24_no_curriculum
  • Matched seed set planned for main: 123,456,789.
  • Large checkpoints are configured under /workspace/large_artifacts/qwen_compiler_multiseed_reattribution/checkpoints.

  • Planned process:

    • Smoke: one tiny run to validate the forked trainer, seed columns, report generation, and artifact paths.
    • Pilot: reduced two-seed, three-arm grid to validate aggregation and charts before the long run.
    • Main: three arms times three seeds, then final Markdown and HTML reports with variance charts.

Smoke

  • Command: python src/run_multiseed_suite.py --phase smoke --seeds 123,456,789 --arms max24_curriculum,expand_copy,max24_no_curriculum
  • Completed run: smoke_max24_curriculum_seed123
  • Outcome: passed.

  • Verified:

    • The forked trainer loads Qwen/Qwen3-4B, attaches QLoRA, trains through staged max-24 curriculum, and writes run CSV/JSON outputs.
    • seed is present in run metadata.
    • The analysis script writes both Markdown and HTML reports.
    • Large artifact path is configured separately; smoke used --save_checkpoints 0, so no large files were produced.

Pilot

  • First pilot attempt caught a runner bug before any result-bearing run: the reduced pilot override supplied three stage-step counts to the one-stage max24_no_curriculum arm.
  • Fix: preserve each arm's stage structure in the runner; one-stage arms now receive a single reduced pilot budget.
  • Corrected pilot command: python src/run_multiseed_suite.py --phase pilot --seeds 123,456,789 --arms max24_curriculum,expand_copy,max24_no_curriculum

  • Completed runs:

    • pilot_max24_curriculum_seed123
    • pilot_expand_copy_seed123
    • pilot_max24_no_curriculum_seed123
    • pilot_max24_curriculum_seed456
    • pilot_expand_copy_seed456
    • pilot_max24_no_curriculum_seed456
  • Outcome: passed after the runner fix.

  • Verified:

    • The analyzer aggregates by arm and seed.
    • Diagnostic Markdown/HTML reports are generated.
    • Figures are generated: mean accuracy with seed spread, per-seed heatmap, standard accuracy by seed, length curve, training loss, and training state accuracy.
  • Interpretation: pilot rows are diagnostic only and are not result-bearing.

Main

  • Planned command: python src/run_multiseed_suite.py --phase main --seeds 123,456,789 --arms max24_curriculum,expand_copy,max24_no_curriculum
  • Main configuration: Qwen/Qwen3-4B QLoRA, 512-wide compiler, LoRA rank 8, batch 8, 512 train examples, 64 examples per single-template split, 32 paired examples per paired split.
  • Completed command: python src/run_multiseed_suite.py --phase main --seeds 123,456,789 --arms max24_curriculum,expand_copy,max24_no_curriculum

  • Completed runs:

    • main_max24_curriculum_seed123
    • main_expand_copy_seed123
    • main_max24_no_curriculum_seed123
    • main_max24_curriculum_seed456
    • main_expand_copy_seed456
    • main_max24_no_curriculum_seed456
    • main_max24_curriculum_seed789
    • main_expand_copy_seed789
    • main_max24_no_curriculum_seed789
  • Outcome: all nine result-bearing runs completed and the analysis step passed.

  • Final standard-L24 executable accuracy, mean +/- seed std:

    • expand_copy: 43.2% +/- 40.9, range 0.0% to 81.2%.
    • max24_curriculum: 14.1% +/- 11.3, range 1.6% to 23.4%.
    • max24_no_curriculum: 0.5% +/- 0.9, range 0.0% to 1.6%.

  • Final paired-L24 executable accuracy, mean +/- seed std:

    • expand_copy: 60.4% +/- 30.9.
    • max24_curriculum: 40.1% +/- 34.7.
    • max24_no_curriculum: 24.5% +/- 25.0.

  • Key interpretation:

    • Copied expansion has the best mean in this seed set, but the seed spread is very large.
    • No-curriculum training is weak on standard L24 even when some wording splits score much higher.
    • High state-prefix recovery does not reliably translate into exact length-24 program recovery.
    • Single-seed results are not sufficient for attribution in this harness.

  • Reports:

    • reports/qwen_compiler_multiseed_reattribution_report.md
    • reports/qwen_compiler_multiseed_reattribution_report.html
    • reports/checkpoint_manifest_all.csv

Figures 6

final accuracy by length
final accuracy by length · reports/figures/
mean accuracy by split
mean accuracy by split · reports/figures/
per seed accuracy heatmap
per seed accuracy heatmap · reports/figures/
standard accuracy by seed
standard accuracy by seed · reports/figures/
training loss by seed
training loss by seed · reports/figures/
training state accuracy by seed
training state accuracy by seed · reports/figures/

Data files 24

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

9 more result files not shown here — browse the full folder on GitHub.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗