Qwen Extrapolation-Bound ABI
The one idea you need
The model acts as a translator: it turns a plain-language request into a short program that a simple step-by-step machine then runs. It only ever practiced stringing together up to three steps, yet it correctly writes and runs sixteen-step programs far past its practice length.
The question
If you only train a small model to string together up to three steps, can it still correctly write and run procedures four to five times longer?
What we found
Yes. A model trained only on procedures up to three steps long reliably writes correct sixteen-step procedures, with accuracy climbing from 61% under single-step training to a perfect 100%. Surprisingly, adding longer training examples out to eight steps added nothing and slightly hurt, slipping to 89%. Short practice generalizes several times deeper than you would expect.
Why it matters
When training a small model to emit runnable programs or tool chains, skip collecting long multi-step examples. Short compositional practice transfers to procedures several times deeper, so spend your data budget on task variety, not example length.
On this page
Results at a glance 4
How to read
Bars show the share of long programs the model got right. The four groups along the bottom are training diets, from single-step-only up to eight-step examples, with separate bars for twelve-step and sixteen-step tests. Taller is better.
Takeaway → Success leaps between single-step and up-to-three-step training, reaching the top; the two deeper diets are no taller, so extra training depth added nothing.
Data table
| training curriculum (max composed depth) | depth 12 | depth 16 |
|---|---|---|
| atomic only (d1) | 66.7% | 61.1% |
| up to depth 3 | 94.4% | 100% |
| up to depth 6 | 100% | 100% |
| up to depth 8 | 88.9% | 88.9% |
Numbers from analysis/summary_by_arm.csv
Technical framing
Deep-composition accuracy by training curriculum (constrained decoding) — Training on compositions only up to depth 3 already yields 94-100% at depths 12-16; deeper curricula add nothing (depth 8 even hurts).
How to read
Each line is one training diet. The horizontal axis is how many steps the test procedure has; the vertical axis is the share solved correctly. Higher and flatter is better.
Takeaway → Only the single-step-trained line droops as procedures lengthen; every line that saw even short compositions stays flat and high out to sixteen steps.
Data table
| evaluation composition depth | atomic only (d1) | up to depth 3 | up to depth 6 | up to depth 8 |
|---|---|---|---|---|
| 1 | 88.9% | 100% | 100% | 100% |
| 3 | 72.2% | 88.9% | 94.4% | 83.3% |
| 6 | 72.2% | 83.3% | 94.4% | 83.3% |
| 8 | 72.2% | 100% | 94.4% | 94.4% |
| 12 | 66.7% | 94.4% | 100% | 88.9% |
| 16 | 61.1% | 100% | 100% | 88.9% |
Numbers from analysis/summary_by_arm.csv
Technical framing
Execution accuracy across evaluation depth, per curriculum — Only atomic-only training decays with depth; any compositional supervision (even max depth 3) extrapolates flat out to depth 16.
How to read
For sixteen-step tasks, paired bars per training diet compare the model writing freely against writing with output-format guardrails that keep the program well-formed. Taller is better.
Takeaway → The guardrail bars tower over the free-writing bars everywhere, showing the model already knew the right steps and messy formatting, not reasoning, was the bottleneck.
Data table
| training curriculum (max composed depth) | constrained | free greedy |
|---|---|---|
| atomic only (d1) | 61.1% | 44.4% |
| up to depth 3 | 100% | 72.2% |
| up to depth 6 | 100% | 61.1% |
| up to depth 8 | 88.9% | 55.6% |
Numbers from report table (reports/qwen_extrapolation_bound_abi_report.md)
Technical framing
Constrained vs free decoding at depth 16 — Finite-state constrained decoding adds 17-39 points over free generation at depth 16; the semantics are there, syntax was the bottleneck.
How to read
For deep tasks, bars split every answer into exactly right, a valid alternative that still works, wrong step order or choice, and wrong number used, grouped by training diet. Taller correct bars and shorter error bars are better.
Takeaway → Single-step training gave wrong-step-order answers about a third of the time; any short compositional training shrank that to a sliver and made most answers exactly right.
Data table
| outcome category | atomic only (d1) | up to depth 3 | up to depth 6 | up to depth 8 |
|---|---|---|---|---|
| correct exact | 47.2% | 88.9% | 88.9% | 86.1% |
| correct semantic variant | 16.7% | 8.3% | 11.1% | 2.8% |
| wrong op order/choice | 30.6% | 2.8% | 0% | 8.3% |
| wrong constant/argument | 5.6% | 0% | 0% | 2.8% |
Numbers from report table (reports/qwen_extrapolation_bound_abi_report.md)
Technical framing
Failure taxonomy at depths 12-16 (constrained decoding) — Compositional supervision cuts wrong-operation-order errors from 31% to at most 8%; atomic-only training composes valid but semantically wrong programs 31% of the time.
In the author’s words from the Report · “Abstract”
This standalone experiment measures how far a constrained stack-ABI compiler extrapolates beyond its maximum supervised composition depth. The model emits a program; a deterministic interpreter executes it. The central question is whether shallow composition training is enough for long procedures, or whether the curriculum must reach roughly half the deployment depth.
Overview
Standalone Track 1 experiment measuring how far a constrained stack-ABI compiler extrapolates past its maximum supervised composition depth.
Question
The compiler can learn composed procedures, but a large ABI corpus needs a practical curriculum rule. This experiment tests whether training up to depth 3 is enough for depth 12 and 16, or whether the curriculum must include deeper composed procedures such as depth 6 or 8.
Curricula
atomic_d1: one-operation tasks only.mix_d1_d2_d3: depths 1, 2, and 3.mix_d1_to_d6: depths 1, 2, 3, 4, and 6.mix_d1_to_d8: depths 1, 2, 3, 4, 6, and 8.
Evaluation
- Standard depth sweep: 1, 3, 6, 8, 12, and 16.
- Wording-shift sweep: 8, 12, and 16.
- Decoder arms: free greedy stack generation and finite-state constrained stack generation.
- Gold ABI sanity arm: execute the reference program through the interpreter.
Primary Metrics
- Constrained external execution accuracy at depths 12 and 16.
- Correct-given-valid accuracy, since constrained decoding should keep validity near 100%.
- Failure taxonomy on depth-12/depth-16 constrained outputs.
- Free versus constrained execution to separate syntax/format effects from semantic composition effects.
Artifacts
- Source:
src/qwen_extrapolation_bound_abi.py - Metrics and details:
analysis/ - Reports:
reports/ - Large checkpoints:
/workspace/large_artifacts/qwen_extrapolation_bound_abi/checkpoints
Report
Rendered from reports/qwen_extrapolation_bound_abi_report.md
Abstract
This standalone experiment measures how far a constrained stack-ABI compiler extrapolates beyond its maximum supervised composition depth. The model emits a program; a deterministic interpreter executes it. The central question is whether shallow composition training is enough for long procedures, or whether the curriculum must reach roughly half the deployment depth.
Method
Four QLoRA adapters are trained with the same ABI target and different maximum curriculum depths:
atomic_d1: only one-operation tasks.mix_d1_d2_d3: a balanced mix of one-, two-, and three-operation tasks.mix_d1_to_d6: a balanced mix of depths 1, 2, 3, 4, and 6.mix_d1_to_d8: a balanced mix of depths 1, 2, 3, 4, 6, and 8.
Evaluation sweeps standard prompts at depths 1, 3, 6, 8, 12, and 16, plus wording-shifted prompts at depths 8, 12, and 16. Each trained adapter is evaluated with free greedy decoding and finite-state constrained decoding. A gold ABI sanity arm checks the interpreter.
The primary criterion is constrained external execution accuracy at depths 12 and 16. Valid-program rate alone is not a success metric; a deeper curriculum must reduce valid-but-wrong composition errors, not only improve syntax.
Run Configuration
- Primary suite:
main. - Seeds:
101,202,303. - Evaluation rows:
243metric rows,1458scored examples across curricula and decoder arms. - QLoRA update steps per adapter:
20. - Large adapters are stored outside the experiment tree.
Primary Results
- Constrained depth-6:
atomic_d172.2%;mix_d1_d2_d383.3%;mix_d1_to_d694.4%;mix_d1_to_d883.3%. Depth-6 deltas: max3 minus atomic 11.1%; max6 minus max3 11.1%; max8 minus max6 -11.1%. - Constrained depth-8:
atomic_d172.2%;mix_d1_d2_d3100.0%;mix_d1_to_d694.4%;mix_d1_to_d894.4%. Depth-8 deltas: max3 minus atomic 27.8%; max6 minus max3 -5.6%; max8 minus max6 0.0%. - Constrained depth-12:
atomic_d166.7%;mix_d1_d2_d394.4%;mix_d1_to_d6100.0%;mix_d1_to_d888.9%. Depth-12 deltas: max3 minus atomic 27.8%; max6 minus max3 5.6%; max8 minus max6 -11.1%. - Constrained depth-16:
atomic_d161.1%;mix_d1_d2_d3100.0%;mix_d1_to_d6100.0%;mix_d1_to_d888.9%. Depth-16 deltas: max3 minus atomic 38.9%; max6 minus max3 0.0%; max8 minus max6 -11.1%. - Template depth-8: atomic 61.1%; max3 83.3%; max6 72.2%; max8 77.8%.
- Template depth-12: atomic 72.2%; max3 83.3%; max6 88.9%; max8 88.9%.
- Template depth-16: atomic 72.2%; max3 94.4%; max6 88.9%; max8 88.9%.
- Gold ABI depth-16 sanity: 100.0% execution and 100.0% validity.
- At depth 16, max-depth-8 beats max-depth-3 on
0/3matched seeds; mean per-seed delta -11.1%.
| train_target | arm | split | depth | runs | n_total | exec_accuracy_mean | exec_accuracy_std | valid_exec_rate_mean | correct_given_valid_mean | divergence_rate_mean | constrained_only_rate_mean | free_only_rate_mean | mean_attempts_mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| atomic_d1 | program_stack_constrained | eval_comp_d12 | 12 | 3 | 18 | 66.7% | 0.0% | 100.0% | 66.7% | 61.1% | 22.2% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_comp_d12 | 12 | 3 | 18 | 44.4% | 9.6% | 55.6% | 80.6% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_constrained | eval_comp_d12 | 12 | 3 | 18 | 94.4% | 9.6% | 100.0% | 94.4% | 27.8% | 16.7% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_comp_d12 | 12 | 3 | 18 | 77.8% | 9.6% | 83.3% | 94.4% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d6 | program_stack_constrained | eval_comp_d12 | 12 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | 27.8% | 22.2% | 0.0% | 1.00 |
| mix_d1_to_d6 | program_stack_free | eval_comp_d12 | 12 | 3 | 18 | 77.8% | 9.6% | 83.3% | 94.4% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d8 | program_stack_constrained | eval_comp_d12 | 12 | 3 | 18 | 88.9% | 19.2% | 100.0% | 88.9% | 38.9% | 27.8% | 0.0% | 1.00 |
| mix_d1_to_d8 | program_stack_free | eval_comp_d12 | 12 | 3 | 18 | 61.1% | 25.5% | 72.2% | 83.3% | n/a | n/a | n/a | 1.00 |
| oracle | gold_abi_constrained | eval_comp_d12 | 12 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| atomic_d1 | program_stack_constrained | eval_comp_d16 | 16 | 3 | 18 | 61.1% | 9.6% | 100.0% | 61.1% | 55.6% | 16.7% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_comp_d16 | 16 | 3 | 18 | 44.4% | 9.6% | 61.1% | 72.2% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_constrained | eval_comp_d16 | 16 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | 38.9% | 27.8% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_comp_d16 | 16 | 3 | 18 | 72.2% | 9.6% | 83.3% | 87.8% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d6 | program_stack_constrained | eval_comp_d16 | 16 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | 44.4% | 38.9% | 0.0% | 1.00 |
| mix_d1_to_d6 | program_stack_free | eval_comp_d16 | 16 | 3 | 18 | 61.1% | 25.5% | 77.8% | 76.7% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d8 | program_stack_constrained | eval_comp_d16 | 16 | 3 | 18 | 88.9% | 19.2% | 100.0% | 88.9% | 44.4% | 33.3% | 0.0% | 1.00 |
| mix_d1_to_d8 | program_stack_free | eval_comp_d16 | 16 | 3 | 18 | 55.6% | 9.6% | 77.8% | 71.7% | n/a | n/a | n/a | 1.00 |
| oracle | gold_abi_constrained | eval_comp_d16 | 16 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| atomic_d1 | program_stack_constrained | eval_comp_d6 | 6 | 3 | 18 | 72.2% | 9.6% | 100.0% | 72.2% | 50.0% | 22.2% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_comp_d6 | 6 | 3 | 18 | 50.0% | 0.0% | 50.0% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_constrained | eval_comp_d6 | 6 | 3 | 18 | 83.3% | 0.0% | 100.0% | 83.3% | 16.7% | 0.0% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_comp_d6 | 6 | 3 | 18 | 83.3% | 0.0% | 83.3% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d6 | program_stack_constrained | eval_comp_d6 | 6 | 3 | 18 | 94.4% | 9.6% | 100.0% | 94.4% | 22.2% | 22.2% | 0.0% | 1.00 |
| mix_d1_to_d6 | program_stack_free | eval_comp_d6 | 6 | 3 | 18 | 72.2% | 19.2% | 77.8% | 91.7% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d8 | program_stack_constrained | eval_comp_d6 | 6 | 3 | 18 | 83.3% | 0.0% | 100.0% | 83.3% | 33.3% | 11.1% | 0.0% | 1.00 |
| mix_d1_to_d8 | program_stack_free | eval_comp_d6 | 6 | 3 | 18 | 72.2% | 9.6% | 72.2% | 100.0% | n/a | n/a | n/a | 1.00 |
| oracle | gold_abi_constrained | eval_comp_d6 | 6 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| atomic_d1 | program_stack_constrained | eval_comp_d8 | 8 | 3 | 18 | 72.2% | 25.5% | 100.0% | 72.2% | 44.4% | 22.2% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_comp_d8 | 8 | 3 | 18 | 50.0% | 0.0% | 61.1% | 83.3% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_constrained | eval_comp_d8 | 8 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | 16.7% | 11.1% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_comp_d8 | 8 | 3 | 18 | 88.9% | 9.6% | 88.9% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d6 | program_stack_constrained | eval_comp_d8 | 8 | 3 | 18 | 94.4% | 9.6% | 100.0% | 94.4% | 11.1% | 11.1% | 0.0% | 1.00 |
| mix_d1_to_d6 | program_stack_free | eval_comp_d8 | 8 | 3 | 18 | 83.3% | 16.7% | 94.4% | 88.9% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d8 | program_stack_constrained | eval_comp_d8 | 8 | 3 | 18 | 94.4% | 9.6% | 100.0% | 94.4% | 16.7% | 11.1% | 0.0% | 1.00 |
| mix_d1_to_d8 | program_stack_free | eval_comp_d8 | 8 | 3 | 18 | 83.3% | 16.7% | 88.9% | 93.3% | n/a | n/a | n/a | 1.00 |
| oracle | gold_abi_constrained | eval_comp_d8 | 8 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| atomic_d1 | program_stack_constrained | eval_template_d12 | 12 | 3 | 18 | 72.2% | 9.6% | 100.0% | 72.2% | 72.2% | 44.4% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_template_d12 | 12 | 3 | 18 | 27.8% | 19.2% | 33.3% | 83.3% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_constrained | eval_template_d12 | 12 | 3 | 18 | 83.3% | 0.0% | 100.0% | 83.3% | 66.7% | 50.0% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_template_d12 | 12 | 3 | 18 | 33.3% | 0.0% | 33.3% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d6 | program_stack_constrained | eval_template_d12 | 12 | 3 | 18 | 88.9% | 9.6% | 100.0% | 88.9% | 66.7% | 55.6% | 5.6% | 1.00 |
| mix_d1_to_d6 | program_stack_free | eval_template_d12 | 12 | 3 | 18 | 38.9% | 34.7% | 38.9% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d8 | program_stack_constrained | eval_template_d12 | 12 | 3 | 18 | 88.9% | 19.2% | 100.0% | 88.9% | 66.7% | 55.6% | 0.0% | 1.00 |
| mix_d1_to_d8 | program_stack_free | eval_template_d12 | 12 | 3 | 18 | 33.3% | 16.7% | 38.9% | 83.3% | n/a | n/a | n/a | 1.00 |
| oracle | gold_abi_constrained | eval_template_d12 | 12 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| atomic_d1 | program_stack_constrained | eval_template_d16 | 16 | 3 | 18 | 72.2% | 9.6% | 100.0% | 72.2% | 77.8% | 44.4% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_template_d16 | 16 | 3 | 18 | 27.8% | 9.6% | 38.9% | 83.3% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_constrained | eval_template_d16 | 16 | 3 | 18 | 94.4% | 9.6% | 100.0% | 94.4% | 72.2% | 66.7% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_template_d16 | 16 | 3 | 18 | 27.8% | 9.6% | 33.3% | 83.3% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d6 | program_stack_constrained | eval_template_d16 | 16 | 3 | 18 | 88.9% | 9.6% | 100.0% | 88.9% | 55.6% | 44.4% | 0.0% | 1.00 |
| mix_d1_to_d6 | program_stack_free | eval_template_d16 | 16 | 3 | 18 | 44.4% | 19.2% | 50.0% | 88.9% | n/a | n/a | n/a | 1.00 |
| mix_d1_to_d8 | program_stack_constrained | eval_template_d16 | 16 | 3 | 18 | 88.9% | 19.2% | 100.0% | 88.9% | 72.2% | 55.6% | 0.0% | 1.00 |
| mix_d1_to_d8 | program_stack_free | eval_template_d16 | 16 | 3 | 18 | 33.3% | 16.7% | 38.9% | 83.3% | n/a | n/a | n/a | 1.00 |
| oracle | gold_abi_constrained | eval_template_d16 | 16 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
Interpretation
This experiment identifies the useful extrapolation span of supervised composition curricula. If max-depth-3 training holds through depth 12 or 16, a large ABI corpus can stay shallow. If max-depth-6 or max-depth-8 training is needed for depth-12 or depth-16 reliability, the corpus should include composed procedures up to roughly half the expected deployment depth. The main result is that depth-3 compositional supervision is sufficient for this depth range: standard depth-16 constrained execution rises from 61.1% with atomic-only training to 100.0% with mix_d1_d2_d3. Training through depth 6 matches it at 100.0%, while training through depth 8 falls to 88.9%. The same pattern holds on the wording-shifted endpoint: mix_d1_d2_d3 reaches 94.4% at template depth 16, versus 88.9% for max-depth-6 and 88.9% for max-depth-8. Because constrained validity is 100% throughout the trained arms, these gains are reductions in valid-but-wrong composition errors rather than syntax improvements. At depth 16, max-depth-8 training changes execution by -11.1% and correct-given-valid by -11.1% relative to max-depth-3 training. For atomic_d1 constrained decoding on depth-12/depth-16 composition splits, procedures break down as: correct_exact 47.2%, wrong_op_order_or_choice 30.6%, correct_semantic_variant 16.7%, wrong_constant_or_argument 5.6%. For mix_d1_d2_d3 constrained decoding on depth-12/depth-16 composition splits, procedures break down as: correct_exact 88.9%, correct_semantic_variant 8.3%, wrong_op_order_or_choice 2.8%. For mix_d1_to_d6 constrained decoding on depth-12/depth-16 composition splits, procedures break down as: correct_exact 88.9%, correct_semantic_variant 11.1%. For mix_d1_to_d8 constrained decoding on depth-12/depth-16 composition splits, procedures break down as: correct_exact 86.1%, wrong_op_order_or_choice 8.3%, correct_semantic_variant 2.8%, wrong_constant_or_argument 2.8%.
Limitations
This experiment tests compilation over a fixed known primitive library. It does not test invention of operations outside the ABI. The finite-state decoder is tied to the task schema and uses task-visible constants and type information. Composed examples in the curricula are supervised generated data, so gains should be read as curriculum effects rather than unsupervised discovery.
Artifacts
- Metrics:
analysis/summary_by_arm.csvandanalysis/all_metrics.csv - Details:
analysis/all_details.csv - Training logs:
analysis/all_train_logs.csv - Checkpoints:
/workspace/large_artifacts/qwen_extrapolation_bound_abi/checkpoints
Experiment log 4
Show the running log (4 entries)
Setup
- Created fresh experiment directory:
/workspace/experiments/qwen_extrapolation_bound_abi - Large artifacts directory:
/workspace/large_artifacts/qwen_extrapolation_bound_abi - Core question: how far constrained ABI compilation extrapolates beyond the maximum supervised composition depth.
- Report format: standalone Markdown and HTML with plots.
Run smoke_v1
- Started: 2026-06-26 20:32:31 UTC
- Suite:
smoke - Model:
Qwen/Qwen3-4B - Seeds:
101 - Training targets:
atomic_d1,mix_d1_d2_d3,mix_d1_to_d6,mix_d1_to_d8 - Training examples per seed:
24 - Eval examples per split:
2 - Eval splits:
eval_indist_d1,eval_comp_d16,eval_template_d16 - Steps:
1 - Resample attempts:
3
Completed smoke_v1 in 462.5s.
- Metric rows: 27
- Detail rows: 54
- Training log rows: 4
Run pilot_v1
- Started: 2026-06-26 20:40:44 UTC
- Suite:
pilot - Model:
Qwen/Qwen3-4B - Seeds:
101 - Training targets:
atomic_d1,mix_d1_d2_d3,mix_d1_to_d6,mix_d1_to_d8 - Training examples per seed:
96 - Eval examples per split:
3 - Eval splits:
eval_indist_d1,eval_comp_d3,eval_comp_d6,eval_comp_d8,eval_comp_d12,eval_comp_d16,eval_template_d8,eval_template_d12,eval_template_d16 - Steps:
8 - Resample attempts:
3
Completed pilot_v1 in 1210.6s.
- Metric rows: 81
- Detail rows: 243
- Training log rows: 20
- Sanity: gold ABI constrained execution and validity were 100% on every pilot split, including depth 16.
- Pilot read: constrained validity stayed at 100%; depth-12/depth-16 execution remained nonzero across curricula, so the main run can measure the extrapolation bound rather than only observing collapse.
Run main_v1
- Started: 2026-06-26 21:01:36 UTC
- Suite:
main - Model:
Qwen/Qwen3-4B - Seeds:
101,202,303 - Training targets:
atomic_d1,mix_d1_d2_d3,mix_d1_to_d6,mix_d1_to_d8 - Training examples per seed:
180 - Eval examples per split:
6 - Eval splits:
eval_indist_d1,eval_comp_d3,eval_comp_d6,eval_comp_d8,eval_comp_d12,eval_comp_d16,eval_template_d8,eval_template_d12,eval_template_d16 - Steps:
20 - Resample attempts:
3
Completed main_v1 in 4804.4s.
- Metric rows: 243
- Detail rows: 1458
- Training log rows: 60
Main read:
- Atomic-only constrained depth-16 execution: 61.1% standard, 72.2% template shift.
- Depth-3 curriculum constrained depth-16 execution: 100.0% standard, 94.4% template shift.
- Depth-6 curriculum constrained depth-16 execution: 100.0% standard, 88.9% template shift.
- Depth-8 curriculum constrained depth-16 execution: 88.9% standard, 88.9% template shift.
- Constrained validity was 100% throughout trained arms, so gains are correct-given-valid composition gains rather than syntax gains.
- Practical corpus implication: include shallow composed examples through depth 3 first; deeper examples are not automatically beneficial at this budget and may add variance.
Figures 6
Data files 5
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv90 kBanalysis/summary_by_arm.csv73 kBruns/main_v1/metrics.csv67 kBruns/pilot_v1/metrics.csv23 kBruns/smoke_v1/metrics.csv5.6 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.