Qwen Compositional Curriculum ABI
The one idea you need
Think of the model as writing a numbered recipe that a robot cook then follows. Trained only on one-action recipes, it fumbles the order of an eight-step dish. Mixing in a handful of two- and three-action recipes teaches it to sequence long recipes correctly.
The question
Does showing the model a few short multi-step examples help it correctly turn long procedures into working programs, instead of putting the steps in the wrong order?
What we found
Yes. Adding a few two- and three-step examples lifted correct answers on unseen six-step problems from 72% to 83%, and on eight-step problems from 78% to 89%. But adding only two-step examples did nothing (72% stayed 72%) — you must reach at least one step past your shortest hard case. Every program already followed the format's rules, so this fixed step ordering, not formatting.
Why it matters
When training a small model to write programs or tool-call sequences, forcing a valid output format is not enough for correct results. Add a few genuine multi-step training examples, reaching deeper than your shortest failing case, to teach correct step order.
On this page
Results at a glance 4
How to read
Two groups, six-step and eight-step unseen problems; each has three bars: trained on single-step only, on a one-and-two-step mix, and on a one-two-and-three-step mix. Height is share solved correctly by the external checker; taller is better.
Takeaway → Only the one-two-and-three-step mix rises in both groups (to 83% and 89%); the other two mixes stall near 72% (atomic-only edges up to 78% at eight steps), so shallow practice alone does not help.
Data table
| held-out procedure depth | atomic only (d1) | mix d1+d2 | mix d1+d2+d3 |
|---|---|---|---|
| depth 6 | 72.2% | 72.2% | 83.3% |
| depth 8 | 77.8% | 72.2% | 88.9% |
Numbers from analysis/summary_by_arm.csv (main suite, program_stack_constrained, eval_comp_d6/d8)
Technical framing
Held-out deep composition: execution accuracy by training curriculum (constrained decoding) — Adding depth-2/3 composition examples lifts held-out depth-6 and depth-8 execution by 11.1 points over atomic-only training.
How to read
Three lines, one per training mix (single-step only, one-and-two-step mix, one-two-and-three-step mix), track the share of correct answers as problem length grows from two to eight steps; higher lines are better.
Takeaway → Only the three-step-mix line stays high and climbs back to 89% at eight steps, while both other lines sag to about 72% by six steps.
Data table
| procedure depth | atomic only (d1) | mix d1+d2 | mix d1+d2+d3 |
|---|---|---|---|
| 2 | 88.9% | 83.3% | 88.9% |
| 3 | 77.8% | 88.9% | 77.8% |
| 4 | 72.2% | 83.3% | 83.3% |
| 6 | 72.2% | 72.2% | 83.3% |
| 8 | 77.8% | 72.2% | 88.9% |
Numbers from analysis/summary_by_arm.csv (main suite, program_stack_constrained, eval_comp splits)
Technical framing
Execution accuracy across procedure depth (constrained decoding) — Only the d1+d2+d3 curriculum recovers as depth grows, reaching 88.9% at depth 8; the other arms fall to 72.2% by depth 6.
How to read
On the longest problems, bars split every valid program into exactly correct, wrong step order or choice, and correct alternate route, grouped by training mix. A taller first bar and shorter middle bar are better.
Takeaway → The three-step mix has the biggest exactly-correct share (83%) and nearly halves wrong-order errors (25% to 14%), so it fixed real reasoning, not neatness.
Data table
| outcome of generated program | atomic only (d1) | mix d1+d2 | mix d1+d2+d3 |
|---|---|---|---|
| exactly correct | 61.1% | 63.9% | 83.3% |
| wrong op order/choice | 25% | 27.8% | 13.9% |
| correct semantic variant | 13.9% | 8.3% | 2.8% |
Numbers from report table (reports/qwen_compositional_curriculum_abi_report.md, Interpretation section)
Technical framing
Failure taxonomy at depths 6-8: where valid programs go wrong — The d1+d2+d3 curriculum nearly halves wrong-operation-order errors (25.0% to 13.9%), showing a semantic gain, not just syntax.
How to read
For eight-step problems, bars compare three ways of producing the program — format-guided decoding, retry-until-valid, and free generation — for two training mixes (single-step only and the three-step mix); taller is better.
Takeaway → Guided decoding is tallest for both mixes; the single best result pairs the three-step mix with guided decoding at 89%.
Data table
| decoding strategy | atomic only (d1) | mix d1+d2+d3 |
|---|---|---|
| constrained | 77.8% | 88.9% |
| resample-to-valid | 66.7% | 83.3% |
| free greedy | 61.1% | 83.3% |
Numbers from analysis/summary_by_arm.csv (main suite, eval_comp_d8 rows)
Technical framing
Decoder comparison at depth 8: constrained decoding beats free and resample — Constrained decoding adds accuracy on top of the curriculum; the best cell combines the d1+d2+d3 mix with finite-state decoding.
In the author’s words from the Report · “Abstract”
This standalone experiment tests whether shallow composed-procedure supervision makes a small language model a more reliable compiler into a deterministic stack ABI. The model never needs to execute the procedure itself; success is measured by executing the generated ABI program in an external interpreter.
Overview
Standalone Track 1 experiment testing whether shallow composed-procedure supervision improves a Qwen stack-ABI compiler.
Question
Constrained ABI decoding can enforce valid syntax, but the remaining hard errors are valid programs that execute to the wrong result. This experiment tests whether those errors shrink when training includes a small amount of composed procedures instead of only atomic one-operation tasks.
Arms
atomic_d1: train only on one-operation procedures.mix_d1_d2: train on a balanced mix of one- and two-operation procedures.mix_d1_d2_d3: train on a balanced mix of one-, two-, and three-operation procedures.
Each trained adapter is evaluated with:
program_stack_free: greedy raw ABI generation.program_stack_constrained: finite-state constrained ABI decoding.program_stack_resample_valid: retry free decoding until the ABI parses, within a fixed attempt budget.gold_abi_constrained: interpreter sanity check using the gold ABI.
Primary Metrics
- External execution accuracy at held-out depths 4, 6, and 8.
- Correct-given-valid accuracy, to distinguish semantic gains from syntax-only gains.
- Template-shift execution accuracy at depths 6 and 8.
- Failure taxonomy over valid/wrong, invalid, wrong operation, and wrong argument cases.
Artifacts
- Source:
src/qwen_compositional_curriculum_abi.py - Metrics and details:
analysis/ - Reports:
reports/ - Large checkpoints:
/workspace/large_artifacts/qwen_compositional_curriculum_abi/checkpoints
Report
Rendered from reports/qwen_compositional_curriculum_abi_report.md
Abstract
This standalone experiment tests whether shallow composed-procedure supervision makes a small language model a more reliable compiler into a deterministic stack ABI. The model never needs to execute the procedure itself; success is measured by executing the generated ABI program in an external interpreter.
Method
Three QLoRA adapters are trained with the same ABI target and different curriculum depths:
atomic_d1: only one-operation tasks.mix_d1_d2: a balanced mix of one- and two-operation tasks.mix_d1_d2_d3: a balanced mix of one-, two-, and three-operation tasks.
Evaluation sweeps procedure depth 1, 2, 3, 4, 6, and 8. Depths above a curriculum's maximum are held-out composition tests. Separate wording-shift splits test whether the compiler is robust to surface phrasing. Each trained adapter is evaluated with free greedy decoding, finite-state constrained decoding, and a resample-to-valid baseline. A gold ABI sanity arm checks the interpreter.
The primary criterion is external execution accuracy on held-out deeper depths, especially depth 6 and depth 8. Valid-program rate alone is not a success metric; a useful curriculum must reduce valid-but-wrong composition errors, not only improve syntax.
Run Configuration
- Primary suite:
main. - Seeds:
101,202,303. - Evaluation rows:
240metric rows,1440scored examples across curricula and decoder arms. - QLoRA update steps per adapter:
16. - Large adapters are stored outside the experiment tree.
Primary Results
- Constrained depth-6 execution:
atomic_d172.2%;mix_d1_d272.2%;mix_d1_d2_d383.3%. - Depth-6 gain from adding d2/d3 examples:
mix_d1_d2minus atomic 0.0%;mix_d1_d2_d3minus atomic 11.1%. - Depth-6 correct-given-valid: atomic 72.2%; d1/d2 mix 72.2%; d1/d2/d3 mix 83.3%.
- Constrained depth-8 execution: atomic 77.8%; d1/d2/d3 mix 88.9%; delta 11.1%.
- Template-shift depth-6 constrained execution: atomic 83.3%; d1/d2/d3 mix 88.9%; delta 5.6%.
- Gold ABI depth-6 sanity: 100.0% execution and 100.0% validity.
- Depth-6 d1/d2/d3 curriculum beats atomic on
2/3matched seeds; mean per-seed delta 11.1%.
| train_target | arm | split | depth | runs | n_total | exec_accuracy_mean | exec_accuracy_std | valid_exec_rate_mean | correct_given_valid_mean | divergence_rate_mean | constrained_only_rate_mean | free_only_rate_mean | mean_attempts_mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| atomic_d1 | program_stack_constrained | eval_comp_d4 | 4 | 3 | 18 | 72.2% | 9.6% | 100.0% | 72.2% | 50.0% | 22.2% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_comp_d4 | 4 | 3 | 18 | 50.0% | 0.0% | 50.0% | 100.0% | n/a | n/a | n/a | 1.00 |
| atomic_d1 | program_stack_resample_valid | eval_comp_d4 | 4 | 3 | 18 | 50.0% | 0.0% | 61.1% | 83.3% | 11.1% | 0.0% | 0.0% | 1.44 |
| mix_d1_d2 | program_stack_constrained | eval_comp_d4 | 4 | 3 | 18 | 83.3% | 0.0% | 100.0% | 83.3% | 38.9% | 22.2% | 0.0% | 1.00 |
| mix_d1_d2 | program_stack_free | eval_comp_d4 | 4 | 3 | 18 | 61.1% | 19.2% | 61.1% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2 | program_stack_resample_valid | eval_comp_d4 | 4 | 3 | 18 | 61.1% | 19.2% | 61.1% | 100.0% | 0.0% | 0.0% | 0.0% | 1.39 |
| mix_d1_d2_d3 | program_stack_constrained | eval_comp_d4 | 4 | 3 | 18 | 83.3% | 16.7% | 100.0% | 83.3% | 22.2% | 5.6% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_comp_d4 | 4 | 3 | 18 | 77.8% | 19.2% | 77.8% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_resample_valid | eval_comp_d4 | 4 | 3 | 18 | 77.8% | 19.2% | 77.8% | 100.0% | 0.0% | 0.0% | 0.0% | 1.22 |
| oracle | gold_abi_constrained | eval_comp_d4 | 4 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| atomic_d1 | program_stack_constrained | eval_comp_d6 | 6 | 3 | 18 | 72.2% | 9.6% | 100.0% | 72.2% | 50.0% | 16.7% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_comp_d6 | 6 | 3 | 18 | 55.6% | 19.2% | 66.7% | 82.2% | n/a | n/a | n/a | 1.00 |
| atomic_d1 | program_stack_resample_valid | eval_comp_d6 | 6 | 3 | 18 | 55.6% | 25.5% | 61.1% | 88.9% | 16.7% | 5.6% | 5.6% | 1.39 |
| mix_d1_d2 | program_stack_constrained | eval_comp_d6 | 6 | 3 | 18 | 72.2% | 9.6% | 100.0% | 72.2% | 50.0% | 22.2% | 0.0% | 1.00 |
| mix_d1_d2 | program_stack_free | eval_comp_d6 | 6 | 3 | 18 | 50.0% | 16.7% | 61.1% | 83.3% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2 | program_stack_resample_valid | eval_comp_d6 | 6 | 3 | 18 | 50.0% | 16.7% | 61.1% | 83.3% | 5.6% | 0.0% | 0.0% | 1.39 |
| mix_d1_d2_d3 | program_stack_constrained | eval_comp_d6 | 6 | 3 | 18 | 83.3% | 0.0% | 100.0% | 83.3% | 27.8% | 11.1% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_comp_d6 | 6 | 3 | 18 | 72.2% | 9.6% | 72.2% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_resample_valid | eval_comp_d6 | 6 | 3 | 18 | 72.2% | 9.6% | 72.2% | 100.0% | 0.0% | 0.0% | 0.0% | 1.28 |
| oracle | gold_abi_constrained | eval_comp_d6 | 6 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| atomic_d1 | program_stack_constrained | eval_comp_d8 | 8 | 3 | 18 | 77.8% | 9.6% | 100.0% | 77.8% | 44.4% | 16.7% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_comp_d8 | 8 | 3 | 18 | 61.1% | 25.5% | 61.1% | 100.0% | n/a | n/a | n/a | 1.00 |
| atomic_d1 | program_stack_resample_valid | eval_comp_d8 | 8 | 3 | 18 | 66.7% | 28.9% | 66.7% | 100.0% | 5.6% | 5.6% | 0.0% | 1.39 |
| mix_d1_d2 | program_stack_constrained | eval_comp_d8 | 8 | 3 | 18 | 72.2% | 9.6% | 100.0% | 72.2% | 50.0% | 16.7% | 0.0% | 1.00 |
| mix_d1_d2 | program_stack_free | eval_comp_d8 | 8 | 3 | 18 | 55.6% | 9.6% | 55.6% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2 | program_stack_resample_valid | eval_comp_d8 | 8 | 3 | 18 | 55.6% | 9.6% | 55.6% | 100.0% | 0.0% | 0.0% | 0.0% | 1.44 |
| mix_d1_d2_d3 | program_stack_constrained | eval_comp_d8 | 8 | 3 | 18 | 88.9% | 9.6% | 100.0% | 88.9% | 22.2% | 5.6% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_comp_d8 | 8 | 3 | 18 | 83.3% | 16.7% | 83.3% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_resample_valid | eval_comp_d8 | 8 | 3 | 18 | 83.3% | 16.7% | 83.3% | 100.0% | 0.0% | 0.0% | 0.0% | 1.17 |
| oracle | gold_abi_constrained | eval_comp_d8 | 8 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| atomic_d1 | program_stack_constrained | eval_template_d6 | 6 | 3 | 18 | 83.3% | 16.7% | 100.0% | 83.3% | 61.1% | 44.4% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_template_d6 | 6 | 3 | 18 | 38.9% | 9.6% | 38.9% | 100.0% | n/a | n/a | n/a | 1.00 |
| atomic_d1 | program_stack_resample_valid | eval_template_d6 | 6 | 3 | 18 | 44.4% | 9.6% | 44.4% | 100.0% | 5.6% | 5.6% | 0.0% | 1.61 |
| mix_d1_d2 | program_stack_constrained | eval_template_d6 | 6 | 3 | 18 | 77.8% | 9.6% | 100.0% | 77.8% | 61.1% | 38.9% | 5.6% | 1.00 |
| mix_d1_d2 | program_stack_free | eval_template_d6 | 6 | 3 | 18 | 44.4% | 19.2% | 44.4% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2 | program_stack_resample_valid | eval_template_d6 | 6 | 3 | 18 | 44.4% | 19.2% | 44.4% | 100.0% | 0.0% | 0.0% | 0.0% | 1.56 |
| mix_d1_d2_d3 | program_stack_constrained | eval_template_d6 | 6 | 3 | 18 | 88.9% | 19.2% | 100.0% | 88.9% | 44.4% | 33.3% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_template_d6 | 6 | 3 | 18 | 55.6% | 25.5% | 55.6% | 100.0% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_resample_valid | eval_template_d6 | 6 | 3 | 18 | 61.1% | 19.2% | 61.1% | 100.0% | 5.6% | 5.6% | 0.0% | 1.44 |
| oracle | gold_abi_constrained | eval_template_d6 | 6 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| atomic_d1 | program_stack_constrained | eval_template_d8 | 8 | 3 | 18 | 77.8% | 9.6% | 100.0% | 77.8% | 50.0% | 27.8% | 0.0% | 1.00 |
| atomic_d1 | program_stack_free | eval_template_d8 | 8 | 3 | 18 | 50.0% | 0.0% | 50.0% | 100.0% | n/a | n/a | n/a | 1.00 |
| atomic_d1 | program_stack_resample_valid | eval_template_d8 | 8 | 3 | 18 | 50.0% | 0.0% | 55.6% | 91.7% | 16.7% | 0.0% | 0.0% | 1.50 |
| mix_d1_d2 | program_stack_constrained | eval_template_d8 | 8 | 3 | 18 | 72.2% | 19.2% | 100.0% | 72.2% | 66.7% | 38.9% | 0.0% | 1.00 |
| mix_d1_d2 | program_stack_free | eval_template_d8 | 8 | 3 | 18 | 33.3% | 0.0% | 38.9% | 88.9% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2 | program_stack_resample_valid | eval_template_d8 | 8 | 3 | 18 | 33.3% | 0.0% | 38.9% | 88.9% | 5.6% | 0.0% | 0.0% | 1.61 |
| mix_d1_d2_d3 | program_stack_constrained | eval_template_d8 | 8 | 3 | 18 | 88.9% | 9.6% | 100.0% | 88.9% | 66.7% | 55.6% | 0.0% | 1.00 |
| mix_d1_d2_d3 | program_stack_free | eval_template_d8 | 8 | 3 | 18 | 33.3% | 0.0% | 38.9% | 88.9% | n/a | n/a | n/a | 1.00 |
| mix_d1_d2_d3 | program_stack_resample_valid | eval_template_d8 | 8 | 3 | 18 | 33.3% | 0.0% | 44.4% | 77.8% | 11.1% | 0.0% | 0.0% | 1.61 |
| oracle | gold_abi_constrained | eval_template_d8 | 8 | 3 | 18 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
Interpretation
The experiment isolates a training-data question. If shallow composition examples improve depth-6 and depth-8 execution without merely increasing valid-rate, the remaining valid-but-wrong errors are partly a coverage problem: the compiler needed to see short composition traces to stabilize longer composition. If the depth curve does not improve, the remaining errors are a deeper sequencing or grounding limit. At depth 6, the d1/d2/d3 curriculum changes execution by 11.1% and correct-given-valid by 11.1% relative to atomic-only training. At depth 8, the same curriculum changes execution by 11.1% relative to atomic-only training. For atomic_d1 constrained decoding on depth-6/depth-8 composition splits, procedures break down as: correct_exact 61.1%, wrong_op_order_or_choice 25.0%, correct_semantic_variant 13.9%. For mix_d1_d2 constrained decoding on depth-6/depth-8 composition splits, procedures break down as: correct_exact 63.9%, wrong_op_order_or_choice 27.8%, correct_semantic_variant 8.3%. For mix_d1_d2_d3 constrained decoding on depth-6/depth-8 composition splits, procedures break down as: correct_exact 83.3%, wrong_op_order_or_choice 13.9%, correct_semantic_variant 2.8%.
Limitations
This experiment tests compilation over a fixed known primitive library. It does not test invention of operations outside the ABI. The finite-state decoder is tied to the task schema and uses task-visible constants and type information. Depth-2 and depth-3 examples in the training curricula are supervised generated data, so gains should be read as curriculum effects rather than unsupervised discovery.
Artifacts
- Metrics:
analysis/summary_by_arm.csvandanalysis/all_metrics.csv - Details:
analysis/all_details.csv - Training logs:
analysis/all_train_logs.csv - Checkpoints:
/workspace/large_artifacts/qwen_compositional_curriculum_abi/checkpoints
Experiment log 4
Show the running log (4 entries)
Setup
- Created fresh experiment directory:
/workspace/experiments/qwen_compositional_curriculum_abi - Large artifacts directory:
/workspace/large_artifacts/qwen_compositional_curriculum_abi - Core question: whether adding depth-2 and depth-3 supervised procedure examples reduces valid-but-wrong errors at held-out deeper depths under constrained ABI decoding.
- Report format: standalone Markdown and HTML with plots.
Run smoke_v2
- Started: 2026-06-26 18:29:58 UTC
- Suite:
smoke - Model:
Qwen/Qwen3-4B - Seeds:
101 - Training targets:
atomic_d1,mix_d1_d2,mix_d1_d2_d3 - Training examples per seed:
18 - Eval examples per split:
2 - Eval splits:
eval_indist_d1,eval_comp_d6,eval_template_d6 - Steps:
1 - Resample attempts:
2
Completed smoke_v2 in 566.9s.
- Metric rows: 30
- Detail rows: 60
- Training log rows: 3
Run pilot_v1
- Started: 2026-06-26 18:40:18 UTC
- Suite:
pilot - Model:
Qwen/Qwen3-4B - Seeds:
101 - Training targets:
atomic_d1,mix_d1_d2,mix_d1_d2_d3 - Training examples per seed:
90 - Eval examples per split:
4 - Eval splits:
eval_indist_d1,eval_comp_d2,eval_comp_d3,eval_comp_d4,eval_comp_d6,eval_comp_d8,eval_template_d6,eval_template_d8 - Steps:
12 - Resample attempts:
2
Completed pilot_v1 in 952.1s.
- Metric rows: 80
- Detail rows: 320
- Training log rows: 15
Run main_v1
- Started: 2026-06-26 18:57:40 UTC
- Suite:
main - Model:
Qwen/Qwen3-4B - Seeds:
101,202,303 - Training targets:
atomic_d1,mix_d1_d2,mix_d1_d2_d3 - Training examples per seed:
120 - Eval examples per split:
6 - Eval splits:
eval_indist_d1,eval_comp_d2,eval_comp_d3,eval_comp_d4,eval_comp_d6,eval_comp_d8,eval_template_d6,eval_template_d8 - Steps:
16 - Resample attempts:
2
Completed main_v1 in 4097.4s.
- Metric rows: 240
- Detail rows: 1440
- Training log rows: 45
Figures 6
Data files 5
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv84 kBanalysis/summary_by_arm.csv69 kBruns/main_v1/metrics.csv67 kBruns/pilot_v1/metrics.csv16 kBruns/smoke_v2/metrics.csv6.1 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.