Qwen Large ABI Nested Compiler
The one idea you need
A small model writes short programs by picking from a fixed menu of building-block operations, and a calculator-like machine runs them, scoring only the final answer. Enlarging the menu is like giving a cook more ingredients: fine for step-by-step recipes, but recipes that call sub-recipes are a separate skill.
The question
When a small model writes programs by picking from a fixed library of operations, does a four-times bigger library hurt it, and can it learn recipes that split into branches?
What we found
Two things. Making the library four times larger, from 32 to 128 operations, did not hurt straight-line programs at all: both stayed perfect on 16-step chains. But branching is a separate skill. Models shown only straight-line programs got the hardest eight-branch tasks almost entirely wrong, about 13% correct, while adding just a handful of small two-to-three-branch examples lifted that to roughly 87%.
Why it matters
If you want a small model to compose tool calls, a bigger tool library alone will not teach structure. Straight-line chaining scales for free, but branching needs explicit examples, and even then reworded prompts can still break it.
On this page
Results at a glance 4
How to read
Left to right the tasks split into more branches, from 2 to 8; higher means more programs produced the right final answer. Two lines are models shown a few small branching examples; two were shown only straight-line programs.
Takeaway → The two branch-trained lines stay near the top even at eight branches while the straight-line-only lines sink to the floor, showing small branch practice generalizes far wider.
Data table
| Nested branches | 32 ops, chain-only training | 128 ops, chain-only training | 32 ops, nested training | 128 ops, nested training |
|---|---|---|---|---|
| 2 | 53.3% | 26.7% | 100% | 100% |
| 3 | 13.3% | 46.7% | 100% | 100% |
| 4 | 26.7% | 33.3% | 100% | 100% |
| 8 | 6.7% | 13.3% | 73.3% | 86.7% |
Numbers from report table (reports/qwen_large_abi_nested_compiler_report.md); analysis/summary_by_arm.csv
Technical framing
Nested supervision transfers to wider held-out branch counts — Shallow nested supervision (2-3 branches) generalizes: nested-trained models hit 100% at 4 branches and 73-87% at 8; chain-only models collapse.
How to read
Left to right the straight-line programs get longer, from 3 to 16 steps; higher means more correct final answers. Lines compare a 32-operation library against one four times larger, for each training style.
Takeaway → Every line hugs the top across all lengths, so quadrupling the operation library leaves long straight-line programs essentially perfect.
Data table
| Chain depth | 32 ops, chain-only training | 128 ops, chain-only training | 32 ops, nested training | 128 ops, nested training |
|---|---|---|---|---|
| 3 | 100% | 100% | 100% | 100% |
| 8 | 100% | 100% | 100% | 100% |
| 16 | 100% | 100% | 100% | 93.3% |
Numbers from report table (reports/qwen_large_abi_nested_compiler_report.md); analysis/summary_by_arm.csv
Technical framing
Growing the operation library from 32 to 128 does not hurt linear chains — Constrained chain compilation stays at or near 100% out to depth 16 even with a 4x larger primitive library.
How to read
Bars are grouped by how each model was trained; taller is better on the hardest eight-branch tasks. One bar per pair uses a guardrail limiting the model to valid steps, the other lets it write freely.
Takeaway → For branch-trained models the guardrail bar is about double the free-writing bar, but no guardrail rescues models never shown branches.
Data table
| Training curriculum | constrained decoding | free greedy decoding |
|---|---|---|
| 32 ops, chain-only | 6.7% | 0% |
| 128 ops, chain-only | 13.3% | 6.7% |
| 32 ops, nested | 73.3% | 40% |
| 128 ops, nested | 86.7% | 46.7% |
Numbers from report table (reports/qwen_large_abi_nested_compiler_report.md), eval_nested_l8 rows
Technical framing
Constrained decoding beats free decoding on hardest nested tasks (8 branches) — Finite-state constraints roughly double nested-8 accuracy for nested-trained models, but cannot rescue chain-only training.
How to read
Bars are grouped by training; taller means more correct answers on prompts worded differently than training. One set is reworded straight-line programs, the other reworded eight-branch programs.
Takeaway → The best model keeps most of its score on reworded straight-line programs but drops sharply on reworded branching ones, so wording robustness is unsolved for branches.
Data table
| Training curriculum | shifted chain depth 16 | shifted nested 8 branches |
|---|---|---|
| 32 ops, chain-only | 46.7% | 0% |
| 128 ops, chain-only | 80% | 0% |
| 32 ops, nested | 33.3% | 46.7% |
| 128 ops, nested | 73.3% | 40% |
Numbers from report table (reports/qwen_large_abi_nested_compiler_report.md), template-shifted rows
Technical framing
Template-shifted wording degrades nested tasks more than chains — Under reworded prompts the best nested model keeps 73% on chains but drops to 40% on nested-8: wording robustness is unsolved for nesting.
In the author’s words from the Report · “Abstract”
This standalone experiment tests whether a constrained stack-ABI compiler remains reliable when the primitive library grows from 32 to 128 unary operations and when tasks require nested branch sub-procedures. The model emits a program; a deterministic interpreter executes it.
Overview
Standalone experiment testing whether a constrained stack-ABI compiler still works when the primitive library becomes large and when tasks require nested sub-procedures rather than only linear chains.
Question
The compiler can compose a small known ABI into long linear procedures. This experiment tests two new bottlenecks before scaling to a real crystallized-skill corpus:
- Operation selection at larger ABI size: does moving from 32 to 128 unary operations degrade linear chain compilation?
- Nested structure: does adding shallow nested supervision let the model compile branch/sub-procedure tasks at larger held-out nesting widths?
Training Targets
abi32_chain_d3: 32 unary operations, chain tasks only, depths 1 to 3.abi128_chain_d3: 128 unary operations, chain tasks only, depths 1 to 3.abi32_nested_d3: 32 unary operations, chain depths 1 to 3 plus nested tasks with 2 to 3 branches.abi128_nested_d3: 128 unary operations, chain depths 1 to 3 plus nested tasks with 2 to 3 branches.
Evaluation
- Chain depth sweep: depths 3, 8, and 16.
- Template-shifted chain endpoint: depth 16.
- Nested branch sweep: 2, 3, 4, and 8 branches.
- Template-shifted nested endpoint: 8 branches.
- Decoder arms: free greedy stack generation and finite-state constrained stack generation.
- Gold ABI sanity arm: execute the reference program through the interpreter.
Primary Metrics
- Constrained external execution accuracy.
- Correct-given-valid accuracy, since constrained decoding should keep validity near 100%.
- Failure taxonomy on chain depth 16 and nested 8-branch outputs.
- Free versus constrained execution to separate syntax effects from semantic operation/structure selection.
Artifacts
- Source:
src/qwen_large_abi_nested_compiler.py - Metrics and details:
analysis/ - Reports:
reports/ - Large checkpoints:
/workspace/large_artifacts/qwen_large_abi_nested_compiler/checkpoints
Report
Rendered from reports/qwen_large_abi_nested_compiler_report.md
Abstract
This standalone experiment tests whether a constrained stack-ABI compiler remains reliable when the primitive library grows from 32 to 128 unary operations and when tasks require nested branch sub-procedures. The model emits a program; a deterministic interpreter executes it.
Method
Four QLoRA adapters are trained over the same numeric stack ABI shape:
abi32_chain_d3: 32 unary operations, chain tasks only, depths 1 to 3.abi128_chain_d3: 128 unary operations, chain tasks only, depths 1 to 3.abi32_nested_d3: 32 unary operations, chain depths 1 to 3 plus nested tasks with 2 to 3 branches.abi128_nested_d3: 128 unary operations, chain depths 1 to 3 plus nested tasks with 2 to 3 branches.
Evaluation sweeps linear chains at depths 3, 8, and 16, plus nested branch tasks with 2, 3, 4, and 8 branches. Template-shifted endpoints test wording robustness. Each adapter is evaluated with free greedy decoding and finite-state constrained decoding. Gold ABI sanity arms check both ABI sizes.
The primary criteria are constrained external execution accuracy on chain depth 16 and nested 8-branch tasks. Valid-program rate alone is not a success metric; the compiler must select the right operations and merge structure, not merely produce parseable syntax.
Run Configuration
- Primary suite:
main. - Seeds:
101,202,303. - Evaluation rows:
270metric rows,1350scored examples across curricula and decoder arms. - QLoRA update steps per adapter:
24. - Large adapters are stored outside the experiment tree.
Primary Results
- Constrained chain depth 3: 32-chain 100.0%; 128-chain 100.0%; 32-nested 100.0%; 128-nested 100.0%.
- Constrained chain depth 8: 32-chain 100.0%; 128-chain 100.0%; 32-nested 100.0%; 128-nested 100.0%.
- Constrained chain depth 16: 32-chain 100.0%; 128-chain 100.0%; 32-nested 100.0%; 128-nested 93.3%.
- Constrained nested 2 branches: 32-chain 53.3%; 128-chain 26.7%; 32-nested 100.0%; 128-nested 100.0%.
- Constrained nested 3 branches: 32-chain 13.3%; 128-chain 46.7%; 32-nested 100.0%; 128-nested 100.0%.
- Constrained nested 4 branches: 32-chain 26.7%; 128-chain 33.3%; 32-nested 100.0%; 128-nested 100.0%.
- Constrained nested 8 branches: 32-chain 6.7%; 128-chain 13.3%; 32-nested 73.3%; 128-nested 86.7%.
- Constrained template chain depth 16: 32-chain 46.7%; 128-chain 80.0%; 32-nested 33.3%; 128-nested 73.3%.
- Constrained template nested 8 branches: 32-chain 0.0%; 128-chain 0.0%; 32-nested 46.7%; 128-nested 40.0%.
- Gold ABI nested-8 sanity: 32-op 100.0% execution, 128-op 100.0% execution.
- At chain depth 16, 128-op chain beats 32-op chain on
0/3matched seeds; mean per-seed delta 0.0%. - On 128-op nested-8 tasks, nested curriculum beats chain-only on
3/3matched seeds; mean per-seed delta 73.3%.
| train_target | arm | split | depth | runs | n_total | exec_accuracy_mean | exec_accuracy_std | valid_exec_rate_mean | correct_given_valid_mean | divergence_rate_mean | constrained_only_rate_mean | free_only_rate_mean | mean_attempts_mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| abi128_chain_d3 | program_stack_constrained | eval_chain_d16 | 16 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | 26.7% | 26.7% | 0.0% | 1.00 |
| abi128_chain_d3 | program_stack_free | eval_chain_d16 | 16 | 3 | 15 | 73.3% | 11.5% | 73.3% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi128_nested_d3 | program_stack_constrained | eval_chain_d16 | 16 | 3 | 15 | 93.3% | 11.5% | 100.0% | 93.3% | 26.7% | 20.0% | 0.0% | 1.00 |
| abi128_nested_d3 | program_stack_free | eval_chain_d16 | 16 | 3 | 15 | 73.3% | 11.5% | 73.3% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi32_chain_d3 | program_stack_constrained | eval_chain_d16 | 16 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | 6.7% | 6.7% | 0.0% | 1.00 |
| abi32_chain_d3 | program_stack_free | eval_chain_d16 | 16 | 3 | 15 | 93.3% | 11.5% | 93.3% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi32_nested_d3 | program_stack_constrained | eval_chain_d16 | 16 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | 0.0% | 0.0% | 1.00 |
| abi32_nested_d3 | program_stack_free | eval_chain_d16 | 16 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 1.00 |
| oracle_abi128 | gold_abi_constrained | eval_chain_d16 | 16 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| oracle_abi32 | gold_abi_constrained | eval_chain_d16 | 16 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| abi128_chain_d3 | program_stack_constrained | eval_chain_d8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | 20.0% | 20.0% | 0.0% | 1.00 |
| abi128_chain_d3 | program_stack_free | eval_chain_d8 | 8 | 3 | 15 | 80.0% | 34.6% | 80.0% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi128_nested_d3 | program_stack_constrained | eval_chain_d8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | 0.0% | 0.0% | 1.00 |
| abi128_nested_d3 | program_stack_free | eval_chain_d8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi32_chain_d3 | program_stack_constrained | eval_chain_d8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | 0.0% | 0.0% | 1.00 |
| abi32_chain_d3 | program_stack_free | eval_chain_d8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi32_nested_d3 | program_stack_constrained | eval_chain_d8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | 0.0% | 0.0% | 1.00 |
| abi32_nested_d3 | program_stack_free | eval_chain_d8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 1.00 |
| oracle_abi128 | gold_abi_constrained | eval_chain_d8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| oracle_abi32 | gold_abi_constrained | eval_chain_d8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| abi128_chain_d3 | program_stack_constrained | eval_chain_template_d16 | 16 | 3 | 15 | 80.0% | 20.0% | 100.0% | 80.0% | 46.7% | 26.7% | 0.0% | 1.00 |
| abi128_chain_d3 | program_stack_free | eval_chain_template_d16 | 16 | 3 | 15 | 53.3% | 23.1% | 53.3% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi128_nested_d3 | program_stack_constrained | eval_chain_template_d16 | 16 | 3 | 15 | 73.3% | 30.6% | 100.0% | 73.3% | 66.7% | 40.0% | 0.0% | 1.00 |
| abi128_nested_d3 | program_stack_free | eval_chain_template_d16 | 16 | 3 | 15 | 33.3% | 23.1% | 33.3% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi32_chain_d3 | program_stack_constrained | eval_chain_template_d16 | 16 | 3 | 15 | 46.7% | 46.2% | 100.0% | 46.7% | 73.3% | 20.0% | 0.0% | 1.00 |
| abi32_chain_d3 | program_stack_free | eval_chain_template_d16 | 16 | 3 | 15 | 26.7% | 30.6% | 40.0% | 62.5% | n/a | n/a | n/a | 1.00 |
| abi32_nested_d3 | program_stack_constrained | eval_chain_template_d16 | 16 | 3 | 15 | 33.3% | 23.1% | 100.0% | 33.3% | 73.3% | 20.0% | 0.0% | 1.00 |
| abi32_nested_d3 | program_stack_free | eval_chain_template_d16 | 16 | 3 | 15 | 13.3% | 11.5% | 46.7% | 29.2% | n/a | n/a | n/a | 1.00 |
| oracle_abi128 | gold_abi_constrained | eval_chain_template_d16 | 16 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| oracle_abi32 | gold_abi_constrained | eval_chain_template_d16 | 16 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| abi128_chain_d3 | program_stack_constrained | eval_nested_l4 | 4 | 3 | 15 | 33.3% | 23.1% | 100.0% | 33.3% | 100.0% | 26.7% | 0.0% | 1.00 |
| abi128_chain_d3 | program_stack_free | eval_nested_l4 | 4 | 3 | 15 | 6.7% | 11.5% | 20.0% | 25.0% | n/a | n/a | n/a | 1.00 |
| abi128_nested_d3 | program_stack_constrained | eval_nested_l4 | 4 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | 26.7% | 20.0% | 0.0% | 1.00 |
| abi128_nested_d3 | program_stack_free | eval_nested_l4 | 4 | 3 | 15 | 80.0% | 20.0% | 86.7% | 91.7% | n/a | n/a | n/a | 1.00 |
| abi32_chain_d3 | program_stack_constrained | eval_nested_l4 | 4 | 3 | 15 | 26.7% | 11.5% | 100.0% | 26.7% | 100.0% | 26.7% | 6.7% | 1.00 |
| abi32_chain_d3 | program_stack_free | eval_nested_l4 | 4 | 3 | 15 | 6.7% | 11.5% | 20.0% | 33.3% | n/a | n/a | n/a | 1.00 |
| abi32_nested_d3 | program_stack_constrained | eval_nested_l4 | 4 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | 0.0% | 0.0% | 1.00 |
| abi32_nested_d3 | program_stack_free | eval_nested_l4 | 4 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 1.00 |
| oracle_abi128 | gold_abi_constrained | eval_nested_l4 | 4 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| oracle_abi32 | gold_abi_constrained | eval_nested_l4 | 4 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| abi128_chain_d3 | program_stack_constrained | eval_nested_l8 | 8 | 3 | 15 | 13.3% | 11.5% | 100.0% | 13.3% | 100.0% | 13.3% | 6.7% | 1.00 |
| abi128_chain_d3 | program_stack_free | eval_nested_l8 | 8 | 3 | 15 | 6.7% | 11.5% | 6.7% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi128_nested_d3 | program_stack_constrained | eval_nested_l8 | 8 | 3 | 15 | 86.7% | 11.5% | 100.0% | 86.7% | 66.7% | 40.0% | 0.0% | 1.00 |
| abi128_nested_d3 | program_stack_free | eval_nested_l8 | 8 | 3 | 15 | 46.7% | 23.1% | 60.0% | 75.0% | n/a | n/a | n/a | 1.00 |
| abi32_chain_d3 | program_stack_constrained | eval_nested_l8 | 8 | 3 | 15 | 6.7% | 11.5% | 100.0% | 6.7% | 100.0% | 6.7% | 0.0% | 1.00 |
| abi32_chain_d3 | program_stack_free | eval_nested_l8 | 8 | 3 | 15 | 0.0% | 0.0% | 20.0% | 0.0% | n/a | n/a | n/a | 1.00 |
| abi32_nested_d3 | program_stack_constrained | eval_nested_l8 | 8 | 3 | 15 | 73.3% | 30.6% | 100.0% | 73.3% | 53.3% | 33.3% | 0.0% | 1.00 |
| abi32_nested_d3 | program_stack_free | eval_nested_l8 | 8 | 3 | 15 | 40.0% | 34.6% | 66.7% | 66.7% | n/a | n/a | n/a | 1.00 |
| oracle_abi128 | gold_abi_constrained | eval_nested_l8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| oracle_abi32 | gold_abi_constrained | eval_nested_l8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| abi128_chain_d3 | program_stack_constrained | eval_nested_template_l8 | 8 | 3 | 15 | 0.0% | 0.0% | 100.0% | 0.0% | 100.0% | 0.0% | 0.0% | 1.00 |
| abi128_chain_d3 | program_stack_free | eval_nested_template_l8 | 8 | 3 | 15 | 0.0% | 0.0% | 26.7% | 0.0% | n/a | n/a | n/a | 1.00 |
| abi128_nested_d3 | program_stack_constrained | eval_nested_template_l8 | 8 | 3 | 15 | 40.0% | 20.0% | 100.0% | 40.0% | 86.7% | 26.7% | 0.0% | 1.00 |
| abi128_nested_d3 | program_stack_free | eval_nested_template_l8 | 8 | 3 | 15 | 13.3% | 23.1% | 13.3% | 100.0% | n/a | n/a | n/a | 1.00 |
| abi32_chain_d3 | program_stack_constrained | eval_nested_template_l8 | 8 | 3 | 15 | 0.0% | 0.0% | 100.0% | 0.0% | 100.0% | 0.0% | 0.0% | 1.00 |
| abi32_chain_d3 | program_stack_free | eval_nested_template_l8 | 8 | 3 | 15 | 0.0% | 0.0% | 20.0% | 0.0% | n/a | n/a | n/a | 1.00 |
| abi32_nested_d3 | program_stack_constrained | eval_nested_template_l8 | 8 | 3 | 15 | 46.7% | 50.3% | 100.0% | 46.7% | 73.3% | 26.7% | 0.0% | 1.00 |
| abi32_nested_d3 | program_stack_free | eval_nested_template_l8 | 8 | 3 | 15 | 20.0% | 34.6% | 26.7% | 50.0% | n/a | n/a | n/a | 1.00 |
| oracle_abi128 | gold_abi_constrained | eval_nested_template_l8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| oracle_abi32 | gold_abi_constrained | eval_nested_template_l8 | 8 | 3 | 15 | 100.0% | 0.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
Interpretation
This experiment separates two questions that matter before scaling a real ABI: whether a larger operation catalog hurts linear chain compilation, and whether branch/sub-procedure structure requires explicit nested supervision. Operation-scale effect on chain depth 16: 128-op chain training changes execution by 0.0% relative to 32-op chain training. Nested-curriculum effect at 32 ops on nested-8 tasks: 66.7%. Nested-curriculum effect at 128 ops on nested-8 tasks: 73.3%. Template-shifted 128-op nested training reaches 73.3% on chain depth 16 but only 40.0% on nested-8, so wording robustness is not solved for nested branch tasks. The central positive result is that shallow nested supervision transfers beyond the trained branch counts: both nested curricula reach 100.0% at nested depth 4, and the 128-op nested curriculum reaches 86.7% at nested depth 8. The operation-catalog result is also positive but narrower: moving from 32 to 128 unary operations does not harm constrained linear-chain compilation, but it does not by itself teach nested structure. Because constrained decoding supplies only syntactic validity, any execution gain in constrained rows should be read as better operation or merge selection rather than better self-execution. For abi128_chain_d3 constrained decoding on chain/nested endpoints, procedures break down as: correct_exact 46.7%, wrong_op_order_or_choice 43.3%, correct_semantic_variant 10.0%. For abi128_nested_d3 constrained decoding on chain/nested endpoints, procedures break down as: correct_exact 86.7%, wrong_op_order_or_choice 10.0%, correct_semantic_variant 3.3%. For abi32_chain_d3 constrained decoding on chain/nested endpoints, procedures break down as: correct_exact 53.3%, wrong_op_order_or_choice 46.7%. For abi32_nested_d3 constrained decoding on chain/nested endpoints, procedures break down as: correct_exact 83.3%, wrong_op_order_or_choice 13.3%, correct_semantic_variant 3.3%.
Limitations
This experiment tests compilation over a known numeric primitive library. It does not test invention of operations outside the ABI. The finite-state decoder is tied to the task schema and uses task-visible constants plus known line kinds, so results measure operation and merge selection inside a valid grammar. Nested tasks are branch-merge programs, not arbitrary loops or recursion.
Artifacts
- Metrics:
analysis/summary_by_arm.csvandanalysis/all_metrics.csv - Details:
analysis/all_details.csv - Training logs:
analysis/all_train_logs.csv - Checkpoints:
/workspace/large_artifacts/qwen_large_abi_nested_compiler/checkpoints
Experiment log 4
Show the running log (4 entries)
Setup
- Created fresh experiment directory:
/workspace/experiments/qwen_large_abi_nested_compiler - Large artifacts directory:
/workspace/large_artifacts/qwen_large_abi_nested_compiler - Core question: whether constrained ABI compilation survives both larger primitive libraries and nested branch/sub-procedure structure.
- Report format: standalone Markdown and HTML with plots.
Run smoke_v1
- Started: 2026-06-26 23:01:46 UTC
- Suite:
smoke - Model:
Qwen/Qwen3-4B - Seeds:
101 - Training targets:
abi32_chain_d3,abi128_chain_d3,abi32_nested_d3,abi128_nested_d3 - Training examples per seed:
24 - Eval examples per split:
1 - Eval splits:
eval_chain_d16,eval_nested_l8,eval_chain_template_d16,eval_nested_template_l8 - Steps:
1 - Resample attempts:
3
Completed smoke_v1 in 617.7s.
- Metric rows: 40
- Detail rows: 40
- Training log rows: 4
Run pilot_v1
- Started: 2026-06-26 23:12:48 UTC
- Suite:
pilot - Model:
Qwen/Qwen3-4B - Seeds:
101 - Training targets:
abi32_chain_d3,abi128_chain_d3,abi32_nested_d3,abi128_nested_d3 - Training examples per seed:
96 - Eval examples per split:
3 - Eval splits:
eval_chain_d3,eval_chain_d8,eval_chain_d16,eval_chain_template_d16,eval_nested_l2,eval_nested_l4,eval_nested_l8,eval_nested_template_l8 - Steps:
8 - Resample attempts:
3
Completed pilot_v1 in 1514.4s.
- Metric rows: 80
- Detail rows: 240
- Training log rows: 20
Pilot read:
- 32-op chain-only constrained chain depth-16 reached 100.0%.
- 128-op chain-only constrained chain depth-16 also reached 100.0%, but template chain depth-16 fell to 33.3%.
- Chain-only curricula transferred to 2-branch nested tasks at 66.7% but collapsed at 4 and 8 branches.
- 32-op nested curriculum produced a nonzero nested-8 row (33.3%) but did not solve nested-4 or template nested-8.
- 128-op nested curriculum did not improve nested-8 in the pilot.
- Added
eval_nested_l3before main so trained nested-boundary performance is measured explicitly.
Run main_v1
- Started: 2026-06-26 23:39:21 UTC
- Suite:
main - Model:
Qwen/Qwen3-4B - Seeds:
101,202,303 - Training targets:
abi32_chain_d3,abi128_chain_d3,abi32_nested_d3,abi128_nested_d3 - Training examples per seed:
240 - Eval examples per split:
5 - Eval splits:
eval_chain_d3,eval_chain_d8,eval_chain_d16,eval_chain_template_d16,eval_nested_l2,eval_nested_l3,eval_nested_l4,eval_nested_l8,eval_nested_template_l8 - Steps:
24 - Resample attempts:
3
Completed main_v1 in 5622.3s.
- Metric rows: 270
- Detail rows: 1350
- Training log rows: 60
Final read:
- The 128-operation catalog did not break linear chain compilation under constrained decoding:
abi128_chain_d3reached 100.0% at chain depth 16, matchingabi32_chain_d3. - Chain-only training did not learn nested branch structure: constrained nested-8 was 6.7% for 32 ops and 13.3% for 128 ops.
- Shallow nested training transferred strongly beyond the trained branch counts: constrained nested-8 reached 73.3% for 32 ops and 86.7% for 128 ops.
- Nested depth 4 was solved by both nested curricula at 100.0%, showing stable near-depth transfer.
- Template-shifted nested-8 remained weak: 46.7% for 32-op nested training and 40.0% for 128-op nested training, even though validity was 100.0%.
- Gold ABI sanity arms were 100.0% across both ABI sizes and all splits, so residual errors are compiler selection/grounding errors rather than interpreter or decoder plumbing errors.
- Read: operation selection at 128 ops is not the immediate blocker for linear pipelines; nested/control-flow structure is trainable with shallow examples; wording-robust nested grounding remains the next bottleneck.
- The 128-operation catalog did not break linear chain compilation under constrained decoding:
Figures 7
Data files 5
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv72 kBanalysis/summary_by_arm.csv61 kBruns/main_v1/metrics.csv50 kBruns/pilot_v1/metrics.csv20 kBruns/smoke_v1/metrics.csv8.0 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.