Qwen Trace Procedure Depth Stress
The one idea you need
Picture a cook who writes a flawless six-step recipe but burns the dish whenever he cooks it himself. Hand that written recipe to a machine and it follows the steps perfectly. This model wrote correct multi-step procedures yet could not carry them out in its own head.
The question
If you train a small model only on single-step operations, can it chain them into correct multi-step procedures it never saw during training?
What we found
Yes. Trained only on single-step tasks, the 4-billion-parameter model wrote six-step procedures that ran correctly 63% of the time once a plain step-follower executed them, versus 0% when the model tried to state the final answer itself. Accuracy slid down gently as tasks lengthened, never collapsing. So it truly composes steps into working procedures, but cannot reliably run them in its own head. Treat it as a recipe-writer, not a calculator.
Why it matters
Have small models emit an explicit step list and run it with a deterministic step-follower rather than trusting their stated answer. The steps hold up on long tasks; the weak spot is reworded requests, so invest in language grounding.
On this page
Results at a glance 4
How to read
Horizontal axis is how many steps the task needs, from two to six; vertical axis is how often it was solved, higher is better. Three upper lines are machine-run procedures; the bottom line is the model answering directly in its head.
Takeaway → The procedure lines slope down gently and stay high, while the direct-answer line falls all the way to zero by six steps.
Data table
| composition depth | compact program (executed) | trace + final (executed) | trace, no final (executed) | answer only (final answer) |
|---|---|---|---|---|
| 2 | 76.7% | 76.7% | 65% | 41.7% |
| 3 | 75% | 75% | 61.7% | 15% |
| 4 | 70% | 63.3% | 68.3% | 5% |
| 6 | 63.3% | 58.3% | 56.7% | 0% |
Numbers from report table (reports/qwen_trace_procedure_depth_stress_report.md); same data in analysis/summary_by_arm.csv
Technical framing
Held-out composition accuracy vs procedure depth (trained on atomic steps only) — Procedure arms degrade gradually with depth (63% at depth 6) while direct answering collapses to 0% — a compiler, not a self-executor.
How to read
Horizontal axis is task length in steps; vertical axis is how often solved, higher is better. The top line is the model's written procedure run by a machine; the bottom line is the final answer the model wrote itself.
Takeaway → The two lines split wide apart: at six steps the machine-run procedure is often right while the model's own answer is essentially never right.
Data table
| composition depth | executed procedure | model's emitted final answer |
|---|---|---|
| 2 | 76.7% | 30% |
| 3 | 75% | 18.3% |
| 4 | 63.3% | 3.3% |
| 6 | 58.3% | 0% |
Technical framing
Trace arm: procedure executes correctly even when its own final answer is wrong — At depth 6 the emitted procedure runs correctly 58% of the time while the model's own final answer is right 0% — compilation beats self-execution.
How to read
Horizontal axis is task length in steps; vertical axis is how often the machine-run procedure solved it, higher is better. The upper line is the familiar wording, the lower line is the same tasks reworded.
Takeaway → The reworded line sits below the familiar-wording line at every length, showing the wording of the request is the remaining hurdle.
Data table
| composition depth | standard prompts | template-shifted prompts |
|---|---|---|
| 2 | 76.7% | 53.3% |
| 4 | 70% | 43.3% |
| 6 | 63.3% | 41.7% |
Numbers from report table (reports/qwen_trace_procedure_depth_stress_report.md), program_stack eval_comp vs eval_template rows
Technical framing
Template-shifted wording cuts compact-program execution at every depth — Rewording the task drops depth-6 execution from 63% to 42%, so the remaining bottleneck is language grounding, not procedure execution.
How to read
Each bar is one outcome type for the written procedures across all task lengths; taller means a bigger share. The first bar is fully correct; the rest are kinds of mistakes, taller meaning more common.
Takeaway → The tallest failure bar is procedures that won't run at all, while picking the wrong steps is a tiny sliver, so parsing not reasoning is the main breakdown.
Data table
| failure category | trace + final, all composition depths |
|---|---|
| correct (exact) | 58% |
| invalid / unexecutable | 33.7% |
| wrong op order or choice | 4.5% |
| correct semantic variant | 3.8% |
Technical framing
Where trace-arm procedures go wrong on composition splits — Most failures are procedures that fail to parse or execute; picking the wrong operations is rare once a procedure is valid.
In the author’s words from the Report · “Abstract”
This standalone experiment tests whether a local 4B model composes known primitives into executable procedures when trained only on atomic procedures. The primary score for procedure arms is external execution of the emitted stack program, not the model's own final answer.
Overview
This standalone experiment tests whether a local 4B model can compile natural language tasks into explicit executable procedures over a fixed stack ABI.
The primary metric for procedure arms is external execution of the generated procedure. The model's emitted final answer is measured separately and is not trusted as the procedure score.
Large adapter checkpoints are stored outside this experiment directory:
/workspace/large_artifacts/qwen_trace_procedure_depth_stressReport
Rendered from reports/qwen_trace_procedure_depth_stress_report.md
Abstract
This standalone experiment tests whether a local 4B model composes known primitives into executable procedures when trained only on atomic procedures. The primary score for procedure arms is external execution of the emitted stack program, not the model's own final answer.
Method
Training examples contain one primitive operation. Evaluation sweeps held-out procedure depths 2, 3, 4, and 6, plus template-shifted prompts at depths 2, 4, and 6. The task families are string, number, table, date, list, and path transformations. Four arms are compared: answer_only, trace_stack_final, trace_stack_no_final, and program_stack.
The load-bearing distinction is compilation versus self-execution. trace_stack_final may emit a FINAL line, but its procedure is also parsed and executed by the interpreter. trace_stack_no_final tests the same numbered procedure format without answer supervision. program_stack tests compact raw instructions.
Run Configuration
- Primary suite:
main. - Seeds:
101,202,303,404,505. - Evaluation examples across trained arms:
1920. - QLoRA update steps per adapter:
32. - Large adapters are stored outside the experiment tree.
Primary Results
- Best held-out composition execution row:
program_stackdepth2at 76.7%. - Compact program execution at depth 6:
program_stack63.3% with seed std 11.2%, versusanswer_onlyfinal accuracy 0.0%. trace_stack_finaldepth-6 composition execution: 58.3% with seed std 8.3%.trace_stack_finaldepth-6 template-shift execution: 36.7%.- Final-answer supervision comparison at depth 4:
trace_stack_finalexecution 63.3%;trace_stack_no_finalexecution 68.3%. - Template-shift depth-6 execution for
program_stack: 41.7%, a 21.7% absolute drop from standard depth-6 composition.
| arm | split | depth | runs | n_total | primary_accuracy_mean | primary_accuracy_std | exec_accuracy_mean | exec_accuracy_std | valid_exec_rate_mean | final_accuracy_mean | no_final_rate_mean | exact_program_rate_mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| answer_only | eval_comp_d2 | 2 | 5 | 60 | 41.7% | 10.2% | 0.0% | 0.0% | 0.0% | 41.7% | 0.0% | 0.0% |
| program_stack | eval_comp_d2 | 2 | 5 | 60 | 76.7% | 3.7% | 76.7% | 3.7% | 78.3% | 0.0% | 100.0% | 75.0% |
| trace_stack_final | eval_comp_d2 | 2 | 5 | 60 | 76.7% | 7.0% | 76.7% | 7.0% | 80.0% | 30.0% | 0.0% | 73.3% |
| trace_stack_no_final | eval_comp_d2 | 2 | 5 | 60 | 65.0% | 16.0% | 65.0% | 16.0% | 75.0% | 0.0% | 100.0% | 63.3% |
| answer_only | eval_comp_d3 | 3 | 5 | 60 | 15.0% | 7.0% | 0.0% | 0.0% | 0.0% | 15.0% | 0.0% | 0.0% |
| program_stack | eval_comp_d3 | 3 | 5 | 60 | 75.0% | 10.2% | 75.0% | 10.2% | 81.7% | 0.0% | 100.0% | 68.3% |
| trace_stack_final | eval_comp_d3 | 3 | 5 | 60 | 75.0% | 5.9% | 75.0% | 5.9% | 83.3% | 18.3% | 0.0% | 70.0% |
| trace_stack_no_final | eval_comp_d3 | 3 | 5 | 60 | 61.7% | 4.6% | 61.7% | 4.6% | 73.3% | 0.0% | 100.0% | 61.7% |
| answer_only | eval_comp_d4 | 4 | 5 | 60 | 5.0% | 7.5% | 0.0% | 0.0% | 0.0% | 5.0% | 0.0% | 0.0% |
| program_stack | eval_comp_d4 | 4 | 5 | 60 | 70.0% | 11.2% | 70.0% | 11.2% | 71.7% | 0.0% | 100.0% | 63.3% |
| trace_stack_final | eval_comp_d4 | 4 | 5 | 60 | 63.3% | 11.2% | 63.3% | 11.2% | 63.3% | 3.3% | 0.0% | 63.3% |
| trace_stack_no_final | eval_comp_d4 | 4 | 5 | 60 | 68.3% | 13.7% | 68.3% | 13.7% | 71.7% | 0.0% | 100.0% | 65.0% |
| answer_only | eval_comp_d6 | 6 | 5 | 60 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| program_stack | eval_comp_d6 | 6 | 5 | 60 | 63.3% | 11.2% | 63.3% | 11.2% | 65.0% | 0.0% | 100.0% | 60.0% |
| trace_stack_final | eval_comp_d6 | 6 | 5 | 60 | 58.3% | 8.3% | 58.3% | 8.3% | 63.3% | 0.0% | 0.0% | 48.3% |
| trace_stack_no_final | eval_comp_d6 | 6 | 5 | 60 | 56.7% | 12.4% | 56.7% | 12.4% | 61.7% | 0.0% | 100.0% | 48.3% |
| answer_only | eval_indist_d1 | 1 | 5 | 60 | 86.7% | 12.6% | 0.0% | 0.0% | 0.0% | 86.7% | 0.0% | 0.0% |
| program_stack | eval_indist_d1 | 1 | 5 | 60 | 98.3% | 3.7% | 98.3% | 3.7% | 98.3% | 0.0% | 100.0% | 98.3% |
| trace_stack_final | eval_indist_d1 | 1 | 5 | 60 | 98.3% | 3.7% | 98.3% | 3.7% | 100.0% | 91.7% | 0.0% | 98.3% |
| trace_stack_no_final | eval_indist_d1 | 1 | 5 | 60 | 98.3% | 3.7% | 98.3% | 3.7% | 98.3% | 0.0% | 100.0% | 96.7% |
| answer_only | eval_template_d2 | 2 | 5 | 60 | 45.0% | 12.6% | 0.0% | 0.0% | 0.0% | 45.0% | 0.0% | 0.0% |
| program_stack | eval_template_d2 | 2 | 5 | 60 | 53.3% | 4.6% | 53.3% | 4.6% | 56.7% | 0.0% | 100.0% | 46.7% |
| trace_stack_final | eval_template_d2 | 2 | 5 | 60 | 50.0% | 5.9% | 50.0% | 5.9% | 53.3% | 40.0% | 0.0% | 43.3% |
| trace_stack_no_final | eval_template_d2 | 2 | 5 | 60 | 48.3% | 9.1% | 48.3% | 9.1% | 58.3% | 0.0% | 100.0% | 43.3% |
| answer_only | eval_template_d4 | 4 | 5 | 60 | 3.3% | 7.5% | 0.0% | 0.0% | 0.0% | 3.3% | 0.0% | 0.0% |
| program_stack | eval_template_d4 | 4 | 5 | 60 | 43.3% | 7.0% | 43.3% | 7.0% | 45.0% | 0.0% | 100.0% | 40.0% |
| trace_stack_final | eval_template_d4 | 4 | 5 | 60 | 40.0% | 7.0% | 40.0% | 7.0% | 41.7% | 8.3% | 1.7% | 35.0% |
| trace_stack_no_final | eval_template_d4 | 4 | 5 | 60 | 33.3% | 0.0% | 33.3% | 0.0% | 43.3% | 0.0% | 100.0% | 31.7% |
| answer_only | eval_template_d6 | 6 | 5 | 60 | 1.7% | 3.7% | 0.0% | 0.0% | 0.0% | 1.7% | 0.0% | 0.0% |
| program_stack | eval_template_d6 | 6 | 5 | 60 | 41.7% | 8.3% | 41.7% | 8.3% | 41.7% | 0.0% | 100.0% | 36.7% |
| trace_stack_final | eval_template_d6 | 6 | 5 | 60 | 36.7% | 12.6% | 36.7% | 12.6% | 38.3% | 0.0% | 11.7% | 30.0% |
| trace_stack_no_final | eval_template_d6 | 6 | 5 | 60 | 38.3% | 9.5% | 38.3% | 9.5% | 41.7% | 0.0% | 100.0% | 33.3% |
Interpretation
A real compiler result should degrade gradually with depth rather than collapse immediately once procedure length exceeds the atomic training distribution. A gap between trace_stack_final execution and final-answer accuracy means the model can emit a correct procedure while failing to self-execute it. A gap between standard composition and template-shift composition localizes the remaining problem to language grounding rather than procedure sequencing. The main result is positive: training only on atomic procedures still produced executable depth-6 compositions at 63.3% for the compact raw stack ABI, while answer-only scoring was 0.0%. This is not a self-execution win; it is a compiler win, because the deterministic interpreter supplies the execution. The final-answer confound persists at the hardest standard composition split: trace_stack_final executed correctly at 58.3% but its emitted final answer was correct at 0.0%. Removing the final-answer line did not kill procedure learning: trace_stack_no_final reached 56.7% at depth 6. Final-answer supervision is therefore not required for the basic compiler effect, though arm ranking varies by depth and template split. The remaining major weakness is prompt wording. program_stack falls from 63.3% on standard depth-6 composition to 41.7% under template shift, so the next bottleneck is language grounding into the ABI, not deterministic execution. For trace_stack_final on composition splits, generated procedures break down as: correct_exact 58.0%, invalid_or_unexecutable 33.7%, wrong_op_order_or_choice 4.5%, correct_semantic_variant 3.8%.
Limitations
This experiment tests composition over a fixed known primitive library. It does not test invention of new operations outside the ABI. The families are synthetic but selected to cover several common deterministic task shapes.
Artifacts
- Metrics:
analysis/summary_by_arm.csvandanalysis/all_metrics.csv - Details:
analysis/all_details.csv - Training logs:
analysis/all_train_logs.csv - Checkpoints:
/workspace/large_artifacts/qwen_trace_procedure_depth_stress/checkpoints
Experiment log 4
Show the running log (4 entries)
Setup
- Created: 2026-06-26 UTC
- Fresh experiment directory:
experiments/qwen_trace_procedure_depth_stress - Large artifact directory:
large_artifacts/qwen_trace_procedure_depth_stress - Base model:
Qwen/Qwen3-4B - Primary question: can a model trained on atomic procedures compose known primitives into deeper executable procedures when the generated procedure is run by a deterministic interpreter?
- Primary readouts: execution accuracy by composition depth, template-shift execution by depth, final-answer-vs-execution gap, and failure taxonomy.
Run smoke_v1
- Started: 2026-06-26 07:14:05 UTC
- Suite:
smoke - Model:
Qwen/Qwen3-4B - Seeds:
101 - Arms:
answer_only,trace_stack_final,trace_stack_no_final,program_stack - Training examples per seed:
24 - Eval examples per split:
6 - Steps:
2
Completed smoke_v1 in 657.7s.
- Metric rows: 32
- Detail rows: 192
- Training log rows: 8
Run pilot_v1
- Started: 2026-06-26 07:25:33 UTC
- Suite:
pilot - Model:
Qwen/Qwen3-4B - Seeds:
101 - Arms:
answer_only,trace_stack_final,trace_stack_no_final,program_stack - Training examples per seed:
120 - Eval examples per split:
12 - Steps:
20
Completed pilot_v1 in 533.6s.
- Metric rows: 32
- Detail rows: 384
- Training log rows: 20
Run main_v1
- Started: 2026-06-26 07:35:04 UTC
- Suite:
main - Model:
Qwen/Qwen3-4B - Seeds:
101,202,303,404,505 - Arms:
answer_only,trace_stack_final,trace_stack_no_final,program_stack - Training examples per seed:
180 - Eval examples per split:
12 - Steps:
32
Completed main_v1 in 2743.2s.
- Metric rows: 160
- Detail rows: 1920
- Training log rows: 100
Figures 5
Data files 5
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv46 kBanalysis/summary_by_arm.csv29 kBruns/main_v1/metrics.csv34 kBruns/pilot_v1/metrics.csv7.4 kBruns/smoke_v1/metrics.csv5.9 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.