Qwen Program-Only Executable ABI
The one idea you need
Picture an accountant who writes flawless spreadsheet formulas but misreads the final total aloud. Here a small model writes a tiny program for each office task and a calculator runs it. Trust the run, not the spoken answer, and accuracy jumps.
The question
If a small model writes a little program to solve each task instead of just stating the answer, does it handle brand-new multi-step problems better?
What we found
Yes. Teaching a small model to write a short runnable program lifted brand-new multi-step accuracy from about 44% (just stating an answer) to 73%. And when it wrote out its steps plus an answer, the steps ran correctly 94% of the time while its stated answer was right only 40%. But the gain swung wildly between training runs, so this is a promising lever, not a finished recipe.
Why it matters
For tasks you can verify by running code — conversions, lookups, date math — have a small model write a short program and grade what it computes, not the answer it types. Expect big run-to-run swings and sensitivity to exact prompt wording.
On this page
Results at a glance 5
How to read
Grouped bars, one cluster per teaching style; the three colored bars in each cluster are familiar tasks, brand-new multi-step tasks, and reworded tasks. Bar height is the share of tasks solved; taller is better.
Takeaway → On brand-new multi-step tasks the program-writing cluster is clearly tallest, well above just stating the answer, so compiled programs compose new procedures better.
Data table
| Supervision arm | in-distribution | held-out composition | template shift |
|---|---|---|---|
| stack program (program-only) | 100% | 72.9% | 56.2% |
| python program (program-only) | 100% | 43.8% | 39.6% |
| trace + final answer | 83.3% | 39.6% | 77.1% |
| answer only | 89.6% | 43.8% | 75% |
| zero-shot answer | 37.5% | 4.2% | 37.5% |
Numbers from experiments/qwen_program_only_executable_abi/analysis/summary_by_arm.csv (main suite, primary_accuracy_mean)
Technical framing
Held-out composition: program-only compilation beats answer-only supervision — Strict program-only stack compilation reaches 72.9% on held-out compositions vs 43.8% for answer-only supervision.
How to read
Two bars per task group; height is share solved, taller is better. One color scores the answer the model typed, the other scores what happens when you actually run the steps it wrote.
Takeaway → On brand-new tasks the run-the-steps bar towers over the typed-answer bar, so the model builds correct steps but fumbles the final number.
Data table
| Evaluation split | final-answer accuracy | externally executed procedure accuracy |
|---|---|---|
| in-distribution | 83.3% | 100% |
| held-out composition | 39.6% | 93.8% |
| template shift | 77.1% | 50% |
Technical framing
Trace + final arm: the procedure is right far more often than the answer token — Executing the generated procedure yields 93.8% on held-out compositions while the emitted FINAL line is right only 39.6% of the time.
How to read
Bars grouped by task type (text patterns, unit conversion, table lookup, dates); height is the share of new combinations solved, taller is better, with one color per teaching style.
Takeaway → The styles have opposite strengths: program-writing aces text and date tasks where answer styles score near zero, and loses on unit conversions.
Data table
| Task family | Program (stack) | Program (python) | Trace + final | Answer only |
|---|---|---|---|---|
| String tasks | 91.7% | 0% | 0% | 8.3% |
| Unit conversion | 50% | 50% | 100% | 100% |
| Table lookup | 50% | 100% | 58.3% | 66.7% |
| Date offset | 100% | 25% | 0% | 0% |
Numbers from experiments/qwen_program_only_executable_abi/analysis/summary_by_arm.csv (eval_composition family columns)
Technical framing
Which task families each supervision style composes — Arms have opposite strengths: stack programs ace string and date compositions where answer-style arms score near zero, and vice versa.
How to read
Two bars per teaching style; height is share solved, taller is better. One color is familiar wording, the other is the same tasks reworded.
Takeaway → Program styles are perfect on familiar wording but fall hardest when prompts are reworded, while answer styles barely dip, so programs overfit surface wording.
Data table
| Arm | In-distribution | Template shift |
|---|---|---|
| Program (stack) | 100% | 56.2% |
| Program (python) | 100% | 39.6% |
| Trace + final | 83.3% | 77.1% |
| Answer only | 89.6% | 75% |
| Zero-shot | 37.5% | 37.5% |
Technical framing
Template shift flips the ranking: programs overfit surface form — Program arms are perfect in-distribution but drop hardest when prompt templates change; answer-style arms degrade far less.
How to read
One bar per teaching style; height is the average amount of text the model generates per task, so shorter is cheaper.
Takeaway → The program-writing bars are roughly three times taller than the bare-answer bar, the price paid for the accuracy gains.
Data table
| Arm | Mean new tokens per example |
|---|---|
| Answer only | 9.92 |
| Zero-shot | 24 |
| Program (stack) | 28.9 |
| Program (python) | 33.3 |
| Trace + final | 34.2 |
Numbers from experiments/qwen_program_only_executable_abi/analysis/summary_by_arm.csv (eval_composition, mean_new_tokens_mean)
Technical framing
Token cost of each output format on composition eval — Executable programs cost roughly 3x the tokens of a bare answer, but that budget buys the composition and procedure gains.
In the author’s words from the Report · “Abstract”
This experiment tests whether a local 4B language model can compile deterministic office-style tasks into executable programs when the final answer is absent from the program-only targets. Program-only outputs are parsed and executed by a deterministic interpreter; correctness is based on the interpreter result.
Overview
This standalone experiment tests whether a local 4B model can compile deterministic office-style tasks into executable programs whose interpreter result, not an emitted answer token, determines correctness.
The experiment compares final-answer supervision, trace-plus-final supervision, and two program-only executable ABIs. Large adapter checkpoints are stored under:
/workspace/large_artifacts/qwen_program_only_executable_abi
Report
Rendered from reports/qwen_program_only_executable_abi_report.md
Abstract
This experiment tests whether a local 4B language model can compile deterministic office-style tasks into executable programs when the final answer is absent from the program-only targets. Program-only outputs are parsed and executed by a deterministic interpreter; correctness is based on the interpreter result.
Method
The task factory creates string, unit-conversion, table-lookup, and date-offset examples. The training split contains atomic operations, while the composition split recombines known primitives into held-out multi-step procedures. Four arms are compared: answer_only, trace_stack_final, program_stack, and program_python.
For program-only arms, the primary metric is strict execution accuracy: the generated program must execute to the correct answer and must not contain a FINAL line. For answer-emitting arms, the primary metric is exact match on the parsed FINAL line.
Run Configuration
- Primary suite:
main. - Adapter seeds:
101,202. - Total trained-arm evaluation examples:
576across arms, splits, and seeds. - QLoRA update steps per adapter:
48. - Large adapters are stored outside the experiment tree.
Primary Results
- Best in-distribution arm:
program_pythonat 100.0%. - Best held-out composition arm:
program_stackat 72.9%. - Best externally executed held-out composition procedure:
trace_stack_finalat 93.8%. - Best strict program-only composition arm:
program_stackat 72.9%.
| suite | arm | split | runs | n_total | primary_accuracy_mean | primary_accuracy_std | exec_accuracy_mean | valid_exec_rate_mean | no_final_rate_mean | mean_new_tokens_mean |
|---|---|---|---|---|---|---|---|---|---|---|
| main | program_stack | eval_composition | 2 | 48 | 72.9% | 32.4% | 72.9% | 72.9% | 100.0% | 28.90 |
| main | answer_only | eval_composition | 2 | 48 | 43.8% | 8.8% | 0.0% | 0.0% | 0.0% | 9.92 |
| main | program_python | eval_composition | 2 | 48 | 43.8% | 8.8% | 43.8% | 58.3% | 100.0% | 33.27 |
| main | trace_stack_final | eval_composition | 2 | 48 | 39.6% | 2.9% | 93.8% | 100.0% | 0.0% | 34.21 |
| main | zero_shot_answer | eval_composition | 1 | 24 | 4.2% | 0.0% | 0.0% | 0.0% | 75.0% | 24.00 |
| main | program_python | eval_indist | 2 | 48 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | 27.56 |
| main | program_stack | eval_indist | 2 | 48 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | 22.08 |
| main | answer_only | eval_indist | 2 | 48 | 89.6% | 2.9% | 0.0% | 0.0% | 0.0% | 9.83 |
| main | trace_stack_final | eval_indist | 2 | 48 | 83.3% | 0.0% | 100.0% | 100.0% | 0.0% | 29.92 |
| main | zero_shot_answer | eval_indist | 1 | 24 | 37.5% | 0.0% | 0.0% | 0.0% | 33.3% | 24.00 |
| main | trace_stack_final | eval_template_shift | 2 | 48 | 77.1% | 8.8% | 50.0% | 52.1% | 0.0% | 29.12 |
| main | answer_only | eval_template_shift | 2 | 48 | 75.0% | 5.9% | 0.0% | 0.0% | 0.0% | 9.92 |
| main | program_stack | eval_template_shift | 2 | 48 | 56.2% | 8.8% | 56.2% | 56.2% | 100.0% | 19.88 |
| main | program_python | eval_template_shift | 2 | 48 | 39.6% | 14.7% | 39.6% | 54.2% | 100.0% | 26.12 |
| main | zero_shot_answer | eval_template_shift | 1 | 24 | 37.5% | 0.0% | 0.0% | 0.0% | 25.0% | 24.00 |
Interpretation
The strongest procedure-level result is the externally executed trace_stack_final program: 93.8% execution accuracy on held-out compositions, with 39.6% final-answer accuracy. The generated procedure can be right while the answer token is wrong, so the emitted FINAL line is a confounded score for procedure arms. The best held-out composition arm was program_stack. Its family accuracies were: string 91.7%, unit 50.0%, table 50.0%, date 100.0%. The strongest strict program-only arm reached 72.9% on held-out compositions, compared with 43.8% for answer-only. This directly measures whether executable compilation improves composition rather than merely producing a plausible answer string. The same row has seed standard deviation 32.4%, so the result is positive but not yet stable enough to treat as a finished recipe. The trace-plus-final procedure execution was more stable than strict program-only emission in this compact run: composition execution standard deviation was 8.8%, versus 32.4% for the best strict program-only row. A program-only win would show that the model learned a useful executable ABI. A program-only loss, especially with high valid-execution rate, indicates that the model can imitate program syntax but still chooses the wrong operations or arguments.
Limitations
This is a compact controlled run. The generated domains are narrow, and the interpreters intentionally support only a small operation set. The result should be read as an ABI and supervision test, not a benchmark of general assistant capability.
Artifacts
- Metrics:
analysis/summary_by_arm.csvandanalysis/all_metrics.csv - Details:
analysis/all_details.csv - Training logs:
analysis/all_train_logs.csv - Checkpoints:
/workspace/large_artifacts/qwen_program_only_executable_abi/checkpoints
Experiment log 4
Show the running log (4 entries)
Setup
- Created: 2026-06-26 UTC
- Fresh experiment directory:
experiments/qwen_program_only_executable_abi - Large artifact directory:
large_artifacts/qwen_program_only_executable_abi - Base model:
Qwen/Qwen3-4B - Primary question: can a model emit an executable program when the final answer is not present in the target and the score comes only from interpreting the generated program?
Run smoke_v1
- Started: 2026-06-26 06:06:48 UTC
- Suite:
smoke - Model:
Qwen/Qwen3-4B - Seeds:
101 - Arms:
answer_only,program_stack - Train examples per seed:
24 - Eval examples per split:
8 - Steps:
2
Completed smoke_v1 in 79.4s.
- Metric rows: 9
- Detail rows: 72
- Training log rows: 4
Run pilot_v1
- Started: 2026-06-26 06:09:13 UTC
- Suite:
pilot - Model:
Qwen/Qwen3-4B - Seeds:
101 - Arms:
answer_only,trace_stack_final,program_stack,program_python - Train examples per seed:
96 - Eval examples per split:
16 - Steps:
24
Completed pilot_v1 in 297.7s.
- Metric rows: 15
- Detail rows: 240
- Training log rows: 28
Run main_v1
- Started: 2026-06-26 06:15:31 UTC
- Suite:
main - Model:
Qwen/Qwen3-4B - Seeds:
101,202 - Arms:
answer_only,trace_stack_final,program_stack,program_python - Train examples per seed:
192 - Eval examples per split:
24 - Steps:
48
Completed main_v1 in 1177.0s.
- Metric rows: 27
- Detail rows: 648
- Training log rows: 56
Figures 5
Data files 22
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv7.9 kBanalysis/main_v1_metrics.csv4.8 kBanalysis/pilot_v1_metrics.csv2.2 kBanalysis/smoke_v1_metrics.csv1.3 kBanalysis/summary_by_arm.csv7.1 kBruns/main_v1_answer_only_s101/metrics.csv745 Bruns/main_v1_answer_only_s202/metrics.csv699 Bruns/main_v1_program_python_s101/metrics.csv709 Bruns/main_v1_program_python_s202/metrics.csv662 Bruns/main_v1_program_stack_s101/metrics.csv689 Bruns/main_v1_program_stack_s202/metrics.csv642 Bruns/main_v1_trace_stack_final_s101/metrics.csv785 Bruns/main_v1_trace_stack_final_s202/metrics.csv774 Bruns/main_v1_zero_shot_answer/metrics.csv746 Bruns/pilot_v1_answer_only_s101/metrics.csv590 Bruns/pilot_v1_program_python_s101/metrics.csv597 Bruns/pilot_v1_program_stack_s101/metrics.csv600 Bruns/pilot_v1_trace_stack_final_s101/metrics.csv642 Bruns/pilot_v1_zero_shot_answer/metrics.csv610 Bruns/smoke_v1_answer_only_s101/metrics.csv570 Bruns/smoke_v1_program_stack_s101/metrics.csv586 Bruns/smoke_v1_zero_shot_answer/metrics.csv585 B
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.