Qwen Crystallized Trace ABI Tournament
The one idea you need
Imagine training a clerk to run fixed procedures like currency conversions, date math, and table lookups. You can teach using only the final number, or using a full written worksheet showing every step. This tests whether the worksheet version actually produces more correct answers.
The question
When you teach a small model to run step-by-step procedures, does making it write out every step give more correct answers than just teaching the final answer?
What we found
Barely. On familiar inputs, writing out each step scored 94% versus 92% for answer-only, a two-point edge that cost three to six times more generated text. On genuinely new combinations of steps the model had never seen, every format collapsed to roughly 30 to 40%, so the extra detail bought almost nothing where it mattered most. Answer-only was actually best on reworded prompts.
Why it matters
For deterministic, familiar tasks, answer-only supervision is cheaper and nearly as accurate. Verbose step-by-step traces do not buy real generalization to new step combinations, so reserve them for cases you have actually measured.
On this page
Results at a glance 5
How to read
Bars show percent correct (taller is better) for five output styles: answer-only, two simple step-by-step styles, a verbose structured-object style, and the untrained model. They are grouped by test type: familiar problems, reworded prompts, and brand-new step combinations.
Takeaway → On familiar and reworded problems the three simple trained styles cluster high near 85 to 94%, but all crash to roughly a third correct on brand-new step combinations; the verbose structured style trails even on familiar problems.
Data table
| Evaluation split | python trace | stack trace | answer-only | JSON trace | zero-shot (untrained) |
|---|---|---|---|---|---|
| fresh values (in-dist) | 93.8% | 93.8% | 91.7% | 25% | 0% |
| template shift | 83.3% | 87.5% | 89.6% | 39.6% | 0% |
| unseen composition | 39.6% | 29.2% | 33.3% | 12.5% | 0% |
Numbers from analysis/summary_by_arm.csv (main suite)
Technical framing
Accuracy by output format across evaluation splits — Python and stack traces edge out answer-only on fresh values, but every format collapses on unseen operation compositions.
How to read
Bars show the average amount of text the model writes to produce an answer on familiar problems (shorter is cheaper), one per output style. Answer-only sits far left near ten words, the verbose structured style far right near sixty-four.
Takeaway → The step-writing styles use three to six times more text than answer-only, a large cost for at most a two-point accuracy gain.
Data table
| Output format | mean generated tokens |
|---|---|
| answer-only | 9.73 |
| stack trace | 29.1 |
| python trace | 37.2 |
| JSON trace | 63.6 |
Numbers from analysis/summary_by_arm.csv (main suite, eval_indist tokens_mean)
Technical framing
Token cost of each output format (fresh-value eval) — Traces cost 3-6.5x more tokens than answer-only for at most a ~2pp accuracy gain on fresh values.
How to read
Bars show the share of responses that yield a readable final value (taller is better) for each style across the three test types, contrasting the simple line-based styles with the verbose structured-object style.
Takeaway → The three simple styles always produce a usable answer, while the verbose structured style fails to give one about a quarter of the time.
Data table
| Evaluation split | answer-only | python trace | stack trace | JSON trace |
|---|---|---|---|---|
| fresh values (in-dist) | 100% | 100% | 100% | 75% |
| template shift | 100% | 100% | 100% | 72.9% |
| unseen composition | 100% | 100% | 100% | 68.8% |
Numbers from analysis/summary_by_arm.csv (main suite, valid_mean)
Technical framing
Format validity: JSON is harder to emit reliably — Line-oriented formats always parse; the verbose JSON object fails to yield a valid FINAL value roughly a quarter of the time.
How to read
Bars show percent correct (taller is better) for the best step-writing style on brand-new step combinations, broken out by task type: text cleanup, unit conversion, table lookup, and date math.
Takeaway → Unit conversion stays fully correct while text cleanup and date math drop to zero, showing failures cluster where a genuinely new combination of steps is required.
Data table
| Task family | python trace |
|---|---|
| string normalization | 0% |
| unit conversion | 100% |
| table calculation | 58.3% |
| date arithmetic | 0% |
Numbers from analysis/summary_by_arm.csv (main, python, eval_composition family columns)
Technical framing
Where composition fails: python trace accuracy by task family (unseen compositions) — Composition failures concentrate in string and date tasks that need new operation combinations; direct unit conversion stays easy.
How to read
Bars show percent correct (taller is better) on reworded prompts, grouped by task type, with one bar per output style: answer-only, two simple step styles, and the verbose structured-object style.
Takeaway → Line-based styles stay above 83% on most task families (dipping as low as 67% on text cleanup), while the verbose structured style drops to zero on table lookup and date math, failing whole categories.
Data table
| Task family | Answer only | Python trace | Stack trace | JSON trace |
|---|---|---|---|---|
| String normalize | 83.3% | 75% | 66.7% | 58.3% |
| Unit convert | 100% | 75% | 100% | 100% |
| Lookup table | 83.3% | 100% | 91.7% | 0% |
| Date arithmetic | 91.7% | 83.3% | 91.7% | 0% |
Numbers from experiments/qwen_crystallized_trace_abi_tournament/analysis/summary_by_arm.csv
Technical framing
Template-shift accuracy by task family: JSON fails whole families — JSON's template-shift weakness is not uniform: it scores 0% on table and date tasks while other formats stay above 83% on most families.
In the author’s words from the Report · “Abstract”
This experiment tests whether a local 4B language model learns practical deterministic procedures better when supervised with compact executable traces instead of final answers alone. The same generated tasks are rendered through four output ABIs: final-answer text, Python-like trace, JSON IR, and stack-style IR. Each trained arm uses the same QLoRA budget and is evaluated on fresh values, unseen operation compositions, and template-shifted prompts.
Overview
This standalone experiment tests whether dense executable traces help a local 4B model learn compact crystallized procedures better than answer-only supervision, and whether the output representation itself is a load-bearing choice.
The experiment generates deterministic tasks from several practical families, trains small QLoRA adapters for multiple output ABIs, evaluates held-out examples, and writes Markdown and HTML reports with charts.
Large adapter checkpoints are stored outside this directory under:
/workspace/large_artifacts/qwen_crystallized_trace_abi_tournament
Report
Rendered from reports/qwen_crystallized_trace_abi_tournament_report.md
Abstract
This experiment tests whether a local 4B language model learns practical deterministic procedures better when supervised with compact executable traces instead of final answers alone. The same generated tasks are rendered through four output ABIs: final-answer text, Python-like trace, JSON IR, and stack-style IR. Each trained arm uses the same QLoRA budget and is evaluated on fresh values, unseen operation compositions, and template-shifted prompts.
Method
The task factory creates examples from four families: string normalization, unit conversion, lookup-table calculation, and date arithmetic. Every example has a deterministic answer and a gold procedural rendering for each ABI. Evaluation uses greedy generation and exact matching of the parsed FINAL value.
Run Configuration
- Primary suite:
main. - Adapter seeds:
101,202. - QLoRA update steps per adapter:
48. - Total trained-arm evaluation examples:
eval_composition=192,eval_indist=192,eval_template_shift=192. - Output ABIs:
answer,python,json, andstack.
Primary Results
- Best fresh-value arm:
pythonat 93.8%. - Best unseen-composition arm:
pythonat 39.6%. - Best template-shift arm:
answerat 89.6%. - Primary suite summarized below:
main.
| suite | arm | split | runs | n_total | accuracy_mean | accuracy_std | valid_mean | tokens_mean |
|---|---|---|---|---|---|---|---|---|
| main | python | eval_composition | 2 | 48 | 39.6% | 2.9% | 100.0% | 44.60 |
| main | answer | eval_composition | 2 | 48 | 33.3% | 0.0% | 100.0% | 9.60 |
| main | stack | eval_composition | 2 | 48 | 29.2% | 11.8% | 100.0% | 31.94 |
| main | json | eval_composition | 2 | 48 | 12.5% | 5.9% | 68.8% | 63.88 |
| main | zero_shot_answer | eval_composition | 1 | 24 | 0.0% | 0.0% | 20.8% | 32.00 |
| main | python | eval_indist | 2 | 48 | 93.8% | 2.9% | 100.0% | 37.25 |
| main | stack | eval_indist | 2 | 48 | 93.8% | 2.9% | 100.0% | 29.08 |
| main | answer | eval_indist | 2 | 48 | 91.7% | 5.9% | 100.0% | 9.73 |
| main | json | eval_indist | 2 | 48 | 25.0% | 0.0% | 75.0% | 63.56 |
| main | zero_shot_answer | eval_indist | 1 | 24 | 0.0% | 0.0% | 4.2% | 32.00 |
| main | answer | eval_template_shift | 2 | 48 | 89.6% | 14.7% | 100.0% | 9.42 |
| main | stack | eval_template_shift | 2 | 48 | 87.5% | 5.9% | 100.0% | 28.71 |
| main | python | eval_template_shift | 2 | 48 | 83.3% | 11.8% | 100.0% | 37.35 |
| main | json | eval_template_shift | 2 | 48 | 39.6% | 2.9% | 72.9% | 62.83 |
| main | zero_shot_answer | eval_template_shift | 1 | 24 | 0.0% | 0.0% | 4.2% | 32.00 |
Interpretation
The composition gap for the best fresh-value arm is 54.2 percentage points. A small or negative gap would indicate that the learned representation transfers across operation combinations; a large positive gap indicates that the model mainly learned the easier in-distribution mapping. On fresh values, the best trace ABI scored 93.8% and answer-only scored 91.7%. The shortest strong trace ABI used 29.1 generated tokens on average, while answer-only used 9.7. This is a weak trace advantage in accuracy and a large trace cost in tokens. For the best unseen-composition arm, family accuracy was: string 0.0%, unit 100.0%, table 58.3%, date 0.0%. The composition failures are concentrated in families that require a new operation combination, while direct unit conversion remains easy. The JSON ABI had an average valid-final rate of 72.2% across primary splits, showing that a verbose structured object can be harder to emit reliably than line-oriented formats.
Limitations
This is a compact experiment. It is designed to reveal representation sensitivity and supervision effects, not to maximize absolute performance. The generated tasks are deterministic and intentionally narrow enough to allow controlled held-out splits. Larger task coverage and longer training would be needed before treating any ABI as a production recipe.
Artifacts
- Metrics:
analysis/summary_by_arm.csvandanalysis/all_metrics.csv - Details:
analysis/all_details.csv - Training logs:
analysis/all_train_logs.csv - Checkpoints:
/workspace/large_artifacts/qwen_crystallized_trace_abi_tournament/checkpoints
Experiment log 4
Show the running log (4 entries)
Setup
- Created: 2026-06-26 UTC
- Fresh experiment directory:
experiments/qwen_crystallized_trace_abi_tournament - Large artifact directory:
large_artifacts/qwen_crystallized_trace_abi_tournament - Base model:
Qwen/Qwen3-4B - Primary question: can dense trace supervision over a practical executable ABI improve held-out exact-answer accuracy over answer-only fine-tuning?
Run smoke_v1
- Started: 2026-06-26 04:20:46 UTC
- Suite:
smoke - Model:
Qwen/Qwen3-4B - Seeds:
101 - Arms:
answer,python - Train examples per seed:
24 - Eval examples per split:
8 - Steps:
2
Completed smoke_v1 in 146.4s.
- Metric rows: 9
- Detail rows: 72
- Training log rows: 4
Run pilot_v1
- Started: 2026-06-26 04:24:02 UTC
- Suite:
pilot - Model:
Qwen/Qwen3-4B - Seeds:
101 - Arms:
answer,python,json,stack - Train examples per seed:
96 - Eval examples per split:
16 - Steps:
24
Completed pilot_v1 in 367.7s.
- Metric rows: 15
- Detail rows: 240
- Training log rows: 28
Run main_v1
- Started: 2026-06-26 04:30:45 UTC
- Suite:
main - Model:
Qwen/Qwen3-4B - Seeds:
101,202 - Arms:
answer,python,json,stack - Train examples per seed:
192 - Eval examples per split:
24 - Steps:
48
Completed main_v1 in 1324.1s.
- Metric rows: 27
- Detail rows: 648
- Training log rows: 56
Figures 4
Data files 22
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv5.7 kBanalysis/main_v1_metrics.csv3.5 kBanalysis/pilot_v1_metrics.csv1.6 kBanalysis/smoke_v1_metrics.csv970 Banalysis/summary_by_arm.csv3.8 kBruns/main_v1_answer_s101/metrics.csv517 Bruns/main_v1_answer_s202/metrics.csv479 Bruns/main_v1_json_s101/metrics.csv449 Bruns/main_v1_json_s202/metrics.csv505 Bruns/main_v1_python_s101/metrics.csv515 Bruns/main_v1_python_s202/metrics.csv485 Bruns/main_v1_stack_s101/metrics.csv528 Bruns/main_v1_stack_s202/metrics.csv450 Bruns/main_v1_zero_shot_answer/metrics.csv469 Bruns/pilot_v1_answer_s101/metrics.csv402 Bruns/pilot_v1_json_s101/metrics.csv397 Bruns/pilot_v1_python_s101/metrics.csv408 Bruns/pilot_v1_stack_s101/metrics.csv396 Bruns/pilot_v1_zero_shot_answer/metrics.csv433 Bruns/smoke_v1_answer_s101/metrics.csv388 Bruns/smoke_v1_python_s101/metrics.csv393 Bruns/smoke_v1_zero_shot_answer/metrics.csv425 B
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.