Qwen Numeric-Copy Compiler
The one idea you need
Instead of doing chained arithmetic in its head and reading the digits back, the four-billion-parameter model only points: which word is the starting number, which words are the operations, which are their arguments, and in what order. An exact calculator then copies those symbols and runs the math flawlessly.
The question
Can a small model reliably run multi-step arithmetic if it just locates the numbers and operations in the text and lets an exact calculator do the math?
What we found
Yes. When the model just points to where each number and operation sits and copies the exact symbols for a hidden calculator to run, it solves four-step problems about 90% of the time. A version trained to write the answer itself never left the floor, at most 3% at any length. A shortcut trained only on final answers aced four steps but crashed to zero at eight, never learning the ordered pointing that stretches to longer chains.
Why it matters
For long step-by-step symbolic work, stop making a small model regenerate exact values from memory. Have it mark the right token positions and copy symbols verbatim into a real executor; reliability then hinges on locating slots, not doing arithmetic.
On this page
Results at a glance 3
How to read
Bars are grouped by chain length (4, 8, 12, and 24 steps); bar height is the share of chains a method solves correctly end to end, so taller is better. The four colors are four methods, from point-and-copy down to writing the answer directly.
Takeaway → The point-and-copy bars tower over the rest at every length past four steps; at four steps the shortcut method spikes to a perfect score, just edging out point-and-copy, while writing the answer directly stays pinned near the floor throughout.
Data table
| program length | QLoRA copy-trace compiler | frozen-backbone copy-trace | answer-only compiler control | direct final-answer QLoRA |
|---|---|---|---|---|
| 4 steps | 89.8% | 83.6% | 100% | 3.1% |
| 8 steps | 72.7% | 46.1% | 0% | 0.8% |
| 12 steps | 46.9% | 16.4% | 1.6% | 0% |
| 24 steps | 20.3% | 1.6% | 2.3% | 0.8% |
Numbers from experiments/qwen_numeric_copy_compiler/analysis/final_metrics.csv
Technical framing
Exact program execution by chain length (standard prompts) — Trace-supervised numeric-copy compiling executes long modular programs; direct answering stays at chance and answer-only training only memorizes length 4.
How to read
Lines track the share of chains solved correctly as they lengthen up to 24 steps, though practice stopped at 12; higher is better. Colors compare the point-and-copy model against a frozen-backbone version, each on plain and reworded prompts.
Takeaway → The point-and-copy line on plain prompts holds near 20% at 24 steps, while the frozen version collapses to almost nothing there.
Data table
| program length (trained on 1-12 steps) | QLoRA, standard | QLoRA, paraphrase | frozen, standard | frozen, paraphrase |
|---|---|---|---|---|
| 4 | 89.8% | 85.9% | 83.6% | 87.5% |
| 8 | 72.7% | 63.3% | 46.1% | 68.8% |
| 12 | 46.9% | 46.1% | 16.4% | 46.1% |
| 24 | 20.3% | 5.5% | 1.6% | 14.8% |
Numbers from experiments/qwen_numeric_copy_compiler/analysis/final_metrics.csv
Technical framing
Length generalization: executor accuracy vs chain length — QLoRA keeps 20% exact execution at 24 steps, twice the training length, where the frozen pilot collapses on standard prompts.
How to read
Lines show the share correct for each program piece — the starting value, the operations, the number arguments, and the whole program together — as chains lengthen; higher is better.
Takeaway → Starting values and operations stay pinned near perfect while the whole-program line sags, showing tiny slips in the number slots are what compound.
Data table
| program length | initial value | operations | arguments | whole program exact |
|---|---|---|---|---|
| 4 | 100% | 100% | 97.5% | 89.8% |
| 8 | 100% | 100% | 96.1% | 72.7% |
| 12 | 100% | 100% | 94.1% | 46.9% |
| 24 | 100% | 99.9% | 92.9% | 18% |
Numbers from experiments/qwen_numeric_copy_compiler/analysis/final_metrics.csv
Technical framing
Where the QLoRA compiler loses accuracy (standard prompts) — Initial values and operations are solved almost perfectly; small per-slot argument errors compound into the whole-program exact-match falloff.
In the author’s words from the Report · “Summary”
The full QLoRA trace condition reached 89.8% exact execution at standard length 4, 72.7% at length 8, 46.9% at length 12, and 20.3% at length 24. Direct final-answer QLoRA stayed at chance. An answer-only numeric-copy compiler learned length-4 programs exactly but did not generalize to longer chains. The key finding is that exact numeric copying changes the failure mode. The model no longer has to infer numeric values from hidden states; it only has to learn where the program tokens are. That makes the latent compiler much more reliable, especially on trained-length and moderately longer programs.
Overview
Standalone numeric-copy compiler experiment for a Qwen causal language model.
The experiment tests whether a model can learn token roles and ordered program slots while numeric and operator symbols are copied from a deterministic token map. The frozen pilot trains only the compiler heads over frozen hidden states. The full condition trains QLoRA adapters plus the same numeric-copy compiler.
Layout
src/qwen_numeric_copy_compiler_experiment.py training and evaluation harness
src/analyze_qwen_numeric_copy_compiler.py run aggregation and plots
runs/ lightweight JSON and CSV outputs
reports/ experiment log and standalone write-up
analysis/ aggregate tables and figuresLarge checkpoints are stored outside the experiment directory:
large_artifacts/qwen_numeric_copy_compiler/checkpoints/Reading Order
reports/qwen_numeric_copy_compiler_paper.mdanalysis/summary.mdanalysis/final_metrics.csvreports/qwen_numeric_copy_compiler_experiment_log.md
Main Artifact
The main trained QLoRA condition is:
large_artifacts/qwen_numeric_copy_compiler/checkpoints/main_qwen3_4b_qlora_numeric_copy_trace_mixed_l12/The checkpoint manifest is:
experiments/qwen_numeric_copy_compiler/checkpoint_manifest.csvReport
Rendered from reports/qwen_numeric_copy_compiler_paper.md
Summary
This experiment tested whether a Qwen causal language model can expose a modular-arithmetic program through learned token roles while exact numeric and operation symbols are copied from a lexical token map. The frozen pilot trained only compiler heads over frozen hidden states. The full condition trained QLoRA adapters and compiler heads live. Both used the same executable modular runtime.
The result is positive. The full QLoRA trace condition reached 89.8% exact execution at standard length 4, 72.7% at length 8, 46.9% at length 12, and 20.3% at length 24. Direct final-answer QLoRA stayed at chance. An answer-only numeric-copy compiler learned length-4 programs exactly but did not generalize to longer chains.
The key finding is that exact numeric copying changes the failure mode. The model no longer has to infer numeric values from hidden states; it only has to learn where the program tokens are. That makes the latent compiler much more reliable, especially on trained-length and moderately longer programs.
Setup
- Base model:
Qwen/Qwen3-4B - Quantization: 4-bit NF4
- Full-condition trainable update: LoRA rank 8, alpha 16, dropout 0.05, target
all-linear - Task: modular arithmetic instructions modulo 97
- Train lengths: 1-12 steps
- Evaluation lengths: 4, 8, 12, 24 steps
- Evaluation templates: standard and paraphrase
- Evaluation size: 128 examples per split
- Large checkpoints:
large_artifacts/qwen_numeric_copy_compiler/checkpoints/
Each prompt describes an initial value and a sequence of add, subtract, and multiply updates. The compiler predicts:
- the token position of the initial value;
- ordered operation token positions;
- ordered argument token positions.
Numeric residues and operation IDs are then copied from deterministic per-token maps. The executor applies the copied program exactly modulo 97.
Runs
| Run | Variant | Purpose |
|---|---|---|
pilot_qwen3_4b_frozen_numeric_copy_trace_mixed_l12 | copy_trace | Frozen-backbone pilot gate with trace and executor loss. |
main_qwen3_4b_qlora_numeric_copy_trace_mixed_l12 | copy_trace | Full QLoRA numeric-copy compiler condition. |
control_qwen3_4b_direct_numeric_copy_distribution_l12 | direct | Final-answer QLoRA control without compiler structure. |
control_qwen3_4b_qlora_numeric_copy_answer_only_l12 | copy_answer_only | Numeric-copy compiler trained only from final-answer loss. |
Smoke runs verify the harness and are not part of the main comparison.
Main QLoRA Result
| Split | Executor | Init | Op | Arg | Arg Pos | Program Exact |
|---|---|---|---|---|---|---|
| Standard L4 | 89.8% | 100.0% | 100.0% | 97.5% | 96.9% | 89.8% |
| Standard L8 | 72.7% | 100.0% | 100.0% | 96.1% | 95.7% | 72.7% |
| Standard L12 | 46.9% | 100.0% | 100.0% | 94.1% | 94.0% | 46.9% |
| Standard L24 | 20.3% | 100.0% | 99.9% | 92.9% | 95.3% | 18.0% |
| Paraphrase L4 | 85.9% | 100.0% | 100.0% | 96.5% | 95.1% | 85.9% |
| Paraphrase L8 | 63.3% | 100.0% | 100.0% | 94.5% | 93.6% | 63.3% |
| Paraphrase L12 | 46.1% | 100.0% | 100.0% | 93.6% | 92.6% | 46.1% |
| Paraphrase L24 | 5.5% | 100.0% | 97.4% | 85.8% | 84.9% | 5.5% |
The compiler solves initial values and operations almost completely. The remaining loss comes from argument slot errors, especially on paraphrase length 24. The standard length-24 result is notable: argument value accuracy stays at 92.9% and program exact reaches 18.0%, even though evaluation length is twice the maximum training length.
Frozen Pilot
| Split | Executor | Init | Op | Arg | Arg Pos | Program Exact |
|---|---|---|---|---|---|---|
| Standard L4 | 83.6% | 100.0% | 100.0% | 95.5% | 94.5% | 82.8% |
| Standard L8 | 46.1% | 100.0% | 100.0% | 90.2% | 88.5% | 45.3% |
| Standard L12 | 16.4% | 100.0% | 100.0% | 86.5% | 84.2% | 16.4% |
| Standard L24 | 1.6% | 100.0% | 93.9% | 77.6% | 75.3% | 0.0% |
| Paraphrase L4 | 87.5% | 100.0% | 100.0% | 96.9% | 94.1% | 87.5% |
| Paraphrase L8 | 68.8% | 100.0% | 100.0% | 95.4% | 91.5% | 68.8% |
| Paraphrase L12 | 46.1% | 100.0% | 100.0% | 93.7% | 90.4% | 45.3% |
| Paraphrase L24 | 14.8% | 100.0% | 99.5% | 90.2% | 88.2% | 13.3% |
The frozen pilot passed the gate. It showed that Qwen hidden states already support a numeric-copy compiler when trained with token-role traces. The QLoRA condition improves standard long-chain performance but does not uniformly improve every paraphrase split.
Controls
| Run | Standard L4 | Standard L8 | Standard L12 | Standard L24 | Paraphrase L4 | Paraphrase L8 | Paraphrase L12 | Paraphrase L24 |
|---|---|---|---|---|---|---|---|---|
| Direct final-answer QLoRA | 3.1% | 0.8% | 0.0% | 0.8% | 1.6% | 3.1% | 3.9% | 1.6% |
| Answer-only numeric-copy compiler | 100.0% | 0.0% | 1.6% | 2.3% | 100.0% | 7.0% | 1.6% | 0.0% |
The direct control stayed at chance. The answer-only compiler control is more interesting: it solved length-4 programs perfectly but did not extend to longer chains. Its slot-position diagnostics were poor despite perfect length-4 execution, indicating that it found a short-horizon solution rather than the ordered slot interface learned by trace supervision.
Interpretation
The experiment isolates a concrete bottleneck. A compiler that classifies numeric values from hidden states is too noisy for exact multi-step execution. A compiler that copies numeric values from selected tokens is much stronger. Once numeric copying is used, performance tracks slot localization accuracy.
The result supports three claims:
- Frozen Qwen hidden states contain enough information for a trace-supervised numeric-copy compiler.
- Live QLoRA training can improve the compiler on standard longer chains.
- Final-answer training alone can find a short numeric-copy solution, but it does not discover a length-generalizing ordered compiler under this budget.
The result does not show a universal intelligence improvement. It shows a practical recipe for this class of serial symbolic tasks: learn where the program tokens are, copy exact symbols, and run an invisible executor.
Recommended Next Experiment
The next experiment should target robust slot generalization rather than numeric value decoding.
Recommended design:
- Keep numeric-copy readout.
- Train a slot-stability objective that penalizes drift in ordered operation and argument slots across paraphrase variants of the same program.
- Add a curriculum that mixes short and long lengths every batch instead of relying on uniform random lengths.
- Save mid-run checkpoints and select by validation program exactness, because the frozen pilot showed different splits peaking at different times.
Success criteria:
- standard and paraphrase L12 program exact above 60%;
- paraphrase L24 above 20%;
- answer-only control remains unable to solve L8+ without traces.
Artifacts
Small files:
- Code:
experiments/qwen_numeric_copy_compiler/src/ - Runs:
experiments/qwen_numeric_copy_compiler/runs/ - Analysis:
experiments/qwen_numeric_copy_compiler/analysis/ - Manifest:
experiments/qwen_numeric_copy_compiler/checkpoint_manifest.csv
Large files:
- Frozen pilot heads:
large_artifacts/qwen_numeric_copy_compiler/checkpoints/pilot_qwen3_4b_frozen_numeric_copy_trace_mixed_l12/ - Main QLoRA adapter and heads:
large_artifacts/qwen_numeric_copy_compiler/checkpoints/main_qwen3_4b_qlora_numeric_copy_trace_mixed_l12/ - Direct control adapter and heads:
large_artifacts/qwen_numeric_copy_compiler/checkpoints/control_qwen3_4b_direct_numeric_copy_distribution_l12/ - Answer-only control adapter and heads:
large_artifacts/qwen_numeric_copy_compiler/checkpoints/control_qwen3_4b_qlora_numeric_copy_answer_only_l12/
Experiment log 7
Show the running log (7 entries, 2026-06-21)
Objective
Test whether a Qwen model can expose a symbolic modular-arithmetic program through learned token roles while exact numeric and operator values are copied from lexical token maps instead of inferred by semantic value classifiers.
Experiment Question
Can a frozen hidden-state parser pass a numeric-copy pilot gate, and does live QLoRA training improve or preserve that compiler interface on standard and paraphrased arithmetic programs?
Planned Gates
- Frozen numeric-copy pilot: train compiler heads over frozen model hidden states, with the backbone kept fixed.
- QLoRA numeric-copy compiler: train LoRA adapters and compiler heads live.
- Controls: final-answer direct adaptation and answer-only compiler training.
Primary Metrics
init_pos_accuracy,op_pos_accuracy,arg_pos_accuracy: role and slot localization.init_accuracy,op_accuracy,arg_accuracy: copied program-symbol correctness.program_exact: exact compiled program correctness.executor_accuracy: exact modular execution accuracy.- Standard and paraphrase splits are evaluated separately at multiple chain lengths.
Artifact Policy
Lightweight outputs stay in experiments/qwen_numeric_copy_compiler/runs/.
Large adapters and head checkpoints stay in large_artifacts/qwen_numeric_copy_compiler/checkpoints/.
Log
2026-06-21
- Created standalone experiment directory and external checkpoint path.
- Implemented the numeric-copy harness and analyzer.
- Tiny frozen-backbone smoke test passed as
runs/smoke_tiny_frozen_copy_trace/. - Qwen frozen micro-smoke passed as
runs/smoke_qwen3_4b_frozen_copy_trace/. Frozen numeric-copy pilot completed as
runs/pilot_qwen3_4b_frozen_numeric_copy_trace_mixed_l12/.- Final standard executor accuracy: L4 83.6%, L8 46.1%, L12 16.4%, L24 1.6%.
- Final paraphrase executor accuracy: L4 87.5%, L8 68.8%, L12 46.1%, L24 14.8%.
- Final copied argument accuracy stayed high through paraphrase L12 at 93.7%, but standard long-chain argument accuracy was lower.
- The frozen pilot gate passed, so the live QLoRA numeric-copy condition was run next.
QLoRA numeric-copy trace condition completed as
runs/main_qwen3_4b_qlora_numeric_copy_trace_mixed_l12/.- Final standard executor accuracy: L4 89.8%, L8 72.7%, L12 46.9%, L24 20.3%.
- Final paraphrase executor accuracy: L4 85.9%, L8 63.3%, L12 46.1%, L24 5.5%.
- Standard L24 copied argument accuracy reached 92.9%, up from 77.6% in the frozen pilot.
- Final-answer controls were run after this condition.
Direct final-answer QLoRA control completed as
runs/control_qwen3_4b_direct_numeric_copy_distribution_l12/.- Final accuracy stayed near chance on every split.
Answer-only numeric-copy compiler control completed as
runs/control_qwen3_4b_qlora_numeric_copy_answer_only_l12/.- It solved length-4 programs exactly, but stayed near chance at length 8 and above.
- This indicates final-answer loss can discover a short numeric-copy shortcut, but did not discover the length-generalizing slot interface under this budget.
- Regenerated aggregate analysis and checkpoint manifest.
- Wrote standalone Markdown and HTML reports.
Completion audit passed:
- Source compilation passed.
- Analyzer regeneration passed.
- Standalone wording scan found no stale references.
- Expected README, report, log, analysis, run, source, and manifest files are present.
checkpoint_manifest.csvindexes 18 large checkpoint files.
Figures 3
Data files 7
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/final_metrics.csv6.1 kBruns/control_qwen3_4b_direct_numeric_copy_distribution_l12/results.json21 kBruns/control_qwen3_4b_qlora_numeric_copy_answer_only_l12/results.json23 kBruns/main_qwen3_4b_qlora_numeric_copy_trace_mixed_l12/results.json24 kBruns/pilot_qwen3_4b_frozen_numeric_copy_trace_mixed_l12/results.json23 kBruns/smoke_qwen3_4b_frozen_copy_trace/results.json4.5 kBruns/smoke_tiny_frozen_copy_trace/results.json7.3 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.