Qwen LoRA Parser Compiler
The one idea you need
Picture tapping into a model's hidden scratchpad. A light, cheap retraining installs a reader that turns its internal states into a written arithmetic recipe: a starting number, each operation, and the number each step uses. The steps come through crisp; the numbers stay blurry.
The question
Can you cheaply retrain a small model so its internal states can be read as a trustworthy, step-by-step arithmetic program you could run yourself?
What we found
Partly. With step-by-step coaching, a small four-billion-parameter model's hidden states became a readable program: it named the starting number every time and picked the right operation about 98 percent of the time, while ordinary answer-only training stayed near zero. But the actual numbers came through only about 80 percent of the time per step, so exact recipes fell to zero by twelve steps.
Why it matters
If you want a small model's reasoning to be machine-readable, explicit per-step coaching really does install the interface, but audit each channel on its own. One noisy field compounds across steps and quietly kills exact multi-step execution.
On this page
Results at a glance 3
How to read
Bars show the share of four-step problems each training approach solves exactly; taller is better. Two colors mark standard versus reworded question wording. The two coached approaches on the left tower over the two plain-training controls on the right.
Takeaway → Coached bars reach 33 to 41 percent while both controls sit near zero, so the readable recipe comes from the coaching, not from ordinary answer-fitting.
Data table
| Training condition | standard prompts | paraphrase prompts |
|---|---|---|
| trace + executor (main) | 32.8% | 40.6% |
| tagger pilot | 17.2% | 32.8% |
| answer-only compiler | 0% | 0% |
| direct final-answer QLoRA | 3.1% | 0% |
Numbers from experiments/qwen_lora_parser_compiler/analysis/final_metrics.csv
Technical framing
Exact-answer accuracy on length-4 programs: trace supervision vs controls — Explicit trace supervision makes hidden states compile to correct answers (33-41%); answer-only training stays near zero.
How to read
Each line tracks the share of problems solved exactly as length grows from four to twenty-four steps; higher is better. The two coached lines start well above the flat, near-floor answer-only control line, then slide down toward it.
Takeaway → Both coached lines fall to near zero by twelve steps, showing tiny per-step errors pile up and wreck long problems even when short ones work.
Data table
| Program length (steps) | trace + executor, standard | trace + executor, paraphrase | answer-only control, standard |
|---|---|---|---|
| 4 | 32.8% | 40.6% | 0% |
| 8 | 10.9% | 6.2% | 1.6% |
| 12 | 0% | 4.7% | 3.1% |
| 24 | 0% | 0% | 3.1% |
Numbers from experiments/qwen_lora_parser_compiler/analysis/final_metrics.csv
Technical framing
Executor accuracy collapses with program length — Per-step argument errors compound: exact execution is near zero by 12 steps even though the interface works at length 4.
How to read
Each line shows how accurately one piece of the recipe reads as problems lengthen; higher is better. The starting-number line stays pinned at the top, the operation line sits just below, and the step-number line trails lowest.
Takeaway → The step-number line sits near 80 percent and keeps sliding while the starting-number line holds perfect and the operation line stays high, pinpointing number-reading as the weak link.
Data table
| Program length (steps) | initial value | operation | argument value |
|---|---|---|---|
| 4 | 100% | 97.7% | 79.7% |
| 8 | 100% | 97.7% | 77.9% |
| 12 | 100% | 94% | 70.7% |
| 24 | 100% | 81.4% | 67.4% |
Numbers from experiments/qwen_lora_parser_compiler/analysis/final_metrics.csv
Technical framing
Which program symbols the parser decodes (main condition, standard prompts) — Initial values and operations decode almost perfectly; noisy argument-value decoding (~80% and falling) is the bottleneck.
In the author’s words from the Report · “Summary”
The full trace+executor condition reached 32.8% standard length-4 executor accuracy and 40.6% paraphrase length-4 accuracy, while direct final-answer QLoRA and answer-only compiler training stayed near chance. The failure mode is also clear: argument value decoding is not accurate enough. It remains near 80% on length-4 programs and falls with length, which compounds into near-zero exact programs by length 12. The experiment supports a narrow claim: explicit parser supervision can make a small QLoRA-adapted model expose a partial symbolic program in hidden states. It does not support a broad claim of large universal intelligence improvement from this recipe.
Overview
Standalone QLoRA-attached parser/compiler experiment.
The experiment tests whether live LoRA updates on a small causal language model can make its hidden states readable as a symbolic modular-arithmetic program. The pilot gate is a token-role and symbol tagger trained through the model; the full condition adds a differentiable executor that compiles predicted program symbols into the final answer.
Layout
src/qwen_lora_parser_compiler_experiment.py training and evaluation harness
src/analyze_qwen_lora_parser_compiler.py run aggregation and plots
runs/ lightweight JSON and CSV outputs
reports/ experiment log and standalone write-up
analysis/ aggregate tables and figuresLarge checkpoints are stored outside the experiment directory:
large_artifacts/qwen_lora_parser_compiler/checkpoints/Reading Order
reports/qwen_lora_parser_compiler_paper.mdanalysis/summary.mdanalysis/final_metrics.csvreports/qwen_lora_parser_compiler_experiment_log.md
Large Artifacts
You only need the large artifact directory to load adapters or resume/evaluate a checkpoint. The main trained condition is:
large_artifacts/qwen_lora_parser_compiler/checkpoints/main_qwen3_4b_qlora_trace_argstrong_mixed_l12/The checkpoint manifest is:
experiments/qwen_lora_parser_compiler/checkpoint_manifest.csvReport
Rendered from reports/qwen_lora_parser_compiler_paper.md
Summary
This experiment tested whether QLoRA posttraining can install a small symbolic interface on top of Qwen/Qwen3-4B. The model was trained live with LoRA adapters while a parser read the final hidden sequence and emitted a modular-arithmetic program: an initial value, ordered operations, and ordered arguments. A differentiable executor then ran the predicted program modulo 97.
The result is mixed. Trace supervision created a real, measurable program interface that ordinary final-answer training did not discover. The full trace+executor condition reached 32.8% standard length-4 executor accuracy and 40.6% paraphrase length-4 accuracy, while direct final-answer QLoRA and answer-only compiler training stayed near chance. The failure mode is also clear: argument value decoding is not accurate enough. It remains near 80% on length-4 programs and falls with length, which compounds into near-zero exact programs by length 12.
The experiment supports a narrow claim: explicit parser supervision can make a small QLoRA-adapted model expose a partial symbolic program in hidden states. It does not support a broad claim of large universal intelligence improvement from this recipe.
Setup
- Base model:
Qwen/Qwen3-4B - Loading: 4-bit NF4 quantization
- Trainable model update: LoRA rank 8, alpha 16, dropout 0.05, target
all-linear - Task: synthetic text instructions for arithmetic modulo 97
- Train lengths: 1-12 steps
- Evaluation lengths: 4, 8, 12, 24 steps
- Evaluation templates: standard and paraphrase
- Evaluation size: 64 examples per split
- Large checkpoint storage:
large_artifacts/qwen_lora_parser_compiler/checkpoints/
The parser predicts:
- token positions for the initial value, each operation, and each argument;
- symbols for the initial value, operation class, and argument value;
- an answer distribution through exact differentiable modular execution.
Runs
| Run | Variant | Purpose |
|---|---|---|
pilot_qwen3_4b_qlora_tagger_mixed_l12 | qlora_tagger | Pilot gate: learn token roles and program symbols with trace supervision only. |
main_qwen3_4b_qlora_trace_argstrong_mixed_l12 | qlora_trace | Full condition: trace supervision plus differentiable executor loss, with stronger argument supervision. |
control_qwen3_4b_direct_mixed_l12 | direct | Ordinary final-answer QLoRA control using an answer-marker hidden-state classifier. |
control_qwen3_4b_qlora_answer_only_mixed_l12 | qlora_answer_only | Compiler/executor trained from final-answer loss only, without trace supervision. |
Smoke runs are present only to verify the harness and checkpoint path.
Main Result
Full Trace+Executor Condition
| Split | Executor | Init | Op | Arg | Program Exact |
|---|---|---|---|---|---|
| Standard L4 | 32.8% | 100.0% | 97.7% | 79.7% | 32.8% |
| Standard L8 | 10.9% | 100.0% | 97.7% | 77.9% | 9.4% |
| Standard L12 | 0.0% | 100.0% | 94.0% | 70.7% | 0.0% |
| Standard L24 | 0.0% | 100.0% | 81.4% | 67.4% | 0.0% |
| Paraphrase L4 | 40.6% | 100.0% | 96.9% | 82.4% | 40.6% |
| Paraphrase L8 | 6.2% | 100.0% | 97.7% | 75.8% | 6.2% |
| Paraphrase L12 | 4.7% | 100.0% | 90.5% | 69.3% | 0.0% |
| Paraphrase L24 | 0.0% | 100.0% | 75.5% | 53.1% | 0.0% |
The parser learned initial values and operations well. It also usually located argument tokens. The blocker is argument value decoding. A per-step argument error rate around 20-45% makes exact execution collapse as program length grows.
Token-Tagger Pilot
| Split | Executor | Init | Op | Arg | Op Pos | Arg Pos | Program Exact |
|---|---|---|---|---|---|---|---|
| Standard L4 | 17.2% | 96.9% | 98.0% | 76.2% | 100.0% | 89.5% | 17.2% |
| Standard L8 | 3.1% | 98.4% | 94.9% | 73.8% | 100.0% | 85.2% | 0.0% |
| Standard L12 | 0.0% | 96.9% | 95.6% | 68.8% | 100.0% | 82.4% | 0.0% |
| Standard L24 | 1.6% | 100.0% | 88.2% | 65.4% | 90.2% | 77.4% | 0.0% |
| Paraphrase L4 | 32.8% | 98.4% | 99.2% | 77.7% | 100.0% | 94.5% | 32.8% |
| Paraphrase L8 | 17.2% | 95.3% | 99.0% | 77.5% | 100.0% | 93.0% | 15.6% |
| Paraphrase L12 | 0.0% | 100.0% | 99.2% | 71.6% | 100.0% | 91.7% | 0.0% |
| Paraphrase L24 | 1.6% | 93.8% | 97.0% | 71.2% | 99.4% | 87.3% | 0.0% |
This pilot passed the narrow gate: QLoRA plus trace heads can make Qwen hidden states parseable. It did not pass the stronger gate needed for long exact execution.
Controls
| Run | Standard L4 | Standard L8 | Standard L12 | Standard L24 | Paraphrase L4 | Paraphrase L8 | Paraphrase L12 | Paraphrase L24 |
|---|---|---|---|---|---|---|---|---|
| Direct final-answer QLoRA | 3.1% | 0.0% | 0.0% | 1.6% | 0.0% | 0.0% | 1.6% | 0.0% |
| Answer-only compiler | 0.0% | 1.6% | 3.1% | 3.1% | 0.0% | 1.6% | 0.0% | 0.0% |
The controls stayed at chance. The learned structure in the trace-supervised runs is therefore not explained by ordinary answer fitting, and the compiler interface was not discovered from final-answer loss alone.
Interpretation
The useful signal is not the final executor score by itself. The useful signal is that explicit trace supervision made internal hidden states reliably expose parts of a program:
- Initial value decoding reached 100.0% in the full condition.
- Operation decoding stayed high through length 12 and degraded at length 24.
- Argument token localization was usually high.
- Argument value decoding was the unresolved bottleneck.
This is an important distinction. The experiment did not fail because the parser could not find the computation. It failed because the numeric argument channel was too noisy for exact multi-step execution. With 8 or 12 steps, even a good but imperfect per-step argument classifier produces very few exact programs.
The result suggests the next experiment should not add more recurrence or a larger executor first. The highest-leverage change is to replace the weak argument-value readout with a stronger numeric interface.
Recommended Next Experiment
Run a numeric-copy compiler variant:
- Keep the same QLoRA parser and role supervision.
- Add an auxiliary numeric-token decoder for argument values, separate from the semantic hidden-state classifier.
- Add a constrained value head that only predicts valid generated argument ranges for each operation class.
- Evaluate whether argument accuracy can exceed 95% at length 8 before running longer chains.
Success criterion:
- argument value accuracy above 95% at L8;
- program exact above 50% at L8;
- direct and answer-only controls still near chance.
This is the shortest path to determining whether the symbolic interface is fundamentally useful or merely limited by the current numeric readout.
Artifacts
Small files:
- Code:
experiments/qwen_lora_parser_compiler/src/ - Runs:
experiments/qwen_lora_parser_compiler/runs/ - Analysis:
experiments/qwen_lora_parser_compiler/analysis/ - Manifest:
experiments/qwen_lora_parser_compiler/checkpoint_manifest.csv
Large files:
- Pilot adapter and heads:
large_artifacts/qwen_lora_parser_compiler/checkpoints/pilot_qwen3_4b_qlora_tagger_mixed_l12/ - Full trace+executor adapter and heads:
large_artifacts/qwen_lora_parser_compiler/checkpoints/main_qwen3_4b_qlora_trace_argstrong_mixed_l12/ - Direct control adapter and heads:
large_artifacts/qwen_lora_parser_compiler/checkpoints/control_qwen3_4b_direct_mixed_l12/ - Answer-only compiler control adapter and heads:
large_artifacts/qwen_lora_parser_compiler/checkpoints/control_qwen3_4b_qlora_answer_only_mixed_l12/
Experiment log 7
Show the running log (7 entries, 2026-06-21)
Objective
Test whether live QLoRA updates can make a small causal language model expose a clean modular-arithmetic program in its hidden states, and whether a learned compiler/executor can use that program to answer longer arithmetic chains.
Experiment Question
Can a LoRA-adapted model learn token-level roles and program symbols for synthetic arithmetic instructions well enough to support a compiled executor, including on paraphrased prompts and longer held-out chains?
Planned Gates
- Token-tagger pilot: train QLoRA plus parser heads on init, operation, argument, and token-position supervision.
- Parser/executor condition: add differentiable execution loss on the predicted modular program.
- Answer-only control: train the same QLoRA substrate without trace supervision.
Primary Metrics
init_pos_accuracy,op_pos_accuracy,arg_pos_accuracy: whether the parser attends to the intended source tokens.init_accuracy,op_accuracy,arg_accuracy: whether those tokens are decoded into correct program symbols.program_exact: whether the whole predicted program is exact.executor_accuracy: whether compiled execution returns the correct answer.- Standard and paraphrase splits are evaluated separately at multiple chain lengths.
Artifact Policy
Lightweight outputs stay in experiments/qwen_lora_parser_compiler/runs/.
Large adapter and head checkpoints stay in large_artifacts/qwen_lora_parser_compiler/checkpoints/.
Log
2026-06-21
- Created standalone experiment directory and large-artifact checkpoint path.
- Added a live QLoRA training harness with a token-tagger pilot variant and parser/executor variants.
- Tightened the synthetic generator so operation-position labels point at the operation token rather than the end of the surrounding phrase.
- Tiny random LoRA smoke test passed and wrote
runs/smoke_tiny_tagger/results.json. - Qwen 3 4B 4-bit LoRA micro-smoke passed and wrote
runs/smoke_qwen3_4b_qlora_tagger/results.json. Qwen QLoRA token-tagger pilot completed as
runs/pilot_qwen3_4b_qlora_tagger_mixed_l12/.- Operation symbols were mostly solved: 88.2-99.2% across final splits.
- Argument positions were mostly found: 77.4-94.5%.
- Argument values remained the limiting error: 65.4-77.7%.
Qwen QLoRA trace+executor condition completed as
runs/main_qwen3_4b_qlora_trace_argstrong_mixed_l12/.- Short-chain executor accuracy improved over the tagger-only pilot.
- Standard length-4 reached 32.8%; paraphrase length-4 reached 40.6%.
- Length-8 remained weak and length-12/24 remained near zero.
- Argument value accuracy remained the main bottleneck.
Direct final-answer QLoRA control completed as
runs/control_qwen3_4b_direct_mixed_l12/.- Final accuracy stayed near chance on every split.
Answer-only compiler control completed as
runs/control_qwen3_4b_qlora_answer_only_mixed_l12/.- Final executor accuracy stayed near chance and the parser did not discover useful symbols.
- Regenerated aggregate analysis and checkpoint manifest.
- Wrote standalone Markdown and HTML reports.
- Recommended next step: a numeric-copy compiler variant that directly targets the argument-value bottleneck.
Figures 3
Data files 7
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/final_metrics.csv5.8 kBruns/control_qwen3_4b_direct_mixed_l12/results.json21 kBruns/control_qwen3_4b_qlora_answer_only_mixed_l12/results.json23 kBruns/main_qwen3_4b_qlora_trace_argstrong_mixed_l12/results.json23 kBruns/pilot_qwen3_4b_qlora_tagger_mixed_l12/results.json23 kBruns/smoke_qwen3_4b_qlora_tagger/results.json4.4 kBruns/smoke_tiny_tagger/results.json7.2 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).