Qwen Register Trace Refiner
The one idea you need
A small model rewrites each math word problem as a tiny step-by-step program a calculator runs exactly. When the program is wrong, a repair loop swaps one or two steps for runner-up guesses and keeps whichever runs cleanest, never seeing the answer. A badly wrong program needs more swaps than that.
The question
If a small model turns a math word problem into a little program and gets it wrong, can trying a batch of nearby edits and keeping the best one fix it?
What we found
Rarely. Even a flawless picker that always grabbed the correct edit reached only 37% on plainly worded problems and 7% on reworded ones, because the correct program usually isn't among the roughly 1,300 nearby edits at all. The trained picker did worse, nudging 23% to 27% and leaving reworded problems stuck near 5%.
Why it matters
Before training a scorer to rescue a small model's outputs, first check whether a perfect chooser could even win: the correct answer must exist in your candidate set. If it rarely does, widen or steer the search instead of tuning the scorer.
On this page
Results at a glance 3
How to read
Grouped bars for three problem types: plainly worded, reworded, and matched pairs. Each group shows three bars: the raw model, the trained repair loop, and a perfect picker. Bar height is the share of problems solved correctly; taller is better.
Takeaway → The repair loop barely clears the raw model and adds nothing on reworded problems, while the perfect-picker bar towers over both, so the ceiling, not the scorer, is the wall.
Data table
| Evaluation split | base | learned refiner | oracle |
|---|---|---|---|
| standard | 23.4% | 26.6% | 37.1% |
| paraphrase | 4.7% | 4.7% | 7% |
| paired | 12.3% | 12.9% | 18.6% |
Numbers from analysis/final_metrics.csv
Technical framing
Executor accuracy on fresh length-24 programs: base compiler vs learned refiner vs oracle selection — The learned refiner recovers only a small slice of the oracle gap (+3.1pp on standard, nothing on paraphrase).
How to read
One bar per split: training, validation, fresh plainly worded, fresh reworded, fresh matched pairs. Height is the share of problems whose correct program appears somewhere among the roughly 1,300 nearby edits; higher is better.
Takeaway → Even on plainly worded problems the correct fix is present only about a third of the time, and on reworded ones almost never (7%), which caps any picker.
Data table
| Split | oracle found in candidates |
|---|---|
| train | 21.9% |
| validation | 20.3% |
| fresh standard | 37.1% |
| fresh paraphrase | 7% |
| fresh paired | 18.6% |
Numbers from analysis/final_metrics.csv
Technical framing
How often the correct program even exists in the local repair set — The key diagnostic: local two-edit repair rarely contains the correct program, worst on paraphrases (7%), capping any selector.
How to read
One bar per fresh split: plainly worded, reworded, matched pairs. Height is the fraction of the gap between raw accuracy and perfect-picker accuracy that the trained repair loop closed; higher means more of the reachable gain captured.
Takeaway → Even where good candidates exist, the loop captures under a quarter of the reachable gain on plainly worded problems, less on pairs, and none on reworded ones.
Data table
| Fresh evaluation split | learned refiner |
|---|---|
| standard | 22.9% |
| paraphrase | 0% |
| paired | 9.4% |
Numbers from analysis/final_metrics.csv
Technical framing
Share of the base-to-oracle gap the learned refiner recovers — Even where good candidates exist, the verifier picks them only sometimes: about 23% of the gap on standard, 0% on paraphrase.
In the author’s words from the Report · “Abstract”
On fresh length-24 programs, the learned guarded refiner improves standard-prompt execution accuracy from 23.4% to 26.6%, with an oracle upper bound of 37.1% inside the same candidate set. It does not improve paraphrase accuracy, where the oracle itself reaches only 7.0%. On paired standard/paraphrase evaluation, the refiner improves 12.3% to 12.9%, with an 18.6% oracle ceiling. The result is positive but narrow: local repair helps, but the correct program is often outside the top-3/two-edit neighborhood, and learned candidate selection remains hard.
Overview
This experiment tests whether a learned verifier can improve a Qwen3-4B register-program compiler by searching local edits around the compiled latent program and selecting a better execution trace.
The experiment is standalone: it packages the fixed input compiler and the trained refiner under large_artifacts/qwen_register_trace_refiner/, while the experiment directory contains only source, logs, metrics, figures, and write-ups.
Layout
src/qwen_register_trace_refiner_experiment.py- builds candidate repairs, trains the verifier, evaluates base/learned/guarded/oracle selection.src/qwen_register_trace_refiner_core.py- local register compiler, data generator, and modular runtime utilities.src/analyze_qwen_register_trace_refiner.py- regenerates CSV summaries and figures.runs/- smoke, pilot, and main run metrics.analysis/- aggregate CSVs, summary, and figures.reports/- standalone experiment log and paper-style report.checkpoint_manifest.csv- exact large artifacts used by the main run.
Large Artifacts
Download or preserve these separately from the experiment folder:
large_artifacts/qwen_register_trace_refiner/checkpoints/input_register_compilerlarge_artifacts/qwen_register_trace_refiner/checkpoints/main_register_trace_refiner_s512/register_trace_refiner.pt
The input compiler directory is about 84 MB. The trained refiner checkpoint is about 2.5 MB.
Main Result
Fresh length-24 modular programs, top-3/two-edit local repair search:
| split | base | learned/guarded | oracle |
|---|---|---|---|
| standard | 23.4% | 26.6% | 37.1% |
| paraphrase | 4.7% | 4.7% | 7.0% |
| paired | 12.3% | 12.9% | 18.6% |
The learned refiner recovers a small part of the available oracle gap. The larger result is diagnostic: the correct program is often not present in the local repair set, especially for paraphrases, and selecting repaired candidates robustly remains difficult.
Reproduce
Smoke:
PYTHONPATH=experiments/qwen_register_trace_refiner/src \
python experiments/qwen_register_trace_refiner/src/qwen_register_trace_refiner_experiment.py \
--run_name smoke_register_trace_refiner_guarded \
--train_examples 8 --val_examples 4 --eval_examples 4 --eval_pairs 4 \
--verifier_epochs 1 --qwen_batch_size 2 --repair_topk 2 --repair_max_edits 1 \
--trace_d_model 64 --trace_layers 1 --trace_heads 4Main:
PYTHONPATH=experiments/qwen_register_trace_refiner/src \
python experiments/qwen_register_trace_refiner/src/qwen_register_trace_refiner_experiment.py \
--run_name main_register_trace_refiner_s512 \
--train_examples 512 --val_examples 128 --eval_examples 256 --eval_pairs 256 \
--verifier_epochs 18 --qwen_batch_size 8 \
--repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 24 \
--trace_d_model 128 --trace_layers 3 --trace_heads 4 --trace_ff_mult 4Analysis:
python experiments/qwen_register_trace_refiner/src/analyze_qwen_register_trace_refiner.pyReport
Rendered from reports/qwen_register_trace_refiner_paper.md
Abstract
This experiment tests a latent-program repair loop for a Qwen3-4B register compiler. A fixed compiler reads a text prompt and emits a register program for modular arithmetic. A deterministic runtime executes the program. The new component enumerates local register-program edits and trains a verifier to select a better candidate from each local repair set without seeing the target answer or target trace at evaluation time.
On fresh length-24 programs, the learned guarded refiner improves standard-prompt execution accuracy from 23.4% to 26.6%, with an oracle upper bound of 37.1% inside the same candidate set. It does not improve paraphrase accuracy, where the oracle itself reaches only 7.0%. On paired standard/paraphrase evaluation, the refiner improves 12.3% to 12.9%, with an 18.6% oracle ceiling. The result is positive but narrow: local repair helps, but the correct program is often outside the top-3/two-edit neighborhood, and learned candidate selection remains hard.
Method
Each example is a length-24 chain of operations modulo 97. The text prompt describes an initial value and a sequence of add, subtract, and multiply updates. A fixed Qwen3-4B register compiler outputs:
- one initial-value register,
- one operation register per step,
- one argument register per step.
The runtime executes the argmax register program exactly. The refiner then constructs a candidate set around that program:
- keep the base program,
- edit the initial value,
- edit one operation,
- edit one argument,
- edit one operation and argument at the same step,
- edit two argument slots.
Each slot uses the top-3 compiler alternatives. For length 24 this creates 1,299 candidates per example.
Candidates are featurized by compiler log-probabilities, edit structure, soft-runtime trace likelihoods, operation/argument statistics, and the candidate execution trace. A small transformer scores the candidate trace plus summary features. Training labels are produced offline by exact execution against the target trace. At evaluation time, the verifier sees only candidate features and traces, not the target answer or target states.
A guarded selector is tuned on validation: use the learned candidate only when it beats the base candidate by a score margin; otherwise retain the base program. The selected validation guard threshold was 0.25.
Results
Primary run: main_register_trace_refiner_s512.
| split | base | learned/guarded | oracle |
|---|---|---|---|
| train_len24 | 14.8% | 16.6% | 21.9% |
| val_len24 | 15.6% | 17.2% | 20.3% |
| fresh_standard_len24 | 23.4% | 26.6% | 37.1% |
| fresh_paraphrase_len24 | 4.7% | 4.7% | 7.0% |
| fresh_paired_len24 | 12.3% | 12.9% | 18.6% |
Fresh paired details:
| metric | base | learned/guarded | oracle |
|---|---|---|---|
| executor accuracy | 12.3% | 12.9% | 18.6% |
| program exact | 12.1% | 12.7% | 18.4% |
| state prefix fraction | 79.5% | 79.7% | 80.7% |
| pair both correct | 1.6% | 1.6% | 1.6% |
| pair state consistency | 1.6% | 1.6% | 1.6% |
Candidate-set profile:
| split | candidates/example | positive candidates/example | oracle found |
|---|---|---|---|
| train_len24 | 1299.0 | 0.45 | 21.9% |
| val_len24 | 1299.0 | 0.36 | 20.3% |
| fresh_standard_len24 | 1299.0 | 0.68 | 37.1% |
| fresh_paraphrase_len24 | 1299.0 | 0.15 | 7.0% |
| fresh_paired_len24 | 1299.0 | 0.47 | 18.6% |
Figures:
../analysis/figures/executor_accuracy_by_split.png../analysis/figures/oracle_gap_recovered.png../analysis/figures/candidate_set_profile.png
Discussion
The experiment supports three conclusions.
First, local repair is a real lever. On fresh standard length-24 programs, the learned refiner recovers 22.9% of the available oracle gap and improves exact execution by 3.2 percentage points.
Second, local top-k repair is not enough for robust paraphrase behavior. The oracle ceiling is only 7.0% on paraphrase examples, which means the correct program usually is not present in the searched neighborhood.
Third, verifier selection remains difficult even when the correct repair is present. The standard split has a 37.1% oracle ceiling, but the learned guarded selector reaches 26.6%. Better candidate scoring, broader search, or iterative edit policies are needed before this becomes a large test-time-compute gain.
The most direct next step is to widen the repair distribution without exploding candidates: use a learned proposal policy over suspect slots, add three-edit candidates only around low-confidence prefixes, and train the verifier/editor jointly on base-wrong repairable cases.
Experiment log 11
Show the running log (11 entries)
Objective
Train a learned repair verifier for a Qwen3-4B register-program compiler. The compiler emits a fixed register program for modular arithmetic. The refiner enumerates local program edits, executes each candidate with a deterministic modular runtime, and learns to select a better candidate without access to the target answer or target trace at evaluation time.
Artifact Discipline
- Experiment code, metrics, reports, and figures are in
experiments/qwen_register_trace_refiner/. - Large model artifacts are in
large_artifacts/qwen_register_trace_refiner/. checkpoint_manifest.csvrecords the input compiler checkpoint and trained main refiner checkpoint.Analysis includes CSV summaries and three figures:
Iteration Notes
1. Scaffold
Created a standalone experiment directory with local source files:
qwen_register_trace_refiner_core.pyqwen_register_trace_refiner_experiment.pyanalyze_qwen_register_trace_refiner.py
Copied the fixed input compiler into:
large_artifacts/qwen_register_trace_refiner/checkpoints/input_register_compiler
2. Smoke Run
Command used a tiny dataset, top-2 one-edit search, and one verifier epoch.
Outcome: the register interface, candidate construction, verifier training, metrics writing, and figure generation all worked.
3. Pilot Run
Run: pilot_register_trace_refiner_s128
Settings:
- 128 train examples
- 64 validation examples
- 64 fresh standard examples
- 64 fresh paraphrase examples
- 64 paired evaluation pairs
- top-3/two-edit repair search
- 6 verifier epochs
Finding: the candidate set had oracle headroom, but the learned verifier kept choosing the base candidate. Fresh standard was 20.3% base and 40.6% oracle, but learned stayed at 20.3%.
4. Oversampled Pilot
Run: pilot_register_trace_refiner_oversample_s128
Change: oversampled repairable training groups where the base candidate was wrong but a local edit was exact.
Finding: validation improved from 15.6% base to 18.8% learned, but fresh standard dropped from 20.3% to 17.2%. This showed the verifier could learn repair choices but needed a base-preserving selection rule.
5. Guarded Selection
Added validation-tuned guarded selection:
Keep the learned candidate only when its score beats the base candidate by a tuned margin; otherwise keep the base program.
Smoke run smoke_register_trace_refiner_guarded verified the guarded path and paired metrics.
6. Main Run
Run: main_register_trace_refiner_s512
Settings:
- 512 train examples
- 128 validation examples
- 256 fresh standard examples
- 256 fresh paraphrase examples
- 256 paired evaluation pairs
- top-3/two-edit repair search, 1,299 candidates per example
- 18 verifier epochs
- repairable-group oversampling = 10
- guard threshold selected on validation = 0.25
Runtime: 1087.1 seconds on NVIDIA RTX 6000 Ada Generation.
Main Metrics
| split | base | learned | guarded | oracle |
|---|---|---|---|---|
| train_len24 | 14.8% | 16.6% | 16.6% | 21.9% |
| val_len24 | 15.6% | 17.2% | 17.2% | 20.3% |
| fresh_standard_len24 | 23.4% | 26.6% | 26.6% | 37.1% |
| fresh_paraphrase_len24 | 4.7% | 4.7% | 4.7% | 7.0% |
| fresh_paired_len24 | 12.3% | 12.9% | 12.9% | 18.6% |
Interpretation
The refiner produced a real but small gain. Standard fresh L24 improved by 3.2 points and recovered 22.9% of the oracle gap. Paired L24 improved by 0.6 points and recovered 9.4% of the oracle gap. Paraphrase L24 did not improve.
The candidate set is the main limiter for paraphrase robustness: oracle availability is only 7.0% on paraphrase and 18.6% on paired evaluation. Selection is still a limiter on standard prompts: oracle reaches 37.1%, while learned/guarded reaches 26.6%.
Figures 3
Data files 12
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv11 kBanalysis/final_metrics.csv4.2 kBruns/main_register_trace_refiner_s512/metrics.csv4.2 kBruns/main_register_trace_refiner_s512/results.json42 kBruns/pilot_register_trace_refiner_oversample_s128/metrics.csv3.3 kBruns/pilot_register_trace_refiner_oversample_s128/results.json30 kBruns/pilot_register_trace_refiner_s128/metrics.csv3.2 kBruns/pilot_register_trace_refiner_s128/results.json20 kBruns/smoke_register_trace_refiner/metrics.csv2.6 kBruns/smoke_register_trace_refiner/results.json12 kBruns/smoke_register_trace_refiner_guarded/metrics.csv3.2 kBruns/smoke_register_trace_refiner_guarded/results.json14 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).