Qwen Learned Repair Verifier
The one idea you need
A wrong program sits in a pool of 1,299 candidate rewrites, itself included, and one of them is correct nearly nine times in ten. It is like a multiple-choice test where the right answer sits on the page most of the time, but you must circle it with no answer key.
The question
If a small model writes a wrong program, can a cheap judge that never sees the correct answer reliably pick a fixed version from many candidates?
What we found
Partly. Without ever seeing the true answer, the judge lifted correct execution from 30% to 47%. But a checker allowed to peek at the answer key found a correct fix already sitting in the candidate pile 88% of the time, so blind picking claimed only about a third of the fixable errors. Comparing two wordings of the same task and rewarding agreement did better, reaching 51%.
Why it matters
When repairing small-model output, spend on the selector, not the candidate generator: the fix is almost always already in the pool. And if you can pose one task two ways, rewarding answers that agree beats a trained scorer alone.
On this page
Results at a glance 2
How to read
Grouped bars for three fresh test splits: standard, reworded, and paired-wordings. Each split shows three bars: the raw model near 29-30%, the answer-blind picker at 44-48%, and a checker peeking at the answer key near 87-91%. Taller is better.
Takeaway → The blind picker clears the raw model on every split but sits far below the peeking checker: a real gain, with most of the fixable headroom still unclaimed.
Data table
| Fresh evaluation split | base compiler | learned verifier | oracle ceiling |
|---|---|---|---|
| standard | 28.5% | 44.1% | 90.6% |
| paraphrase | 28.5% | 48% | 86.7% |
| paired | 30.3% | 47.3% | 88.1% |
Numbers from experiments/qwen_learned_repair_verifier/analysis/final_metrics.csv
Technical framing
Learned repair verifier lifts executor accuracy on fresh length-24 tasks — Reranking local repairs with a small learned verifier lifts accuracy from ~29% to 44-48%, recovering roughly a third of the oracle gap.
How to read
Four selection methods along the bottom: raw model, answer-blind picker, agreement reranker, answer-peeking checker. Two bars each: single-task accuracy and the stricter both-wordings-correct rate. Higher is better; the checker tops out near 88% and 86%.
Takeaway → The agreement reranker beats the blind picker on both bars, and its jump is largest on both-wordings-correct: consistency repairs the disagreements the picker itself created.
Data table
| Selection method | executor accuracy | both paraphrases correct |
|---|---|---|
| base compiler | 30.3% | 28.1% |
| learned verifier | 47.3% | 34.4% |
| pair rerank | 51% | 46.5% |
| oracle ceiling | 88.1% | 85.9% |
Numbers from experiments/qwen_learned_repair_verifier/analysis/final_metrics.csv
Technical framing
Paired split: pair-consistency reranking adds further gains over the learned verifier — Enforcing answer consistency across paraphrase pairs raises paired accuracy to 51% and both-correct to 46.5%, well above the learned verifier alone.
In the author’s words from the Report · “Abstract”
On fresh paired length-24 programs, base execution accuracy is 30.3%. The learned verifier reaches 47.3%. A paired consistency reranker reaches 51.0%. The oracle state-verifier ceiling is 88.1%. Thus the learned verifier recovers 29.4% of the measured base-to-oracle gap on the paired split.
Overview
This experiment trains a small non-oracle verifier to rerank local repairs of programs compiled by a fixed Qwen-attached numeric compiler.
The compiler reads a modular-arithmetic prompt, copies an initial value, a sequence of operations, and operation arguments, then executes the copied program modulo 97. The learned verifier does not see the true answer or true state trajectory at test time. It receives candidate-edit features and chooses one candidate from a local top-k/two-edit repair neighborhood.
Main Result
Primary run: main_rich_learned_verifier_s512
Fresh length-24 results:
| Split | Base | Learned verifier | Pair rerank | Oracle ceiling |
|---|---|---|---|---|
| Standard | 28.5% | 44.1% | n/a | 90.6% |
| Paraphrase | 28.5% | 48.0% | n/a | 86.7% |
| Paired | 30.3% | 47.3% | 51.0% | 88.1% |
The learned verifier recovers 29.4% of the base-to-oracle gap on the fresh paired split. A paired consistency reranker raises paired executor accuracy to 51.0% and paired both-correct accuracy to 46.5%.
Layout
experiments/qwen_learned_repair_verifier/
src/ experiment and analysis scripts
reports/ standalone paper, HTML report, and experiment log
analysis/ regenerated CSVs, summary, and figures
runs/ per-run metrics and train logs, without large checkpoints
checkpoint_manifest.csv
large_artifacts/qwen_learned_repair_verifier/checkpoints/
fixed_compiler_step00800/
main_rich_learned_verifier_s512/learned_verifier.ptReproduction
Run the main experiment:
python experiments/qwen_learned_repair_verifier/src/qwen_learned_repair_verifier_experiment.py \
--run_name main_rich_learned_verifier_s512 \
--train_examples 512 \
--val_examples 128 \
--eval_examples 256 \
--eval_pairs 256 \
--repair_topk 3 \
--repair_max_edits 2 \
--verifier_epochs 18 \
--qwen_batch_size 8 \
--verifier_width 192Regenerate analysis:
python experiments/qwen_learned_repair_verifier/src/analyze_qwen_learned_repair_verifier.pyKey Files
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/executor_accuracy.pngreports/qwen_learned_repair_verifier_experiment_log.mdreports/qwen_learned_repair_verifier_paper.md
Report
Rendered from reports/qwen_learned_repair_verifier_paper.md
Abstract
This experiment tests whether a small learned verifier can recover exact long-chain execution from local repairs of programs compiled by a Qwen-attached numeric compiler. Each prompt describes a sequence of modular arithmetic updates. The compiler copies executable slots from the prompt, and a deterministic runtime executes the copied program modulo 97.
The repair neighborhood contains 1,299 candidates for each length-24 example: the base compiled program plus top-3 local edits of the initial value, operations, arguments, same-step operation/argument pairs, and pairs of argument edits. An oracle state verifier can identify the correct candidate when the full target trajectory is available. This experiment trains a non-oracle verifier to choose among the same candidates using only candidate features available at test time.
The result is positive but partial. On fresh paired length-24 programs, base execution accuracy is 30.3%. The learned verifier reaches 47.3%. A paired consistency reranker reaches 51.0%. The oracle state-verifier ceiling is 88.1%. Thus the learned verifier recovers 29.4% of the measured base-to-oracle gap on the paired split.
Question
Can a lightweight verifier learn to select useful local repairs of a compiled program without seeing the true answer or the true state trajectory at test time?
This is the deployment-relevant version of verifier-guided repair. The oracle state verifier is useful as a training label source and as a ceiling, but it is not a runtime method. A learned verifier must infer which candidate is plausible from model confidence geometry, edit type, candidate program structure, and execution features.
Method
The experiment freezes a QLoRA-attached Qwen/Qwen3-4B numeric compiler. The compiler predicts:
- the initial modular value;
- one operation per step from add, subtract, multiply;
- one numeric argument per step.
The copied program is executed exactly modulo 97. All evaluated programs have length 24.
For each example, the experiment enumerates a local repair neighborhood:
| Candidate class | Budget |
|---|---|
| Base compiled program | 1 |
| Alternate initial values | top-3 |
| Alternate operations | top-3 per active step |
| Alternate arguments | top-3 per active step |
| Same-step operation and argument edits | top-3 by slot |
| Two-argument edits | top-3 by slot, up to 24 slots |
The resulting length-24 neighborhood has 1,299 candidates per example.
Offline labels mark candidates whose full execution trajectory matches the true state trajectory. The learned verifier is a two-layer MLP trained with a groupwise ranking loss:
loss = logsumexp(all candidate scores) - logsumexp(correct candidate scores)For examples with no correct candidate in the neighborhood, a small auxiliary term selects the base program. At test time, labels are not used.
Candidate features include:
- candidate prior and prior delta from compiler logits;
- edit count, edit position, edit type, and changed-slot margins;
- changed values and argument-change magnitudes;
- candidate operation mix and argument statistics;
- differentiable executor support for the candidate's final answer and state path;
- agreement with the base program's answer and state path.
The primary main run uses 512 verifier-training examples, 128 validation examples, and fresh held-out length-24 standard, paraphrase, and paired splits.
Results
Main Fresh Results
| Split | Base | Learned verifier | Pair rerank | Oracle ceiling |
|---|---|---|---|---|
| Fresh standard L24 | 28.5% | 44.1% | n/a | 90.6% |
| Fresh paraphrase L24 | 28.5% | 48.0% | n/a | 86.7% |
| Fresh paired L24 | 30.3% | 47.3% | 51.0% | 88.1% |
The learned verifier improves all fresh splits. On the paired split it recovers 29.4% of the base-to-oracle gap:
(47.3 - 30.3) / (88.1 - 30.3) = 29.4%The paired consistency reranker is only available when two renderings of the same latent program are evaluated together. It selects from the learned verifier's top candidates while rewarding program, state, and answer agreement between renderings. On the paired split it improves executor accuracy from 47.3% to 51.0%.
Paired Split Details
| Metric | Base | Learned verifier | Pair rerank | Oracle ceiling |
|---|---|---|---|---|
| Executor accuracy | 30.3% | 47.3% | 51.0% | 88.1% |
| Program exact | 30.3% | 47.3% | 51.0% | 87.7% |
| State prefix fraction | 58.6% | 71.2% | 73.4% | 90.7% |
| Pair both-correct | 28.1% | 34.4% | 46.5% | 85.9% |
| Pair state consistency | 71.1% | 55.1% | 82.8% | 92.6% |
The individual learned verifier raises exact accuracy but lowers paired state consistency, because it can choose different plausible repairs for two surface forms. The paired reranker fixes much of that problem: paired state consistency rises to 82.8%, and paired both-correct rises to 46.5%.
Training Dynamics
The best validation epoch is epoch 15:
| Split | Base | Learned | Oracle |
|---|---|---|---|
| Train L24 | 28.7% | 58.8% | 86.7% |
| Validation L24 | 32.0% | 50.8% | 85.2% |
The train/validation gap is visible but not dominant. Fresh held-out performance lands between the validation result and the lower pilot results, which is consistent with a real but incomplete learned signal.
Iterations
The first smoke run used a tiny top-2/one-edit neighborhood to verify loading, candidate generation, metric writing, and verifier checkpointing.
The first full-neighborhood pilot used fragment-mixed training and validation renderings. That produced very low oracle ceilings on train and validation while clean fresh standard/paraphrase ceilings stayed high. The setup was corrected to use clean standard/paraphrase renderings for verifier training and validation.
A clean balanced pilot showed a measurable learned-verifier gain, but the gain was weak on paraphrased prompts. A paraphrase-weighted pilot helped slightly but reduced validation quality. The decisive improvement came from richer candidate features that exposed edit values, operation mix, and argument-change magnitudes. That configuration was used for the main run.
Interpretation
The experiment answers the immediate question positively: a small learned verifier can convert some oracle repair headroom into actual non-oracle accuracy. The result is not just a validation artifact; it appears on fresh standard, paraphrase, and paired length-24 splits.
The ceiling remains much higher than the learned result. The oracle verifier finds correct repairs for 88.1% of fresh paired examples, while the learned verifier reaches 47.3% and the paired reranker reaches 51.0%. This means the candidate neighborhood is strong enough, but the learned selection signal is still weak relative to exact trajectory verification.
The paired reranker result is important. When multiple renderings of the same latent task are available, consistency is a strong non-oracle signal. It improves both exact accuracy and state consistency. However, it is not a single-prompt method.
Limitations
- The task is synthetic modular arithmetic.
- The compiler and executor are specialized to this task family.
- The verifier labels are generated from true state trajectories offline.
- The learned verifier uses hand-engineered candidate features rather than a deeper model over prompt tokens and candidate traces.
- The main result is from one fixed compiler checkpoint and one main verifier seed.
- Paired consistency reranking requires two renderings of the same latent task.
Conclusion
Learned repair verification is a high-value direction. It improves fresh paired length-24 exact execution from 30.3% to 47.3%, and paired consistency reranking raises it to 51.0%. This is a meaningful conversion of oracle repair headroom into a non-oracle mechanism, but most of the 88.1% oracle ceiling remains unclaimed.
The next step should make the verifier less hand-engineered and more task-native: train a candidate-trace transformer or Qwen-conditioned reranker over compact execution traces, and combine it with consistency training across multiple prompt renderings.
Artifacts
Small files live in:
experiments/qwen_learned_repair_verifier/Large checkpoints live in:
large_artifacts/qwen_learned_repair_verifier/checkpoints/Primary files:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/executor_accuracy.pngruns/main_rich_learned_verifier_s512/metrics.csvruns/main_rich_learned_verifier_s512/verifier_train_log.csvcheckpoint_manifest.csv
Experiment log 11
Show the running log (11 entries)
Objective
Train a small learned verifier/reranker that chooses among local repairs of a fixed Qwen-compiled modular-arithmetic program. The verifier is trained with offline labels from exact state trajectories, but at test time it receives only non-oracle candidate features.
Success Criteria
- Build a standalone experiment directory with source, runs, analysis, reports, and a checkpoint manifest.
- Keep large model artifacts under
large_artifacts/. - Evaluate base compiler selection, learned verifier selection, paired consistency reranking, and oracle state-verifier selection on fresh length-24 standard, paraphrase, and paired splits.
- Treat the oracle verifier as a ceiling, not as a deployable method.
Runs
Smoke
smoke_learned_verifier
- Used tiny verifier datasets with top-2/one-edit repair.
- Verified Qwen checkpoint loading, candidate generation, verifier training, metric writing, and checkpoint writing.
- The low oracle ceiling was expected because the smoke used a deliberately restricted repair neighborhood.
Pilot: Fragment-Mixed Train/Val
pilot_learned_verifier_s128
- Used the full top-3/two-edit search.
- Train and validation prompts used per-fragment mixed renderings.
- Result: train/validation oracle ceilings were only 17.2% and 9.4%, while clean fresh standard/paraphrase ceilings were near 90%.
- Interpretation: fragment-mixed renderings were outside the useful compiler distribution. The dataset construction was patched to train and validate on clean standard/paraphrase renderings.
Pilot: Clean Standard/Paraphrase
pilot_clean_learned_verifier_s128
- Used 128 train examples and 64 validation examples.
- Validation improved from 34.4% base to 43.8% learned, with an 84.4% oracle ceiling.
- Fresh standard improved from 29.7% to 40.6%.
- Fresh paraphrase and paired gains were small, so the next pilot tested data balance and richer features.
Pilot: Paraphrase-Weighted
pilot_para_weighted_learned_verifier_s192
- Overweighted paraphrase renderings in verifier training.
- Fresh standard improved from 29.7% to 43.8%.
- Fresh paraphrase improved from 20.3% to 26.6%.
- Validation was weaker than the balanced clean pilot, so paraphrase weighting was not used for the main run.
Pilot: Rich Candidate Features
pilot_rich_features_s128
- Added non-oracle features for edit values, operation mix, argument-change magnitudes, candidate argument statistics, and base/candidate operation mix.
- Validation improved from 34.4% base to 51.6% learned.
- Fresh standard improved from 29.7% to 50.0%.
- Fresh paired improved from 30.5% to 45.3%.
- This configuration was selected for the main run.
Main
main_rich_learned_verifier_s512
- 512 train examples, 128 validation examples.
- 256 fresh standard examples, 256 fresh paraphrase examples, and 256 paired latent programs rendered twice.
- Top-3/two-edit candidate neighborhood, 1,299 candidates per length-24 example.
- Verifier width 192, 18 epochs, selected by validation learned executor accuracy.
Main fresh results:
| Split | Base | Learned | Pair rerank | Oracle |
|---|---|---|---|---|
| Standard L24 | 28.5% | 44.1% | n/a | 90.6% |
| Paraphrase L24 | 28.5% | 48.0% | n/a | 86.7% |
| Paired L24 | 30.3% | 47.3% | 51.0% | 88.1% |
Paired details:
| Metric | Base | Learned | Pair rerank | Oracle |
|---|---|---|---|---|
| Executor accuracy | 30.3% | 47.3% | 51.0% | 88.1% |
| Program exact | 30.3% | 47.3% | 51.0% | 87.7% |
| State prefix fraction | 58.6% | 71.2% | 73.4% | 90.7% |
| Pair both-correct | 28.1% | 34.4% | 46.5% | 85.9% |
| Pair state consistency | 71.1% | 55.1% | 82.8% | 92.6% |
Interpretation
The learned verifier substantially improves over the fixed compiler, but it does not close most of the oracle gap. The result is a positive but partial conversion of oracle repair headroom into a non-oracle mechanism.
The paired consistency reranker is useful when two renderings of the same latent program are available: it raises paired executor accuracy from 47.3% to 51.0% and paired both-correct accuracy from 34.4% to 46.5%.
Artifacts
- Small files:
experiments/qwen_learned_repair_verifier/ - Large checkpoints:
large_artifacts/qwen_learned_repair_verifier/checkpoints/
Figures 1
Data files 14
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv13 kBanalysis/final_metrics.csv3.4 kBruns/main_rich_learned_verifier_s512/metrics.csv3.4 kBruns/main_rich_learned_verifier_s512/results.json37 kBruns/pilot_clean_learned_verifier_s128/metrics.csv3.3 kBruns/pilot_clean_learned_verifier_s128/results.json24 kBruns/pilot_learned_verifier_s128/metrics.csv3.1 kBruns/pilot_learned_verifier_s128/results.json21 kBruns/pilot_para_weighted_learned_verifier_s192/metrics.csv3.5 kBruns/pilot_para_weighted_learned_verifier_s192/results.json29 kBruns/pilot_rich_features_s128/metrics.csv3.2 kBruns/pilot_rich_features_s128/results.json25 kBruns/smoke_learned_verifier/metrics.csv2.7 kBruns/smoke_learned_verifier/results.json12 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).