Qwen Progressive Repair Compiler
The one idea you need
A frozen model turns each prompt into a small arithmetic program that is often slightly wrong. The system lists a handful of tiny edits that might fix it, then a separate small judge circles the one it thinks will run correctly, never seeing the true answer.
The question
If a small model's first attempt at a program is usually wrong, can a separate judge pick the correct fix from a short list of candidate edits?
What we found
Partly. The correct fix sits in the candidate list almost nine times in ten, yet the judge finds it only about half the time, lifting exactly-correct programs from 30% to 49% against the 88% a flawless chooser would reach. Generating candidate fixes is easy; reliably recognizing the right one is the real wall.
Why it matters
When a repair pipeline stalls, measure two things separately: whether a correct fix exists among your candidates, and whether your selector picks it. Here candidates were fine, so invest in the chooser, not more edits, and skip easy warm-ups.
On this page
Results at a glance 3
How to read
Grouped bars for three fresh test splits (standard, paraphrase, paired). Each split shows base compiler (low), learned selector (middle), and a flawless-chooser ceiling (high). Bar height is the share of programs that run to the exactly correct answer; higher is better.
Takeaway → The learned selector's middle bar lands roughly halfway up in every split, clearly above the base near 30% but far below the roughly 90% ceiling, so most fixable errors stay unselected.
Data table
| Fresh evaluation split | base compiler | learned verifier | oracle ceiling |
|---|---|---|---|
| fresh standard | 28.5% | 47.7% | 90.6% |
| fresh paraphrase | 28.5% | 44.9% | 86.7% |
| fresh paired | 30.3% | 48.6% | 88.1% |
Numbers from analysis/summary.md table (mirrored in reports/qwen_progressive_repair_compiler_paper.md)
Technical framing
Learned repair selector vs base compiler and oracle on fresh splits — The learned verifier lifts exact execution ~16-19 points over the base compiler but recovers only ~30% of the gap to the oracle.
How to read
Line of validation accuracy per training epoch for the learned selector, with flat reference lines for the base compiler (bottom, near 32%) and the flawless chooser (top, near 85%). Epochs 1-3 use one-edit sets, 4-7 medium, 8-18 the full two-edit space; higher is better.
Takeaway → The line hugs the base rate flat through the small and medium stages, then jumps only after full-space training begins around epoch eight: the warm-up bought nothing.
Data table
| Verifier training epoch (small: 1-3, medium: 4-7, full: 8-18) | learned verifier | base compiler | oracle ceiling |
|---|---|---|---|
| 1 | 32% | 32% | 85.2% |
| 2 | 32% | — | — |
| 3 | 32% | — | — |
| 4 | 32% | — | — |
| 5 | 32% | — | — |
| 6 | 32% | — | — |
| 7 | 32.8% | — | — |
| 8 | 39.8% | — | — |
| 9 | 47.7% | — | — |
| 10 | 39.1% | — | — |
| 11 | 46.1% | — | — |
| 12 | 49.2% | — | — |
| 13 | 46.9% | — | — |
| 14 | 53.1% | — | — |
| 15 | 51.6% | — | — |
| 16 | 46.9% | — | — |
| 17 | 41.4% | — | — |
| 18 | 48.4% | 32% | 85.2% |
Numbers from runs/main_progressive_repair_s512/verifier_train_log.csv
Technical framing
Validation accuracy only improves once training reaches the full repair space — The staged curriculum was not the mechanism: accuracy stays at the base rate through the small and medium stages and rises only in the full stage.
How to read
Grouped bars on the paired split for three measures: exact-run accuracy, both-phrasings-correct, and cross-phrasing consistency. Four bars each compare base compiler, learned selector, pair reranking, and the flawless-chooser ceiling; higher is better.
Takeaway → Pair reranking tops the single-prompt selector on every measure and, unlike it, lifts consistency above the base rate rather than dropping it below (71% to 57%).
Data table
| Metric | base compiler | learned verifier | pair rerank | oracle ceiling |
|---|---|---|---|---|
| executor accuracy | 30.3% | 48.6% | 53.7% | 88.1% |
| pair both-correct | 28.1% | 36.7% | 50.4% | 85.9% |
| pair state consistency | 71.1% | 57% | 87.5% | 92.6% |
Numbers from report table (reports/qwen_progressive_repair_compiler_paper.md, Fresh Paired Details)
Technical framing
Fresh paired split: pair reranking beats the single-prompt learned verifier — Pair reranking adds accuracy and restores cross-rendering consistency, which the single-prompt learned verifier actually degrades below base.
In the author’s words from the Report · “Abstract”
This experiment tests a deployable repair-selection layer for a frozen Qwen-attached numeric compiler. The compiler converts a prompt into an executable modular-arithmetic program. Around that program, the system enumerates local edits and trains a small transformer verifier to choose among candidate execution traces without seeing the true answer or true state trajectory at test time. The new variable is a progressive candidate-space curriculum. The verifier starts on small one-edit neighborhoods and then trains on the full two-edit repair space. Final evaluation uses the full candidate space for all methods.
Overview
This experiment trains a non-oracle repair selector for a frozen Qwen-attached numeric program compiler. Each prompt is compiled into an executable modular-arithmetic program, local candidate repairs are enumerated, and a small transformer verifier chooses among candidate execution traces without access to the true answer or true state trajectory at test time.
The experiment-specific small files live here:
experiments/qwen_progressive_repair_compiler/Large checkpoints live separately:
large_artifacts/qwen_progressive_repair_compiler/checkpoints/Primary Command
python experiments/qwen_progressive_repair_compiler/src/qwen_progressive_repair_compiler_experiment.py \
--run_name main_progressive_repair_s512 \
--train_examples 512 \
--val_examples 128 \
--eval_examples 256 \
--eval_pairs 256 \
--candidate_curriculum small:2:1:8:3,medium:3:1:16:4,full:3:2:24:11Analysis
python experiments/qwen_progressive_repair_compiler/src/analyze_qwen_progressive_repair_compiler.pyExpected outputs:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/executor_accuracy.pnganalysis/figures/paired_details.pnganalysis/figures/training_curve.pngreports/qwen_progressive_repair_compiler_paper.mdreports/qwen_progressive_repair_compiler_paper.htmlcheckpoint_manifest.csv
Report
Rendered from reports/qwen_progressive_repair_compiler_paper.md
Abstract
This experiment tests a deployable repair-selection layer for a frozen Qwen-attached numeric compiler. The compiler converts a prompt into an executable modular-arithmetic program. Around that program, the system enumerates local edits and trains a small transformer verifier to choose among candidate execution traces without seeing the true answer or true state trajectory at test time.
The new variable is a progressive candidate-space curriculum. The verifier starts on small one-edit neighborhoods and then trains on the full two-edit repair space. Final evaluation uses the full candidate space for all methods.
Setup
- Primary run:
main_progressive_repair_s512 - Qwen substrate:
Qwen/Qwen3-4B - Modulus:
97 - Max program length:
24 - Best verifier epoch:
14
Candidate labels are computed offline from exact trajectories. At evaluation time, the learned verifier receives only candidate-local information: copied slots, edit metadata, compiler probabilities, predicted states, and soft-executor support.
Results
Fresh Splits
| Split | Base | Learned | Pair rerank | Oracle | Gap recovered |
|---|---|---|---|---|---|
| fresh_standard_len24 | 28.5% | 47.7% | n/a | 90.6% | 30.8% |
| fresh_paraphrase_len24 | 28.5% | 44.9% | n/a | 86.7% | 28.2% |
| fresh_paired_len24 | 30.3% | 48.6% | 53.7% | 88.1% | 31.8% |
Fresh Paired Details
| Metric | Base | Learned | Pair rerank | Oracle |
|---|---|---|---|---|
| Executor accuracy | 30.3% | 48.6% | 53.7% | 88.1% |
| Program exact | 30.3% | 48.6% | 53.7% | 87.7% |
| State prefix fraction | 58.6% | 71.9% | 76.2% | 90.7% |
| Pair both-correct | 28.1% | 36.7% | 50.4% | 85.9% |
| Pair state consistency | 71.1% | 57.0% | 87.5% | 92.6% |
Training Dynamics
The training curve tracks validation accuracy on the full candidate set after every verifier epoch. Stage labels mark the candidate budget used for that part of training.
Interpretation
On the fresh paired split, the learned verifier moves exact execution from 30.3% to 48.6%. The oracle ceiling is 88.1%, so the learned selector recovers 31.8% of the measured base-to-oracle gap. Pair reranking is a separate consistency control for paired prompt renderings; it reaches 53.7% executor accuracy on the paired split.
The key bottleneck is still selection quality. The candidate space often contains a correct executable program, but the learned verifier does not always identify it. The experiment therefore supports continuing toward verifier-to-compiler distillation or joint training, because the current external selector converts only part of the available local-repair headroom.
The staged curriculum is not, by itself, the breakthrough mechanism in this run. Validation accuracy stayed near the base compiler through the small and medium stages, then improved only after the verifier trained on the full candidate neighborhood. That points to a practical next step: use the full repair space from the start or distill full-space successful choices into the compiler rather than relying on a long warm-up over easier neighborhoods.
Limitations
- The task is synthetic modular arithmetic.
- The compiler and runtime are specialized.
- Candidate labels use exact trajectories during training.
- The verifier consumes engineered trace features rather than only raw hidden states.
- Pair reranking requires paired prompt renderings and is not a single-prompt method.
Artifacts
Small experiment files live in:
experiments/qwen_progressive_repair_compiler/Large artifacts live in:
large_artifacts/qwen_progressive_repair_compiler/checkpoints/Primary files:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/executor_accuracy.pnganalysis/figures/paired_details.pnganalysis/figures/training_curve.pngruns/main_progressive_repair_s512/metrics.csvruns/main_progressive_repair_s512/verifier_train_log.csvreports/qwen_progressive_repair_compiler_paper.mdreports/qwen_progressive_repair_compiler_paper.htmlcheckpoint_manifest.csv
Experiment log 7
Show the running log (7 entries)
Objective
Train a deployable repair-selection layer for a frozen Qwen-attached numeric compiler. The learned verifier should choose local executable-program repairs without test-time access to the true answer or true trajectory.
Success Criteria
- Keep this experiment in its own directory with source, runs, analysis, reports, and checkpoint manifest.
- Store bulky checkpoints under
large_artifacts/qwen_progressive_repair_compiler/. - Evaluate base compiler selection, learned verifier selection, pair reranking, and oracle trajectory selection.
- Produce CSVs, figures, a markdown report, and an HTML report.
- Run smoke/pilot/main configurations and record outcomes.
Planned Runs
Smoke
smoke_progressive_repair
- Tiny data and a short two-stage candidate curriculum.
- Purpose: verify checkpoint loading, candidate generation, staged verifier training, metric writing, checkpoint writing, and report generation.
- Completed successfully. Metrics are not used for conclusions because the top-2/one-edit neighborhood and tiny held-out sets were intentionally too small.
Pilot
pilot_progressive_repair_s96
- Moderate data with full two-edit validation.
- Purpose: test whether progressive candidate-space training is stable before running the larger configuration.
- Completed successfully.
- Fresh paired length-24 result: base 28.1%, learned verifier 40.6%, pair rerank 42.2%, oracle 87.5%.
- Validation result: base 37.5%, learned verifier 45.8%, oracle 85.4%.
- This justified running the larger configuration.
Main
main_progressive_repair_s512
- 512 verifier-training examples, 128 validation examples, and fresh held-out standard, paraphrase, and paired splits.
- Candidate curriculum:
small:2:1:8:3,medium:3:1:16:4,full:3:2:24:11. - Completed successfully in 874.9 seconds.
- Best validation epoch: 14.
- Fresh standard length-24: base 28.5%, learned verifier 47.7%, oracle 90.6%.
- Fresh paraphrase length-24: base 28.5%, learned verifier 44.9%, oracle 86.7%.
- Fresh paired length-24: base 30.3%, learned verifier 48.6%, pair rerank 53.7%, oracle 88.1%.
- The small and medium curriculum stages did not materially improve full-space validation accuracy; most of the gain appeared after full-neighborhood training began.
Running Notes
Initialized the scaffold with self-contained source files, a copied Qwen compiler checkpoint under the experiment's large-artifact root, and an analyzer that generates markdown and HTML reports.
Figures 3
Data files 8
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv6.8 kBanalysis/final_metrics.csv3.4 kBruns/main_progressive_repair_s512/metrics.csv3.4 kBruns/main_progressive_repair_s512/results.json41 kBruns/pilot_progressive_repair_s96/metrics.csv3.4 kBruns/pilot_progressive_repair_s96/results.json25 kBruns/smoke_progressive_repair/metrics.csv2.7 kBruns/smoke_progressive_repair/results.json14 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).