Qwen On-Policy Repair-to-Compiler
The one idea you need
The model turns a plain request into a tiny arithmetic program a computer can run. It drafts one, an automatic checker tries small edits until a version runs correctly, then the model retrains on those fixed drafts, studying its own corrected work.
The question
If a small model writes buggy little arithmetic programs, can teaching it its own automatically-corrected drafts make it reliable on brand-new requests?
What we found
Yes, but the self-correction isn't the secret ingredient. On brand-new requests the model climbed from about 29% correct programs to 99% after training on its own corrected drafts. The catch: training on plain correct programs alone also reached 99%, and training on corrections alone hit 91%. So the corrections give a usable signal without any answer key, but they aren't uniquely responsible for the jump.
Why it matters
A tiny dose of step-by-step fine-tuning turns a weak program generator near-perfect. Self-correction earns its keep only when you lack labeled answers; if you already have correct examples, they train just as well with less machinery.
On this page
Results at a glance 6
How to read
Bars grouped by test set (validation plus three sets of brand-new requests). Height is the share of programs the model gets fully correct. Three bars per group: before training, after training, and the best any small automatic edit could reach. Taller is better.
Takeaway → After-training bars leap from under a third to almost the top, matching the best-reachable line, so training nearly closes the gap on new requests.
Data table
| Evaluation split | baseline compiler | trained compiler | final repair ceiling |
|---|---|---|---|
| validation | 26.6% | 96.9% | 100% |
| fresh standard | 28.5% | 99.6% | 100% |
| fresh paraphrase | 28.9% | 97.3% | 100% |
| fresh paired | 29.1% | 99.2% | 100% |
Technical framing
Compiler accuracy before vs after on-policy repair training — One round of trace-level fine-tuning lifts the deployable compiler from ~29% to ~99% on fresh prompts, nearly closing the repair-ceiling gap.
How to read
One group per training recipe on brand-new requests. The main bar is the share of programs correct; a companion bar shows the best any small automatic edit could reach. Compares mixing plain examples with self-corrections, plain examples alone, and self-corrections alone. Taller is better.
Takeaway → Plain examples alone tie the mixed recipe near the top; self-corrections alone trail well behind, so the corrections help but aren't uniquely responsible.
Data table
| Training condition | compiler accuracy | repair ceiling |
|---|---|---|
| repair-or-gold (main) | 99.2% | 100% |
| gold-only control | 99.2% | 100% |
| repair-only control | 91% | 99.2% |
Numbers from experiments/qwen_onpolicy_repair_compiler/analysis/summary.md (Controls table)
Technical framing
Attribution controls: what training signal drives the gain (fresh paired split) — Gold trace supervision alone matches the main run at 99.2%; verified repairs alone still reach 91.0%, so repairs are a real but weaker deployable signal.
How to read
Bars grouped by test set; height is the share of cases where a small automatic edit could reach a fully correct program. Two bars per group, before and after training. Higher means more failures were fixable.
Takeaway → After training every remaining slip is fixable to correct, versus only about two-thirds before, because the model now starts much closer to right.
Data table
| Evaluation split | baseline repair ceiling | final repair ceiling |
|---|---|---|
| validation | 69.5% | 100% |
| fresh standard | 67.2% | 100% |
| fresh paraphrase | 66.8% | 100% |
| fresh paired | 64.5% | 100% |
Numbers from experiments/qwen_onpolicy_repair_compiler/analysis/summary.md (Baseline To Final table)
Technical framing
Local repair headroom at baseline vs after training — The target-aware repair ceiling itself rises from ~65-70% to 100% after training, since local edits repair a near-correct compiler far more easily.
How to read
Bars grouped by test set; height is the share of programs correct after training. Four bars per group, one per recipe: the mixed recipe, plain-examples-only, self-corrections-only, and a smaller-data run. Taller is better.
Takeaway → Recipes using plain examples stay near the top everywhere, while self-corrections-only and the smaller run dip on reworded requests.
Data table
| Evaluation split | Repair-or-gold (main) | Gold-only control | Repair-only control | Pilot (96 train examples) |
|---|---|---|---|---|
| Validation | 96.9% | 96.9% | 89.8% | 90.6% |
| Fresh standard | 99.6% | 99.6% | 99.2% | 92.2% |
| Fresh paraphrase | 97.3% | 97.3% | 83.2% | 79.7% |
| Fresh paired | 99.2% | 99.2% | 91% | 91.4% |
Numbers from analysis/all_final_metrics.csv (executor_accuracy)
Technical framing
Final compiler accuracy by evaluation split and training recipe — Repair-only training lags most on paraphrased prompts (83% vs 97%); gold-only matches the mixed recipe on every split.
How to read
Each group is a way of checking pairs of requests that mean the same thing (both correct, same program, same answer, same steps). Height is the share of pairs that match. Two bars per group, before and after training. Higher is better.
Takeaway → After training, pairs correct on both wordings jump from about a quarter to nearly all, closing the wording-robustness gap.
Data table
| Paired-prompt metric (fresh paired split) | Baseline compiler | After repair training |
|---|---|---|
| Both paraphrases correct | 26.6% | 98.4% |
| Same program emitted | 72.3% | 98.4% |
| Same answer | 72.3% | 98.4% |
| Same state trace | 72.3% | 98.4% |
Numbers from runs/main_onpolicy_repair_s256/train_log.csv (fresh_paired_len24 pair metrics, baseline and train rows)
Technical framing
Paraphrase-pair consistency before vs after on-policy training — Training nearly closes the paraphrase-robustness gap: pairs answered correctly on both phrasings jump from 27% to 98%.
How to read
Bars grouped by test set; height is the share of programs a small automatic edit still had to change, meaning still wrong. Three bars per group: mixed recipe, self-corrections-only, and the smaller-data run. Lower is better.
Takeaway → Leftover mistakes are few overall but cluster on reworded requests, worst for the self-corrections-only recipe at about one in six.
Data table
| Evaluation split | Repair-or-gold (main) | Repair-only control | Pilot (96 train examples) |
|---|---|---|---|
| Validation | 3.1% | 8.6% | 7.8% |
| Fresh standard | 0.4% | 0.8% | 7.8% |
| Fresh paraphrase | 2.7% | 16.4% | 15.6% |
| Fresh paired | 0.8% | 8.2% | 7% |
Numbers from analysis/all_final_metrics.csv (repair_changed_fraction)
Technical framing
Residual fixable errors after training: how often repair still edits the program — Remaining locally-repairable errors concentrate on paraphrased prompts, especially for the repair-only control (16%) and the small pilot.
In the author’s words from the Report · “Abstract”
This experiment tests whether verified local program repairs can be converted into a better Qwen-attached compiler policy. A QLoRA compiler emits an executable modular-arithmetic program from each prompt. The training loop runs the current compiler on its own prompts, enumerates nearby program edits, keeps targets that pass an exact execution verifier, and fine-tunes the same compiler toward those repaired targets.
Overview
This experiment tests whether verified local program repairs can be distilled into the Qwen-attached compiler policy itself.
The compiler emits an executable modular-arithmetic program from a prompt. For each on-policy compiler output, the training loop enumerates nearby edits, keeps locally verified repaired programs, and fine-tunes the same QLoRA/compiler head toward those targets. The central measurement is whether the deployable compiler improves on fresh prompts, not whether target-aware repair search has headroom.
Layout
src/qwen_onpolicy_repair_compiler_experiment.py: training and evaluation entrypoint.src/qwen_onpolicy_repair_compiler_core.py: compiler, executor, task generation, and local verifier utilities.analysis/analyze_qwen_onpolicy_repair_compiler.py: aggregates runs, writes figures, markdown, HTML, and summary files.runs/: per-run CSV and JSON outputs.reports/: standalone writeups.checkpoint_manifest.csv: generated manifest for large checkpoint artifacts.
Large model artifacts are stored under:
large_artifacts/qwen_onpolicy_repair_compiler/checkpoints/Primary Question
Can target-aware local repair headroom become a policy-weight improvement after one or more on-policy fine-tuning rounds?
The strongest positive signal is a fresh paired compiler accuracy gain after training, with a remaining local repair ceiling that explains the available headroom.
Report
Rendered from reports/qwen_onpolicy_repair_compiler_paper.md
Abstract
This experiment tests whether verified local program repairs can be converted into a better Qwen-attached compiler policy. A QLoRA compiler emits an executable modular-arithmetic program from each prompt. The training loop runs the current compiler on its own prompts, enumerates nearby program edits, keeps targets that pass an exact execution verifier, and fine-tunes the same compiler toward those repaired targets.
Setup
- Primary run:
main_onpolicy_repair_s256 - Qwen substrate:
Qwen/Qwen3-4B - Modulus:
97 - Max program length:
24 - Train examples:
256 - On-policy rounds:
1 - Epochs per round:
1 - Target mode:
repair_or_gold - Repair budget: top-k
3, max edits2
The local repair column is a ceiling measured with target-aware verification during analysis and target construction. It is not a deployable inference path. The deployable model is the compiler row after fine-tuning.
Results
Final Splits
| Split | Compiler | Local repair ceiling | Program exact | State prefix | Repair found |
|---|---|---|---|---|---|
| val_len24 | 96.9% | 100.0% | 96.9% | 98.9% | 100.0% |
| fresh_standard_len24 | 99.6% | 100.0% | 99.6% | 99.6% | 100.0% |
| fresh_paraphrase_len24 | 97.3% | 100.0% | 97.3% | 98.2% | 100.0% |
| fresh_paired_len24 | 99.2% | 100.0% | 99.2% | 99.6% | 100.0% |
Baseline To Final
| Split | Baseline compiler | Final compiler | Baseline repair ceiling | Final repair ceiling |
|---|---|---|---|---|
| val_len24 | 26.6% | 96.9% | 69.5% | 100.0% |
| fresh_standard_len24 | 28.5% | 99.6% | 67.2% | 100.0% |
| fresh_paraphrase_len24 | 28.9% | 97.3% | 66.8% | 100.0% |
| fresh_paired_len24 | 29.1% | 99.2% | 64.5% | 100.0% |
On-Policy Target Quality
| Round | Epoch | Verified repair targets | Changed targets | Active rows | Avg candidates | Avg verified |
|---|---|---|---|---|---|---|
| 1 | 1 | 67.6% | 36.7% | 100.0% | 307.00 | 1.02 |
Fresh Paired Details
| Metric | Compiler | Local repair ceiling |
|---|---|---|
| Executor accuracy | 99.2% | 100.0% |
| Program exact | 99.2% | 100.0% |
| State prefix fraction | 99.6% | 100.0% |
| Pair both-correct | 98.4% | 100.0% |
| Pair state consistency | 98.4% | 100.0% |
Run Summary
| Run | Fresh paired compiler | Fresh paired repair ceiling | Repair found | Program exact |
|---|---|---|---|---|
| control_gold_only_s256 | 99.2% | 100.0% | 100.0% | 99.2% |
| control_repair_only_s256 | 91.0% | 99.2% | 99.2% | 91.0% |
| main_onpolicy_repair_s256 | 99.2% | 100.0% | 100.0% | 99.2% |
| pilot_onpolicy_repair_s96_r1 | 91.4% | 98.4% | 98.4% | 90.6% |
Interpretation
On the fresh paired split, the compiler moves from 29.1% at the initial evaluation point to 99.2% after on-policy repair training. The measured local repair ceiling at the end is 100.0%, so the compiler recovers 98.9% of the initial compiler-to-repair gap.
Attribution is sharper with controls. The gold-only control reaches 99.2% fresh paired accuracy, matching the mixed repair-or-gold run. The repair-only control, with gold auxiliary losses disabled and unverified rows skipped, still reaches 91.0%. So the headline gain is a real compiler-policy improvement, but it is not uniquely caused by repaired targets; dense gold trace supervision is sufficient under this budget, while verified local repairs alone provide a strong but weaker training signal.
The key result is therefore narrower and more useful: a small amount of trace-level posttraining can turn a weak executable-program compiler into a near-ceiling compiler on fresh prompts, and target-aware local repairs provide a deployable training signal even when gold fallback is removed.
Limitations
- The task is synthetic modular arithmetic.
- Target construction uses exact execution verification.
- The compiler head and deterministic runtime are specialized to copied numeric programs.
- The local repair ceiling is target-aware and should be read only as headroom.
- The primary result is one run unless more runs are added to the directory.
Artifacts
Small experiment files live in:
experiments/qwen_onpolicy_repair_compiler/Large artifacts live in:
large_artifacts/qwen_onpolicy_repair_compiler/checkpoints/Primary files:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/executor_accuracy.pnganalysis/figures/paired_details.pnganalysis/figures/training_curve.pnganalysis/figures/target_quality.pnganalysis/figures/iteration_summary.pngruns/main_onpolicy_repair_s256/metrics.csvruns/main_onpolicy_repair_s256/train_log.csvreports/qwen_onpolicy_repair_compiler_paper.mdreports/qwen_onpolicy_repair_compiler_paper.htmlcheckpoint_manifest.csv
Experiment log 6
Show the running log (6 entries, 2026-06-23)
2026-06-23
Created standalone on-policy repair-to-compiler experiment.
Initial design:
- Start from a QLoRA Qwen numeric compiler checkpoint.
- Generate targets from the current compiler policy, not from an independent editor.
- Enumerate local init/op/arg repairs around each emitted program.
- Use verified repairs when found; fall back to gold trace targets by default so the training set remains dense.
- Fine-tune the same compiler policy and evaluate fresh standard, paraphrase, and paired prompts after each round.
- Generate markdown and HTML reports with figures from the analyzer.
Initial implementation files:
src/qwen_onpolicy_repair_compiler_experiment.pysrc/qwen_onpolicy_repair_compiler_core.pyanalysis/analyze_qwen_onpolicy_repair_compiler.py
Next steps:
- Compile the scripts.
- Copy the fixed compiler checkpoint into this experiment's large-artifact namespace.
- Run a small smoke test.
- If the smoke test passes, run a pilot with enough examples to detect whether on-policy targets improve the compiler or damage it.
Smoke Run
Command shape:
python experiments/qwen_onpolicy_repair_compiler/src/qwen_onpolicy_repair_compiler_experiment.py --run_name smoke_onpolicy_repair --train_examples 8 --val_examples 4 --eval_examples 4 --eval_pairs 4 --onpolicy_rounds 1 --epochs_per_round 1 --train_batch_size 1 --qwen_batch_size 2 --repair_topk 2 --repair_max_edits 1 --repair_max_pair_arg_slots 4 --max_length 384 --lr 2e-5 --gold_trace_loss_weight 0.05 --executor_loss_weight 0.0 --state_loss_weight 0.0 --seed 72 --eval_seed 72001Outcome:
- Passed model loading, target generation, fine-tuning, evaluation, CSV/JSON writing, and checkpoint saving.
- Validation compiler accuracy stayed at 25.0% on the tiny smoke split.
- Validation local-repair ceiling was also 25.0% with the deliberately tiny repair budget.
- Fresh paired compiler accuracy was 0.0% on the tiny paired split, which is not meaningful beyond path validation.
Pilot Run
Command shape:
python experiments/qwen_onpolicy_repair_compiler/src/qwen_onpolicy_repair_compiler_experiment.py --run_name pilot_onpolicy_repair_s96_r1 --train_examples 96 --val_examples 64 --eval_examples 64 --eval_pairs 64 --onpolicy_rounds 1 --epochs_per_round 1 --train_batch_size 2 --qwen_batch_size 8 --repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 8 --max_length 384 --lr 5e-5 --gold_trace_loss_weight 0.15 --executor_loss_weight 0.1 --state_loss_weight 0.0 --seed 73 --eval_seed 73001Outcome:
- Baseline validation compiler accuracy was 32.8%; validation local-repair ceiling was 68.8%.
- After one on-policy round, validation compiler accuracy reached 90.6%.
- Fresh paired compiler accuracy reached 91.4%.
- Verified repair targets were found for 67.7% of training rows; 37.5% of targets changed at least one slot.
Main Run
Command shape:
python experiments/qwen_onpolicy_repair_compiler/src/qwen_onpolicy_repair_compiler_experiment.py --run_name main_onpolicy_repair_s256 --train_examples 256 --val_examples 128 --eval_examples 256 --eval_pairs 256 --onpolicy_rounds 1 --epochs_per_round 1 --train_batch_size 2 --qwen_batch_size 8 --repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 8 --max_length 384 --lr 5e-5 --gold_trace_loss_weight 0.15 --executor_loss_weight 0.1 --state_loss_weight 0.0 --seed 74 --eval_seed 74001Outcome:
- Baseline fresh paired compiler accuracy was 29.1%; baseline local-repair ceiling was 64.5%.
- After one round, fresh paired compiler accuracy reached 99.2%; local-repair ceiling reached 100.0%.
- Fresh standard reached 99.6%; fresh paraphrase reached 97.3%.
- Verified repair targets were found for 67.6% of training rows; 36.7% of targets changed at least one slot.
Attribution Controls
Gold-only control:
python experiments/qwen_onpolicy_repair_compiler/src/qwen_onpolicy_repair_compiler_experiment.py --run_name control_gold_only_s256 --train_examples 256 --val_examples 128 --eval_examples 256 --eval_pairs 256 --onpolicy_rounds 1 --epochs_per_round 1 --train_batch_size 2 --qwen_batch_size 8 --repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 8 --max_length 384 --lr 5e-5 --target_mode gold_only --gold_trace_loss_weight 0.15 --executor_loss_weight 0.1 --state_loss_weight 0.0 --seed 74 --eval_seed 74001Repair-only control:
python experiments/qwen_onpolicy_repair_compiler/src/qwen_onpolicy_repair_compiler_experiment.py --run_name control_repair_only_s256 --train_examples 256 --val_examples 128 --eval_examples 256 --eval_pairs 256 --onpolicy_rounds 1 --epochs_per_round 1 --train_batch_size 2 --qwen_batch_size 8 --repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 8 --max_length 384 --lr 5e-5 --target_mode repair_only --gold_trace_loss_weight 0.0 --executor_loss_weight 0.0 --state_loss_weight 0.0 --seed 74 --eval_seed 74001Control outcome:
- Gold-only matched the mixed main run at 99.2% fresh paired accuracy.
- Repair-only reached 91.0% fresh paired accuracy with 173 active repaired rows and 83 skipped rows.
- Interpretation: the headline compiler improvement is real, but dense gold trace supervision is sufficient under this budget; verified local repairs alone are useful but weaker.
Report Generation
Generated:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/*.pngreports/qwen_onpolicy_repair_compiler_paper.mdreports/qwen_onpolicy_repair_compiler_paper.html
Figures 5
Data files 12
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv7.3 kBanalysis/final_metrics.csv2.1 kBruns/control_gold_only_s256/metrics.csv2.1 kBruns/control_gold_only_s256/results.json21 kBruns/control_repair_only_s256/metrics.csv2.3 kBruns/control_repair_only_s256/results.json21 kBruns/main_onpolicy_repair_s256/metrics.csv2.1 kBruns/main_onpolicy_repair_s256/results.json21 kBruns/pilot_onpolicy_repair_s96_r1/metrics.csv2.2 kBruns/pilot_onpolicy_repair_s96_r1/results.json21 kBruns/smoke_onpolicy_repair/metrics.csv2.0 kBruns/smoke_onpolicy_repair/results.json20 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.