Qwen Slot Repair Distillation
The one idea you need
A model writes short number-crunching programs that run correctly only about a quarter of the time. A fix almost always sits one or two small edits away. But a helper must spot which pieces are wrong from the program alone, with no answer key to check against.
The question
If a computer usually writes a slightly-wrong program, can a small model learn to spot and fix the few wrong pieces on its own, without ever seeing the answer?
What we found
No. A correct program almost always sits one or two edits away — a search that peeks at the answer lifts solve rates from about a quarter to roughly 86%. But the blind helper couldn't pick which edits to make: on reworded problems it dropped accuracy from 25% to 19%, and on fresh problems it barely moved. It only helped on the exact problems it was tuned on.
Why it matters
Before bolting a learned fixer onto a model's output, test it on fresh and reworded inputs, and keep a do-nothing baseline. A high "a fix exists nearby" ceiling proves fixes are reachable — not that a blind one-shot patcher can find them.
On this page
Results at a glance 6
How to read
Grouped bars of how often programs run correctly across four problem sets — the tuning set plus three sets of fresh problems. Each set has three bars: the untouched base program, the trained editor's fix, and the answer-key repair ceiling. Higher is better.
Takeaway → The editor bar tops the base only on the tuning set; on every fresh set it ties or dips below, all far under the roughly 85% ceiling.
Data table
| Evaluation split | frozen base compiler | distilled slot editor | local repair oracle |
|---|---|---|---|
| validation | 32.8% | 35.2% | 85.2% |
| fresh standard | 28.5% | 27.7% | 87.1% |
| fresh paraphrase | 25% | 18.8% | 85.2% |
| fresh paired | 25.4% | 24.8% | 86.1% |
Numbers from experiments/qwen_slot_repair_distillation/analysis/final_metrics.csv (run main_slot_repair_distill_s512)
Technical framing
Slot-repair editor vs frozen compiler vs local-repair oracle — The editor beats the frozen base only on validation; on fresh prompts it matches or hurts, far below the ~85% oracle ceiling.
How to read
One bar per problem set showing the share of the base-to-ceiling gap the editor captured. Positive means it fixed programs; negative means it broke them; zero is doing nothing. Higher is better.
Takeaway → Bars are positive on the training and tuning sets but turn negative on all three fresh sets, worst at minus ten percent on rewordings — the skill doesn't transfer.
Data table
| Evaluation split | gap recovered |
|---|---|
| train | 6.4% |
| validation | 4.5% |
| fresh standard | -1.3% |
| fresh paraphrase | -10.4% |
| fresh paired | -1% |
Numbers from experiments/qwen_slot_repair_distillation/analysis/summary.md table
Technical framing
Share of the base-to-oracle gap the editor recovers, by split — Positive gap recovery on train/validation flips negative on every fresh distribution - the learned repair signal does not transfer.
How to read
A line of the editor's tuning-set accuracy across 18 training rounds against a flat line for the untouched base program. Above the flat line is better; higher means more programs run correctly.
Takeaway → The editor line bounces between 13% and 35% and only clears the flat base line on the last round — the win is fragile, not a settled skill.
Data table
| Training epoch | distilled slot editor | frozen base compiler |
|---|---|---|
| 1 | 13.3% | 32.8% |
| 2 | 26.6% | — |
| 3 | 22.7% | — |
| 4 | 28.1% | — |
| 5 | 31.2% | — |
| 6 | 29.7% | — |
| 7 | 29.7% | — |
| 8 | 30.5% | — |
| 9 | 27.3% | — |
| 10 | 22.7% | — |
| 11 | 30.5% | — |
| 12 | 18.8% | — |
| 13 | 15.6% | — |
| 14 | 30.5% | — |
| 15 | 19.5% | — |
| 16 | 32.8% | — |
| 17 | 27.3% | — |
| 18 | 35.2% | 32.8% |
Numbers from experiments/qwen_slot_repair_distillation/runs/main_slot_repair_distill_s512/editor_train_log.csv
Technical framing
Editor validation accuracy across training epochs — The editor only overtakes the frozen base at the final epoch, and the curve is noisy - the validation gain is marginal, not a stable capability.
How to read
For pairs of prompts asking the same question two ways, bars for the base program, the answer-key repairs, and the editor. Two series: giving the same answer to both wordings, and getting both right. Higher is better.
Takeaway → The editor's same-answer bar falls to about 38% from the base's 70%, while answer-key repairs lift it to 88% — the editor's patches make rewordings disagree.
Data table
| Program source (fresh paired split) | Same answer on both phrasings | Both phrasings correct |
|---|---|---|
| Base compiler | 69.5% | 23.4% |
| Local-repair oracle | 88.3% | 84% |
| Slot editor | 37.9% | 16.8% |
Numbers from experiments/qwen_slot_repair_distillation/analysis/all_final_metrics.csv
Technical framing
Editing breaks paraphrase consistency on paired prompts — The editor nearly halves cross-paraphrase answer agreement (69.5% to 37.9%), while oracle repairs raise it to 88.3%.
How to read
For six editor setups, paired bars compare tuning-set accuracy against fresh-problem accuracy. Higher is better; a good recipe would raise both bars together.
Takeaway → No setup wins both — the main run tops tuning accuracy at 35% yet a smaller value-stabilized pilot beats it on fresh problems, 36% versus 28%.
Data table
| Editor configuration | Validation accuracy | Fresh standard accuracy |
|---|---|---|
| distill b3 w8 (96ex) | 18.8% | 10.9% |
| distill b5 (96ex) | 29.2% | 31.2% |
| gated (96ex) | 31.2% | 34.4% |
| gated oracle-base (96ex) | 22.9% | 23.4% |
| gated value-stabilized (96ex) | 27.1% | 35.9% |
| main distill (512ex) | 35.2% | 27.7% |
Numbers from experiments/qwen_slot_repair_distillation/analysis/all_final_metrics.csv
Technical framing
Pilot ablations: no editor variant wins on both validation and fresh prompts — The 512-example main run is best on validation (35.2%) but pilots like value-stabilized beat it on fresh prompts (35.9% vs 27.7%).
How to read
Two lines across 18 training rounds: how precisely the editor targets the parts to change (of edits it makes, how many were right) and how completely (of needed edits, how many it made). Higher and steadier is better.
Takeaway → Precision swings between 45% and 71% round to round and never settles — the choice of which few parts to edit stays unreliable throughout training.
Data table
| Training epoch | Gate precision (val) | Gate recall (val) |
|---|---|---|
| 1 | 37.3% | 62.5% |
| 2 | 60.3% | 59.9% |
| 3 | 51% | 66.4% |
| 4 | 65.3% | 63.2% |
| 5 | 61.4% | 51.3% |
| 6 | 71% | 57.9% |
| 7 | 66.2% | 61.8% |
| 8 | 65.6% | 56.6% |
| 9 | 59% | 62.5% |
| 10 | 51% | 65.1% |
| 11 | 60.4% | 63.2% |
| 12 | 45.3% | 70.4% |
| 13 | 48.9% | 70.4% |
| 14 | 57.1% | 61.2% |
| 15 | 45.2% | 67.8% |
| 16 | 68.7% | 59.2% |
| 17 | 66.4% | 56.1% |
| 18 | 66% | 65.1% |
Numbers from experiments/qwen_slot_repair_distillation/runs/main_slot_repair_distill_s512/editor_train_log.csv
Technical framing
Edit-gate precision never stabilizes during training — Which-slot-to-edit precision oscillates between 45% and 71% across 18 epochs, so the sparse edit decision never becomes reliable.
In the author’s words from the Report · “Abstract”
This experiment tests whether local repair headroom can be distilled into a gated slot editor for a frozen Qwen-attached numeric compiler. The compiler first emits an executable modular-arithmetic program. Offline candidate search identifies a corrected local program when one exists. A small transformer editor is then trained to decide which init/op/arg slots to edit and what replacement values to emit from the base compiler trace, without candidate enumeration at evaluation time.
Overview
This standalone experiment tests whether local repair headroom can be distilled into a deployable gated slot editor for a frozen Qwen-attached numeric compiler. The compiler emits an executable modular-arithmetic program. Offline local repair search supplies corrected-program labels when available, and the editor learns to decide which init/op/arg slots to edit plus what replacement values to emit from the base compiler trace.
At evaluation time, the editor emits one program. It does not receive target answers, target states, or a candidate list to rerank.
Small experiment files live here:
experiments/qwen_slot_repair_distillation/Large checkpoints live separately:
large_artifacts/qwen_slot_repair_distillation/checkpoints/Main Result
The primary run was main_slot_repair_distill_s512.
| Split | Base | Editor | Oracle |
|---|---|---|---|
| validation | 32.8% | 35.2% | 85.2% |
| fresh standard | 28.5% | 27.7% | 87.1% |
| fresh paraphrase | 25.0% | 18.8% | 85.2% |
| fresh paired | 25.4% | 24.8% | 86.1% |
The editor learned a validation signal but did not transfer cleanly to fresh prompt distributions. The local oracle ceiling stayed high, so nearby corrected programs usually exist; the failure is the one-shot policy's ability to select the right sparse edits and values from the base trace alone.
Primary Command
python experiments/qwen_slot_repair_distillation/src/qwen_slot_repair_distillation_experiment.py \
--run_name main_slot_repair_distill_s512 \
--train_examples 512 \
--val_examples 128 \
--eval_examples 256 \
--eval_pairs 256 \
--repair_topk 3 \
--repair_max_edits 2 \
--repair_max_pair_arg_slots 24 \
--qwen_batch_size 8 \
--editor_d_model 128 \
--editor_layers 3 \
--editor_heads 4 \
--editor_ff_mult 4 \
--editor_epochs 18 \
--editor_lr 0.001 \
--editor_target_mode oracle_or_gold \
--changed_slot_weight 4.0 \
--unchanged_value_loss_weight 0.05 \
--edit_gate_pos_weight 8.0 \
--edit_threshold_grid 0.15,0.2,0.25,0.3,0.4,0.5,0.6,0.7 \
--max_length 384 \
--seed 66 \
--eval_seed 66001Analysis
python experiments/qwen_slot_repair_distillation/src/analyze_qwen_slot_repair_distillation.pyKey outputs:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/executor_accuracy.pnganalysis/figures/paired_details.pnganalysis/figures/training_curve.pnganalysis/figures/iteration_summary.pngreports/qwen_slot_repair_distillation_paper.mdreports/qwen_slot_repair_distillation_paper.htmlreports/qwen_slot_repair_distillation_experiment_log.mdcheckpoint_manifest.csv
Report
Rendered from reports/qwen_slot_repair_distillation_paper.md
Abstract
This experiment tests whether local repair headroom can be distilled into a gated slot editor for a frozen Qwen-attached numeric compiler. The compiler first emits an executable modular-arithmetic program. Offline candidate search identifies a corrected local program when one exists. A small transformer editor is then trained to decide which init/op/arg slots to edit and what replacement values to emit from the base compiler trace, without candidate enumeration at evaluation time.
Setup
- Primary run:
main_slot_repair_distill_s512 - Qwen substrate:
Qwen/Qwen3-4B - Modulus:
97 - Max program length:
24 - Best editor epoch:
18 - Best validation threshold:
0.7 - Editor target mode:
oracle_or_gold - Unchanged-slot value loss weight:
0.05
Training labels come from exact offline trajectories. At evaluation time, the editor receives only the base compiler trace and predicts a single program. The oracle column is the local candidate-search ceiling, not an inference method.
Results
Fresh Splits
| Split | Base | Editor | Oracle | Gap recovered | Editor in candidates |
|---|---|---|---|---|---|
| fresh_standard_len24 | 28.5% | 27.7% | 87.1% | -1.3% | 49.6% |
| fresh_paraphrase_len24 | 25.0% | 18.8% | 85.2% | -10.4% | 34.8% |
| fresh_paired_len24 | 25.4% | 24.8% | 86.1% | -1.0% | 43.9% |
Fresh Paired Details
| Metric | Base | Editor | Oracle |
|---|---|---|---|
| Executor accuracy | 25.4% | 24.8% | 86.1% |
| Program exact | 25.4% | 24.8% | 85.7% |
| State prefix fraction | 52.3% | 53.4% | 88.7% |
| Pair both-correct | 23.4% | 16.8% | 84.0% |
| Pair state consistency | 69.5% | 37.1% | 88.3% |
The final paired editor uses 1.17 edits per program on average. Against its training target definition, gate precision is 66.9% and gate recall is 59.7%.
Training Dynamics
The training curve tracks validation accuracy after each editor epoch.
Iteration Summary
The direct full-slot editor either copied the base program or damaged too many slots. The gated editor fixed that interface problem, and a small unchanged-slot value loss made one pilot improve fresh paired accuracy. The larger primary run did not preserve that fresh-split gain.
| Run | Validation | Fresh paired base | Fresh paired editor | Fresh paired oracle | Avg edits |
|---|---|---|---|---|---|
| pilot_slot_repair_distill_s96_b5 | 29.2%->29.2% | 25.8% | 25.8% | 88.3% | 0.00 |
| pilot_slot_repair_distill_s96_b3_w8 | 33.3%->18.8% | 28.9% | 10.9% | 83.6% | 2.40 |
| pilot_slot_repair_gated_s96 | 25.0%->31.2% | 30.5% | 28.1% | 86.7% | 0.18 |
| pilot_slot_repair_gated_s96_oracle_base | 20.8%->22.9% | 24.2% | 20.3% | 87.5% | 0.84 |
| pilot_slot_repair_gated_s96_value_stabilized | 20.8%->27.1% | 28.1% | 32.8% | 88.3% | 0.39 |
| main_slot_repair_distill_s512 | 32.8%->35.2% | 25.4% | 24.8% | 86.1% | 1.17 |
Interpretation
On the fresh paired split, the editor moves exact execution from 25.4% to 24.8%. The local oracle ceiling is 86.1%, so the editor recovers -1.0% of the measured base-to-oracle gap.
The main result is therefore not a successful distillation of the local oracle. The editor learns a real validation signal, but its selected edits do not transfer robustly to fresh prompt distributions. The high oracle ceiling shows that nearby corrected programs usually exist; the failure is the one-shot policy's ability to choose the right sparse edits and values from the base trace alone.
Limitations
- The task is synthetic modular arithmetic.
- Candidate labels use exact trajectories during training.
- The frozen compiler and deterministic runtime are specialized.
- The editor consumes engineered base-trace features, not only raw Qwen hidden states.
- The main result is one primary seed.
- The deployable editor emits one program and does not get to evaluate candidate executions at inference time.
Artifacts
Small experiment files live in:
experiments/qwen_slot_repair_distillation/Large artifacts live in:
large_artifacts/qwen_slot_repair_distillation/checkpoints/Primary files:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/executor_accuracy.pnganalysis/figures/paired_details.pnganalysis/figures/training_curve.pnganalysis/figures/iteration_summary.pngruns/main_slot_repair_distill_s512/metrics.csvruns/main_slot_repair_distill_s512/editor_train_log.csvreports/qwen_slot_repair_distillation_paper.mdreports/qwen_slot_repair_distillation_paper.htmlcheckpoint_manifest.csv
Experiment log 9
Show the running log (9 entries)
Objective
Train a deployable slot editor that distills local repair headroom into direct corrected-program predictions for a frozen Qwen-attached numeric compiler. At evaluation time, the editor emits one program and does not receive target answers, target states, or a candidate list to rerank.
File Layout
- Source:
experiments/qwen_slot_repair_distillation/src/ - Runs:
experiments/qwen_slot_repair_distillation/runs/ - Analysis CSVs and figures:
experiments/qwen_slot_repair_distillation/analysis/ - Reports:
experiments/qwen_slot_repair_distillation/reports/ - Large checkpoints:
large_artifacts/qwen_slot_repair_distillation/checkpoints/ - Manifest:
experiments/qwen_slot_repair_distillation/checkpoint_manifest.csv
Success Criteria
- Keep source, runs, analysis, reports, and manifest in this experiment directory.
- Store bulky checkpoints under
large_artifacts/qwen_slot_repair_distillation/. - Evaluate base compiler, deployable editor, and local oracle ceiling.
- Run smoke, pilot, and main configurations.
- Iterate when a design fails rather than one-shotting the final run.
- Produce CSVs, figures, a markdown report, and an HTML report.
Implementation Iterations
Direct Full-Slot Editor
The first editor predicted every init/op/arg slot directly.
smoke_slot_repair_distill: path check passed, but the model changed many slots and destroyed accuracy.smoke_slot_repair_distill_bias: adding base-copy logit bias made the editor copy the base program.pilot_slot_repair_distill_s96_b5: copied the base program; fresh paired stayed 25.8% -> 25.8%.pilot_slot_repair_distill_s96_b3_w8: edited too aggressively; fresh paired fell 28.9% -> 10.9%.
Conclusion: full-slot prediction is the wrong deployable interface. It lacks an explicit decision about whether a slot should be edited.
Gated Edit Policy
The second editor predicts edit gates separately from replacement values. It starts from the base program, then applies only slots whose gate probability passes a validation-selected threshold.
smoke_slot_repair_gated: path check passed.pilot_slot_repair_gated_s96: validation improved 25.0% -> 31.2%, but fresh paired fell 30.5% -> 28.1%.pilot_slot_repair_gated_s96_oracle_base: using copy-base fallback for no-oracle examples was too conservative and still hurt fresh paired 24.2% -> 20.3%.pilot_slot_repair_gated_s96_value_stabilized: adding a small unchanged-slot value loss stabilized false-positive edits. Fresh standard improved 28.1% -> 35.9%, paraphrase 23.4% -> 25.0%, and paired 28.1% -> 32.8%.
Conclusion: gated editing plus weak unchanged-slot value supervision was the only pilot worth scaling.
Primary Run
Run: main_slot_repair_distill_s512
Configuration:
- 512 train examples
- 128 validation examples
- 256 fresh standard examples
- 256 fresh paraphrase examples
- 256 paired programs, evaluated as 512 paired prompt variants
- top-3/two-edit local teacher
- gated edit policy, 128 hidden width, 3 layers
editor_target_mode=oracle_or_goldunchanged_value_loss_weight=0.05- validation threshold grid:
0.15,0.2,0.25,0.3,0.4,0.5,0.6,0.7
Final selected checkpoint:
- best epoch: 18
- best threshold: 0.7
Main metrics:
| Split | Base | Editor | Oracle | Gap recovered |
|---|---|---|---|---|
| train_len24 | 27.5% | 31.2% | 85.4% | 6.4% |
| val_len24 | 32.8% | 35.2% | 85.2% | 4.5% |
| fresh_standard_len24 | 28.5% | 27.7% | 87.1% | -1.3% |
| fresh_paraphrase_len24 | 25.0% | 18.8% | 85.2% | -10.4% |
| fresh_paired_len24 | 25.4% | 24.8% | 86.1% | -1.0% |
Fresh paired diagnostics:
- editor average edits per program: 1.17
- editor gate precision: 66.9%
- editor gate recall: 59.7%
- editor pair both-correct: 16.8% versus base 23.4%
- editor pair state consistency: 37.1% versus base 69.5%
Interpretation
The main run is not a successful local-oracle distillation. The editor learned a validation signal but did not improve fresh distributions. The high local oracle ceiling means the repair headroom is real; this experiment shows that a single feed-forward gate/value policy over engineered base traces is not enough to recover that headroom robustly.
The most useful next direction is likely not a larger version of this exact editor. A stronger follow-up should keep some form of execution-grounded selection at inference time, or train the Qwen-attached compiler on on-policy repair traces so the base compiler itself moves toward repairable/correct programs instead of relying on a small post-hoc editor.
Final Artifacts
- Markdown report:
reports/qwen_slot_repair_distillation_paper.md - HTML report:
reports/qwen_slot_repair_distillation_paper.html - Summary:
analysis/summary.md - Main metrics:
runs/main_slot_repair_distill_s512/metrics.csv - Main train log:
runs/main_slot_repair_distill_s512/editor_train_log.csv - Figures:
analysis/figures/ - Checkpoint manifest:
checkpoint_manifest.csv
Figures 4
Data files 20
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv16 kBanalysis/final_metrics.csv3.1 kBruns/main_slot_repair_distill_s512/metrics.csv3.1 kBruns/main_slot_repair_distill_s512/results.json50 kBruns/pilot_slot_repair_distill_s96_b3_w8/metrics.csv2.5 kBruns/pilot_slot_repair_distill_s96_b3_w8/results.json22 kBruns/pilot_slot_repair_distill_s96_b5/metrics.csv2.4 kBruns/pilot_slot_repair_distill_s96_b5/results.json22 kBruns/pilot_slot_repair_gated_s96/metrics.csv3.1 kBruns/pilot_slot_repair_gated_s96/results.json34 kBruns/pilot_slot_repair_gated_s96_oracle_base/metrics.csv3.2 kBruns/pilot_slot_repair_gated_s96_oracle_base/results.json34 kBruns/pilot_slot_repair_gated_s96_value_stabilized/metrics.csv3.1 kBruns/pilot_slot_repair_gated_s96_value_stabilized/results.json33 kBruns/smoke_slot_repair_distill/metrics.csv1.8 kBruns/smoke_slot_repair_distill/results.json12 kBruns/smoke_slot_repair_distill_bias/metrics.csv1.9 kBruns/smoke_slot_repair_distill_bias/results.json12 kBruns/smoke_slot_repair_gated/metrics.csv2.2 kBruns/smoke_slot_repair_gated/results.json14 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).