Qwen3.5-4B Trained vs Frozen Repair MDP Report
The one idea you need
A small coding model writes a program; it fails the tests. You can either train a specialist to patch the broken code, or throw the attempt away and write a few fresh ones. Here, the cheap fresh rewrites won.
The question
When a small coding model writes a program that fails its tests, is it better to train a dedicated fixer or just retry from scratch?
What we found
Retry from scratch. Given the same compute budget, a few more fresh attempts rescued 5 of 57 stuck problems, while a fixer trained to patch broken code rescued only 2 — and its repairs secretly failed the hidden tests nearly a third of the time. An untrained fixer helped slightly (3 of 57), but neither repair approach beat plain retries.
Why it matters
Before building a trained repair model, spend that budget on more diverse first-draft attempts — they recovered more failures here. And distrust code that passes the visible tests: repairs often quietly break on the hidden ones.
On this page
Results at a glance 3
How to read
Of 57 problems with no working solution, each bar counts how many one strategy rescued: an untrained fixer, a trained fixer, and simply making more fresh attempts. Taller is better.
Takeaway → More-fresh-attempts is the tallest bar and the trained fixer the shortest, so retrying beat repair and training the fixer actually made it worse.
Data table
| arm (each spent ~80k extra forward tokens) | tasks recovered |
|---|---|
| frozen repair | 3 |
| SFT-trained repair | 2 |
| sample more (token-matched) | 5 |
Numbers from report table (reports/qwen35_4b_trained_vs_frozen_repair_mdp_report.md, Main Held-Out Results)
Technical framing
Zero-coverage tasks recovered per arm (out of 57) — At matched token budget, simply sampling more direct solutions recovered more zero-coverage tasks than either repair loop; training made repair worse.
How to read
Each bar shows the share of 150 problems solved. The leftmost is four fresh tries alone; the other three add extra effort in different ways. Higher is better.
Takeaway → All extra-effort bars edge above the four-tries baseline, but more fresh attempts sits highest, adding the most coverage overall.
Data table
| arm | coverage |
|---|---|
| direct x4 | 62% |
| frozen repair | 64% |
| SFT repair | 63.3% |
| sample more | 65.3% |
Numbers from report table (reports/qwen35_4b_trained_vs_frozen_repair_mdp_report.md, Main Held-Out Results)
Technical framing
Held-out coverage by arm (150 MBPP tasks) — Every budget extension beats the direct-x4 base, but sample-more tops both repair arms at essentially the same forward-token cost.
How to read
Each bar shows how often a fixer produced code that passed the visible checks yet still failed the hidden ones — a misfire. Untrained versus trained fixer. Lower is better.
Takeaway → The trained fixer's bar is taller, meaning training made it more likely to output code that looks correct but quietly fails.
Data table
| repair arm | false repair rate |
|---|---|
| frozen repair | 25% |
| SFT repair | 29.2% |
Numbers from report table (reports/qwen35_4b_trained_vs_frozen_repair_mdp_report.md, Main Held-Out Results)
Technical framing
False-repair rate: visible-pass candidates that fail hidden tests — Training raised the false-repair rate: SFT repair more often produced candidates that pass public tests but fail hidden ones.
In the author’s words from the Overview · “Summary”
This experiment tested whether a trained repair policy can expand held-out coding coverage beyond frozen Qwen self-repair, under a fair comparison against spending the same estimated model-forward-token budget on more direct samples.
Overview
This top-level README was generated during repository normalization because the imported experiment did not include one.
- Source track:
track-z - Primary report: reports/qwen35_4b_trained_vs_frozen_repair_mdp_report.md
- Metadata: metadata.yaml
How To Read
Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.
Summary
This experiment tested whether a trained repair policy can expand held-out coding coverage beyond frozen Qwen self-repair, under a fair comparison against spending the same estimated model-forward-token budget on more direct samples.
Report
Rendered from reports/qwen35_4b_trained_vs_frozen_repair_mdp_report.md
Date: 2026-06-25
Summary
This experiment tested whether a trained repair policy can expand held-out coding coverage beyond frozen Qwen self-repair, under a fair comparison against spending the same estimated model-forward-token budget on more direct samples.
The result is negative for trained repair. On 150 held-out MBPP tasks, direct sampling covered 62.0% of tasks, leaving 57 zero-coverage tasks. Frozen repair recovered 3 of those 57 tasks. The SFT repair adapter recovered only 2 of 57 and had a higher false-repair rate. A token-matched sample-more baseline recovered 5 of 57, beating both repair arms at essentially the same estimated model-forward-token cost.
The practical read is: in this setup, the best use of extra model budget was more diverse direct generation, not trained repair. Frozen repair produced a small useful lift, but trained repair did not improve it.
Main Held-Out Results
| Arm | N | Coverage | Zero-to-one | Zero-to-one rate | False repair rate | Candidates/task | Distinct behavior | Forward tokens |
|---|---|---|---|---|---|---|---|---|
| Direct x4 | 150 | 62.0% | 0 / 57 | 0.0% | - | 3.45 | 0.770 | 138,902 |
| Frozen repair | 150 | 64.0% | 3 / 57 | 5.3% | 25.0% | 4.01 | 0.754 | 79,614 |
| SFT repair | 150 | 63.3% | 2 / 57 | 3.5% | 29.2% | 3.98 | 0.766 | 79,325 |
| Sample more | 150 | 65.3% | 5 / 57 | 8.8% | - | 5.18 | 0.708 | 79,861 |
Zero-To-One Tasks
| Arm | Zero-to-one task IDs |
|---|---|
| Direct x4 | - |
| Frozen repair | 112, 137, 147 |
| SFT repair | 112, 147 |
| Sample more | 36, 42, 67, 129, 148 |
Commit / Selection Summaries
The following table uses the final budget available in each candidate pool. Oracle coverage is a ceiling: it selects a hidden-correct candidate if one exists in the pool. Other policies use only public/visible candidate behavior.
| Arm | Commit policy | Budget | Coverage ceiling | Selected hidden-pass | Coverage captured |
|---|---|---|---|---|---|
| Direct x4 | First visible-pass | 8 | 62.0% | 60.0% | 96.8% |
| Direct x4 | Public-signature majority | 8 | 62.0% | 60.0% | 96.8% |
| Direct x4 | Shortest visible-pass | 8 | 62.0% | 61.3% | 98.9% |
| Direct x4 | Oracle coverage | 8 | 62.0% | 62.0% | 100.0% |
| Frozen repair | First visible-pass | 8 | 64.0% | 62.0% | 96.9% |
| Frozen repair | Public-signature majority | 8 | 64.0% | 62.0% | 96.9% |
| Frozen repair | Shortest visible-pass | 8 | 64.0% | 63.3% | 99.0% |
| Frozen repair | Oracle coverage | 8 | 64.0% | 64.0% | 100.0% |
| SFT repair | First visible-pass | 8 | 63.3% | 61.3% | 96.8% |
| SFT repair | Public-signature majority | 8 | 63.3% | 61.3% | 96.8% |
| SFT repair | Shortest visible-pass | 8 | 63.3% | 62.7% | 98.9% |
| SFT repair | Oracle coverage | 8 | 63.3% | 63.3% | 100.0% |
| Sample more | First visible-pass | 8 | 65.3% | 62.0% | 94.9% |
| Sample more | Public-signature majority | 8 | 65.3% | 62.0% | 94.9% |
| Sample more | Shortest visible-pass | 8 | 65.3% | 64.7% | 99.0% |
| Sample more | Oracle coverage | 8 | 65.3% | 65.3% | 100.0% |
Training Details
- SFT training examples: 17.
- Max steps: 80.
- Batch size / grad accumulation: 1 / 4.
- Learning rate: 0.0001.
- Final logged SFT loss: 0.01351678092032671.
- DPO was skipped because the SFT repair arm failed the held-out gate.
Figures
- Coverage by arm
- Zero-to-one by arm
- False repair rate
- Diversity by arm
- Tokens vs zero-to-one
- Repair SFT loss
Interpretation
The headline test was trained repair versus frozen repair on tasks with no hidden-correct direct sample. Trained repair did not pass that test: it recovered fewer zero-base tasks than frozen repair and produced a worse visible-pass-but-hidden-fail profile.
The sample-more baseline is the decisive comparator. It spent approximately the same model-forward-token budget as frozen repair and recovered more zero-base tasks. That means the repair loop did not justify its extra prompt structure or training in this run.
The false-repair rates matter. Frozen repair had 28 visible-passing repair candidates, 7 of which failed hidden tests. SFT repair had 24 visible-passing repair candidates, also with 7 hidden failures. Repair can create plausible candidates that satisfy public evidence but do not generalize, so aggregate visible pass rates would overstate its value.
Limitations
- This is one held-out MBPP run, not a multi-seed estimate.
- The SFT adapter trained on only 17 mined repair examples, so the trained-arm negative should be read as a result for this small verified-repair recipe, not as a proof that repair training cannot work.
- Repair was conservative: it repaired visible-failing parsed candidates and did not repair candidates that already passed visible tests but failed hidden tests.
- No transfer benchmark was run in this package; the held-out MBPP comparison is the primary readout.
- Hidden tests were used for evaluation and train-side label mining, but not included in repair prompts.
Conclusion
The experiment does not support trained repair as the next deployable posttraining lever. The best observed intervention was to preserve generation diversity and spend the matched budget on more direct samples. A stronger future repair experiment would need either a much larger verified repair set, a process objective that reduces false repairs, or a repair policy aimed at visible-pass hidden-fail near misses rather than only visible failures.
Experiment log 12
Show the running log (12 entries)
Date: 2026-06-25
Experiment directory: /workspace/experiments/qwen35_4b_trained_vs_frozen_repair_mdp
Large artifacts directory: /workspace/large_artifacts/qwen35_4b_trained_vs_frozen_repair_mdp
Objective
Test whether a trained repair policy expands held-out generation coverage beyond frozen Qwen self-repair at honestly matched model-forward-token budget.
The headline metric is zero-to-one hidden coverage lift: among held-out tasks where the direct sample pool contains no hidden-correct candidate, how many become covered after repair.
Design Commitments
- The frozen Qwen repair loop is the primary baseline.
- The sample-more baseline is matched by estimated forward tokens, not by candidate count.
- Training labels may use full train-task tests, but repair prompts contain only task text, public tests, candidate code, and visible execution traces.
- Hidden tests are used for evaluation and train-side labeling only; they are never included in repair prompts.
- False repair rate is tracked: visible-pass repair candidates that fail hidden tests.
- Diversity is tracked with behavior signatures after each arm.
- Rounds beyond the smoke gate are skipped if the pre-registered gates fail.
Initial Package
- Created package-local source utilities, direct sampler, repair rollout runner, repair dataset builder, SFT trainer, DPO trainer, token-matched sample-more baseline, and commit evaluator.
- Large LoRA artifacts will be stored outside the experiment directory.
Smoke Gate
Smoke direct sampling on 12 MBPP train tasks completed:
- Hidden coverage: 50.0%.
- Mean candidates/task: 2.92.
- Mean hidden-pass candidates/task: 1.33.
- Estimated forward tokens: 8,525.
Frozen repair on the same smoke train pool completed:
- Hidden coverage remained 50.0%.
- Zero-base records: 6.
- Zero-to-one repairs: 0.
- Visible-passing repair candidates: 2.
- False repairs among visible-passing repairs: 1.
- Estimated repair forward tokens: 6,020.
The smoke repair dataset had only 1 SFT example and 1 DPO pair, so the SFT gate was not met. Decision: expand train repair mining before training rather than fitting a meaningless adapter.
Train Repair Mining
Main direct sampling on 100 MBPP train tasks completed:
- Hidden coverage: 72.0%.
- Zero-coverage train tasks: 28.
- Mean candidates/task: 3.42.
- Mean hidden-pass candidates/task: 1.89.
- Estimated forward tokens: 93,141.
One-attempt frozen repair mining on this train pool completed:
- Hidden coverage: 75.0%.
- Zero-to-one repairs: 3 / 28.
- Visible-passing repair candidates: 15.
- False repair rate among visible-passing repairs: 13.3%.
- Estimated repair forward tokens: 50,830.
- Dataset: 13 SFT examples, 13 DPO pairs.
Expanded train-only repair mining with more sources and two attempts/source completed:
- Hidden coverage: 76.0%.
- Zero-to-one repairs: 4 / 28.
- Visible-passing repair candidates: 26.
- False repair rate among visible-passing repairs: 30.8%.
- Estimated repair forward tokens: 136,948.
- Dataset: 17 SFT examples, 15 DPO pairs.
Decision: train the repair SFT adapter on the expanded mined set. The signal is thin, so the trained-arm interpretation must be conservative and judged primarily against frozen repair.
Repair SFT
Repair SFT completed on the expanded mined training set:
- Train examples: 17.
- Max steps: 80.
- Batch size / grad accumulation: 1 / 4.
- Learning rate: 1e-4.
- Max sequence length: 1,536.
- Final logged loss: 0.0135.
- Adapter directory:
/workspace/large_artifacts/qwen35_4b_trained_vs_frozen_repair_mdp/models/repair_sft_lora.
Interpretation note: this adapter fit a very small, mined repair set. The loss confirms the optimizer can fit the examples, but the experiment's useful readout is whether the adapter beats frozen repair on held-out zero-to-one coverage without increasing false repairs.
Held-Out Direct Pool
Direct sampling on 150 MBPP held-out tasks completed:
- Hidden coverage: 62.0% (93 / 150 tasks).
- Zero-coverage held-out denominator: 57 / 150 tasks.
- Visible coverage: 62.0%.
- Mean candidates/task: 3.45.
- Mean hidden-pass candidates/task: 1.63.
- Mean visible-pass candidates/task: 1.96.
- Mean distinct behavior rate: 0.77.
- Estimated forward tokens: 138,902.
This locks the headline denominator. Repair arms are judged by how many of the 57 zero-coverage tasks become hidden-covered, and by whether they do so beyond frozen repair at comparable model-forward-token cost.
Held-Out Frozen Repair
Frozen Qwen repair on the held-out direct pool completed:
- Hidden coverage: 64.0% (96 / 150 tasks).
- Zero-to-one repairs: 3 / 57.
- Zero-to-one rate: 5.3%.
- Visible-passing repair candidates: 28.
- False repairs among visible-passing repairs: 7.
- False repair rate: 25.0%.
- Mean candidates/task: 4.01.
- Mean hidden-pass candidates/task: 1.77.
- Mean distinct behavior rate: 0.754.
- Estimated repair forward tokens: 79,614.
Interpretation note: frozen repair creates a small but real held-out frontier lift, but one quarter of visible-passing repairs fail hidden tests. A trained arm must improve the zero-to-one count without worsening this false-repair profile.
Held-Out Repair SFT
SFT-adapter repair on the same held-out direct pool completed:
- Hidden coverage: 63.3% (95 / 150 tasks).
- Zero-to-one repairs: 2 / 57.
- Zero-to-one rate: 3.5%.
- Visible-passing repair candidates: 24.
- False repairs among visible-passing repairs: 7.
- False repair rate: 29.2%.
- Mean candidates/task: 3.98.
- Mean hidden-pass candidates/task: 1.75.
- Mean distinct behavior rate: 0.766.
- Estimated repair forward tokens: 79,325.
Gate decision: skip DPO. The SFT adapter underperformed frozen repair on the headline metric (2 vs 3 zero-to-one repairs) and had a worse false-repair rate (29.2% vs 25.0%). Running DPO from this checkpoint would add variance without a positive SFT signal.
Held-Out Token-Matched Sample-More Baseline
Token-matched sample-more on the same held-out direct pool completed, using the frozen repair budget as the target:
- Target forward-token budget: 79,614.
- Actual estimated forward tokens: 79,861.
- Extra direct-sampling calls: 344.
- Hidden coverage: 65.3% (98 / 150 tasks).
- Zero-to-one additions: 5 / 57.
- Zero-to-one rate: 8.8%.
- Mean candidates/task: 5.18.
- Mean hidden-pass candidates/task: 2.39.
- Mean distinct behavior rate: 0.708.
Interpretation note: at matched estimated model-forward-token cost, spending the budget on additional direct samples beat both repair arms on the headline metric: sample-more recovered 5 zero-base tasks, frozen repair recovered 3, and SFT repair recovered 2.
Analysis And Report
Commit-policy summaries were generated for direct, frozen repair, SFT repair, and token-matched sample-more pools under first-visible, public-signature-majority, shortest-visible, and oracle-coverage policies.
The final report was generated at reports/qwen35_4b_trained_vs_frozen_repair_mdp_report.md with figures under reports/figures/.
Final readout:
- Trained repair did not beat frozen repair: SFT recovered 2 / 57 zero-base tasks versus frozen repair's 3 / 57.
- SFT repair had a worse false-repair profile: 29.2% visible-pass-hidden-fail repairs versus frozen repair's 25.0%.
- DPO was skipped by gate.
- Token-matched sample-more was the strongest held-out arm: 5 / 57 zero-base tasks at approximately the same estimated forward-token budget as frozen repair.
Conclusion: this package does not support trained repair as a deployable posttraining lever under the tested small verified-repair recipe. The stronger observed use of extra model budget was additional diverse direct sampling.
Final Audit
python -m py_compile src/*.py scripts/*.pypassed.- No files larger than 10 MB are present inside the experiment directory.
- Large LoRA artifacts are stored separately under
/workspace/large_artifacts/qwen35_4b_trained_vs_frozen_repair_mdp. - Report figures were generated as valid PNG files.
- Generated
__pycache__directories were removed. - Standalone-reference scan found no external-paper/citation hooks or cross-track narrative references; only the date-like seed value
20260625matched the broad scan.
Figures 6
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.