Qwen3.5-4B: Learn from Your Own Failures (DPO)
The one idea you need
A code reviewer can reliably tell a working program from a broken one. Reward it purely for widening that gap, and it finds a shortcut: push down its own odds of writing either program. Soon it can barely produce working code at all.
The question
If a small coding model can reliably tell its own correct answers from its wrong ones, can you train it to prefer the correct ones and write better code?
What we found
No. The model already ranked its own correct answer above its wrong one 81% of the time — a sharp internal judge. But training it to favor the correct ones destroyed its writing: single-best-guess solving peaked near 5% then cratered toward zero. The dull alternative won — simply training twice as long on the correct examples alone tripled first-try solving to about 11%.
Why it matters
If a small model already ranks its good outputs above its bad ones, don't spend compute on preference training to exploit that — it can silently collapse generation. First exhaust plain training on the verified-correct examples; the ceiling is often just undertraining.
On this page
Results at a glance 3
How to read
Each cluster is one training method, from the untrained base through training longer on correct examples to preference-trained versions. One bar per cluster is the best-of-16-tries rate, the other the single-best-guess rate; taller is better.
Takeaway → The train-twice-as-long cluster stands tallest on both bars; apart from one tiny early single-shot bump, every preference-trained cluster falls back to or below plain training — preferring correct over wrong added nothing.
Data table
| condition | coverage@16 (sampling ceiling) | greedy@1 (deployable single-shot) |
|---|---|---|
| base | 0% | 0% |
| SFT (banking, 3ep) | 11.3% | 3.7% |
| SFT_2x (6ep) | 21.2% | 11.3% |
| DPO (0.25ep) | 11.3% | 5% |
| DPO (3ep) | 1.3% | 1.3% |
| DPO-shuffled | 6.2% | 3.7% |
Numbers from
Technical framing
Learn from failures: DPO collapses; the coverage->deployable gap closes with MORE SFT — Every prior training in the arc was SFT-on-positives, which raises coverage@16 but not deployable greedy@1. Does DPO on the model's OWN (correct, incorrect) samples raise greedy@1? The SFT model ALREADY discriminates its own correct-vs-wrong at 2AFC=0.81 (strong latent verifier), BUT preference-optimizing it COLLAPSES generation: DPO greedy@1 bumps to 0.050 at 0.25 epochs (within noise of SFT 0.037), then craters -- 0.000 by 0.5ep, 0.013 by 3ep (both greedy AND coverage crash; classic over-optimization, margin blew to ~61). DPO never beats SFT_2x. The EFFECTIVE lever is just MORE SFT: SFT_2x (6 vs 3 epochs) TRIPLES greedy@1 (0.037->0.113) and doubles coverage. Shuffled control (0.037) confirms it's not pure loss-shape; real DPO (0.013) is even worse. NET: you can't close the coverage->deployable gap by preferring correct-over-wrong (that destroys generation) -- train longer on the correct ones (SFT). Strong latent sample-discrimination (0.81) is READ-ONLY; it doesn't transfer to a generation gain via DPO. (Extends prior MBPP DPO work here to the controlled depth-3 substrate.)
How to read
The horizontal axis is how long preference training continues; the two lines are the single-best-guess and best-of-16 solve rates, and a flat line marks the train-longer winner. Higher is better.
Takeaway → Single-best-guess gives a tiny early bump while best-of-16 holds flat, then both plunge toward zero, ending far below the flat winner line — more preference optimization steadily destroys generation.
Data table
| DPO epochs (continued from SFT) | greedy@1 | coverage@16 | SFT_2x greedy (the winner) |
|---|---|---|---|
| 0 | 3.7% | 11.3% | 11.3% |
| 0.25 | 5% | 11.3% | — |
| 0.5 | 0% | 7.5% | — |
| 3 | 1.3% | 1.3% | 11.3% |
Numbers from
Technical framing
DPO collapse trajectory: greedy@1 and coverage crash with more preference optimization — DPO from the SFT model: a within-noise greedy@1 bump at 0.25 epochs (0.037->0.050, coverage flat), then catastrophic collapse -- greedy 0.000 by 0.5 epochs, both greedy and coverage cratered to 0.013 by 3 epochs. The reward margin (logp_chosen - logp_rejected) grew unchecked to ~61: classic DPO over-optimization / distribution collapse. It never reaches SFT_2x's greedy@1 of 0.113 (just training SFT twice as long). Preference training on the model's own failures is fragile and does not close the deployable gap.
How to read
The horizontal axis strengthens a stabilizer meant to hold up the correct answers while pushing away wrong ones. The two rising lines are the preference-trained solve rates; two flat lines mark plain and longer training. Higher is better.
Takeaway → Stronger stabilizing lifts the preference lines slightly, confirming the diagnosis, but single-best-guess never even reaches plain training and neither line approaches the longer-training winner — the fix mitigates without ever recovering.
Data table
| NLL(chosen) anchor weight lambda (3 epochs DPO) | DPO greedy@1 vs anchor weight lambda | DPO coverage@16 vs lambda | plain SFT greedy (0.037) | SFT_2x greedy (0.113, the winner) |
|---|---|---|---|---|
| 0.05 | 1.3% | 1.3% | 3.7% | 11.3% |
| 0.3 | 1.3% | 3.7% | — | — |
| 1 | 2.5% | 6.2% | 3.7% | 11.3% |
Numbers from
Technical framing
Anchoring both distributions reduces the DPO collapse but never recovers to SFT — User question: does anchoring BOTH distributions equally (strong NLL(chosen) anchor) prevent the DPO collapse? The collapse is the 'both-go-down' pathology -- DPO grows the margin by dropping logp(rejected) AND logp(chosen), so the model stops GENERATING the correct code. Strengthening the anchor DIRECTIONALLY helps (greedy 0.013->0.025, coverage 0.013->0.062 as lambda 0.05->1.0), confirming the diagnosis. But it's NOT enough: even fully balanced (lambda=1.0) at 3 epochs ends BELOW plain SFT (0.025<0.037) and far below SFT_2x (0.113); the margin still blew to ~53. The accumulated negative push over 522 steps degrades generation faster than the anchor holds it. Safe regime = strong anchor + FEW steps, but there the DPO nudge is tiny and it ~= plain SFT. Conclusion stands: preference training can't beat 'just more SFT'.
In the author’s words from the Overview · “Results”
DPO does NOT close the gap: pre-DPO 2AFC=0.81 (strong latent verifier) but preference-optimizing it COLLAPSES generation (greedy 0.050@0.25ep -> 0.000@0.5ep -> 0.013@3ep; coverage crashes too). The lever is MORE SFT: SFT_2x triples greedy@1 (0.037->0.113). See reports/report.md, analysis/learn_from_failures.png.
Overview
Research Program
- Program:
posttraining_and_adaptation/evidence_conditioned_selection - Question: does preference/contrastive training on the model's OWN (correct, incorrect) samples raise deployable greedy@1 beyond SFT-on-positives (closing the coverage->deployable gap)?
- Extends prior MBPP DPO work here (constrained_coverage_dpo, offline_hard_negative_coverage_dpo) to the controlled list-DSL depth-3 substrate, targeting the greedy@1 gap.
Setup
- Harvest 174 same-task (chosen=verified-correct, rejected=verified-wrong) pairs from banked_1280's own no-think samples (disjoint from held-out).
- Arms (fresh QLoRA from base, matched correct data): base; SFT (positives, 3ep); SFT_2x (6ep, compute ctrl); DPO (cached-ref-SFT + NLL anchor, various epochs); DPO-shuffled (rejecteds deranged, loss-shape ctrl).
- Eval no-think greedy@1 (deployable) + coverage@16, frozen held-out depth-3, n=80.
Run
python scripts/harvest_pairs.py --adapter <banked_1280> --pool 500 --k 16 then build_shuffled.py, train_lora.py (SFT/SFT_2x), train_dpo.py, eval_ladder.py, analyze.py.
Results
DPO does NOT close the gap: pre-DPO 2AFC=0.81 (strong latent verifier) but preference-optimizing it COLLAPSES generation (greedy 0.050@0.25ep -> 0.000@0.5ep -> 0.013@3ep; coverage crashes too). The lever is MORE SFT: SFT_2x triples greedy@1 (0.037->0.113). See reports/report.md, analysis/learn_from_failures.png.
Interpretation
Strong latent sample-discrimination (2AFC 0.81) is READ-ONLY; it does not transfer to a generation gain via preference training (DPO destroys the model). Close the coverage->deployable gap with more SFT-on-positives, not preference-on-failures.
Knowledgebase Update
- Claim ledger: C29
Artifacts
scripts/harvest_pairs.py,scripts/build_shuffled.py,scripts/train_lora.py,scripts/train_dpo.py(manual cached-ref DPO on QLoRA),scripts/eval_ladder.py,scripts/analyze.pydata/pairs.jsonl,data/pairs_shuffled.jsonl,data/sft.jsonl,runs/eval_*.json,runs/verdict.json,analysis/learn_from_failures.png,reports/{report,design_review}.md- Adapters (~180MB each) moved out of repo.
Report
Rendered from reports/report.md
Summary
Across the whole arc, SFT-on-positives (banking) raises depth-3 coverage@16 (~0.3–0.5) but not deployable greedy@1 (stuck 0.05–0.19). Does preference/contrastive training on the model's OWN (correct, incorrect) samples — learning from its failures — raise greedy@1? Design hardened by an adversarial review (which noted this extends prior MBPP DPO work here — constrained_coverage_dpo, offline_hard_negative_coverage_dpo — which found constrained DPO preserved pass@1 but didn't beat sample-more on coverage; and demanded the compute control + the load-bearing shuffled control + early-stopping).
Result (no-think depth-3, frozen held-out, n=80)
| arm | greedy@1 | cov@16 |
|---|---|---|
| base | 0.000 | 0.000 |
| SFT (banking, 3 epochs) | 0.037 | 0.113 |
| SFT_2x (6 epochs, compute control) | 0.113 | 0.212 |
| DPO learn-from-failures (0.25 ep) | 0.050 | 0.113 |
| DPO (0.5 ep) | 0.000 | 0.075 |
| DPO (3 ep) | 0.013 | 0.013 |
| DPO-shuffled (loss-shape control) | 0.037 | 0.062 |
Harvested 174 (chosen=verified-correct, rejected=verified-wrong) same-task pairs from banked_1280's own no-think samples; chosen/rejected identical median length (151 chars — no length heuristic).
- The model is a strong latent verifier of its own samples: pre-DPO 2AFC = 0.810 (the SFT model assigns higher logp to its correct sample than its wrong one 81% of the time; matches C13's ~0.73).
- But preference-optimizing that discrimination COLLAPSES generation. DPO greedy@1 bumps to 0.050 at 0.25 epochs (within noise of SFT's 0.037; coverage flat = within-support), then craters: 0.000 by 0.5 ep, 0.013 by 3 ep — both greedy@1 AND coverage@16 crash. Classic DPO over-optimization (the margin logp_c−logp_r blew to ~61 unchecked). It never beats SFT_2x at any point.
- The effective lever is just MORE SFT: SFT_2x (6 vs 3 epochs) triples greedy@1 (0.037 → 0.113) and doubles coverage (0.113 → 0.212). The "gap" was partly UNDERTRAINING.
- The shuffled control (0.037) confirms it's not pure loss-shape — and real DPO (0.013) is even worse, so the same-task correct-vs-wrong signal made the collapse worse, not better.
Implication
You cannot close the coverage→deployable gap by teaching the model to PREFER its correct over its wrong samples (DPO) — that destroys its generation. Just train longer on the correct samples (SFT). The model's strong latent sample-discrimination (2AFC 0.81) is a "read-only" verifier ability that does not transfer to a "write" (generation) improvement via preference training. This extends the prior MBPP DPO finding (DPO didn't beat sample-more) to the controlled depth-3 substrate, adding: DPO is fragile/collapses, and the deployable gap is best closed by more SFT-on-positives.
Honest limits
My DPO recipe (cached-ref-SFT DPO, β=0.05, NLL anchor 0.05, LR 2e-5, no dev-early-stop) was not heavily constrained; the prior MBPP run used ~10 steps + heavier anchoring and avoided collapse. So a more carefully constrained DPO might not collapse — but across the tested range (0.25–3 epochs) DPO never beat SFT_2x, and collapsed by 0.5 epochs, so the negative is fairly robust for "does preference-on-failures beat more-SFT." Single seed, n=80.
Artifact Manifest
See reports/artifact_manifest.yaml. Adapters (~180MB each) moved out of repo.
Addendum: does anchoring BOTH distributions equally prevent the collapse? (partial, still loses)
Follow-up (user question): the collapse is the "both-go-down" pathology (DPO grows the margin by dropping logp_rejected AND logp_chosen). Fix attempt: strengthen the NLL(chosen) anchor so the positive distribution is held up while the negative is pushed away — "balance both so neither collapses." Swept the anchor weight λ at 3 epochs:
| DPO anchor λ | greedy@1 | cov@16 |
|---|---|---|
| 0.05 (original) | 0.013 | 0.013 |
| 0.3 | 0.013 | 0.037 |
| 1.0 (balanced) | 0.025 | 0.062 |
Directionally the fix works — stronger anchor → less collapse (greedy 0.013 → 0.025, coverage 0.013 → 0.062). This confirms the both-go-down diagnosis. But it is not enough: even fully balanced (λ=1.0) it ends below plain SFT (0.025 < 0.037) and far below SFT_2x (0.113), and the margin still blew to ~53 — the accumulated negative push over 522 steps degrades generation faster than the anchor holds it. The only safe regime is strong-anchor + few steps, but there the DPO term is a tiny nudge and the objective ≈ plain SFT. So the conclusion stands: preference training can't beat "just more SFT" here; anchoring mitigates but does not overcome the collapse. (Untested: strong anchor + early-stop + DPOP positive-constraint together — likely ≈ SFT.)
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 9
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/eval_DPO.json9.1 kBruns/eval_DPObal0.3.json9.5 kBruns/eval_DPObal1.0.json9.5 kBruns/eval_DPOe0.25.json9.5 kBruns/eval_DPOe0.5.json9.4 kBruns/eval_DPOshuf.json9.4 kBruns/eval_SFT.json9.1 kBruns/eval_SFT2x.json9.2 kBruns/eval_base.json9.2 kB
Reproduce
Smoke test
python scripts/harvest_pairs.py --adapter <banked_1280> --pool 32 --k 8Full run
harvest_pairs (pool 500 k 16) + build_shuffled + train SFT/SFT2x + train_dpo (0.25/0.5/3 ep) + shuffled + eval all + analyzeRun steps are documented inside the experiment folder (README and scripts).