Qwen3.5-4B Constrained Coverage DPO
The one idea you need
Two ways to crack a hard coding problem: coach the model so each attempt is smarter, or just let it fire off twice as many attempts. Here the coaching was leashed to the untrained model, so it could sharpen answers without drifting into broken code.
The question
Can gently retraining a small coding model to prefer better answers solve more problems than simply letting it generate twice as many attempts?
What we found
Not quite. The nudged model solved 62% of problems within four tries versus 58% for the untrained model, and kept its first-try accuracy — a real, cheap gain. But simply doubling to eight tries solved 67%, still more. The consolation: the nudge reached its score on about half the compute, and cracked one problem the brute-force run missed.
Why it matters
A light preference tune buys small local models cheaper coverage but will not out-solve simply sampling more. The real win is combining both: route each problem to whichever approach tends to crack it, since each solves cases the other misses.
On this page
Results at a glance 3
How to read
For each of four setups, two bars out of 24 problems: the share solved by at least one of several tries (coverage, taller is better) and the share solved on the very first try. Setups: plain 4 tries, plain 8 tries, nudged 4 tries, and a scrambled-nudge control.
Takeaway → The nudged 4-try bars top the plain 4-try bars and the scrambled control on both measures, yet stay below the plain 8-try bar — cheaper gain, not a coverage win.
Data table
| arm | coverage@K | pass@1 proxy |
|---|---|---|
| base greedy-hot K=4 | 58.3% | 37.5% |
| base sample-more K=8 | 66.7% | 41.7% |
| constrained DPO K=4 | 62.5% | 41.7% |
| shuffled-label control K=4 | 58.3% | 25% |
Numbers from reports/report_summary.json (eval arms), also README/report table
Technical framing
Coverage and pass@1 on held-out MBPP: constrained DPO vs base sampling — Constrained DPO at K=4 beats base K=4 and the shuffled control without hurting pass@1, but stays below the K=8 sample-more reference.
How to read
Bars show the total words the model processed across all 24 problems for each of the four setups; shorter is cheaper and better. The plain 8-try setup is the tallest, the nudged 4-try setup among the shortest.
Takeaway → The nudged model uses about half the compute of the plain 8-try run, so its coverage comes far cheaper even though it solves slightly fewer problems.
Data table
| arm | forward tokens |
|---|---|
| base greedy-hot K=4 | 23.43k |
| base sample-more K=8 | 45.41k |
| constrained DPO K=4 | 22.41k |
| shuffled-label control K=4 | 25.57k |
Numbers from reports/report_summary.json (eval arms, token_usage.forward_tokens), also report table
Technical framing
Token cost of each arm: coverage gains vs forward tokens spent — Constrained DPO reaches 62.5% coverage at roughly half the tokens of K=8 sampling (66.7%), a better token point but not a coverage win.
How to read
Two lines track how hard the tuning task stays across ten steps: one trained on genuine good-versus-bad answer pairs, one on deliberately scrambled pairs. Flatter and lower means the model is learning steadily rather than breaking down.
Takeaway → The real-pair line holds flat and steady near its floor across all ten steps, showing the gentle nudge trained cleanly instead of collapsing into broken output.
Data table
| optimizer step | real pairs | shuffled pairs |
|---|---|---|
| 1 | 0.71 | 0.728 |
| 2 | 0.704 | 0.718 |
| 3 | 0.716 | 0.741 |
| 4 | 0.706 | 0.713 |
| 5 | 0.706 | 0.787 |
| 6 | 0.704 | 0.705 |
| 7 | 0.702 | 0.702 |
| 8 | 0.711 | 0.728 |
| 9 | 0.72 | 0.723 |
| 10 | 0.72 | 0.701 |
Numbers from reports/report_summary.json (training metrics)
Technical framing
Training loss over 10 optimizer steps: real vs shuffled preference pairs — Both LoRA runs were deliberately stopped at 10 steps; the constrained objective keeps loss near the DPO floor rather than collapsing.
In the author’s words from the Report · “Interpretation”
The real constrained adapter improves over base K4 and over the shuffled constrained adapter while preserving pass@1 and parseability. That means the constrained preference signal did not collapse into the usual parse/pass@1 failure mode, and it is not explained by label shuffling. It does not beat the K8 sample-more coverage reference. So this exact scalar constrained-DPO sampler should not be scaled as the next main bet. The useful new clue is complementarity: constrained K4 recovers a task that K8 base sampling missed, while K8 base recovers tasks constrained K4 missed. … Read the full result →
Overview
Standalone experiment package for a constrained coverage preference objective on Qwen3.5-4B code generation.
The experiment asks whether a small local adapter can improve coverage/sample efficiency without paying for it through parse collapse or first-sample degradation. The objective combines hard-negative DPO with reference-logprob anchoring and positive NLL anchoring. Success is defined on the coverage/pass@1/token Pareto frontier against inference-only sample-more baselines, not merely against a smaller K baseline.
Large model artifacts are stored outside this directory under:
/workspace/large_artifacts/qwen35_4b_constrained_coverage_dpo
Final report: reports/final_report.md
Pilot readout: constrained DPO preserved pass@1/parseability and beat the shuffled control, but did not reach the K=8 sample-more coverage reference. The useful next lead is policy-portfolio scheduling, not scaling this single constrained sampler unchanged.
Report
Rendered from reports/final_report.md
Question
Can a weak hard-negative DPO coverage signal be made useful by explicitly constraining it with a reference anchor, positive NLL anchor, and short early-stopped training, so coverage improves without sacrificing pass@1 or parseability?
The meaningful comparison is not only base K4. The gate is whether constrained DPO is on a better coverage/pass@1/token Pareto point than simply sampling more from the base model.
Pair Mining
| pair set | pairs | tasks with pairs | visible-wrong pair rate | source records |
|---|---|---|---|---|
| real | 58 | 20 | 13.8% | 36 |
| shuffled | 58 | 20 | 13.8% | 36 |
Held-Out Results
| arm | K | coverage@K | pass@1 proxy | parse / task | visible coverage | functional diversity | forward tokens |
|---|---|---|---|---|---|---|---|
| base_hot_k4 | 4 | 58.3% | 37.5% | 3.54 | 58.3% | 51.0% | 23434 |
| base_hot_k8_sample_more | 8 | 66.7% | 41.7% | 7.50 | 66.7% | 32.1% | 45406 |
| constrained_dpo_k4 | 4 | 62.5% | 41.7% | 3.58 | 62.5% | 54.9% | 22411 |
| constrained_shuffled_dpo_k4 | 4 | 58.3% | 25.0% | 3.54 | 58.3% | 55.6% | 25575 |
Task-Level Overlap
| arm | covered tasks | pass@1 tasks |
|---|---|---|
| base_hot_k4 | [12, 13, 14, 17, 18, 19, 21, 22, 23, 27, 28, 29, 30, 32] | [13, 17, 18, 19, 22, 27, 28, 29, 32] |
| base_hot_k8_sample_more | [11, 12, 13, 14, 17, 18, 19, 22, 23, 27, 28, 29, 30, 32, 33, 34] | [11, 12, 14, 17, 18, 19, 27, 28, 29, 30] |
| constrained_dpo_k4 | [11, 12, 13, 14, 17, 18, 19, 22, 23, 25, 27, 28, 29, 30, 32] | [12, 14, 17, 18, 19, 22, 23, 27, 28, 32] |
| constrained_shuffled_dpo_k4 | [11, 12, 13, 14, 17, 18, 19, 22, 23, 27, 28, 29, 30, 32] | [12, 17, 18, 19, 27, 32] |
Training
The constrained trainer used a DPO margin term plus a positive-sample NLL anchor and a reference-logprob drift penalty. Both the real and shuffled adapters were limited to ten optimizer steps.
Gate Readout
Constrained DPO vs base K4: coverage delta 4.2%, pass@1 delta 4.2%, parse-rate delta 1.0%. Constrained DPO vs shuffled constrained control: coverage delta 4.2%. Constrained DPO vs sample-more K8: coverage delta -4.2% at 22411 vs 45406 forward tokens. Task overlap: base K4 union constrained K4 covers 16 tasks; base K8 union constrained K4 covers 17 tasks. Gate readout: no scale-up yet. The constrained adapter preserves pass@1 and beats shuffled, but it does not reach the K8 sample-more coverage reference.
Interpretation
This is a small pilot on 24 MBPP-test tasks, so one recovered task is not enough to claim a robust method. Still, the control structure is informative. The real constrained adapter improves over base K4 and over the shuffled constrained adapter while preserving pass@1 and parseability. That means the constrained preference signal did not collapse into the usual parse/pass@1 failure mode, and it is not explained by label shuffling.
It does not beat the K8 sample-more coverage reference. So this exact scalar constrained-DPO sampler should not be scaled as the next main bet. The useful new clue is complementarity: constrained K4 recovers a task that K8 base sampling missed, while K8 base recovers tasks constrained K4 missed. The next direction is therefore a sampler portfolio or learned scheduler over multiple generation policies, with the same coverage/pass@1/token Pareto gate, rather than simply pushing one LoRA harder.
Experiment log 5
Show the running log (5 entries, 2026-06-26)
2026-06-26
- Created standalone constrained coverage-DPO experiment package.
- Primary question: can the coverage-efficiency signal from hard-negative preference learning survive explicit pass@1, parseability, and reference-distribution constraints?
- Primary gate: constrained DPO must beat or match sample-more on the coverage/pass@1/forward-token Pareto frontier. Beating K=4 alone is not enough.
Guardrails:
- parse successes per task must stay near base;
- pass@1 proxy must stay within 2 percentage points of tuned-hot base;
- shuffled constrained-DPO must not match the real constrained-DPO arm.
- If the gate fails, the result will be used to choose the next mechanism, not to stop the research program.
Setup and Training
- Copied the real-code sampling/evaluation utilities into this standalone package and redirected all model outputs to
/workspace/large_artifacts/qwen35_4b_constrained_coverage_dpo. Rebuilt hard-negative preference pairs from the local training sample pool:
- real pairs: 58 pairs across 20 tasks;
- shuffled control: same pairs with labels shuffled;
- visible-wrong pair rate: 13.8%.
Trained two ten-step QLoRA adapters:
constrained_dpo_lora;constrained_shuffled_dpo_lora.
- Objective: DPO margin term plus positive-sample NLL anchor plus reference-logprob drift penalty.
Evaluation
- Evaluated on MBPP test, offset 0, 24 tasks, one visible test, temperature 1.0, top-p 0.98.
Arms:
- base hot K=4;
- base hot K=8 sample-more reference;
- constrained DPO K=4;
- constrained shuffled-DPO K=4.
Result
| arm | K | coverage@K | pass@1 proxy | parse / task | functional diversity | forward tokens |
|---|---|---|---|---|---|---|
| base_hot_k4 | 4 | 58.3% | 37.5% | 3.54 | 51.0% | 23434 |
| base_hot_k8_sample_more | 8 | 66.7% | 41.7% | 7.50 | 32.1% | 45406 |
| constrained_dpo_k4 | 4 | 62.5% | 41.7% | 3.58 | 54.9% | 22411 |
| constrained_shuffled_dpo_k4 | 4 | 58.3% | 25.0% | 3.54 | 55.6% | 25575 |
- Constrained DPO beat base K=4 by one task and beat shuffled by one task while preserving pass@1 and parseability.
- It did not reach the K=8 sample-more coverage reference.
Task overlap showed complementarity:
- constrained K=4 recovered task 25, missed by base K=8;
- base K=8 recovered tasks 33 and 34, missed by constrained K=4;
- base K=8 union constrained K=4 covered 17/24 tasks.
Readout
- Formal gate: no scale-up of this exact scalar constrained-DPO sampler yet, because it did not beat sample-more coverage.
- Useful signal: this constrained update preserved pass@1/parseability and beat the shuffled control.
- Next direction: test a sampler portfolio or learned scheduler over base-hot, constrained-DPO, and other generation policies, judged by the same coverage/pass@1/forward-token Pareto gate.
Figures 5
Data files 3
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/pairs_real_summary.json920 Breports/pairs_shuffled_summary.json919 Breports/report_summary.json19 kB
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.