Qwen3.5-4B Offline Hard Negative Coverage DPO

Finished2026-06-26imported · line ZAlgorithmic Memory and Retrieval Evidence-Conditioned Selection Posttraining and Adaptation Reliability and Safety GitHub ↗

The one idea you need

A small coding model writes several attempts per task. Some pass the one visible example test but secretly fail the hidden checks — convincing traps. This training mines pairs of a truly-correct solution versus such a trap and teaches the model to prefer the real one.

The question

Can a light preference-training pass teach a small coding model to solve more tasks without having to generate so many candidate answers?

What we found

It depends — and even the gentle version failed its own bar. A light touch matched the task-solving reach of generating twice as many candidate answers at roughly half the compute, and beat a shuffled-pair placebo. But it cut first-try correctness from 50% to 37.5%. Pushed harder, the model collapsed and stopped producing runnable code at all.

Why it matters

If you preference-tune a small model for broader coverage, watch first-answer quality and whether the output still runs. This objective quietly reshuffles probability mass — buying breadth by sacrificing the accuracy of any single generation.

Tasks eventually solved62.5% → 66.7%gentle update vs the fixed model, matching twice-the-samples sampling

First answer correct50% → 37.5%the catch: single-answer accuracy dropped after the update

Compute for the same reachabout halfvs solving more by generating twice as many candidate answers

Aggressive version0% solvedtrained too hard, the model stopped producing runnable code

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 3

Tasks solved versus first-answer accuracy across all training recipes

How to read

Each recipe shows two bars: the share of tasks eventually solved after several attempts, and how often the very first answer is correct. Higher is better for both. The leftmost pair is the fixed, untrained model.

Takeaway → The gentle recipe's solved-bar edges above the untrained model while its first-answer bar sits lower, and the two aggressive recipes flatline at zero — coverage bought by trading first-try quality.

Data table

arm	coverage@K	pass@1 proxy
tuned-hot K=4	62.5%	50%
sample-more K=8	66.7%	29.2%
aggressive DPO (60 steps)	0%	0%
shuffled DPO (60 steps)	0%	0%
positive-only SFT	54.2%	45.8%
conservative DPO (10 steps)	66.7%	37.5%
conservative shuffled control	58.3%	37.5%

Numbers from report table (reports/final_report.md; matches reports/report_summary.json)

Technical framing

Held-out MBPP coverage vs first-sample quality across training arms — Conservative DPO lifts coverage to match K=8 sampling but drops pass@1 by 12.5 points; aggressive DPO collapses entirely.

Computing effort to reach the same share of tasks solved

How to read

Two bars whose height is the computing effort spent. Both recipes solve the same share of tasks, so a shorter bar is better because it means less effort for the same result.

Takeaway → The gentle-update bar is under half the height of the sample-more bar — the same tasks solved for roughly half the compute.

Data table

arm	forward tokens
conservative DPO K=4	20.93k
sample-more K=8	46.47k

Numbers from report table (reports/final_report.md; matches reports/report_summary.json)

Technical framing

Token cost to reach 66.7% coverage: conservative DPO vs sampling more — Both arms hit 66.7% held-out coverage, but conservative DPO at K=4 uses less than half the forward-token budget of K=8 sampling.

Runnable answers produced per task, by training recipe

How to read

One bar per recipe; height is how many runnable, non-broken answers it produces per task. Higher is better. Two of the bars are the aggressively trained versions.

Takeaway → The two aggressively trained bars collapse to zero runnable answers, while every light-touch recipe holds between three and four — overtraining destroys usable output.

Data table

arm	parse successes / task
tuned-hot K=4	3.71
aggressive DPO (60 steps)	0
shuffled DPO (60 steps)	0
positive-only SFT	3.54
conservative DPO (10 steps)	3.83
conservative shuffled control	3.83

Numbers from report table (reports/final_report.md; matches reports/report_summary.json)

Technical framing

Parse successes per task: aggressive DPO destroys parseability — The 60-step DPO adapters emit zero parseable candidates on held-out tasks, failing the parse guardrail; 10-step training stays safe.

In the author’s words from the Report · “Interpretation”

This pilot is intentionally a gate, not a final benchmark. The important result is mixed: Aggressive DPO is unsafe in this setup: the 60-step adapter learned the preference set but destroyed parseability on held-out generation. Conservative DPO has a real coverage-efficiency signal: K=4 matches the K=8 sample-more coverage at less than half the forward-token budget, and beats the matched conservative shuffled control. The formal gate still fails because conservative DPO regresses pass@1 proxy by 12.5 points versus tuned-hot K=4. That means the current objective improves set coverage by moving probability mass around, but it does not preserve first-sample quality. … Read the full result →

Overview

Standalone experiment package for an offline hard-negative preference objective aimed at improving coverage from a fixed Qwen3.5-4B generator.

The experiment mines task-local preference pairs from fresh sampled candidate pools: hidden-correct candidates are preferred over hard negatives that are visible-passing but hidden-wrong, or otherwise high-order parsed failures. A small QLoRA adapter is trained with a DPO-style contrastive loss and evaluated on held-out MBPP tasks against tuned inference-only sampling baselines.

Large model artifacts are stored outside this directory under:

/workspace/large_artifacts/qwen35_4b_offline_hard_negative_coverage_dpo

Report

Rendered from reports/final_report.md

Question

Can a small offline preference update, trained to prefer hidden-correct code over hard hidden-wrong candidates from the same task, improve held-out coverage from a fixed Qwen3.5-4B generator without collapsing useful sampling diversity?

Pair Mining

pair set	pairs	tasks with pairs	visible-wrong pair rate
real	58	20	13.8%
shuffled	58	20	13.8%

pair mining

Held-Out Results

arm	K	coverage@K	pass@1 proxy	parse / task	visible coverage	functional diversity	forward tokens
base_hot_k4	4	62.5%	50.0%	3.71	62.5%	53.8%	21542
base_hot_k8_sample_more	8	66.7%	29.2%	7.58	66.7%	33.9%	46467
hard_negative_dpo_k4	4	0.0%	0.0%	0.00	0.0%	66.7%	12660
shuffled_dpo_k4	4	0.0%	0.0%	0.00	0.0%	26.0%	28182
positive_sft_k4	4	54.2%	45.8%	3.54	54.2%	44.1%	20327
conservative_dpo_k4	4	66.7%	37.5%	3.83	66.7%	58.0%	20933
conservative_shuffled_dpo_k4	4	58.3%	37.5%	3.83	58.3%	50.3%	20877

coverage and pass1

diversity

pareto

Training

training loss

Gate Readout

Aggressive 60-step DPO failed the parse guardrail: coverage 0.0%, parse successes/task 0.00. Conservative 10-step DPO coverage 66.7% vs tuned-hot K=4 62.5%, delta 4.2%. Guardrails for conservative DPO: pass@1 delta -12.5%, functional-diversity delta 4.2%, parse successes/task 3.83. Sample-more K=8 reference: 66.7%. Matched conservative shuffled-pair control coverage: 58.3%. Positive-only SFT coverage: 54.2%. Gate readout: fail for the pilot; this configuration should not be scaled without changing the mechanism.

Interpretation

This pilot is intentionally a gate, not a final benchmark. The important result is mixed:

Aggressive DPO is unsafe in this setup: the 60-step adapter learned the preference set but destroyed parseability on held-out generation.
Conservative DPO has a real coverage-efficiency signal: K=4 matches the K=8 sample-more coverage at less than half the forward-token budget, and beats the matched conservative shuffled control.
The formal gate still fails because conservative DPO regresses pass@1 proxy by 12.5 points versus tuned-hot K=4. That means the current objective improves set coverage by moving probability mass around, but it does not preserve first-sample quality.

The next version, if any, should be explicitly regularized for pass@1 and parseability rather than scaling this objective as-is.

Experiment log 1

Show the running log (1 entry, 2026-06-26)

2026-06-26

Created standalone experiment directory with separate large-artifact storage.
Scope: offline hard-negative coverage DPO for Qwen3.5-4B on MBPP code generation.
Primary gate: a DPO adapter must beat tuned-hot base sampling at matched K on held-out coverage without collapsing pass@1 proxy or functional diversity.
Mechanism controls: shuffled-pair DPO and positive-only SFT if pair mining produces enough usable data.
Stop condition: if pair mining cannot produce a nontrivial task-diverse pair set, record that as the gate result rather than training a degenerate adapter.
Smoke sampling on 2 train tasks passed: records, candidate execution, and manifests were written correctly.
First pair-mining pilot on 12 train tasks at K=8 produced only 7 pairs across 3 tasks, below the training gate. Expanding the mining pool before training.
Expanded mining with 24 additional train tasks at K=12. Combined pool has 58 preference pairs across 20 tasks, which clears the pilot training gate.
Trained three pilot adapters in large-artifact storage: real hard-negative DPO, shuffled-pair DPO control, and positive-only SFT control.
Held-out K=4 evaluation:
- base hot: 62.5% coverage, 50.0% pass@1 proxy.
- aggressive hard-negative DPO: 0.0% coverage, 0 parse successes/task.
- aggressive shuffled DPO: 0.0% coverage, 0 parse successes/task.
- positive-only SFT: 54.2% coverage.
Rescue iteration:
- conservative hard-negative DPO (10 steps, lower LR/beta): 66.7% coverage, matching base hot K=8 at less than half the forward tokens, but pass@1 dropped to 37.5%.
- conservative shuffled DPO: 58.3% coverage.
- Formal gate fails due pass@1 regression, but the conservative DPO arm gives a weak coverage-efficiency signal worth only a multi-seed/pass@1-regularized follow-up.