Qwen3.5-4B Pass@k Coverage RL
The one idea you need
To cover more coding problems, let the model take several tries at each and count a win if any one passes — like buying several lottery tickets. More variety across tries covers more. You can retrain it to diversify, or just turn up its randomness dial.
The question
If you want a model to solve more coding problems across a handful of tries, is it worth retraining it to diversify, or just cranking up randomness?
What we found
No. Retraining the model to spread its tries across different problems actually lowered coverage: it solved about 44% of unseen coding problems within four tries, versus about 69% from simply raising the randomness dial at the same number of tries. It even trailed the plain, untouched model at 50%, and its first-try success sank too. The training collected its reward but never converted it into wider coverage.
Why it matters
Before paying for reinforcement-style fine-tuning to broaden what a small model can solve, exhaust the free knob first: raise sampling randomness. Here a simple randomness bump beat a full training run outright, at zero extra training cost.
On this page
Results at a glance 3
How to read
Four bars, one per way of drafting; bar height is the share of unseen problems where at least one try passes, so taller is better. The retrained model is the third bar; the other three are the untouched model at different randomness levels or try counts.
Takeaway → The retrained bar is shortest — below even the plain untouched model — while the untouched model at highest randomness stands tallest.
Data table
| sampling arm (test split) | coverage@K |
|---|---|
| base t=0.9, K=4 | 50% |
| base hot (t=1.0/1.2), K=4 | 68.8% |
| Pass@K RL adapter, K=4 | 43.8% |
| base t=0.9, K=8 (sample more) | 62.5% |
Numbers from experiments/qwen35_4b_passk_coverage_rl/reports/report_summary.json
Technical framing
Held-out MBPP coverage@K: Pass@K RL adapter loses to tuned inference-only sampling — The RL adapter (43.8%) undercuts even its own base temperature (50%); tuned-hot sampling at the same K wins outright (68.8%).
How to read
For each drafting method, one bar shows how often the first try is correct and another shows how varied the tries are from each other; higher is better on both. The retrained model is the third group of bars.
Takeaway → The retrained model has the lowest first-try quality of any bar, and it made its tries more alike than its own base — the opposite of the goal (only the sample-more arm ends up less varied still).
Data table
| sampling arm (test split) | pass@1 proxy | functional diversity |
|---|---|---|
| base t=0.9, K=4 | 37.5% | 46.9% |
| base hot (t=1.0/1.2), K=4 | 43.8% | 48.4% |
| Pass@K RL adapter, K=4 | 31.2% | 40.6% |
| base t=0.9, K=8 (sample more) | 37.5% | 27.3% |
Numbers from experiments/qwen35_4b_passk_coverage_rl/reports/report_summary.json
Technical framing
Guardrail metrics: adapter also degrades first-sample quality and diversity — The adapter is worst on both guardrails at K=4; K=8 sampling keeps pass@1 but collapses functional diversity.
How to read
Each point is one training round; the line shows how many of eight tries passed. Rounds sitting at zero (all failed) or at eight (all passed) teach nothing; only in-between rounds give a usable signal.
Takeaway → Most points sit flat at zero or eight, so only a handful of rounds gave any teaching signal — explaining why the training stalled.
Data table
| rollout group (training attempt) | positive rollouts per group |
|---|---|
| 1 | 0 |
| 2 | 0 |
| 3 | 0 |
| 4 | 7 |
| 5 | 8 |
| 6 | 8 |
| 7 | 6 |
| 8 | 0 |
| 9 | 0 |
| 10 | 3 |
| 11 | 7 |
| 12 | 0 |
| 13 | 6 |
| 14 | 8 |
| 15 | 0 |
| 16 | 7 |
Numbers from experiments/qwen35_4b_passk_coverage_rl/reports/pilot_unsat_passk_rl_training.json
Technical framing
Why training stalled: most rollout groups were all-fail or all-pass — 7 of 16 groups had zero passes and 3 were fully saturated, so only 6 groups gave a usable gradient signal for online Pass@K RL.
In the author’s words from the Overview · “Summary”
Can a small online QLoRA update aimed at set-level coverage make cheap K-sample generation cover more held-out MBPP tasks than tuned inference-only sampling?
Overview
This top-level README was generated during repository normalization because the imported experiment did not include one.
- Source track:
track-z - Primary report: reports/final_report.md
- Metadata: metadata.yaml
How To Read
Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.
Summary
Can a small online QLoRA update aimed at set-level coverage make cheap K-sample generation cover more held-out MBPP tasks than tuned inference-only sampling?
Report
Rendered from reports/final_report.md
Question
Can a small online QLoRA update aimed at set-level coverage make cheap K-sample generation cover more held-out MBPP tasks than tuned inference-only sampling?
Results
| arm | split | K | coverage@K | pass@1 proxy | visible coverage | functional diversity | forward tokens |
|---|---|---|---|---|---|---|---|
| base_t09_k4 | test | 4 | 50.0% | 37.5% | 75.0% | 46.9% | 14193 |
| base_hot_k4 | test | 4 | 68.8% | 43.8% | 75.0% | 48.4% | 14959 |
| passk_rl_unsat_k4 | test | 4 | 43.8% | 31.2% | 68.8% | 40.6% | 13650 |
| base_t09_k8_sample_more | test | 8 | 62.5% | 37.5% | 87.5% | 27.3% | 28487 |
Readout
Best observed held-out coverage arm: base_hot_k4.
- Adapter vs tuned-hot K=4: 43.8% vs 68.8%, delta -25.0%.
- Adapter vs same-temperature base K=4: 43.8% vs 50.0%, delta -6.2%.
- Sample-more reference K=8: 62.5% at 28487 forward tokens.
- Adapter guardrails: pass@1 proxy 31.2%, functional diversity 40.6%.
Training Diagnostics
Training attempted 16 rollout groups and took 6 updates. It skipped 7 zero-positive groups and 3 saturated-positive groups. Among update groups, positive rollouts ranged from 3 to 7; mean Pass@K utility was 0.988.
Interpretation
The decisive pilot comparison is the Pass@K adapter versus tuned-hot base sampling at matched K. A positive result requires the adapter to improve coverage without collapsing first-sample quality or functional diversity. In this pilot, the adapter did not pass that gate if tuned-hot K=4 is available and higher. The useful finding is diagnostic: online Pass@K reward can be obtained, but sparse or saturated rollout groups dominate, and the small adapter did not convert that reward into held-out coverage.
Gate Decision
The pilot gate failed, so the experiment stops here rather than scaling a larger training run. Scaling this exact online RL variant would spend compute on a configuration that is already dominated by inference-only hot sampling at matched K in the pilot.
Experiment log 3
Show the running log (3 entries)
Purpose
This standalone experiment tests a posttraining objective aimed at coverage rather than single-sample imitation. The adapter is trained with online sampled groups, execution rewards, and a Pass@K-style set utility. The primary readout is whether a small adapter improves held-out coverage at matched K against tuned inference-only sampling.
Protocol
- Build MBPP train and held-out records with one public test and hidden tests.
- Run base default and base hot sampling on a small held-out smoke split.
- Run a tiny online Pass@K RL smoke.
- If smoke works, run a pilot with more train and held-out tasks.
- Compare coverage@K, diversity, pass@1 proxy, and token usage.
- Write a report with figures.
Running Notes
- 2026-06-26: Created fresh standalone experiment and large-artifact directories.
- 2026-06-26: Smoke baseline on 4 held-out test tasks: base default K=4 covered 4/4; base hot K=4 covered 3/4.
- 2026-06-26: First online Pass@K RL smoke found zero full-pass rollouts across 4 train groups and hurt held-out smoke coverage (2/4). This exposed sparse reward starvation.
- 2026-06-26: Added positive-gated training that skips zero-positive groups. Smoke v2 took 4 updates after 6 attempts and tied matched-temperature held-out smoke coverage (3/4), but functional diversity dropped.
- 2026-06-26: Added unsaturated-positive gating to skip both zero-positive and all-positive groups. Pilot training took 6 updates after 16 attempts, skipping 7 zero-positive groups and 3 saturated-positive groups.
- 2026-06-26: Pilot held-out comparison on 16 test tasks: base t=0.9 K=4 covered 8/16, tuned-hot K=4 covered 11/16, Pass@K RL adapter K=4 covered 7/16, and base t=0.9 K=8 covered 10/16. The adapter failed the matched-K gate and reduced pass@1 proxy and functional diversity.
Figures 3
Data files 10
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/pilot_base_hot_k4_summary.json634 Breports/pilot_base_t09_k4_summary.json626 Breports/pilot_base_t09_k8_summary.json646 Breports/pilot_passk_rl_unsat_k4_summary.json723 Breports/report_summary.json12 kBreports/smoke_base_default_k4_summary.json634 Breports/smoke_base_hot_k4_summary.json618 Breports/smoke_base_t07_k4_summary.json614 Breports/smoke_passk_rl_adapter_k4_summary.json702 Breports/smoke_v2_passk_rl_k4_summary.json707 B
Reproduce
Run steps are documented inside the experiment folder (README and scripts).