Qwen3.5-4B Diversity-Keyed Coverage Gate
The one idea you need
Think of asking one coder to solve the same problem again and again while dialing their creativity from cautious to wild. Each setting surfaces solutions the others miss. Pooling all the moods, rather than any single one, cracks problems that a few careful tries left unsolved.
The question
When a small model flunks a coding problem on its first few tries, is it truly incapable, or did it just not explore enough options?
What we found
Mostly the second. Of 24 Python problems a 4-billion-parameter model missed on four tries, spending more and more varied sampling recovered 15, lifting the share solved from 70% to nearly 89%. Mixing three creativity settings beat any single one, 13 recovered versus 10, because each cracked different problems. Nine stayed unsolved: genuine capability gaps.
Why it matters
Before training a small model to be more creative, exhaust cheap tricks at answer time: sample more, and pool several creativity settings instead of tuning just one. But budget for a hard core of problems no sampling fixes.
On this page
Results at a glance 3
How to read
Bars, left to right: the four-try base pool, three separate creativity settings, their pooled union, that union plus a code-repair pass, and a larger union. Bar height is the share of tasks solved on hidden checks; taller is better.
Takeaway → Every increase in sampling budget lifts the bar, though gains shrink; the repair pass adds nothing over the union, pointing to varied sampling as the real lever.
Data table
| Sampling arm | hidden coverage |
|---|---|
| base K4 | 70% |
| default K32 | 80% |
| hot K32 | 82.5% |
| diverse K32 | 81.2% |
| union K32 | 86.2% |
| union + repair | 86.2% |
| union K128 | 88.8% |
Numbers from experiments/qwen35_4b_diversity_keyed_coverage_gate/reports/summary.json
Technical framing
Hidden-test coverage: diverse sampling recovers most base misses — More and more-diverse sampling lifts coverage from 70% to 88.75%; visible-test repair adds nothing over the union.
How to read
Bars count how many of the 24 first-pass misses each strategy rescued: three single creativity settings, their union, the union plus repair, and a larger union. Higher is better.
Takeaway → The union (13) beats the best single setting (10), and the largest budget reaches 15 - the settings recover different problems, so their gains stack.
Data table
| Sampling arm | base misses recovered (of 24) |
|---|---|
| default K32 | 8 |
| hot K32 | 10 |
| diverse K32 | 9 |
| union K32 | 13 |
| union + repair | 13 |
| union K128 | 15 |
Numbers from experiments/qwen35_4b_diversity_keyed_coverage_gate/reports/summary.json
Technical framing
Base-miss recovery: complementary decoding policies stack — The union of three decoding policies recovers 13/24 misses vs 10 for the best single arm - the policies recover different tasks.
How to read
The line plots share of tasks solved (vertical) against total compute spent generating candidate answers (horizontal). Points climb up and to the right as budget grows; up-and-to-the-left would be more efficient.
Takeaway → Coverage keeps rising with spend but flattens; this no-training curve is the bar any future trained model must beat at equal cost.
Data table
| Total forward tokens | hidden coverage |
|---|---|
| 69.64k | 70% |
| 235.5k | 80% |
| 243.3k | 81.2% |
| 243.3k | 82.5% |
| 582.8k | 86.2% |
| 595.6k | 86.2% |
| 698.8k | 88.8% |
Numbers from experiments/qwen35_4b_diversity_keyed_coverage_gate/reports/summary.json
Technical framing
Coverage vs forward-token budget — Coverage keeps rising with sampling budget but with diminishing returns; this no-training curve is the bar any adapter must beat.
In the author’s words from the Overview · “Summary”
This experiment tests whether held-out MBPP tasks missed by a small direct sample pool are diversity-limited or capability-limited. The practical question is whether a small posttraining objective should try to reshape the model into a better ensemble sampler, or whether inference-time diverse sampling already captures the available headroom.
Overview
This top-level README was generated during repository normalization because the imported experiment did not include one.
- Source track:
track-z - Primary report: reports/qwen35_4b_diversity_keyed_coverage_gate_report.md
- Metadata: metadata.yaml
How To Read
Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.
Summary
This experiment tests whether held-out MBPP tasks missed by a small direct sample pool are diversity-limited or capability-limited. The practical question is whether a small posttraining objective should try to reshape the model into a better ensemble sampler, or whether inference-time diverse sampling already captures the available headroom.
Report
Rendered from reports/qwen35_4b_diversity_keyed_coverage_gate_report.md
Date: 2026-06-25
Question
This experiment tests whether held-out MBPP tasks missed by a small direct sample pool are diversity-limited or capability-limited. The practical question is whether a small posttraining objective should try to reshape the model into a better ensemble sampler, or whether inference-time diverse sampling already captures the available headroom.
Setup
- Model: Qwen3.5-4B, used as the generator.
- Dataset: 80 MBPP held-out tasks.
- Public evidence in the prompt: one visible assert per task.
- Evaluation: all remaining MBPP asserts and challenge asserts.
- Base pool: 4 direct samples per task.
- Ladder arms: for tasks missed by the base pool, add 28 samples under default, hot, or tuned-diverse decoding.
- High-budget extension: after merging all K~32 arms, add 40 hot samples only to tasks still uncovered.
- Frozen repair check: after the K~32 union, repair up to two visible-failing candidates on each remaining miss.
Coverage means at least one candidate in the pool passes all hidden evaluation tests. Base-miss recovery is measured only over the 24 tasks where the K=4 pool had no hidden-correct candidate.
Results
| Arm | Hidden coverage | Base-miss recovery | Total forward tokens | Mean candidates | Mean functional diversity |
|---|---|---|---|---|---|
| Base K4 | 70.0% | 0 / 24 (0.0%) | 69,645 | 3.45 | 0.477 |
| Default K32 | 80.0% | 8 / 24 (33.3%) | 235,491 | 10.28 | 0.358 |
| Hot K32 | 82.5% | 10 / 24 (41.7%) | 243,343 | 11.39 | 0.359 |
| Diverse K32 | 81.2% | 9 / 24 (37.5%) | 243,268 | 11.04 | 0.362 |
| Union K32 | 86.2% | 13 / 24 (54.2%) | 582,812 | 24.09 | 0.340 |
| Union+Repair | 86.2% | 13 / 24 (54.2%) | 595,586 | 24.26 | 0.340 |
| Union K128 | 88.8% | 15 / 24 (62.5%) | 698,783 | 29.31 | 0.338 |
Main Findings
The K=4 base pool covered 56 / 80 tasks (70.0%), leaving 24 base misses.
More inference-time sampling recovered a large fraction of those misses. The best single K~32 policy was hot decoding, recovering 10 / 24 base misses and raising coverage to 82.5%. Default and tuned-diverse decoding were slightly lower individually, but they recovered different tasks.
The union result is the core signal. Merging default, hot, and tuned-diverse K~32 pools recovered 13 / 24 base misses (54.2%) and raised coverage to 86.2%. This is stronger evidence for diversity-limited misses than any single arm, because the policies are complementary rather than redundant.
The high-budget extension recovered two more tasks, ending at 15 / 24 recovered base misses (62.5%) and 88.8% total coverage. The newly recovered tasks were 73 and 84. The final remaining base misses were: 16, 26, 31, 39, 43, 44, 48, 60, 77.
Frozen repair did not help on the residual slice. It recovered 0 additional tasks after the K~32 union, and its two visible-passing repairs were both hidden-wrong. This run therefore points to diverse direct sampling, not visible-test repair, as the useful inference-time lever for this setup.
Interpretation
The central result is positive for the diversity hypothesis: many failures of the small K=4 pool are not hard capability absences. They are reachable by changing the sampling distribution and spending more sample budget. At the same time, the strongest no-training baseline is already substantial, so a diversity-keyed adapter should not be considered successful unless it beats hot/diverse sampling and the union strategy at matched forward-token budget.
This package did not train an adapter. That is intentional: the diagnostic first established the no-training ceiling and the tuned sampling baselines that any adapter must beat. Training directly against stylistic clusters would be risky unless it improves functional coverage, because surface diversity alone is not the target.
Decision
Do not run blind verified self-training or visible-test repair training from this result. If a follow-up trains strategy keys or diversity tokens, its primary bar should be:
- Recover more than 10 / 24 base misses at the same budget as the best single K~32 arm.
- Approach or beat the union K~32 recovery of 13 / 24 while using fewer total forward tokens than the full union.
- Preserve or improve functional diversity, measured by distinct failure signatures.
- Avoid lowering base pass@1 or increasing visible-pass/hidden-fail repairs.
The most defensible next training experiment is therefore not ordinary SFT on successful samples. It is a budget-matched diversity-control objective whose output is judged by hidden-test coverage and functional failure-set diversity, with hot sampling and the K~32 union as mandatory baselines.
Experiment log 11
Show the running log (11 entries)
Date: 2026-06-25
Experiment directory: /workspace/experiments/qwen35_4b_diversity_keyed_coverage_gate
Large artifact directory: /workspace/large_artifacts/qwen35_4b_diversity_keyed_coverage_gate
Objective
Before training any strategy-token or diversity-keyed adapter, measure whether held-out coding misses are diversity-limited or capability-limited.
The headline question is: among held-out MBPP tasks where a small base candidate pool contains no hidden-correct solution, how many become covered when we spend more sampling budget under default, hot, and tuned-diverse decoding?
Design Commitments
- Use Qwen3.5-4B only.
- Keep this package standalone: its own config, scripts, logs, reports, data, and figures.
- Do not train a LoRA unless the high-K diagnostic shows meaningful recoverable headroom.
- Use hidden tests for evaluation only, never in prompts.
- Measure functional diversity with per-test failure bitstrings, not only surface/AST diversity.
- Compare tuned diverse decoding against default sampling before crediting any future training objective.
- Run frozen repair only as a combined inference-time recipe after sampling, and track false repairs separately.
Initial Package
- Created standalone experiment directories.
- Added package-local execution/model utilities.
- Added
sample_base_pool.py,sample_zero_base_ladder.py, andrun_frozen_repair.py. - Added per-test failure-bit evaluation for MBPP candidates so functional diversity is measurable.
Smoke
Base smoke on 8 MBPP held-out tasks completed:
- Samples/task: 4.
- Hidden coverage: 75.0%.
- Zero-base tasks: 2.
- Mean candidates/task after dedupe: 3.38.
- Mean functional diversity rate: 0.385.
- Estimated forward tokens: 6,122.
Hot-ladder smoke on the 2 base-missed tasks completed:
- Extra samples per zero-base task: 4.
- Hidden coverage stayed 75.0%.
- Zero-to-one: 0 / 2.
- Estimated extra forward tokens: 2,004.
Smoke decision: scripts, manifests, execution, and functional-diversity fields are working. Proceed to main diagnostic with staged sampling.
Main Baseline: K=4 Direct Sampling
Completed main_base_k4 on 80 MBPP held-out tasks.
- Candidate samples/task: 4.
- Hidden coverage: 56 / 80 = 70.0%.
- Base-missed denominator for the gate: 24 / 80 tasks.
- Base-missed task IDs: 15, 16, 22, 26, 31, 34, 35, 36, 39, 42, 43, 44, 48, 55, 59, 60, 67, 70, 73, 77, 81, 83, 84, 87.
- Mean deduped candidates/task: 3.45.
- Mean visible-pass candidates/task: 2.26.
- Mean hidden-pass candidates/task: 1.88.
- Mean behavior diversity rate: 0.756.
- Mean functional diversity rate: 0.477.
- Estimated forward tokens: 69,645.
Decision: the zero-base denominator is large enough for the intended gate. Proceed to K~32 ladder arms on only those 24 missed tasks: default-more, hot, and tuned-diverse decoding.
Main Ladder: Default-More K~32
Completed main_default_extra_k32: added 28 default-style samples to each of the 24 base-missed tasks.
- Hidden coverage: 64 / 80 = 80.0%.
- Zero-to-one recovery: 8 / 24 = 33.3%.
- Recovered task IDs: 22, 35, 36, 42, 67, 70, 81, 87.
- Mean deduped candidates/task: 10.28.
- Mean hidden-pass candidates/task: 2.21.
- Mean behavior diversity rate: 0.670.
- Mean functional diversity rate: 0.358.
- Estimated incremental forward tokens: 165,846.
Decision: extra sampling produced meaningful zero-to-one recovery. Continue the same K~32 diagnostic for hot and tuned-diverse decoding arms before deciding whether any training objective is justified.
Main Ladder: Hot K~32
Completed main_hot_extra_k32: added 28 high-temperature samples to each of the 24 base-missed tasks.
- Hidden coverage: 66 / 80 = 82.5%.
- Zero-to-one recovery: 10 / 24 = 41.7%.
- Recovered task IDs: 15, 22, 35, 36, 42, 55, 59, 67, 70, 81.
- Mean deduped candidates/task: 11.39.
- Mean hidden-pass candidates/task: 2.31.
- Mean behavior diversity rate: 0.685.
- Mean functional diversity rate: 0.359.
- Estimated incremental forward tokens: 173,698.
Observation: hot decoding recovered two more base-missed tasks than default-more, but the recovered set changed rather than strictly containing default-more. It recovered 15, 55, and 59 that default-more missed, while default-more recovered 87 that hot missed.
Main Ladder: Tuned-Diverse K~32
Completed main_diverse_extra_k32: added 28 mixed-temperature/wide-nucleus samples to each of the 24 base-missed tasks.
- Hidden coverage: 65 / 80 = 81.25%.
- Zero-to-one recovery: 9 / 24 = 37.5%.
- Recovered task IDs: 22, 34, 35, 36, 42, 59, 67, 81, 83.
- Mean deduped candidates/task: 11.04.
- Mean hidden-pass candidates/task: 2.23.
- Mean behavior diversity rate: 0.685.
- Mean functional diversity rate: 0.362.
- Estimated incremental forward tokens: 173,623.
Observation: tuned-diverse underperformed hot by one recovered task but recovered tasks 34 and 83 that hot missed.
Main Ladder: Union K~32
Merged base K=4 plus all three K~32 ladder arms into main_union_k32.
- Hidden coverage: 69 / 80 = 86.25%.
- Zero-to-one recovery: 13 / 24 = 54.2%.
- Recovered task IDs: 15, 22, 34, 35, 36, 42, 55, 59, 67, 70, 81, 83, 87.
- Remaining base-missed task IDs: 16, 26, 31, 39, 43, 44, 48, 60, 73, 77, 84.
- Mean deduped candidates/task: 24.09.
- Mean hidden-pass candidates/task: 2.88.
- Mean behavior diversity rate: 0.636.
- Mean functional diversity rate: 0.340.
- Estimated total forward tokens for merged pool: 582,812.
Decision: the base misses are substantially diversity-limited at this budget. Do not train a diversity-keyed adapter in this package; the no-training tuned sampling baselines are already strong and must be the benchmark for any future training. Run a small frozen-repair pass on the remaining union misses to measure complementarity.
Combined Recipe: Frozen Repair After Union K~32
Completed main_union_k32_repair: one frozen repair attempt from each of up to two visible-failing sources on the 11 union-missed tasks.
- Hidden coverage stayed: 69 / 80 = 86.25%.
- Zero-to-one recovery relative to union misses: 0 / 11.
- Visible-passing repairs: 2.
- Hidden-wrong visible-passing repairs: 2 / 2 = 100%.
- Estimated repair forward tokens: 12,774.
Decision: frozen repair is not a useful complement on this slice. It added no hidden-correct candidates and introduced visible-pass/hidden-fail failures. Since the K~32 union still leaves 11 misses, run one adaptive high-budget sampling extension on the remaining misses to approximate a K~128 diagnostic.
Adaptive High-Budget Extension: Union Hot K~128
Completed main_union_hot_extra_k128: added 40 hot samples to each of the 11 tasks still uncovered by the union K~32 pool.
- Hidden coverage: 71 / 80 = 88.75%.
- Zero-to-one recovery relative to union misses: 2 / 11 = 18.2%.
- Cumulative zero-to-one recovery relative to base misses: 15 / 24 = 62.5%.
- Newly recovered task IDs: 73, 84.
- Final remaining base-missed task IDs: 16, 26, 31, 39, 43, 44, 48, 60, 77.
- Mean deduped candidates/task: 29.31.
- Mean hidden-pass candidates/task: 2.91.
- Mean behavior diversity rate: 0.629.
- Mean functional diversity rate: 0.338.
- Estimated incremental forward tokens: 115,971.
Decision: more sampling still recovers some residual misses, but with diminishing returns. Stop generation here and write the standalone report. The main finding is that many K=4 misses are diversity-limited under inference-time sampling, while a smaller residual appears harder at the tested budget.
Figures 5
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/summary.json9.2 kB
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.