Qwen3.5-4B Coverage Banking: does banking shift the proposal distribution?
The one idea you need
Think of the model as a dart thrower aiming at correct little programs. Sampling more just throws more darts with the same aim. Training it on its own past bullseyes re-aims the arm — sometimes tightening onto a spot it already hits, sometimes reaching a new region of the board.
The question
If you fine-tune a small model only on the correct answers it produced itself, can it start solving problems it could never solve before?
What we found
Yes, but it depends on difficulty. On easy one-step problems it just pulls answers it already knew into its top guess (60% to 80%), gaining no new ground. On harder two-step problems something new happens: on brand-new tasks it proposes correct programs it had never found, tripling its solve rate from 15% to 45%. On the hardest problems, with no wins to learn from, nothing moves.
Why it matters
Before reaching for a bigger model or more sampling, collect your small model's own verified wins and fine-tune on them — it can raise its own ceiling, but only at difficulty levels where sampling already found some correct examples to learn from.
On this page
Results at a glance 2
How to read
Four difficulty levels along the bottom, from one to four chained steps. Bar height is the fraction of brand-new problems solved, higher is better. Four bars per level: base single guess, base within sixteen tries, self-trained single guess, self-trained within sixteen tries.
Takeaway → At one step the single guess jumps but the ceiling stays flat (it just concentrates); at two steps the ceiling itself triples (genuine expansion); three-plus steps sit empty.
Data table
| condition | base single-shot | sample-more (base cov@16) | banked single-shot | banked cov@16 |
|---|---|---|---|---|
| depth 1 | 60% | 100% | 80% | 90% |
| depth 2 | 5% | 15% | 10% | 45% |
| depth 3 | 0% | 0% | 0% | 0% |
| depth 4 | 0% | 0% | 0% | 0% |
Numbers from
Technical framing
Banking: CONCENTRATION at depth 1, EXPANSION at depth 2 — Depth 1: banking pulls coverage into the single-shot (0.60→0.80), ceiling flat = concentration. Depth 2: the coverage ceiling itself rises 0.15→0.45 (3×) on held-out tasks = expansion, proposing compositions the base never sampled.
How to read
Number of tries from one to sixteen along the bottom; fraction of brand-new two-step problems solved up the side; higher is better. Two lines: the original model (lower) and the self-trained model (upper).
Takeaway → The self-trained line sits above the original at every try count, ending near 45% versus 15% — it reaches correct programs the original never proposed.
Data table
| k (samples) | base (think) | banked (think) |
|---|---|---|
| 1 | 3.1% | 10.9% |
| 2 | 5.3% | 15.7% |
| 3 | 6.9% | 19.6% |
| 4 | 8.2% | 22.8% |
| 5 | 9.1% | 25.5% |
| 6 | 9.9% | 27.9% |
| 7 | 10.7% | 30% |
| 8 | 11.3% | 31.9% |
| 9 | 11.9% | 33.7% |
| 10 | 12.5% | 35.4% |
| 11 | 13% | 37.1% |
| 12 | 13.5% | 38.7% |
| 13 | 13.9% | 40.3% |
| 14 | 14.3% | 41.9% |
| 15 | 14.7% | 43.4% |
| 16 | 15% | 45% |
Numbers from
Technical framing
Depth-2 EXPANSION: banking triples the coverage ceiling (held-out) — The banked model proposes correct depth-2 compositions the base never sampled — unique-program count even drops, so banking moved the proposal mass onto correct compositions (C17's lever).
In the author’s words from the Overview · “Results”
Banking does BOTH. Depth 1: CONCENTRATION (think greedy@1 0.60→0.80, ceiling flat). Depth 2: EXPANSION (coverage@16 0.15→0.45, 3×, on held-out tasks — proposes compositions the base never sampled; diversity even drops, so proposal mass moved onto correct programs). Depths 3–4: no move (too few/no examples). Does not beat think sample-more at k=1, but banking+sample-more > base+sample-more. See reports/report.md, analysis/banking_coverage.png, runs/verdict.json.
Overview
Research Program
- Program:
structured_execution_and_compilers - Program question: C17 said the wall is proposal-COVERAGE and only shifting the proposal distribution can beat sample-more. Does banking the model's own verified solutions do that — by CONCENTRATION or EXPANSION?
- Prior anchors: C17 (wall is coverage, selection free), C11/C12 (banking lifts pass@k).
Question
Does QLoRA-SFT on the fixed 4B's OWN execution-verified solutions CONCENTRATE existing coverage into the greedy sample, or EXPAND the coverage ceiling (propose programs the base never sampled — cross the wall)?
Hypothesis
Pre-registered (reports/prereg.md): banking lifts single-shot at trained depths (P1); does not beat sample-more at k=1 (P2); CONCENTRATION dominates EXPANSION (P3); no lift at untrained depth 4 (P4).
Setup
- Model: Qwen3.5-4B (only permitted model). No teacher — targets are the model's own verified outputs.
- Harvest (TRAIN, list depths 1–3, 90 tasks): sample K=40 think, keep hidden-correct, cap 12/task → 80
{prompt, code}SFT pairs (by depth {1:49, 2:24, 3:7}). - Bank: QLoRA-SFT r32/alpha64, 3 epochs, single-shot prompt→code (no-think).
- Eval (HELD-OUT, disjoint, depths 1–4, 20/depth): 4 arms {base, banked} × {no-think, think}, greedy@1 + coverage@16. Sample-more baseline = base-think coverage@16.
Run
Smoke: python scripts/harvest.py --smoke then python scripts/train_lora.py --train data/train.jsonl --out runs/smoke_adapter --smoke then python scripts/eval_ladder.py --tag base --smoke Full: see runs/launch.sh (harvest→train→eval base/banked no-think→analyze) + runs/launch2.sh (base/banked think).
Results
Banking does BOTH. Depth 1: CONCENTRATION (think greedy@1 0.60→0.80, ceiling flat). Depth 2: EXPANSION (coverage@16 0.15→0.45, 3×, on held-out tasks — proposes compositions the base never sampled; diversity even drops, so proposal mass moved onto correct programs). Depths 3–4: no move (too few/no examples). Does not beat think sample-more at k=1, but banking+sample-more > base+sample-more. See reports/report.md, analysis/banking_coverage.png, runs/verdict.json.
Interpretation
The coverage wall is not immovable by self-training: banking expands the ceiling for held-out tasks where it has enough verified examples (C17's proposal-shift lever working). To push deeper (depth 3+, where sampling harvests ≈0), seed with tool-augmented harvest (C12). Refuted own P3 (concentration-only) optimistically.
Knowledgebase Update
- Program evidence updated:
research_programs/structured_execution_and_compilers/evidence.md(C18) - Claim ledger updated: C18 added
Artifacts
scripts/common.py,scripts/harvest.py,scripts/train_lora.py,scripts/eval_ladder.py,scripts/analyze.pydata/train.jsonl(verified SFT pairs),data/{train,eval}_tasks.jsonlruns/eval_{base,banked,base_think,banked_think}.json,runs/verdict.jsonanalysis/banking_coverage.png,reports/prereg.md,reports/report.mdruns/banked_adapter/— trained QLoRA adapter (omitted from git; regenerate via harvest+train)
Report
Rendered from reports/report.md
Summary
C17 proved the fixed 4B's generation wall is COVERAGE, not selection: sample+filter recovers what single-shot misses but is sample-more, and the only lever that can beat sample-more is shifting the PROPOSAL distribution. This experiment tests whether banking the model's OWN execution-verified solutions (QLoRA-SFT, no teacher) does that — and decomposes the mechanism: does banking CONCENTRATE existing coverage into the greedy sample, or EXPAND the coverage ceiling (propose programs the base never sampled — cross the wall)?
Answer: BOTH, depth-dependent — and the expansion is real and generalizes to held-out tasks.
- Depth 1 — CONCENTRATION. Banking pulls coverage into the single-shot: think greedy@1 0.60 → 0.80 (+0.20) while the coverage ceiling stays flat (1.00 → 0.90). Cheap no-think greedy@1 also lifts 0.20 → 0.45.
- Depth 2 — EXPANSION. The banked coverage ceiling rises 0.15 → 0.45 (3×) on HELD-OUT tasks: the banked model proposes correct depth-2 compositions the base never sampled (even with thinking + 16 tries). Diversity did not rise (unique programs 11.5 → 10.85) — banking concentrated the proposal distribution onto the correct region of depth-2 program space, exactly the "shift the proposal distribution" lever C17 named. This partially pushes back the coverage wall.
- Depth 3–4 — no move. Too few (depth-3: 7 pairs) or no (depth-4: untrained) training examples; the wall holds (base and banked coverage both 0).
Banking does not beat think-mode sample-more at k=1 (banked greedy 0.80 < base sample-more 1.00 at depth 1; 0.10 < 0.15 at depth 2). But banking + sample-more > base + sample-more (banked cov@16 = 0.45 vs base 0.15 at depth 2): if you will sample anyway, sample the banked model. Banking expands what sample-more can reach.
Research Program Fit
The correctly-aimed follow-through to C17 (wall is proposal-coverage) and the C11/C12 banking thread. Decomposes banking's mechanism (concentration vs expansion) — which C11/C12 measured only as a pass@k lift.
Method
Substrate list. Fresh verified-depth, collapse-rejected tasks, disjoint TRAIN / EVAL splits.
- Harvest (TRAIN): depths [1,2,3], 90 tasks. Sample K=40 identification completions/task (think, budget 512, no op-menu — the canonical C17 prompt). Keep execution-verified (hidden-correct) programs, cap 12/task. → 80 verified
{prompt, code}SFT pairs (the model's OWN code, no teacher) from 33/90 solved tasks; by depth {1: 49, 2: 24, 3: 7}. (57/90 tasks yielded nothing — C17's coverage wall.) - Bank: QLoRA-SFT (r32/alpha64, bnb 4-bit, 3 epochs), single-shot prompt→code, thinking off.
- Eval (HELD-OUT, disjoint): depths [1,2,3,4], 20/depth. Four arms in one grading harness: {base, banked} × {no-think, think}, greedy@1 + coverage@16. The sample-more baseline is base-think coverage@16.
Results
| d | base-think greedy@1 | sample-more (base-think cov@16) | banked-nt greedy@1 | banked-think greedy@1 | banked-think cov@16 | verdict |
|---|---|---|---|---|---|---|
| 1 | 0.60 | 1.00 | 0.45 | 0.80 | 0.90 | concentration |
| 2 | 0.05 | 0.15 | 0.05 | 0.10 | 0.45 | EXPANSION (3×) |
| 3 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | no-move |
| 4 (untrained) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | no-move |
Diversity (unique programs / 16): depth-1 base-think 9.45 → banked 7.95; depth-2 11.5 → 10.85; depth-3 13.0 → 13.45 — no collapse (consistent with C11).
Pre-registered verdicts
- P1 (banking lifts greedy): SUPPORTED at depth 1 (think greedy +0.20); WEAK at depth 2 (+0.05, below the predicted +0.15) — the depth-2 gain landed in the coverage tail, not the greedy mode.
- P2 (banked greedy beats sample-more at depth 2): REFUTED (0.10 < 0.15). Single-shot does not beat sample-more at k=1 in any depth.
- P3 (concentration NOT expansion): REFUTED in the optimistic direction — there IS expansion (depth-2 ceiling 0.15 → 0.45), alongside concentration at depth 1. Banking does both.
- P4 (no lift at untrained depth 4): HELD — depth 4 (and depth 3, only 7 pairs) do not move.
Interpretation
- The coverage wall is not immovable by self-training. Banking the fixed 4B's OWN verified solutions EXPANDS the coverage ceiling for HELD-OUT tasks at a depth with enough training signal (depth-2, 24 pairs → 3×). This is genuine cross-task generalization, not memorization, and it is C17's predicted lever (shift the proposal distribution) actually working — the proposal mass moves onto correct compositions (diversity even drops), tripling the depth-2 hit rate.
- Two mechanisms, cleanly separated by depth: where the base already covers well (depth 1), banking CONCENTRATES coverage into the deployable single-shot (greedy 0.60 → 0.80); where the base barely covers (depth 2), banking EXPANDS the ceiling (0.15 → 0.45). C11/C12's "banking lifts pass@k" is both of these.
- Bounded, honest win: banking does not beat think-mode sample-more at k=1, and it cannot expand a depth it has no verified examples for (depths 3–4 stay at 0 — plain sampling never harvested them). The recipe that follows: to push the wall deeper you need verified examples at that depth, which plain sampling cannot provide (coverage ≈ 0) — so seed the training set with tool-augmented harvest (C12 decompose-search). This is the concrete path from C18 back to C12.
- Deployment recipe: bank verified self-solutions, then sample-more on the banked model — the coverage ceiling itself rises where you have training signal (depth-2 3×), and single-shot improves at easy depths.
Next Experiments
- Tool-seeded banking: harvest depth-3 solutions via C12 decompose-search (where sampling gets ~0), bank them, and test whether the depth-3 coverage ceiling expands on held-out tasks (the wall-crossing test).
- Concentrate the depth-2 expansion into greedy: iterate banking (expert iteration) — does a second round pull the expanded depth-2 coverage into greedy@1?
- Dose–response: vary the number of depth-2 training pairs (4/12/24/48) to map expansion vs. example count.
Artifact Manifest
See reports/artifact_manifest.yaml. Key: scripts/harvest.py, scripts/train_lora.py, scripts/eval_ladder.py, scripts/analyze.py, scripts/common.py, data/train.jsonl, runs/eval_{base,banked,base_think,banked_think}.json, runs/verdict.json, analysis/banking_coverage.png. The trained adapter (runs/banked_adapter) is omitted from git.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 4
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/eval_banked.json9.3 kBruns/eval_banked_think.json9.8 kBruns/eval_base.json9.2 kBruns/eval_base_think.json9.7 kB
Reproduce
Smoke test
python scripts/harvest.py --smoke && python scripts/train_lora.py --train data/train.jsonl --out runs/smoke_adapter --smoke && python scripts/eval_ladder.py --tag base --smokeFull run
bash runs/launch.sh && bash runs/launch2.shRun steps are documented inside the experiment folder (README and scripts).