Qwen3.5-4B Wall Climbing: does banking shallow composition unlock deeper coverage?
The one idea you need
Self-training here means fine-tuning a model only on problems it already solved by itself. Picture a climber drilling every hold she can reach: practice makes those moves rock-solid, but she never touches the higher hold, so there is nothing to learn from and her reach never extends.
The question
If you fine-tune a small model on the multi-step problems it can already solve, does it gain the ability to solve problems needing one more step?
What we found
No. Fine-tuning the model on the two-step solutions it could already produce tripled its two-step success on fresh tasks, from 12% to 36%. But its three-step success stayed at exactly zero, unchanged from before: both models solved none of the three-step tasks within sixteen tries. Becoming fluent at one difficulty tier taught it nothing about the tier just above.
Why it matters
Self-training on a model's own correct outputs sharpens skills it already demonstrates; it cannot invent a harder skill it never produces. To reach a new difficulty tier, first generate correct examples at that tier with search or tools, then fine-tune on those.
On this page
Results at a glance 2
How to read
Grouped bars compare the untrained model against the self-trained model at three difficulties — tasks needing two, three, or four chained steps. Bar height is the share of fresh tasks solved within sixteen tries; taller is better.
Takeaway → Only the two-step bar grows, from 12% to 36%. The three- and four-step bars are identical between models, three-step pinned flat at zero — training never reached the next rung.
Data table
| condition | base coverage@16 | banked1 coverage@16 (depth 1+2) |
|---|---|---|
| depth 2 | 12% | 36% |
| depth 3 (unlock test) | 0% | 0% |
| depth 4 | 4% | 4% |
Numbers from
Technical framing
Banking depth-1+2 installs depth-2 (3x) but unlocks ZERO depth-3 coverage — Apex bootstrapping test: bank ONLY depth-1+2 self-solutions (83 depth-2 pairs, no depth-3 examples), does the banked model now sample depth-3? Depth 2 install works and generalizes to held-out tasks (0.12->0.36, tripled). But depth 3 stays at exactly 0.00 -- a strong depth-2 composition skill does NOT length-generalize up. DEPTH-LOCAL: self-banking installs only depths already samplable; it cannot climb the wall. To extend the frontier you must SEED each rung externally with tool-search (C12), then bank.
How to read
Lines track the cumulative share of tasks solved as attempts rise from one to sixteen. Two flat lines sit at zero: three-step tasks for both the untrained and self-trained models. One rising line is two-step for the self-trained model. Higher is better.
Takeaway → The two-step line climbs steadily to 36% while both three-step lines stay glued to zero across all sixteen tries — extra sampling never surfaces a single three-step solution.
Data table
| k (samples) | depth 3 base | depth 3 banked1 | depth 2 banked1 (for contrast) |
|---|---|---|---|
| 1 | 0% | 0% | 12.5% |
| 2 | 0% | 0% | 17.1% |
| 3 | 0% | 0% | 20% |
| 4 | 0% | 0% | 22.1% |
| 5 | 0% | 0% | 23.8% |
| 6 | 0% | 0% | 25.3% |
| 7 | 0% | 0% | 26.6% |
| 8 | 0% | 0% | 27.8% |
| 9 | 0% | 0% | 28.9% |
| 10 | 0% | 0% | 30% |
| 11 | 0% | 0% | 31% |
| 12 | 0% | 0% | 32% |
| 13 | 0% | 0% | 33% |
| 14 | 0% | 0% | 34% |
| 15 | 0% | 0% | 35% |
| 16 | 0% | 0% | 36% |
Numbers from
Technical framing
Depth-3 coverage@k: base and banked1 both pinned to zero (no unlock) — The banked model's depth-2 coverage climbs to 0.36 with more samples, but its depth-3 coverage is flat at 0.00 -- identical to base. Banking a depth-2 skill buys nothing at depth 3.
In the author’s words from the Overview · “Results”
DEPTH-LOCAL. Depth 2: base 0.12 → banked1 0.36 (install works, tripled, held-out). Depth 3 (unlock test): base 0.00 → banked1 0.00 — zero unlock. Depth 4 unchanged (0.04). Self-banking installs only depths already samplable; it cannot climb the wall. See reports/report.md, analysis/wall_climbing.png, runs/verdict.json.
Overview
Research Program
- Program:
structured_execution_and_compilers - Program question: can banking be iterated to CLIMB the compositional wall — does installing depth-1+2 unlock depth-3 sampling?
- Prior anchors: C18 (banking installs/expands within a depth), C17 (wall is coverage), C12 (tool-search extends the frontier), C11-M4 (banking is coverage-bounded).
Question
If we bank ONLY depth-1+2 self-solutions, does the banked model now sample depth-3 compositions the base never could — bootstrapping the frontier upward by pure self-training?
Hypothesis
Pre-registered (reports/prereg.md): install works (P1); depth-3 unlocks ≥ +0.05 → CLIMBABLE, else DEPTH-LOCAL (P2); no two-rung leap to depth-4 (P3); Round-2 climb if unlocked (P4).
Setup
- Model: Qwen3.5-4B (only permitted model). No teacher — targets are the model's own execution-verified code.
- Harvest depth-1+2 only (20 d1 + 90 d2 tasks, K=40 think) → 130 pairs {d1:47, d2:83}, no depth-3 examples.
- Bank: QLoRA-SFT r32/alpha64, 3 epochs, single-shot prompt→code → banked1.
- Eval: coverage@16 (think, held-out, disjoint) at depths 2/3/4, base vs banked1, n=25/depth.
Run
Smoke: python scripts/harvest.py --smoke Full: bash runs/launch_r1.sh (harvest → train banked1 → eval base → eval banked1) then python scripts/analyze.py
Results
DEPTH-LOCAL. Depth 2: base 0.12 → banked1 0.36 (install works, tripled, held-out). Depth 3 (unlock test): base 0.00 → banked1 0.00 — zero unlock. Depth 4 unchanged (0.04). Self-banking installs only depths already samplable; it cannot climb the wall. See reports/report.md, analysis/wall_climbing.png, runs/verdict.json.
Interpretation
Composition skill does not length-generalize across a depth. Completes the wall picture: depth-3 is not represented (C19), not steerable (C20), not reachable by banking-shallow (C21). The only way up is to seed each rung externally with tool-search (C12), then bank — self-training is the installer, not the explorer.
Knowledgebase Update
- Program evidence updated:
research_programs/structured_execution_and_compilers/evidence.md(C21) - Claim ledger updated: C21 added
Artifacts
scripts/harvest.py(depth-configurable, adapter-loadable for round 2),scripts/train_lora.py,scripts/eval_ladder.py,scripts/analyze.py,scripts/common.pydata/train.jsonl(130 verified depth-≤2 pairs),data/{train_tasks,eval_tasks}.jsonlruns/eval_{base,banked1}.json,runs/verdict.json,analysis/wall_climbing.pngruns/banked1_adapter/— trained adapter (~180MB, moved out of repo; regenerate via harvest+train)
Report
Rendered from reports/report.md
Summary
C18 showed banking self-verified solutions installs and even expands composition coverage within a depth. This tests the mission's holy grail — can banking be iterated to climb the wall? Specifically: if we bank ONLY depth-1+2 solutions (which the base can harvest), does the proposal distribution shift enough that the banked model now SAMPLES depth-3 compositions the base never could, creating depth-3 coverage from nothing and enabling a second banking round?
Answer: NO. DEPTH-LOCAL. Banking a strong depth-2 composition skill produces exactly zero depth-3 coverage.
| depth | base cov@16 | banked1 cov@16 | Δ | base greedy@1 | banked1 greedy@1 |
|---|---|---|---|---|---|
| 2 | 0.12 | 0.36 | +0.24 | 0.04 | 0.08 |
| 3 (UNLOCK test) | 0.00 | 0.00 | +0.00 | 0.00 | 0.00 |
| 4 | 0.04 | 0.04 | +0.00 | 0.00 | 0.00 |
- The install worked, strongly: banking 83 depth-2 verified solutions (no depth-3 examples) tripled held-out depth-2 coverage (0.12 → 0.36) — a clean replication of C18's within-depth expansion, with a larger set and no depth-3 contamination.
- The unlock failed, completely: depth-3 coverage stayed at exactly 0.00. A strong depth-2 composition skill does not length-generalize "up" to make even one depth-3 task samplable. There is nothing to harvest for a Round 2, so the climb halts at the first rung.
Research Program Fit
The apex of the C13–C20 compositional-wall arc, and the direct test of the mission's "extend capability by a lot" hope. Sharpens C18 (banking) and C11-M4 (expert-iteration is coverage-bounded) into a hard cross-depth wall, and dovetails with C12 (tool-search extends the frontier).
Method
Substrate list. No teacher — all training targets are the fixed 4B's OWN execution-verified solutions.
- Harvest (depth-1+2 ONLY): 20 depth-1 + 90 depth-2 tasks, K=40 think samples/task, keep hidden-correct, cap 12/task → 130 verified
{prompt, code}pairs, {depth-1: 47, depth-2: 83} from 46/110 solved tasks. A clean depth-≤2 SFT set with 3× C18's depth-2 examples. - Bank: QLoRA-SFT (r32/alpha64, 3 epochs, single-shot prompt→code, no-think) →
banked1. - Eval: coverage@16 (think, greedy@1 + 16 sampled) on HELD-OUT tasks (disjoint from harvest) at depths 2, 3, 4, n=25/depth, base vs banked1, one identical harness.
Pre-registered verdicts
- P1 (install sanity, banked1 d2 ≥ base d2 + 0.10): HELD — +0.24.
- P2 (THE unlock, banked1 d3 ≥ base d3 + 0.05): REFUTED — Δ = 0.00 (base and banked1 both exactly 0.00 at depth 3). DEPTH-LOCAL.
- P3 (no two-rung leap, banked1 d4 ≈ 0): HELD — banked1 depth-4 = 0.04 (unchanged from base).
- P4 (Round-2 climb): N/A — no depth-3 coverage was unlocked, so there is nothing to harvest and bank for a second rung.
Interpretation
- Self-banking is coverage-seed-bounded. Banking installs — and generalizes well within a depth (depth-2 tripled on held-out tasks) — but it installs only depths the base can already sample. Composition skill does not length-generalize across a depth: a model that now covers 36% of depth-2 tasks still covers 0% of depth-3. You cannot bootstrap the frontier upward by self-training alone.
- This completes the mechanistic picture of the wall. Depth-3 composition is: not represented (C19 — the first-op representation thins to a thread at depth 3), not steerable (C20 — adding the latent direction is inert), and not reachable by banking-shallow (C21 — the depth-2 skill doesn't generalize up). All three test-time / self-training shortcuts fail at the deep wall by the same underlying fact — the composition simply is not in the model's reach at depth.
- The only way up is to seed each rung externally. To install depth-3 you first need depth-3 solutions to train on, and plain sampling harvests ≈ 0 of them. So the required proposal source is tool-augmented harvest (C12 decompose-and-compose search, which cracks depth-3 that monolithic sampling can't) → execution-verify → bank. The precise deployment recipe: tools reach the next rung, banking installs it, and only then does the base sample it — repeat. Self-training is the installer, not the explorer.
- Consistent with, and sharper than, C11-M4 ("banking compounds but is coverage-bounded"): the coverage bound is not gradual diminishing returns but a hard wall at the depth frontier — 0.36 at depth 2, a cliff to exactly 0.00 at depth 3.
Honesty notes / limits
- Depth-3 coverage is measured at K=16; a much larger K might surface rare depth-3 samples — but the base is also 0.00 at K=16, so the comparison (banking added nothing) is fair, and the depth-2 install is plainly visible at the same K. The claim is "no detectable unlock," not "provably zero at infinite K."
- Diversity did not collapse (unique depth-2 programs 11.0 → 9.2), so the null is not a diversity artifact.
- Single substrate (list), single banking round from the base. A tool-seeded Round 2 (harvest depth-3 via decompose-search, then bank) is the natural positive-control follow-up — it should install depth-3 where self-banking couldn't.
Next Experiments
- Tool-seeded banking (the positive control this predicts): harvest depth-3 solutions via C12 decompose-search (not plain sampling), bank them, and confirm depth-3 held-out coverage rises — demonstrating that the missing ingredient was the explorer, not the installer.
- Representation re-probe: does banking depth-2 raise the depth-2 first-op probe (C19) while leaving depth-3 a thread? Confirms banking installs representation exactly at the trained depth.
Artifact Manifest
See reports/artifact_manifest.yaml. Key: scripts/harvest.py, scripts/train_lora.py, scripts/eval_ladder.py, scripts/analyze.py, scripts/common.py, data/train.jsonl, runs/eval_{base,banked1}.json, runs/verdict.json, analysis/wall_climbing.png. The trained adapter (runs/banked1_adapter, ~180MB) is omitted from git.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/eval_banked1.json8.8 kBruns/eval_base.json8.6 kB
Reproduce
Smoke test
python scripts/harvest.py --smokeFull run
bash runs/launch_r1.sh && python scripts/analyze.pyRun steps are documented inside the experiment folder (README and scripts).