Qwen3.5-4B Depth Scaling & Controls: saturation, data-vs-compute, depth-4
The one idea you need
A tireless robot bolts together the tools in its kit until it finds combinations that solve a puzzle, then writes each solution into a cookbook the model studies. What teaches the model is how many DIFFERENT recipes fill the cookbook, not how often it rereads them.
The question
When you teach a small model from a solver's worked solutions, does piling on more eventually stop helping, and is it the variety or just the extra training that matters?
What we found
Variety, decisively, and the gains never plateaued. Rereading the same 40 solved examples sixteen times over barely moved the solve rate (9% to 16%, within noise), but replacing them with more distinct examples at the exact same training cost jumped it (16% to 38%). And coverage kept climbing to 54% as distinct examples grew past a thousand, with no ceiling in sight.
Why it matters
When self-training a small model on a solver's verified solutions, spend your budget harvesting more DISTINCT solved problems, not more passes over the ones you have. Track distinct-solution count as the real lever, and keep collecting, because the curve never flattened.
On this page
Results at a glance 3
How to read
The horizontal axis is the number of distinct solved problems banked, from 39 up to 1,156. The upper line is the share solved within 16 tries; the lower line is the share solved on the very first try. Up and to the right is better.
Takeaway → Both lines climb steadily to the last point with no bend toward flat, so the model keeps gaining and there is still headroom past 1,156 distinct examples.
Data table
| distinct depth-3 functions banked (N=40/160/640/1280 tool-pairs) | depth-3 think coverage@16 | think single-shot greedy@1 (deployable) |
|---|---|---|
| 39 | 8.7% | 1.3% |
| 153 | 21.2% | 3.7% |
| 555 | 37.5% | 13.8% |
| 1156 | 53.7% | 18.8% |
Numbers from
Technical framing
No saturation through 1280: depth-3 coverage keeps rising with distinct tool-found functions — C23 showed the depth-3 install is data-limited; this extends the dose curve. Coverage@16 climbs 0.087->0.212->0.375->0.537 at 39/153/555/1156 DISTINCT depth-3 functions (N=40/160/640/1280 tool-pairs), and deployable single-shot greedy@1 rises to 0.188 -- NO saturation through 1156 distinct functions. Distinct functions grow near-linearly with pairs, so this is real capacity, not running out of functions to harvest. (Adjacent-dose Wilson CIs marginally overlap at n=80; the overall trend is unmistakable. 2560 dose dropped: training ran 2x slower than budgeted.)
How to read
Three bars show the share of unseen puzzles solved within 16 tries: 40 examples trained briefly, the same 40 trained sixteen times harder, and a larger, more varied set at that same heavy training. Taller is better.
Takeaway → The middle bar barely clears the left one (more training, no real gain), but the right bar more than doubles it at equal cost, so variety is the lever.
Data table
| condition | depth-3 coverage@16 |
|---|---|
| N=40 (40 funcs, 120 visits) | 8.7% |
| up40 (40 funcs, 1920 visits) | 16.3% |
| train_640 (640 funcs, 1920 visits) | 37.5% |
Numbers from
Technical framing
The gain is data-DIVERSITY, not compute (2x2 at matched steps/mixture) — Deconfounding C23's 'more data' gain. Holding compute/mixture fixed and adding DIVERSITY (up40->train_640, both 1920 depth-3 example-visits, 40 vs 640 distinct functions): 0.163->0.375 -- CLEANLY significant (non-overlapping Wilson CIs). Holding diversity fixed and adding COMPUTE (N=40->up40, same 40 functions, 120->1920 visits): 0.087->0.163 -- positive but WITHIN NOISE (overlapping CIs). So the gain is genuinely data-DIVERSITY-driven: banking more DISTINCT explorer-found functions, not just more gradient steps.
How to read
For harder, deeper puzzles, three models each show two bars, solved-within-16-tries and solved-on-first-try. Left is the untrained model, middle is trained only on shallower puzzles, right adds 320 deeper examples. Taller is better.
Takeaway → Adding deeper examples nearly triples the 16-try coverage (7% to 18%), but the first-try bar stays flat at 3%, so the skill is present yet not reliable enough to deploy.
Data table
| condition | depth-4 coverage@16 | depth-4 single-shot greedy@1 (deployable) |
|---|---|---|
| raw base | 0% | 0% |
| scaffold (640 d3, no d4 data) | 6.7% | 3.3% |
| banked_d4 (+320 depth-4 pairs) | 18.3% | 3.3% |
Numbers from
Technical framing
The recipe repeats one rung deeper (weakly): depth-4 installs over the depth-3 transfer baseline — Does the tool-search+banking recipe repeat one compositional rung deeper? Raw base can't do depth-4 (0.00). The depth-3 scaffold (banked_640, NO depth-4 data) already TRANSFERS to 0.067 -- so the correct attribution baseline is the scaffold, not base. Adding 320 depth-4 tool-pairs nearly TRIPLES coverage to 0.183. But deployable greedy@1 stays flat (0.033) = test-time-only, and n=60 CIs marginally overlap -- SUGGESTIVE not conclusive, the same weak stage depth-3 showed at low doses. No scaffold forgetting (depth-3 held at 0.425). A depth-4 dose ladder should strengthen it (per C23's logic).
In the author’s words from the Overview · “Results”
Arm 1: NO saturation through 1280. cov@16 0.00/0.087/0.212/0.375/0.537 at N=0/40/160/640/1280 (distinct funcs 39/153/555/1156); deployable greedy@1 -> 0.188. (Adjacent CIs marginally overlap at n=80.) Arm 2: data-DIVERSITY, not compute. N=40 0.087 -> up40 0.163 (+compute, within noise) -> train_640 0.375 (+diversity, cleanly significant). Arm 3: recipe repeats one rung deeper, weakly. base 0.00, scaffold-transfer 0.067, banked_d4 0.183 (~3x); greedy flat 0.033 (test-time-only), CIs marginally overlap at n=60. Depth-3 guardrail 0.425 (no forgetting). See reports/report.md, analysis/scaling_controls.png.
Overview
Research Program
- Program:
structured_execution_and_compilers - Three follow-ups to C23 (depth-3 install is data-limited): (1) does the dose curve saturate past 640? (2) is the gain data-diversity or compute? (3) does the recipe repeat one rung deeper (depth-4)?
- Anchors: C23 (data-limited dose curve), C22 (weak crossing), C13-C21 (the recipe).
Setup
- Model: Qwen3.5-4B only. Explorer: CPU interpreter brute-search over the 16-op DSL (no external model).
- Arm 1: doses 640/1280 depth-3 tool-pairs (nested; 555/1156 distinct functions). Reuse C23 base/40/160/640.
- Arm 2: banked_up40 (40 distinct x16=640 examples, matched size/mixture/steps to train_640) vs train_640 vs N=40 dose.
- Arm 3: banked_d4 (d12+640d3+320d4) vs SCAFFOLD-only (banked_640) on a fresh depth-4 held-out set.
- QLoRA r32/a64 epochs=3; frozen paired held-out, func-sig + op-composition dedup (0 leakage: d3 0/2305, d4 0/318).
Run
Smoke: python scripts/harvest2.py (CPU, builds all data). Full: harvest2 + train the 3 adapters + bash runs/launch_eval.sh + python scripts/analyze.py.
Results
- Arm 1: NO saturation through 1280. cov@16 0.00/0.087/0.212/0.375/0.537 at N=0/40/160/640/1280 (distinct funcs 39/153/555/1156); deployable greedy@1 -> 0.188. (Adjacent CIs marginally overlap at n=80.)
- Arm 2: data-DIVERSITY, not compute. N=40 0.087 -> up40 0.163 (+compute, within noise) -> train_640 0.375 (+diversity, cleanly significant).
- Arm 3: recipe repeats one rung deeper, weakly. base 0.00, scaffold-transfer 0.067, banked_d4 0.183 (~3x); greedy flat 0.033 (test-time-only), CIs marginally overlap at n=60. Depth-3 guardrail 0.425 (no forgetting).
See reports/report.md, analysis/scaling_controls.png.
Interpretation
The C13->C24 ladder-climbing recipe is diversity-driven (banking DISTINCT explorer-found solutions) and rung-repeatable (depth-4 installs weakly over the depth-3 transfer baseline). No saturation through 1156 distinct depth-3 functions.
Knowledgebase Update
- Program evidence:
research_programs/structured_execution_and_compilers/evidence.md(C24) - Claim ledger: C24 added
Artifacts
scripts/harvest2.py,scripts/train_lora.py,scripts/eval_ladder.py,scripts/analyze.pydata/train_{1280,up40,d4}.jsonl,data/eval_frozen_d{3,4}.jsonl,data/tool_depth{3_2560,4}.jsonlruns/eval_*.json,runs/verdict.json,analysis/scaling_controls.png,reports/{prereg,report,design_review}.md- Adapters (~180MB each) moved out of repo; regenerable.
Report
Rendered from reports/report.md
Summary
Three follow-ups to C23 (depth-3 install is data-limited), each hardened by an adversarial workflow review (reports/design_review.md). All on the list 16-op DSL, interpreter brute-search explorer (no external model), QLoRA r32/α64 epochs=3, frozen paired held-out deduped by function-signature AND op-composition (0 leakage verified: depth-3 0/2305, depth-4 0/318).
Arm 1 — No saturation through 1280 tool-pairs
| N (depth-3 pairs) | 0 | 40 | 160 | 640 | 1280 |
|---|---|---|---|---|---|
| distinct functions | 0 | 39 | 153 | 555 | 1156 |
| think cov@16 (Wilson) | 0.00 | 0.087 | 0.212 | 0.375 [.28,.48] | 0.537 [.43,.64] |
| deployable greedy@1 | 0.00 | 0.013 | 0.037 | 0.138 | 0.188 |
The depth-3 dose curve keeps climbing past 640 — cov@16 0.375 → 0.537 and deployable single-shot greedy@1 0.138 → 0.188. Distinct functions grow near-linearly (so a flat curve would be real saturation, not harvest-diversity exhaustion — it isn't flat). The data-limited regime extends at least to 1156 distinct depth-3 functions with no plateau. (Caveat: adjacent-dose Wilson CIs marginally overlap at n=80 — the overall trend is unmistakable and non-adjacent doses separate cleanly, but adjacent-point significance is not established. The 2560 dose was dropped: its training ran ~2× slower than budgeted and was blocking the eval phase.)
Arm 2 — The gain is data-DIVERSITY, not compute
A 2×2 at matched size/mixture/steps splits C23's "more data" gain:
| 40 distinct funcs | 640 distinct funcs | |
|---|---|---|
| 120 depth-3 visits | 0.087 (N=40 dose) | — |
| 1920 visits | 0.163 (up40) | 0.375 (train_640) |
- Diversity at fixed compute (up40 → train_640, both 1920 visits): 0.163 → 0.375 — cleanly significant (non-overlapping Wilson CIs: up40 [.10,.26], train_640 [.28,.48]).
- Compute at fixed diversity (N=40 → up40, same 40 functions): 0.087 → 0.163 — positive point estimate but within noise (overlapping CIs).
So C23's "data-limited" is genuinely data-diversity-limited: banking more distinct explorer-found functions is what drives the gain, not merely more gradient steps on the same functions.
Arm 3 — The recipe repeats one rung deeper, weakly
Depth-4 rung (cov@16, n=60), against the scaffold-only baseline (banked_640 = depth-1+2+640d3, no depth-4 data — the correct attribution reference, since depth-3 skill transfers):
| raw base | scaffold (640 d3) | banked_d4 (+320 d4) | |
|---|---|---|---|
| depth-4 cov@16 | 0.000 | 0.067 | 0.183 |
| depth-4 greedy@1 | 0.000 | 0.033 | 0.033 |
Adding 320 depth-4 tool-pairs nearly triples depth-4 coverage over the transfer baseline (0.067 → 0.183, +0.116). But deployable greedy@1 stays flat (0.033) — a weak/test-time-only install, exactly the signature depth-3 showed at low doses (C22). At n=60 the Wilson CIs marginally overlap (scaffold [.03,.16] vs banked_d4 [.11,.30]), so this is suggestive, not conclusive. The depth-3 guardrail held (banked_d4 scores 0.425 on depth-3 vs the scaffold's 0.375 — no forgetting, slight improvement). By C23's logic, a depth-4 dose ladder should strengthen it.
Research Program Fit
Refines and stress-tests C23. Arm 1 shows the data-limited regime is deep (no saturation through 1156 distinct functions). Arm 2 attributes the gain to genuine data-diversity, tightening "self-training installs what the explorer finds" — it's the distinct verified solutions that matter. Arm 3 shows the whole recipe (tool-search explorer + banking installer) repeats one compositional rung deeper, at the same weak-then-scales efficiency per rung. Together: the ladder-climbing recipe is diversity-driven and rung-repeatable.
Method
scripts/harvest2.py (extend depth-3 to 2560 nesting C23's 640 excluding the held-out; upsampled-40; depth-4 harvest + held-out), scripts/train_lora.py, scripts/eval_ladder.py (func-sig + op-composition dedup + leakage report), scripts/analyze.py (Wilson CIs + CI-overlap verdicts). Reused C23's depth-3 frozen held-out
- base/40/160/640 evals + banked_640 adapter (bit-identical → same numbers, no re-run).
Pre-registered verdicts
- P1 (saturation): curve still rising at 1280 (0.375 → 0.537); no saturation through 1156 distinct functions (adjacent-dose significance marginal).
- P2 (data vs compute): data-diversity dominates — diversity effect cleanly significant, compute effect within noise.
- P3 (depth-4): recipe repeats, weakly — banked_d4 0.183 vs scaffold 0.067 (~3×), but test-time-only (greedy flat) and CIs marginally overlap at n=60; suggestive, needs a depth-4 dose ladder.
Honest limits
Single training seed at every point (Wilson CIs are eval-noise only; no seed error bars). n=60–80 underpowers adjacent-dose and depth-4 significance — point estimates are clear, adjacent CIs overlap. Fixed epochs (data confounded with gradient exposure except in the deconfounding up40 control). Depth-4 is a single dose (installs-vs-installs-a-little unresolved). 2560 dose dropped (training too slow). Harvested pairs are the search-easy tail.
Next Experiments
- Depth-4 dose ladder (80/320/1280) to resolve installs-vs-installs-a-little and test whether depth-4 becomes deployable with more data (the C23 test one rung deeper).
- ≥3 training seeds at the decision points (saturation top, up40, depth-4) for seed error bars.
- Push depth-3 past 1280 with faster training to locate the eventual saturation.
Artifact Manifest
See reports/artifact_manifest.yaml. Adapters (~180MB each) moved out of repo.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 10
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/eval_b1280_th3.json9.6 kBruns/eval_b160_th3.json9.5 kBruns/eval_b40_th3.json9.4 kBruns/eval_b640_th3.json9.5 kBruns/eval_b_up40_th3.json9.7 kBruns/eval_banked_d4_d3guard.json10 kBruns/eval_banked_d4_d4.json7.4 kBruns/eval_base_d4.json7.1 kBruns/eval_base_th3.json9.5 kBruns/eval_scaffold_d4.json7.3 kB
Reproduce
Smoke test
python scripts/harvest2.pyFull run
python scripts/harvest2.py && (train 3 adapters) && bash runs/launch_eval.sh && python scripts/analyze.pyRun steps are documented inside the experiment folder (README and scripts).