Research log Small Model Experimentation
GitHub

Qwen3.5-4B Depth-3 Dose-Response: data-limited or representational cap?

Finished2026-07-04Algorithmic Memory and RetrievalEvidence-Conditioned SelectionStructured Execution and CompilersGitHub ↗
The reasoning wall was data, not a ceiling

The one idea you need

Picture puzzles built as three stacked layers of nested steps — a recipe that calls sub-recipes that call basic moves. The small model cannot write them alone, but a tireless search tool grinds out correct worked examples. Train on those, and the skill sinks into the model's weights.

The question

When a small model keeps failing at deeply nested, multi-step puzzles, is it hitting a hard limit of its size, or just starved for training examples?

What we found

Just starved for examples. Feeding a fixed 4-billion-parameter model more search-found solutions lifted its solve rate on fresh three-layer puzzles from 0% to 38% when given sixteen tries — climbing steadily at every dose, with no sign of leveling off. It even began solving 1 in 10 fresh puzzles on a single blind attempt with no reasoning shown, versus essentially zero before.

Why it matters

Before declaring a small model too small for a hard compositional task, use a cheap search tool to auto-generate verified solutions and chart solve rate against how many you train on. A still-rising curve means you are data-limited, not capped.

Solve rate on fresh puzzles, sixteen tries allowed0% → 38%as training examples grew from none to 640
Steady climb across doses, no plateau9% → 21% → 38%solve rate after 40, 160, then 640 worked examples
Single blind attempt, no reasoning shown0% → 10%deployable one-shot accuracy at the top dose, previously essentially zero
Fresh test puzzles overlapping the training set0 of 80no shared structure with 750 training solutions — genuine generalization
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Pre-registered verdicts
    5. Interpretation
    6. Honesty notes / limits (from the review)
    7. Next Experiments
    8. Artifact Manifest
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 2

More worked examples, steadily more three-layer puzzles solved

How to read

Bars grouped by training amount banked (0, 40, 160, 640 worked examples); bar height is solve rate on fresh three-layer puzzles, higher is better. Four bars per group: sixteen-try and single-shot rates, each shown with and without a reasoning scratchpad.

0%10%20%30%40%0%0%0%0%N=0 (base)8.7%1.3%0%0%N=4021.2%3.7%0%0%N=16037.5%13.8%33.8%10%N=640

Takeaway → Every bar climbs left to right with no flattening; the two deployable no-scratchpad bars stay pinned at zero until the largest dose, then lift off — capability tracks the amount of data.

Data table
conditionthink coverage@16think single-shot greedy@1no-think coverage@16 (deployable)no-think single-shot greedy@1 (deployable)
N=0 (base)0%0%0%0%
N=408.7%1.3%0%0%
N=16021.2%3.7%0%0%
N=64037.5%13.8%33.8%10%

Numbers from

Technical framing

Depth-3 install is DATA-LIMITED: coverage rises with #tool-found pairs (no plateau) — C22 crossed the depth-3 wall but weakly (0.125) and test-time-only, leaving open: data-limited or representational cap? Bank N tool-found depth-3 solutions (interpreter search over the substrate's own DSL, no external model). Depth-3 think coverage rises MONOTONICALLY 0.00->0.087->0.212->0.375 across N=0/40/160/640 with NON-OVERLAPPING Wilson CIs (top-dose lower CI 0.28 > low-dose upper CI 0.17) -- DATA-LIMITED, not a cap. Crucially the DEPLOYABLE single-shot install scales too: no-think greedy@1 0.00->0.10 at N=640 (~0 at C22's N=130). 0-leakage held-out set (novel depth-3 rules by function AND op-composition) => genuine generalization. The deep wall is a DATA bottleneck, not a hard representational cap: tool-seeded banking scales with data into deployable capability.

Solve rate rises with both attempts allowed and training amount

How to read

Each line is one training dose (0, 40, 160, 640 examples); horizontal axis is attempts allowed per puzzle (1 to 16); vertical axis is solve rate on fresh three-layer puzzles. Higher is better; the untrained base line stays flat on the floor at zero.

0%20%40%60%51015banked N=640banked N=160banked N=40base (N=0)

Takeaway → The lines fan out cleanly in dose order, each larger dose sitting entirely above the smaller ones and none crossing — more training data lifts the whole curve, with no saturation.

Data table
k (samples)base (N=0)banked N=40banked N=160banked N=640
10%1.6%4.5%10.8%
20%3%7.7%17.3%
30%4.1%10.1%21.6%
40%5.1%12%24.7%
50%5.9%13.4%27%
60%6.5%14.6%28.9%
70%7%15.6%30.4%
80%7.4%16.4%31.8%
90%7.7%17.2%32.9%
100%8%17.9%33.9%
110%8.2%18.6%34.7%
120%8.3%19.2%35.5%
130%8.5%19.7%36.1%
140%8.6%20.3%36.6%
150%8.7%20.8%37.1%
160%8.7%21.2%37.5%

Numbers from

Technical framing

Depth-3 coverage@k fans out with dose (base flat at 0) — Each dose's coverage@k curve sits progressively higher: base pinned at 0, N=40 to 0.09, N=160 to 0.21, N=640 to 0.375. More tool-found depth-3 solutions -> more installed depth-3 capability on novel held-out rules, with no sign of saturation through N=640.

In the author’s words from the Overview · “Results”

DATA-LIMITED. Depth-3 think cov@16: 0.00 -> 0.087 -> 0.212 -> 0.375 (N=0/40/160/640), monotone, top-dose CI [0.28,0.48] non-overlapping with low-dose [0.04,0.17]. Deployable scales: no-think cov 0.00->0.338, greedy@1 0.00->0.10 at N=640. 0 leakage (novel rules). See reports/report.md, analysis/dose_response.png.

Overview

Research Program

  • Program: structured_execution_and_compilers
  • Question: C22 crossed the depth-3 wall but weakly — is the install data-limited or a representational cap?
  • Anchors: C22 (tool-seeded banking crossed weakly), C19 (depth-3 rep is a thread), C21 (self-banking can't seed).

Hypothesis

Pre-registered (reports/prereg.md, hardened by reports/design_review.md): DATA-LIMITED if depth-3 think coverage rises with N (top-dose CI above low-dose CI); CAP if it plateaus.

Setup

  • Model: Qwen3.5-4B only. Explorer: CPU interpreter brute-search over the 16-op DSL (640/640 solved, no external model).
  • Doses (nested): C21 depth-1+2 pairs + N tool-found depth-3, N in {40,160,640}; QLoRA r32/a64, epochs=3.
  • Eval: ONE frozen paired held-out set, dedup by function-sig AND op-composition (0 leakage), n=80 depth-3. Think coverage@16 (Wilson CIs) + no-think deployable at top dose; depth-2 guardrail.

Run

Smoke: python scripts/tool_harvest.py --smoke Full: python scripts/tool_harvest.py --n-depth3 640 && (build train_{40,160,640}.jsonl) && bash runs/launch.sh && python scripts/analyze.py

Results

DATA-LIMITED. Depth-3 think cov@16: 0.00 -> 0.087 -> 0.212 -> 0.375 (N=0/40/160/640), monotone, top-dose CI [0.28,0.48] non-overlapping with low-dose [0.04,0.17]. Deployable scales: no-think cov 0.00->0.338, greedy@1 0.00->0.10 at N=640. 0 leakage (novel rules). See reports/report.md, analysis/dose_response.png.

Interpretation

The deep wall is a DATA bottleneck, not a hard representational cap: tool-seeded banking scales with #solutions into deployable single-shot. Completes the C13->C23 recipe (explorer + installer + data throttle).

Knowledgebase Update

  • Program evidence: research_programs/structured_execution_and_compilers/evidence.md (C23)
  • Claim ledger: C23 added

Artifacts

  • scripts/tool_harvest.py, scripts/train_lora.py, scripts/eval_ladder.py (frozen + func-sig & op-comp dedup + leakage), scripts/analyze.py (Wilson CIs)
  • data/train_{40,160,640}.jsonl, data/eval_frozen.jsonl, runs/eval_*.json, runs/verdict.json, analysis/dose_response.png, reports/{prereg,report,design_review}.md
  • Adapters (~180MB each) moved out of repo; regenerable.

Report

Rendered from reports/report.md

Summary

C22 crossed the depth-3 wall with tool-seeded banking but only WEAKLY (think coverage@16 0.125, deployable no-think ≈0). Was that a data limit or a representational cap? This dose-response answers it: DATA-LIMITED, decisively. Banking N tool-found depth-3 pairs installs depth-3 coverage that rises monotonically with N and does not plateau — and, at scale, banks into deployable single-shot.

N distinct depth-3 pairsthink cov@16 (Wilson 95%)think greedy@1no-think cov@16no-think greedy@1
0 (base)0.000 [0.00, 0.05]0.0000.0000.000
400.087 [0.04, 0.17]0.013——
1600.212 [0.14, 0.31]0.037——
6400.375 [0.28, 0.48]0.1380.3380.100
  • Monotone rise, no plateau. Think coverage@16 climbs 0.00 → 0.087 → 0.212 → 0.375 across N ∈ {0,40,160,640}. The top-dose lower CI (0.277) sits above the low-dose upper CI (0.17) — a clean, non-overlapping increase (not a bare non-significant p). Rise 40→640 = +0.29.
  • The deployable install scales too. At N=640, no-think coverage@16 = 0.338 and no-think single-shot greedy@1 = 0.10 — vs C22 (N=130) where deployable was ≈0 (test-time-only). So C22's "test-time-dominated" weakness was also just insufficient data; more tool-found solutions bank depth-3 into the weights.
  • Genuine generalization, not memorization. The 80 held-out depth-3 tasks have 0 leakage — none share a function-signature OR an op-composition with the 750 training rules — so every solved task is a NOVEL depth-3 composition. This closes the finite-DSL memorization confound the review flagged as make-or-break.
  • Scaffold intact. Depth-2 guardrail rose (base 0.175 → banked_640 0.50); the depth-1+2 decomposition scaffold did not collapse under the depth-3-heavy mixture.

Research Program Fit

Resolves C22's central open question and, with it, the mission's "extend capability by a lot" hope: the tool-seeded-banking recipe is not just directionally valid (C22) but data-scalable — more explorer-found solutions → progressively more installed, deployable capability. Design hardened by an adversarial multi-agent review (reports/design_review.md).

Method

Substrate list, families' own 16-op DSL. No external model. Explorer = CPU interpreter brute-search (640/640 depth-3 solved). Doses (NESTED, log-spaced): C21's exact depth-1+2 pairs + N tool-found depth-3 pairs, N ∈ {40,160,640}; QLoRA r32/α64, epochs=3 held constant. Eval on ONE frozen paired held-out set, deduped vs the 640-superset by function-signature AND op-composition (0 leakage verified), n=80 depth-3. Primary: think coverage@16 with Wilson 95% CIs + a dense per-sample solve rate. Deployable: no-think coverage@16 + greedy@1 at the top dose. Guardrail: depth-2 think coverage.

Pre-registered verdicts

  • P1 (monotone): HELD — think coverage@16 strictly increases 0 → 0.087 → 0.212 → 0.375.
  • P2 (decision): DATA-LIMITED — rise 40→640 = +0.29 (≫ +0.10) and top-dose lower CI (0.28) > low-dose upper CI (0.17), i.e. a clean non-overlapping increase. NOT a cap through N=640.
  • P3 (deployable): HELD — at N=640 the depth-3 install reaches deployable no-think coverage 0.338 and single-shot greedy@1 0.10 (both ≈0 at C22's N=130); the deployable read scales with data too.
  • P4 (memorization control): HELD — 0 eval solutions (function-sig or op-composition) in training; the rise is generalization to novel depth-3 rules.

Interpretation

  • The deep wall is not a hard representational bottleneck at the depths tested — it is a data bottleneck. C19 showed the base's depth-3 inverse is barely represented and C20 showed it is not steerable; C22 showed one small banking round installs only a thread. But this dose-response shows that thread thickens smoothly with more tool-found training data — through N=640 the model keeps absorbing more novel-depth-3 compositional capability, and it converts into deployable single-shot. The apparent "cap" of C22 was a small-data artifact.
  • The complete, validated recipe (C13→C23), now quantitative: to extend the frontier a depth — an explorer the base lacks (tool-search) reaches the rung and produces verified solutions; banking those installs the capability; and the amount installed scales with the number of explorer-found solutions, into deployable single-shot. Self-training is the installer, external search is the explorer, and data is the throttle.
  • Mission read: this is "extend capability by a lot without a larger model" demonstrated and scaling — the only external ingredient is an interpreter-backed search (a tool), and everything installed is the fixed 4B's own verified compositions.

Honesty notes / limits (from the review)

  • Single training seed (nested doses): the Wilson CIs capture eval noise only, not training/seed variance; a lone helpful/harmful example or data-order roll is not averaged out. ≥3 seeds would put error bars on the dose effect itself (deferred).
  • Fixed epochs: "more depth-3 data" is physically the same event as "more depth-3 gradient exposure"; this study does not separate data-diversity from compute via a fixed-step + upsampled-40 control (deferred). The claim is the applied "more distinct tool-found pairs → more installed capability," not a pure information bound.
  • Search-easy bias: brute search harvests the shorter/cheaper depth-3 programs first; the curve could cap on harder compositions past N=640 (untested). No plateau is observed through 640.

Next Experiments

  • Push doses past 640 (e.g. 1280/2560) to find where (if) the curve saturates, and characterize the program-length/search-difficulty of solved-vs-unsolved held-out tasks.
  • Fixed-step + upsampled-40 control to split data-diversity from gradient exposure.
  • Iterate the rung: with depth-3 now deployable at N=640, does tool-search harvest depth-4 more cheaply?

Artifact Manifest

See reports/artifact_manifest.yaml. Key: scripts/tool_harvest.py, scripts/train_lora.py, scripts/eval_ladder.py (frozen paired set + function-sig & op-composition dedup + leakage report), scripts/analyze.py (Wilson CIs + CI-overlap decision), data/train_{40,160,640}.jsonl, data/eval_frozen.jsonl, runs/eval_*.json, runs/verdict.json, analysis/dose_response.png, reports/design_review.md. Adapters (~180MB each) omitted from git.

Experiment log 1

Show the running log (1 entry)

Scaffold

Created as a new experiment scaffold.

Figures 1

dose response
dose response · analysis/

Data files 8

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

Smoke test

python scripts/tool_harvest.py --smoke

Full run

python scripts/tool_harvest.py --n-depth3 640 && bash runs/launch.sh && python scripts/analyze.py

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗