Qwen3.5-4B Decompose-and-Compose Frontier
The one idea you need
Picture a locked maze where each door has 23 possible keys. Instead of guessing all three doors from the entrance in one leap, you try one key, walk through, and let the room reveal itself before choosing again. The tool is that room-revealing step.
The question
If a small model can't write a correct three-step program in one shot, can it get there by taking one step at a time with a tool running each?
What we found
Yes — but not because the model got smarter. Taking three steps one at a time, with a tool that runs each and shows the result, solves about 2 in 5 versus 1 in 8 in one shot. The catch: blindly trying all 23 operations does just as well. The step-by-step structure and the tool crack the wall — not the model's judgment about which move is best.
Why it matters
When a small model stalls on multi-step tasks, wrap it in a step-running tool and search the moves — but always benchmark against blind enumeration, which can match clever ranking. Then fine-tune on the solutions found to bank the gain into the weights.
On this page
Results at a glance 4
How to read
Bars grouped by task depth — two-step and three-step. Each group has three bars: one-shot sampling, model-guided step-by-step search, and blind brute-force search. Height is the share of unseen tasks solved; taller is better.
Takeaway → At three steps both search bars stand about three times taller than one-shot, but guided and blind sit nearly level — the search structure, not the model's guidance, cracks the wall.
Data table
| task depth | monolithic sampling (k=8) | model-guided search | brute-force search |
|---|---|---|---|
| depth 2 | 32.5% | 57.5% | 87.5% |
| depth 3 | 12.5% | 40% | 42.5% |
Numbers from runs/search_summary.json
Technical framing
Depth-3 solve rate: one-shot sampling vs decompose-and-compose search — Decompose-and-compose search cracks depth-3 at 3.2-3.4x the monolithic rate, but brute force matches guided search: the crack is structure + interpreter.
How to read
Horizontal axis is the tool-call budget (rightward means more); vertical is the share of unseen tasks solved (higher is better). Four lines: model-guided and blind brute-force, each at two-step and three-step depth. Rising higher and sooner is better.
Takeaway → Guided lines jump early then flatten; blind lines start lower but keep climbing to match or overtake — the model's guidance buys a cheaper search, not more solutions.
Data table
| interpreter-call budget | guided, depth 2 | brute, depth 2 | guided, depth 3 | brute, depth 3 |
|---|---|---|---|---|
| 50 | 25% | 10% | 7.5% | 2.5% |
| 100 | 27.5% | 17.5% | 15% | 7.5% |
| 200 | 30% | 20% | 20% | 17.5% |
| 400 | 57.5% | 30% | 32.5% | 22.5% |
| 800 | 57.5% | 52.5% | 35% | 30% |
| 1500 | 57.5% | 75% | 37.5% | 32.5% |
| 3000 | 57.5% | 82.5% | 40% | 40% |
| 4000 | 57.5% | 87.5% | 40% | 42.5% |
Numbers from runs/search_summary.json
Technical framing
Solve rate vs interpreter-call budget: guidance buys efficiency, not coverage — Model guidance wins at low budgets but plateaus (the planner-wall); brute enumeration keeps climbing and matches or beats it at high budget.
How to read
Four one-shot measures on unseen tasks along the bottom; taller is better. Each has three bars: the frozen model, the self-trained model, and a repeat training run. Height is solve rate.
Takeaway → Every trained bar clears the frozen one — best-of-five roughly doubles and three-step solving quadruples — and the repeat run confirms it, so search-found solutions transfer into the weights.
Data table
| monolithic eval metric | frozen | self-trained | self-trained (replicate) |
|---|---|---|---|
| think greedy@1 | 7.5% | 12.5% | 12.5% |
| think pass@5 | 12.5% | 23.7% | 26.3% |
| depth-3 pass@5 | 2.5% | 10% | 17.5% |
| no-think greedy@1 | 0% | 6.2% | 10% |
Numbers from runs/eval_frozen.json, runs/eval_trained.json, runs/eval_trained2.json
Technical framing
Banking search-found solutions lifts the model's own one-shot sampling (held-out, n=80) — QLoRA-SFT on 327 search-harvested solutions (no teacher) lifts pass@5 +0.11 and depth-3 pass@5 4x — the frontier extension M4 could not achieve.
How to read
Two bars split the nominal three-step tasks by real difficulty: 16 that secretly collapse to two steps or fewer, and 24 that are genuinely three-step. Bar height is the share the step-by-step search solved; taller means more solved.
Takeaway → Search solved all 16 secretly-shallow tasks but only 4 of 24 truly three-step ones — a real frontier gain, but far smaller than the headline 40% suggests.
Data table
| nominal depth-3 task slice | decompose search |
|---|---|
| collapsed (behavioral depth <=2), n=16 | 100% |
| true depth-3, n=24 | 16.7% |
Numbers from report table
Technical framing
Retro-audit: how much of the depth-3 crack is truly depth-3? — 40% of nominal depth-3 tasks collapse to depth <=2; decompose solves all of those but only 17% of true depth-3 (monolithic: 0/24).
In the author’s words from the Overview · “Results”
Search cracks the frontier: hidden-generalizing depth-3 solve rate — monolithic 0.125 → decompose 0.40+ (3.4×). But held to the brute-force bar, the model's guidance buys efficiency, not coverage: guided solves with ~2.5× fewer interpreter calls and wins at low budget, but plateaus (planner-wall) while brute-force enumeration matches/beats it. The crack is the composition-structure + interpreter. Banking extends the frontier into the weights: QLoRA-SFT on 327 search-found solutions (no teacher) lifts monolithic held-out pass@5 0.125 → 0.237 (+0.112, ~2.6 SE), depth-3 pass@5 4× (0.025→0.10), greedy@1 +0.05 — the bound M4 could not break. Replicated (fresh harvest seed: greedy identical, pass@5 0.263, depth-3 pass@5 0.175). … Read the full result →
Overview
Research Program
- Program:
structured_execution_and_compilers. Mission: extend the fixed Qwen3.5-4B frontier — no teacher, no scaling, no contaminated data ([[unearth-latent-capability-mission]]). - Attacks the open problem C11/M4 posed: self-training saturates at the model's sampling frontier (depth-3 uncrackable). Can the model give itself serial depth it lacks in one forward pass by composing its own reliable primitives through the interpreter?
Question
The depth-3 wall may be a serial-composition wall: the frozen 4B reliably does depth-1/2 sub-steps but can't compose 3 in one shot. A decompose-and-compose search turns a depth-3 task into sequential depth-1 decisions — at each node the 4B ranks the next primitive (letter-logit read of current-state → target), the interpreter executes it to materialize the intermediate state, recurse/backtrack over the 23 primitives. Two things to establish, both with sharp controls:
- Does model guidance beat matched-budget BRUTE-FORCE enumeration? With 23 primitives, blind search already cracks depth-3, so guidance only elicits latent capability if it solves more per interpreter-call. We report the full solve-rate-vs-call-budget curve, not just accuracy.
- Does it crack depth-3 that MONOLITHIC sampling can't (the frontier win) — and if so, does banking the found solutions (QLoRA-SFT) lift monolithic single-shot depth-3 (the bound M4 couldn't break)?
The planner-wall risk: the model never sees intermediate states, so per-step proposal from I/O may be as weak as monolithic writing — the wall may just move to the planner. The brute-force comparison IS that test.
Setup
- Fresh procedurally-generated tasks (10 visible + 8 hidden I/O examples; more visible than M1-M4 to constrain found pipelines to generalize). Held-out seed 777. Graded on hidden examples.
src/decompose_lib.py: letter-logit primitive ranking + beam search + brute baseline. Reuses the substrate generator (gen_tasks.py), sandbox (code_env.py), runtime (gen_lib.py), trainer.
Run
../../.venv/bin/python scripts/run_search.py --per-depth 40 --depths 2 3 --beam 50 --top-p 8 # search
../../.venv/bin/python analysis/search_curve.py # figure
# if depth-3 solutions found: bank them
../../.venv/bin/python scripts/train_lora.py --train data/found_solutions.jsonl --out runs/frontier_adapterResults
Full write-up in reports/report.md.
- Search cracks the frontier: hidden-generalizing depth-3 solve rate — monolithic 0.125 → decompose 0.40+ (3.4×). But held to the brute-force bar, the model's guidance buys efficiency, not coverage: guided solves with ~2.5× fewer interpreter calls and wins at low budget, but plateaus (planner-wall) while brute-force enumeration matches/beats it. The crack is the composition-structure + interpreter.
- Banking extends the frontier into the weights: QLoRA-SFT on 327 search-found solutions (no teacher) lifts monolithic held-out pass@5 0.125 → 0.237 (+0.112, ~2.6 SE), depth-3 pass@5 4× (0.025→0.10), greedy@1 +0.05 — the bound M4 could not break. Replicated (fresh harvest seed: greedy identical, pass@5 0.263, depth-3 pass@5 0.175).
- Retro-audit (behavioral min-depth): 40% of nominal depth-3 tasks are behaviorally depth ≤2 (shallower-equivalent compositions). Re-sliced: decompose solved 16/16 collapsed but only 4/24 (17%) true depth-3; monolithic true depth-3 = 0/24 (and 0 across the whole corpus). The frontier extension is real but far more modest than nominal numbers suggest; banking eval is ~30% collapsed at d3 (caveat).
Answer to C11's open problem: the frontier extends without a teacher via tool-augmented search (composition + interpreter) → harvest frontier-exceeding solutions → bank them. Modest but real — and the min-depth audit shows every prior "depth-3" number in the arc was inflated by shallower-equivalent tasks.
Report
Rendered from reports/report.md
Summary
C11/M4 showed self-training saturates at the fixed 4B's sampling frontier (depth-3 uncrackable). This experiment attacks that frontier without a teacher: a decompose-and-compose search gives the model the serial depth it lacks in one forward pass — at each node the 4B ranks the next primitive (letter-logit read of current-state → target), the interpreter executes it to materialize the intermediate state, recurse/backtrack over the 23 primitives. Two findings. (1) It cracks the frontier monolithic sampling can't — hidden-generalizing depth-3 solve rate: monolithic 0.125 → decompose 0.40–0.43 (3.4×). (2) But against the brute-force bar, the model's guidance buys efficiency, not coverage: guided solves with far fewer interpreter calls (depth-3: 350 vs 895) and wins at low budgets, but plateaus (the planner-wall — where its ranking misses it never recovers) while brute-force enumeration keeps climbing to match/beat it. So the frontier crack comes from the composition structure + interpreter, not the model's planning. (3) Banking EXTENDS the frontier into the weights — QLoRA-SFT on 327 search-found solutions (no teacher) lifts the model's OWN monolithic sampling on held-out: pass@5 0.125 → 0.237 (+0.112, ~2.6 SE), depth-3 pass@5 0.025 → 0.100 (4×), greedy@1 0.075 → 0.125 (+0.05, suggestive), and a no-think one-shot path emerges (0 → 0.062). Search+interpreter harvested solutions from outside the sampling distribution and banking pulled them into it — the exact bound M4 (confined to the sampling distribution) could not break. Modest in absolute terms but a genuine, teacher-free frontier extension.
Research Program Fit
structured_execution_and_compilers. Directly attacks the open problem C11 posed. On-mission: the interpreter is a calculator and brute-force search is a deterministic algorithm — no teacher, no other model, no contaminated data. The brute-force bar is the mission's honesty criterion: with only 23 primitives, blind enumeration already cracks depth-3, so model guidance only elicits latent capability if it beats matched-budget brute force.
Method
- Fresh procedurally-generated tasks (10 visible + 8 hidden I/O examples — more visible than M1–M4 to force found pipelines to generalize). Held-out seed 777 (search), 404 (bank eval), 888 (harvest). Graded on hidden.
- Search (
src/decompose_lib.py): beam search over primitive pipelines. Guided = the 4B ranks the next primitive via a 23-way letter-logit read (one forward/node); brute = all primitives. Both share an interpreter-call budget; each application = 1 call. Solve-rate-vs-call-budget curve from one run/task. - Monolithic baseline: M2-style — sample k=8 whole
transforms (thinking), select by visible pass, grade hidden. - Bank: harvest hidden-generalizing solutions via brute search on a large fresh pool → QLoRA-SFT (no teacher) → eval monolithic depth-2/3 frozen vs trained.
Results
Hidden-generalizing solve rate (n=40/depth):
| depth | monolithic (k=8) | guided (mean calls) | brute (mean calls) |
|---|---|---|---|
| 2 | 0.325 | 0.575 (158) | 0.875 (825) |
| 3 | 0.125 | 0.400 (350) | 0.425 (895) |
Solve-rate vs interpreter-call budget (depth 3): guided 0.08→0.20→0.33→0.40 at {50,200,400,3000}; brute 0.03→0.18→0.23→0.40→0.43 at {50,200,400,3000,4000}. Figure: analysis/search_curve.png.
- Frontier cracked: both decompose methods solve depth-3 at ~3.4× the monolithic rate (0.40+ vs 0.125) — externalized serial computation reaches solutions the model can't produce in one shot.
- Guidance = efficiency, not coverage: guided reaches its plateau with ~2.5× fewer calls than brute and dominates the low-budget regime, but caps below brute at high budget (depth-2: guided 0.575 vs brute 0.875) — the planner-wall. At depth-3 they converge (0.40 vs 0.43).
- So by the strict brute-force bar, the model is not out-eliciting search on coverage; it makes the search cheaper. The frontier crack is the composition-structure-plus-interpreter, not model foresight.
Banking — frontier extension into the weights
Harvested 327 hidden-generalizing solutions (242 d2 + 85 d3) via brute search + interpreter (no teacher); QLoRA-SFT → monolithic frozen vs trained on held-out (n=80, seed 404):
| metric (monolithic held-out) | frozen | trained | Δ |
|---|---|---|---|
| think greedy@1 | 0.075 | 0.125 | +0.050 (~1.5 SE) |
| think pass@5 | 0.125 | 0.237 | +0.112 (~2.6 SE) |
| depth-3 pass@5 | 0.025 | 0.100 | 4× |
| no-think greedy@1 | 0.000 | 0.062 | +0.062 |
Banking search-found solutions the model could NOT monolithically sample raised its own sampling coverage (pass@5 +0.112 significant; depth-3 4×) and its single-shot rate (greedy +0.05, suggestive), plus a no-think one-shot path from ~0. This is the frontier extension M4 could not achieve (M4 banked only sampleable solutions). Replicated with a fresh harvest seed (999): greedy@1 0.125 (identical), pass@5 0.263 (vs 0.237), depth-3 pass@5 0.175, no-think 0.100 — robust to the harvest data. Honest limits: absolutes remain low; n=80 held-out; and see the retro-audit below for a collapsed-task caveat.
Retro-audit: behavioral min-depth (added post-hoc; full treatment in the follow-up experiment)
The substrate generator excluded degenerate compositions but not shallower-equivalent ones (e.g. sort_asc∘reverse ≡ sort_desc). An exact behavioral min-depth audit (BFS over all primitive pipelines against all 18 examples) found 40% of this experiment's nominal depth-3 search tasks are behaviorally depth ≤2. Re-slicing the search results by true structure:
| nominal d3 slice | decompose solved | monolithic solved |
|---|---|---|
| collapsed (min-depth ≤2), n=16 | 16/16 | (mixed) |
| full depth-3, n=24 | 4/24 (17%) | 0/24 |
So the headline "decompose cracks depth-3 at 0.40" decomposes into: all of the behaviorally-shallow tasks plus 17% of true depth-3 — still a strict frontier extension over monolithic's 0/24, but far more modest than the nominal number. The banking eval set is similarly ~30% collapsed at depth-3, so the banked "depth-3" gains are over a mixed population (per-task attribution deferred to the follow-up experiment, which uses min-depth-verified tasks only). Corpus-wide corollary: across ALL prior data (M1/M2/C12), the frozen model has never once solved a true full-depth-3 task monolithically.
Controls
Brute-force enumeration at matched call-budget (the honesty bar). Hidden-generalization grading (10 visible constrain the pipeline; still, ~some visible-solves fail hidden — reported only hidden). Monolithic sampling (k=8 + visible-select) is the frontier baseline the decompose loop must beat.
Oracle Versus Deployable Evidence
Search is a deployable procedure (uses only visible I/O + the interpreter); solve rates are hidden-graded. The reference oracle (100% solvable) bounds all methods.
Interpretation
The frontier is crossable — but by giving the fixed model tools (composition + an interpreter), not by its own planning. The model's compositional foresight is real but weak (right primitive in top-8 ~2/3 of the time at the root) and it degrades over serial steps, so guided search plateaus where brute keeps climbing: the depth-3 wall relocates to the planner rather than disappearing. But banking answers the decisive question positively: the 4B can internalize solutions that only search+interpreter could find, extending its monolithic frontier into the weights (pass@5 +0.112, depth-3 4×) — which M4, confined to the sampling distribution, could not. So the answer to C11's open problem ("what extends the frontier without a teacher?") is tool-augmented search (composition + interpreter) to harvest frontier-exceeding solutions, then bank them — the interpreter is a calculator and search is a deterministic algorithm, no teacher involved. The extension is real but modest; the natural next step is to iterate it (harvest→bank→re-harvest with the improved model) as a frontier flywheel.
Next Experiments
- (banking, running) does SFT on search-found depth-3 lift monolithic depth-3? Iterate (harvest→bank→harvest with the improved model) as a frontier flywheel.
- Larger primitive vocabulary where brute-force explodes (so model guidance must carry more of the search).
Artifact Manifest
See artifact_manifest.yaml. Small tasks/solutions/summaries + figure in-repo; LoRA adapter regenerable.
Experiment log 5
Show the running log (5 entries)
Design
Attacks C11/M4's open problem: self-training saturates at the sampling frontier (depth-3 uncrackable). Give the fixed 4B serial depth via a decompose-and-compose search (rank next primitive -> execute -> materialize intermediate state -> recurse), measured against BOTH monolithic sampling (the frontier baseline) and matched-budget brute-force enumeration (the honesty bar). Then bank found solutions.
Build notes
- First proposal design (parse the model's generated next-op) FAILED -- the model reasons verbosely and gets truncated. Switched to a 23-way letter-logit read (like the C10 verifier): one forward/node, ranked primitives, params enumerated for the top-p. Fast + reliable.
- 10 visible examples (vs 6 in M1-M4) to constrain found pipelines to generalize to hidden.
Search results (see runs/search_summary.json, analysis/search_curve.png)
Hidden-generalizing solve rate (n=40/depth): d2 monolithic 0.325 / guided 0.575 (158 calls) / brute 0.875 (825 calls); d3 monolithic 0.125 / guided 0.400 (350 calls) / brute 0.425 (895 calls). Root ranking: true first-op in model top-8 for 2/3 of tasks.
FRONTIER CRACKED but by STRUCTURE not planning: decompose solves depth-3 at ~3.4x monolithic (0.40+ vs 0.125). Model guidance = call-EFFICIENCY (2.5x fewer calls, wins low-budget) but PLATEAUS (planner-wall); brute-force enumeration matches/beats it at high budget. By the brute-force bar the model does not out-elicit search on coverage. Harvested 327 hidden-gen solutions (242 d2 + 85 d3) for banking.
Banking results (see runs/eval_frozen.json vs eval_trained.json)
QLoRA-SFT on 327 search-found solutions (no teacher) -> monolithic held-out (n=80, seed 404), frozen->trained: think_greedy@1 0.075->0.125 (+0.05, ~1.5 SE); think_pass@5 0.125->0.237 (+0.112, ~2.6 SE); depth-3 pass@5 0.025->0.100 (4x); nothink_greedy 0.000->0.062. FRONTIER EXTENDED INTO THE WEIGHTS: banking solutions the model couldn't monolithically sample raised its own sampling coverage -- the bound M4 could not break. Modest absolutes; greedy gain suggestive, pass@5 gain significant. Replication (harvest seed 999) running via scripts/repl_bank.sh. Adapters (~170MB) gitignored + removed before commit.
Replication + retro-audit (post-close additions)
- Banking REPLICATED with fresh harvest seed 999 (327 solutions): greedy@1 0.125 (identical to seed-888), pass@5 0.263 (vs 0.237), depth-3 pass@5 0.175, no-think 0.100. Robust to harvest data.
- BEHAVIORAL MIN-DEPTH RETRO-AUDIT (exact BFS over all pipelines vs all 18 examples): the generator did not exclude shallower-equivalent compositions; 40% of nominal-d3 search tasks are behaviorally depth<=2. Re-sliced: decompose solved 16/16 collapsed but only 4/24 (17%) TRUE depth-3; monolithic true-d3 0/24 -- and 0 across the entire corpus (M1/M2/C12). Banking eval ~30% collapsed at d3 (mixed-population caveat). Full treatment + verified-depth substrate in the follow-up experiment (depth-wall anatomy).
Figures 1
Data files 4
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/eval_frozen.json366 Bruns/eval_trained.json393 Bruns/eval_trained2.json391 Bruns/search_summary.json1.3 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).