Qwen3.5-4B: When Does the Model's Structure Beat Brute Search? (depth-4)
The one idea you need
Picture a lock whose combination is a chain of dials, each with sixteen positions. The model tries to recall the right sequence from memory. Brute force instead tries every combination on the real lock and keeps only the ones that actually open it.
The question
When checking every possible answer gets too expensive, does the model's built-in sense of program structure finally beat blindly trying everything?
What we found
No. Adding one extra dial — a sixteen-times-larger space of combinations — did not flip things. Exhaustive search stayed near-perfect at about 97 percent, while the model's knack for guessing the right combination from memory collapsed from 51 percent to 10 percent. The gap did not shrink toward a crossover; it widened, from 46 to 87 points. Structure fades faster than search slows.
Why it matters
If you can actually run candidate programs and check their output, spend compute on enumerate-and-test search plus a cheap output filter, not on training a small model to internalize program structure — that internal structure erodes as tasks get deeper.
On this page
Results at a glance 2
How to read
The bottom axis is composition depth, three then four chained steps. The vertical axis is percent correct; higher is better. The upper line is brute-force search's solved rate; the lower line is how often the model guessed the right program shape from memory.
Takeaway → The two lines spread apart instead of meeting: the model drops from 51 to 10 percent while search holds near 97, so no crossover ever appears.
Data table
| composition depth | model structure-coverage (banked, forward pass) | brute-force structure-search DEPLOY (tool) |
|---|---|---|
| 3 | 51.2% | 97.5% |
| 4 | 10% | 96.7% |
Numbers from
Technical framing
When does the model's structure beat brute-force? Never: the scissors WIDENS with depth (0.51->0.10 model vs 0.98->0.97 brute) — C34 left open: brute-force wins at depth-3 because the 4096-skeleton space is enumerable; the model's structure-pruning would only win when the space is too large to brute-force. Tested at depth-4 (space 16^4=65536, 16x larger) with a depth-4-banked model + held-out depth-4 tasks (n=60). RESULT: the hypothesized regime does NOT appear -- the scissors widens, never crosses. (1) Banking's structure-installation COLLAPSES with depth: banked_d4 structure-coverage at depth-4 = 0.10 vs 0.51 for banked_1280 at depth-3. (2) Brute-force structure-search + value-fill + execution-consensus STAYS near-perfect: deploy 0.967 at depth-4 (vs 0.975 at depth-3), because the 8-visible filter is depth-invariant (~6 skeletons survive at depth-4 vs ~2 at depth-3). (3) The model-vs-brute deploy gap GROWS from -0.46 to -0.87. The model never wins because banking's structure degrades faster than brute's exponential cost (16^depth) grows intractable, AND the model's structure can't be cheaply injected into a search (behavioral inference costs a full enumeration; op-seq generation broken, C32). With the interpreter, the TOOL is the deployable lever; the weights are never the better structure-proposer.
How to read
The bottom axis is composition depth: three, four, five steps. The vertical axis counts program shapes to enumerate, sixteen raised to the depth — about four thousand, then sixty-five thousand, then a million. Fewer shapes favors search.
Takeaway → The count multiplies sixteen-fold per step, only turning punishing near depth five — yet the model's structure has already collapsed at depth four, before search runs out of room.
Data table
| composition depth | brute-force structure space |
|---|---|
| 3 | 4096 |
| 4 | 65.54k |
| 5 | 1.049M |
Numbers from
Technical framing
Brute-force's cost is exponential (16^depth) -- its ceiling is ~depth-5, but banking's structure collapses BEFORE brute becomes intractable — Brute-force structure-search enumerates 16^depth skeletons: 4096 (d3) -> 65536 (d4) -> ~1M (d5). It becomes intractable around depth-5. BUT banking's forward-pass structure has already collapsed to 0.10 at depth-4 (would be ~0 at depth-5). So there is a depth-5+ regime where NEITHER works cheaply -- brute intractable, model collapsed -- but the model never WINS. The model degrades with depth faster than brute's cost forces you to abandon it, so the crossover where the model's structure-pruning would help never materializes on this substrate.
In the author’s words from the Overview · “Results”
The scissors WIDENS, never crosses: banked_d4 structure-cov 0.10 (vs 0.51 at depth-3) while brute-full deploys 0.967 (vs 0.975). Banking's structure collapses with depth faster than brute's exponential cost grows intractable. See reports/report.md, analysis/structure_search_scaling.png.
Overview
Research Program
- Program:
structured_execution_and_compilers/evidence_conditioned_selection - Question (C34 follow-up): C34 found brute-force structure-search dominates the model at deploy at depth-3 (enumerable space). Does the model's structure-pruning win at depth-4 (16^4=65536, too large to enumerate cheaply)?
Setup
- banked_d4 (depth-4-banked) model + held-out depth-4 tasks (n=60, min-depth-verified).
- Gate: banked_d4 structure-coverage at depth-4 (bank_structure.py). Brute-full structure-search deploy (brute_only.py: enumerate 65536, value-fill, execution-consensus).
Run
python scripts/bank_structure.py --tag banked_d4 --adapter <banked_d4> --eval-file data/eval_frozen_d4.jsonl (gate); python scripts/brute_only.py (brute-full deploy); python scripts/analyze.py.
Results
The scissors WIDENS, never crosses: banked_d4 structure-cov 0.10 (vs 0.51 at depth-3) while brute-full deploys 0.967 (vs 0.975). Banking's structure collapses with depth faster than brute's exponential cost grows intractable. See reports/report.md, analysis/structure_search_scaling.png.
Interpretation
The model never beats brute-force structure-search on this substrate. Closes the C32->C33->C34->C35 arc: the wall is structure; banking installs it into the forward pass (collapsing with depth); with an interpreter, brute structure-search dominates the weights outright.
Knowledgebase Update
- Claim ledger: C35
Artifacts
scripts/bank_structure.py(gate),scripts/bank_fill.py(model-guided, superseded by brute),scripts/brute_only.py(brute-full deploy),scripts/analyze.pydata/eval_frozen_d4.jsonl,runs/{bank_banked_d4,brute_d4,verdict}.json,analysis/structure_search_scaling.png,reports/report.md- Uses external banked_d4 adapter (scratchpad).
Report
Rendered from reports/report.md
Motivation
C34 established that at depth-3, brute-force structure-search + value-fill + execution-select deploys at 0.975, dominating the banked model (0.46), and left open: "brute-force wins because the depth-3 structure space (4096) is enumerable; the model's structure-pruning would only become a deployable lever when the space is too large to brute-force." This tests that hypothesis at depth-4 (structure space 16^4 = 65536, 16× larger), with a depth-4-banked model (banked_d4) and held-out depth-4 tasks.
Result (held-out, min-depth-verified)
| depth | structure space | model structure-cov (banked, forward pass) | brute-full deploy (tool) | model − brute gap |
|---|---|---|---|---|
| 3 | 4,096 | 0.512 (banked_1280) | 0.975 | −0.46 |
| 4 | 65,536 | 0.100 (banked_d4) | 0.967 | −0.87 |
The hypothesized regime does not appear in the measured depth-3/4 cells:
- The tested depth-4 banked model has much lower structure coverage: banked_d4's structure-coverage at depth-4 is only 0.10, vs 0.51 for a different model, banked_1280, at depth-3. The comparison is not dose- or curriculum-matched, so it is a two-model trend rather than a controlled causal depth effect.
- Brute-force stays near-perfect: deploy 0.967 at depth-4 (vs 0.975 at depth-3). The 8-visible filter is depth-invariant — still only ~6 skeletons survive it at depth-4 (vs ~2 at depth-3), so structure-search + execution-consensus near-solves while the space is enumerable.
- So the model-vs-brute deploy gap grows from −0.46 (depth-3) to −0.87 (depth-4).
Why there is no measured crossover through depth 4
- The tested banked coverage drops while brute remains effective. Brute's cost is exponential (16^depth: 4096 → 65536 → 1M), so depth 5 is the first projected stress regime. No depth-5 guided-search or banked-model result was run here; whether a model-guided method wins there is explicitly open.
- The model's structure can't be cheaply injected into a search anyway. Recovering it from behavior costs a full 16^depth enumeration (the model-guided run took 45+ min at depth-4 before it was killed and the brute number taken directly), and direct op-sequence generation is broken (0.00 even at depth-1, C32).
Implication
Through depth 4 on this list DSL, with an interpreter available, brute-force structure-search + value-fill + execution-select dominates the tested banked models. The 0.51→0.10 banked comparison is suggestive but crosses two differently trained models, so it does not by itself establish a causal depth-collapse law. Banking's demonstrated value here is forward-pass-only; the unmeasured depth-5 regime remains the relevant test for cheap model-guided pruning.
Honest scope
- List DSL, 16 op-types; depth-3 (from C33/C34) vs depth-4 (n=60). banked_d4 is one depth-4-banked model with an undocumented/dose-unmatched recipe relative to banked_1280.
- The "model can't be cheaply injected into search" point is about the behavioral-inference and op-seq-generation routes; a step-wise decompose-search guided by the model's op-ranking (C25) is a different (linear-cost) route — but C25 found the model's step-1 ranking is ~chance (planner wall), so it is bottlenecked at seeding; and here banked_d4's structure-cov (0.10) shows the forward-pass structure has collapsed anyway.
Artifact Manifest
See reports/artifact_manifest.yaml. Uses external banked_d4 adapter (scratchpad).
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Reproduce
Smoke test
python scripts/brute_only.py # brute-full deploy, no modelFull run
bank_structure --tag banked_d4 --adapter <banked_d4> --eval-file data/eval_frozen_d4.jsonl (gate); brute_only.py; analyzeRun steps are documented inside the experiment folder (README and scripts).