Qwen3.5-4B: Do the Structure Findings Generalize? (string/register)
The one idea you need
Each task needs a tiny three-step program, like a short recipe: the order of actions (chop, then fry, then salt) plus the exact amounts. Getting the amounts right turns out to be easy — the model's whole struggle is choosing the right order of actions.
The question
When a small model fails to write a short multi-step program, is it choosing the wrong sequence of actions, or just getting the numbers inside them wrong?
What we found
The wrong sequence. Across three completely different kinds of programming tasks, a 4-billion-parameter model almost never solved one on its own — 0 to 2 percent — and its misses were wrong-order, not right-order-wrong-number. Hand it the correct sequence of actions and it fills the numbers perfectly every time (100 percent). A blind tool that tries candidate sequences and checks each solved nearly everything, 98 to 100 percent — beating the model outright.
Why it matters
Don't burn compute sampling a small model more, or fine-tune it hoping it plans multi-step logic — its weights can't propose the right structure. If you can check answers with an interpreter, enumerate candidate sequences and let cheap value-search plus run-and-check win.
On this page
Results at a glance 2
How to read
Four measures left to right: the model's single best answer, whether its program's actions match the right sequence, solve rate given the correct sequence, and a blind search tool. Three colors mark the task types (character edits, register machine, list operations). Taller means higher solve rate, which is better.
Takeaway → The first two bars sit near the floor for all three task types while the last two hit the ceiling — the model cannot plan, but supplying or searching the sequence solves it.
Data table
| condition | string (space 2,197) | register (space 1,728) | list (space 4,096) |
|---|---|---|---|
| base greedy@1 | 0% | 2% | 0% |
| base STRUCTURE-cov | 1% | 4% | 0% |
| oracle-skeletonfill | 100% | 100% | 100% |
| brute-force search DEPLOY | 100% | 100% | 98% |
Numbers from
Technical framing
Do the structure findings generalize? YES -- same pattern on string/register/list (base ~0, oracle=1.0, brute-search near-solves) — C16 cross-substrate-tested the EARLY ladder (C13-15); the recent sharp findings (C32 wall-is-structure, C34 brute-dominates) were never generalized. Family-generic replication on STRING (char edits), REGISTER (3-register machine), LIST (anchor), depth-3, min-depth-verified, n=100. RESULT: C32+C34 are MODEL-LEVEL LAWS -- identical pattern on all three. (1) The wall is STRUCTURE: base structure-coverage = concrete-coverage (value tax +0.01 string, +0.00 register/list) -- failures are wrong-skeleton, not right-skeleton-wrong-param. (2) Values trivially searchable given structure: oracle-skeletonfill = 1.000 everywhere. (3) Structure matters (random-skeletonfill low, see 2nd chart). (4) Brute-force structure-search + value-fill + execution-consensus DOMINATES the model: brute-deploy ~1.0 (string 1.00, register 1.00, list 0.98) vs base ~0. So the fixed 4B is a VALUE-computer, not a deep-STRUCTURE-proposer, across genuinely different substrates. The deployable lever is the TOOL, not the weights, everywhere.
How to read
Horizontal axis counts how many random sequences are tried (8, 50, 200), each filled with searched values and run to check; vertical axis is solve rate, higher is better. One line per task type: character edits, register machine, list operations. Compare against the near-100% from the correct sequence.
Takeaway → All three lines climb only modestly, topping out between 9 and 32 percent, far below the near-perfect result from the correct sequence — the right structure is genuinely required, not incidental.
Data table
| R = random op-type skeletons tried (each value-filled + execute-checked) | string random-skeletonfill | register random-skeletonfill | list random-skeletonfill |
|---|---|---|---|
| 8 | 0% | 2% | 0% |
| 50 | 5% | 11% | 5% |
| 200 | 17% | 32% | 9% |
Numbers from
Technical framing
Structure genuinely matters on all 3: random op-type skeletons rarely solve (far below brute-deploy ~1.0) — Value-fungibility control across substrates: do RANDOM op-type skeletons + value-search crack the tasks (which would make the wall trivially value-fungible)? On all three, random structure stays far below the brute-full deploy (~1.0 with the TRUE structure enumerated) -- so the correct STRUCTURE is genuinely required everywhere. Register is somewhat more fungible (R200=0.32) than string (0.17) and list (0.09) because its structure space is smaller (1728) and its arithmetic ops have more behaviorally-equivalent orderings. But the pattern -- structure is the bottleneck, not values -- holds on all three.
In the author’s words from the Overview · “Results”
C32 + C34 are MODEL-LEVEL LAWS: identical pattern on all three (base ~0, structure-cov = concrete-cov, oracle-skelfill 1.0, random low, brute-deploy ~1.0). See reports/report.md, analysis/crosssubstrate_structure.png.
Overview
Research Program
- Program:
structured_execution_and_compilers/interpretability_and_diagnostics - Question: C16 tested the EARLY ladder (C13-15) cross-family. Do the RECENT findings -- C32 (wall-is-structure), C34 (brute-search dominates) -- hold on STRING and REGISTER, or are they list-DSL artifacts?
Setup
- Family-generic replication on STRING (char edits, 13 prims), REGISTER (3-register machine, 12 prims), LIST (anchor, 16 prims), depth-3, min-depth-verified, n=100. Base model + search (no banking).
- Metrics: base greedy@1/cov@8 + format-immune structure-coverage; oracle-skeletonfill; random-skeletonfill@R; brute-full structure-search + value-fill + execution-consensus deploy.
Run
python scripts/cross_substrate.py --family {string,register,list} --n 100 --k 8 --randR 8 50 200 then python scripts/analyze.py.
Results
C32 + C34 are MODEL-LEVEL LAWS: identical pattern on all three (base ~0, structure-cov = concrete-cov, oracle-skelfill 1.0, random low, brute-deploy ~1.0). See reports/report.md, analysis/crosssubstrate_structure.png.
Interpretation
The fixed 4B is a value-computer, not a deep-structure-proposer, across substrates; the wall is structure-proposal everywhere; with an interpreter, brute-force structure-search dominates the weights outright everywhere. Establishes the compositional arc as model-level, not list-DSL-specific.
Knowledgebase Update
- Claim ledger: C36
Artifacts
scripts/cross_substrate.py(family-generic C32+C34),scripts/analyze.pyruns/cs_{string,register,list}.json,runs/verdict.json,analysis/crosssubstrate_structure.png,reports/report.md
Report
Rendered from reports/report.md
Motivation
C16 cross-substrate-tested the early compositional ladder (C13–15) on STRING and REGISTER. But the recent, sharper findings were never generalized: C32 (the wall is STRUCTURE not values), C34 (brute-force structure-search DOMINATES the model at deploy). This tests whether they are model-level laws or list-DSL artifacts.
Method
Family-generic replication on STRING (char edits, 13 primitives), REGISTER (3-register int machine, 12 primitives), and LIST (anchor, 16 primitives), at depth-3 (min-depth-verified, n=100 each): base model greedy@1/cov@8 + format-immune STRUCTURE-coverage (does the model program's behavior match the true op-type skeleton with any params?); oracle-skeletonfill (true structure + value-search); random-skeletonfill@R (value-fungibility control); brute-full structure-search + value-fill + execution-consensus deploy.
Result (depth-3, n=100)
| substrate | space | base greedy@1 | base cov@8 | STRUCTURE-cov | value tax | oracle-skelfill | random R200 | brute-deploy |
|---|---|---|---|---|---|---|---|---|
| string | 2,197 | 0.000 | 0.000 | 0.010 | +0.010 | 1.000 | 0.170 | 1.000 |
| register | 1,728 | 0.020 | 0.040 | 0.040 | +0.000 | 1.000 | 0.320 | 1.000 |
| list | 4,096 | 0.000 | 0.000 | 0.000 | +0.000 | 1.000 | 0.090 | 0.980 |
C32 + C34 are model-level laws — the pattern is essentially identical on all three substrates:
- The wall is STRUCTURE. Base structure-coverage = concrete-coverage (value tax ≈ 0 everywhere): the model's depth-3 failures are wrong-skeleton, not right-skeleton-wrong-param. No hidden pool of right-structure-wrong-value solutions on any substrate.
- Values are trivially searchable given structure. oracle-skeletonfill = 1.000 on all three.
- Structure genuinely matters. random-skeletonfill stays low (R200: 0.17 string, 0.09 list, 0.32 register). Register is somewhat more value-fungible (smaller space, arithmetic ops alias more), but still far from oracle.
- Brute-force structure-search DOMINATES the model. brute-deploy ≈ 1.0 (1.00 string, 1.00 register, 0.98 list) while the base model is ~0.
Implication
The fixed Qwen3.5-4B is a value-computer, not a deep-structure-proposer, across genuinely different substrates (string edits, register machines, list DSLs). The compositional wall is structure-proposal everywhere; and with an interpreter, brute-force structure-search dominates the weights outright everywhere. Combined with C16 (early ladder cross-family) and C33/C35 (banking installs structure but collapses with depth), the entire compositional arc is established as model-level, not an artifact of one hand-built DSL. The deployable "beat sample-more" lever is the TOOL (structure-search + value-fill + execution-select), not the weights, on every substrate tested.
Honest scope
- Depth-3, n=100 per substrate, base model (banking not re-run per substrate — C33's banking-installs-structure and C35's depth-collapse were tested on list only). C31 (op-type-latent vs param-surface via activation probing) not re-run cross-substrate (would need per-substrate probes).
- Register is more value-fungible (random 0.32 at R200) — a substrate property, not a break in the pattern.
Artifact Manifest
See reports/artifact_manifest.yaml.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Reproduce
Smoke test
python scripts/cross_substrate.py --family string --n 12 --k 4 --randR 4 30Full run
for f in string register list: cross_substrate.py --family $f --n 100 --k 8 --randR 8 50 200; analyze.pyRun steps are documented inside the experiment folder (README and scripts).