Qwen3.5-4B Cross-Family Laws
The one idea you need
Picture a cook. Hand them a written recipe and they nail the dish every time, however long. Ask them to predict the dish in their head and it depends on the ingredients. Ask them to reconstruct the recipe from taste alone, and they fail fast.
The question
If a small model can follow a step-by-step plan perfectly, can it also work out that plan on its own, or run it in its head?
What we found
No. Handed the exact steps, this fixed 4-billion-parameter model wrote correct code almost every time across three unrelated task types. But asked to infer the same procedure from example inputs and outputs alone, it fell from about 80% at one step toward zero by three stacked steps. Running a plan and discovering one are separate abilities.
Why it matters
Never spend tool calls or search budget making the model run a plan you already have — it never fails there. Spend that compute finding the plan. Offload state-tracking to code only when the state is bulky text, not compact numbers.
On this page
Results at a glance 3
How to read
The horizontal axis is how many steps are chained into the plan, one to four; the vertical axis is the share of correct code written from a supplied plan. Three lines: integer lists, text strings, a three-number machine. Higher is better.
Takeaway → All three lines sit at or near the top across every depth — handed the exact steps, the model writes correct code regardless of task type or length.
Data table
| composition depth | list | string | register |
|---|---|---|---|
| 1 | 100% | 100% | 100% |
| 2 | 92% | 100% | 100% |
| 3 | 100% | 100% | 100% |
| 4 | 100% | 100% | 100% |
Numbers from
Technical framing
Transcription is a model-level LAW: ≈1.0 in every family — Plan-given execution is near-perfect across three unrelated substrates — the fixed 4B is a universal reliable compiler.
How to read
Same three task types and one-to-four step axis; the vertical axis is how often the model tracked the running state correctly in its head, with no code. The three-number machine stays high, integer lists sag, text strings sit at the floor. Higher is better.
Takeaway → The number-machine line holds near three-quarters while the string line starts low and falls to zero — head simulation is set by how heavy the state is to hold, not by depth.
Data table
| composition depth | list | string | register |
|---|---|---|---|
| 1 | 76% | 24% | 92% |
| 2 | 100% | 12% | 80% |
| 3 | 84% | 0% | 72% |
| 4 | 56% | 0% | 72% |
Numbers from
Technical framing
Simulation is SUBSTRATE-DEPENDENT (C15's decay constant was list-specific) — Register (compact 3-int state) simulates robustly; list decays; a mutating char string is floored — simulation fidelity is set by the state representation, not depth.
How to read
The horizontal axis chains one to four steps; the vertical axis is how often the model reconstructed the hidden procedure from example inputs and outputs alone. Three lines for integer lists, text strings, the three-number machine. Higher is better.
Takeaway → Every line dives from high at one step toward zero by three — only the number machine keeps a small nonzero foothold, so inventing a novel procedure is a wall almost everywhere.
Data table
| composition depth | list | string | register |
|---|---|---|---|
| 1 | 80% | 84% | 56% |
| 2 | 12% | 16% | 36% |
| 3 | 0% | 0% | 16% |
| 4 | 0% | 0% | 8% |
Numbers from
Technical framing
The generation wall replicates: identification collapses in every family — Identifying a novel composition collapses toward chance with depth everywhere — 'the model executes what it cannot invent' is model-level.
In the author’s words from the Overview · “Results”
Verdict: SCOPED. Transcription is one invariant flat line at ~1.00 across all families (compiler LAW). Identification walls in all families, gap ≥ 0.84 at depth ≥ 3 (generation-wall LAW). Simulation is substrate-dependent: register robust (0.92→0.72), list decays (1.00→0.56), string floored (0.24→0.00) — C15's decay constant was list-specific. See reports/report.md and analysis/crossfamily_ladder.png.
Overview
Research Program
- Program:
structured_execution_and_compilers - Program question: are the C13–C15 ladder constants properties of the model or of the list substrate?
- Prior anchors: C13 (compiler/generation split), C15 (context composes; simulation length-fragile).
Question
C11–C15 all rest on ONE substrate (integer-list pipelines). Is transcription ≈ 1.0 / simulation-decays / identification-walls a model-level law or a list artifact? Rerun the ladder on genuinely different fresh families and see which constants replicate.
Hypothesis
Pre-registered (reports/prereg.md): transcription ≥ 0.85 all depths both new families; a family-invariant normalized simulation-decay constant; bare identification ≈ 0 at depth ≥ 3; ordering trans > sim > bare.
Setup
- Model: Qwen3.5-4B (only permitted model), thinking on, budget 512. Inference only, no training.
- Task source: three fresh families — list (16 int-list prims, anchor), string (13 char-edit prims), register (12 ops on a 3-int machine). Depth-graded, execution-verified, behavioral min-depth-BFS collapse-rejected. 100 verified tasks/family (25 × depth 1–4).
- Baseline / anchor: the list family measured through the identical harness.
- Controls: identical collapse-rejection across families; oracle 100% pass; family-aware
Step:parser (unit-tested; caught a spurious string-sim-0.00 artifact before scoring). - Primary metric: per-depth accuracy on transcription (plan→code, pass@1), simulation (final-state exact-match), bare identification (I/O→code, pass@4).
- Hidden-label boundary: identification graded by executing the model's
transformagainst hidden I/O.
Run
Smoke: python scripts/run_family.py --family string --smoke Full: for f in list string register; do python scripts/run_family.py --family $f --n-per-depth 25 --depths 1 2 3 4 --budget 512 --seed 303; done && python scripts/analyze.py
Results
Verdict: SCOPED. Transcription is one invariant flat line at ~1.00 across all families (compiler LAW). Identification walls in all families, gap ≥ 0.84 at depth ≥ 3 (generation-wall LAW). Simulation is substrate-dependent: register robust (0.92→0.72), list decays (1.00→0.56), string floored (0.24→0.00) — C15's decay constant was list-specific. See reports/report.md and analysis/crossfamily_ladder.png.
Interpretation
C13 is promoted to a model-level law across substrates ("tools identify, the model compiles" is general). C15 is narrowed: externalize simulation to a tool only where the state representation is expensive to track; for compact integer state the model simulates reliably. New sub-law: the wall's floor ≈ f(hypothesis-space size, simulability).
Knowledgebase Update
- Program evidence updated:
research_programs/structured_execution_and_compilers/evidence.md(C16) - Program backlog updated: representation-swap + op-menu-size floor tests (see report Next Experiments)
- Claim ledger updated: C16 added
Artifacts
src/families.py— the three families (prims, state, input gen, collapse-rejection BFS, oracle)scripts/run_family.py— family-generic ladder runnerscripts/analyze.py— cross-family table, normalized decay, verdict, figuredata/tasks_{list,string,register}.jsonl— verified tasksruns/ladder_{list,string,register}.json,runs/verdict.jsonanalysis/crossfamily_ladder.pngreports/prereg.md,reports/report.md,reports/artifact_manifest.yaml
Report
Rendered from reports/report.md
Summary
Six claims (C11–C15) about the fixed Qwen3.5-4B rested on ONE substrate: integer-list pipelines. This experiment ran the core capability ladder on two genuinely different fresh, execution-verified families — string (char edits) and register (a 3-register integer machine) — alongside the list anchor, to ask whether the ladder constants are model-level laws or list artifacts. Pre-registered in prereg.md.
Verdict: SCOPED — and the scoping is the finding. Two rungs are substrate-invariant laws; one is substrate-dependent.
- LAW — transcription/compiler: plan-given execution ≈ 1.00 at every depth in every family. The three curves collapse to one flat line. The fixed 4B is a universal reliable compiler.
- LAW — generation wall: bare identification of novel compositions collapses toward chance with depth in every family (transcription−identification gap ≥ 0.84 at depth ≥ 3). "The model executes what it cannot invent" is model-level; the rule tools identify, the model compiles is substrate-general.
- SUBSTRATE-DEPENDENT — simulation: C15's simulation-decay "constant" was list-specific. Mental simulation fidelity is set by the cost of tracking the state representation: register (compact 3-int state) is robust (0.92→0.72, ~flat); list decays (1.00→0.56); string is floored (0.24→0.00).
Research Program Fit
Directly stress-tests the structured-execution program's central result (C13 compiler/generation split) and the context-composition triptych (C15) for cross-substrate generality — the highest-leverage open question after the C9–C15 arc, since all six claims shared one family.
Method
All three families share the identical depth-graded, execution-verified, collapse-rejected structure as lists (behavioral min-depth BFS ⇒ nominal depth = real depth), and run through one harness (scripts/run_family.py). Each family defines primitives as Python snippets over a single state variable, so reference code composes trivially and identification grades the same way (execute the model's transform against hidden I/O).
- list — variable-length int lists, 16 primitives (anchor).
- string — lowercase strings, 13 char-edit primitives (reverse, sort_chars, dedup, shift_k caesar, …).
- register — 3-register int machine, 12 primitives (a+=b, rotate, neg_a, mod_a, …); fixed 3-tuple state.
100 verified tasks per family (25 each at depth 1–4). Three rungs, thinking on, budget 512:
- Transcription — plan given as the exact op sequence + step definitions → write the function (pass@1 greedy).
- Simulation — apply the pipeline to one input in your head; emit the state after each step; graded on the final state (greedy). Family-aware parser (see Controls).
- Bare identification — infer
transformfrom I/O examples only (pass@4 sampled).
Results
| rung | family | d1 | d2 | d3 | d4 |
|---|---|---|---|---|---|
| transcription | list | 1.00 | 0.92 | 1.00 | 1.00 |
| string | 1.00 | 1.00 | 1.00 | 1.00 | |
| register | 1.00 | 1.00 | 1.00 | 1.00 | |
| simulation | list | 0.76 | 1.00 | 0.84 | 0.56 |
| string | 0.24 | 0.12 | 0.00 | 0.00 | |
| register | 0.92 | 0.80 | 0.72 | 0.72 | |
| identification | list | 0.80 | 0.12 | 0.00 | 0.00 |
| string | 0.84 | 0.16 | 0.00 | 0.00 | |
| register | 0.56 | 0.36 | 0.16 | 0.08 |
Normalized simulation (each family ÷ its own peak): list 0.76/1.00/0.84/0.56; string 1.00/0.50/0.00/0.00; register 1.00/0.87/0.78/0.78. Cross-family spread up to 0.84 ⇒ no invariant decay constant.
Controls
- Collapse rejection applied identically to all families (behavioral min-depth BFS, probe on 6 inputs, precompiled ops), so any residual shallow-equivalent bias is family-shared and does not confound the cross-family comparison. Oracle: reference code passes visible+hidden for 100% of accepted tasks.
- Parser artifact caught pre-run. A first smoke reported string simulation 0.00 at every depth — a false "law." The model had written
Step 1: nfmic(correct, unquoted) but the list-oriented regex required quotes. Fixed to a family-aware, per-Step i:parser (unit-tested on the exact failing case + register brackets + prose-embedded values) before any scored run. Without this catch the report would have claimed a spurious cross-family simulation collapse. - Same harness, same budget, same thinking setting across families; identification graded by execution, not string match.
Oracle Versus Deployable Evidence
Transcription and identification are deployable (executed code, hidden-set graded). Simulation is a mental-capability microbenchmark (no code, final-state exact match) — it measures whether the model can track state internally, which is what a test-time generate-and-test loop relies on. The register foothold (nonzero deep identification) is deployable and reproduces the C11/C12 pattern that self-search gains traction only where the model can verify its own guesses.
Interpretation
- C13 promoted, not scoped. Its two operative claims — plan-given execution is nearly free, and the compositional deficit is inverse-inference (generation) not execution — now hold across three unrelated substrates ⇒ model-level laws. "Spend tools on hypothesis search, never on execution" is general.
- C15 narrowed. Its simulation-decay curve is list-specific. Sharper deployment corollary: externalize simulation to a tool when the state representation is expensive to track (strings: even single steps; lists: past depth ~3), but for compact integer state the model simulates reliably to depth 4+ — a tool call there is wasted. Representation choice is itself a capability lever.
- New sub-law: the generation wall's floor ≈ f(hypothesis-space size, simulability). Register is the only family both small-enough-to-search (12 ops) and simulable (sim ~0.72), and the only one with a nonzero deep-identification floor (0.16/0.08). List has high simulation but a large space ⇒ zero; string has a small-ish space but zero simulation ⇒ zero. Both factors are jointly necessary. This predicts where a test-time self-search can gain traction, tying together C11 (coverage-bounded banking) and C12 (decompose-search edge).
Next Experiments
- Add a 4th, non-Python-expressible substrate (e.g. a named-graph walk the model must describe, not code) to test whether the transcription law survives when the plan cannot be a code snippet.
- Directly test the floor sub-law: hold simulability fixed, vary op-menu size on the register family, and measure the deep-identification floor vs. |ops|.
- Representation-swap: re-encode string tasks as integer-tuple state (char→ordinal lists) and test whether simulation fidelity jumps to register-like levels — would confirm representation, not task, drives sim.
Artifact Manifest
See reports/artifact_manifest.yaml. Key artifacts: scripts/run_family.py, scripts/analyze.py, src/families.py, runs/ladder_{list,string,register}.json, runs/verdict.json, analysis/crossfamily_ladder.png, data/tasks_{list,string,register}.jsonl.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Reproduce
Smoke test
python scripts/run_family.py --family string --smokeFull run
for f in list string register; do python scripts/run_family.py --family $f --n-per-depth 25 --depths 1 2 3 4 --budget 512 --seed 303; done && python scripts/analyze.pyRun steps are documented inside the experiment folder (README and scripts).