Qwen3.5-4B Depth-Wall Anatomy
The one idea you need
Picture a cook who follows any written recipe flawlessly. But set a finished dish before that cook and they cannot deduce which steps produced it, nor mentally re-cook a guessed recipe to check it against the plate. This model executes; it cannot infer.
The question
When a small model fails at chaining several data transformations together, is the hard part running the long chain of steps, or figuring out which steps produced the result?
What we found
It's figuring out the steps. Handed the exact sequence of operations, this 4-billion-parameter model writes correct code almost every time, even four steps deep, with zero execution deficit. Left to infer that sequence from before-and-after examples, it solves almost nothing past two steps, spotting each added operation only about twice as well as guessing among 63 candidates.
Why it matters
Don't fine-tune a small model to discover multi-step transformations from examples, its weakest skill. Let cheap external tools enumerate and propose candidate pipelines, and use the model for what it does near-perfectly: turning a stated plan into working code.
On this page
Results at a glance 4
How to read
Six task types along the bottom (chain length two to four, each with clean or information-destroying steps); bar height is share of tasks solved. Three bars each: from bare examples, from the stated plan, and with intermediate states shown. Higher is better.
Takeaway → The stated-plan bars stay pinned near the top through chain length four, while both guessing bars sit on the floor. The wall is finding the plan, not running it.
Data table
| task cell (composition depth × destructive-op count) | bare I/O (identify + execute) | plan given (execute only) | intermediates shown |
|---|---|---|---|
| depth 2, clean ops | 5% | 100% | 30% |
| depth 2, lossy ops | 10% | 100% | 15% |
| depth 3, clean ops | 0% | 100% | 10% |
| depth 3, lossy ops | 0% | 100% | 0% |
| depth 4, clean ops | 0% | 100% | 5% |
| depth 4, lossy ops | 0% | 90% | 0% |
Numbers from experiments/qwen35_4b_depth_wall_anatomy/data/discriminator_records.jsonl
Technical framing
Told the pipeline, the 4B executes near-perfectly; left to identify it, it fails — The compositional wall is 100% identification, 0% execution: plan-given stays 0.90-1.00 through depth 4; seeing intermediates barely helps.
How to read
Bottom axis is chain length (one to five steps); height is share of tasks solved. Four lines mark how many information-destroying steps are present, from zero to three. Upper-left is better.
Takeaway → Every line falls off a cliff between one step and two and hits zero by three, overlapping regardless of step type, so depth alone kills it, walling at length two.
Data table
| verified composition depth (ops) | 0 destructive ops | 1 destructive op | 2 destructive ops | 3 destructive ops |
|---|---|---|---|---|
| 1 | 88% | 72% | — | — |
| 2 | 16% | 4% | 8% | — |
| 3 | 0% | 4% | 0% | 0% |
| 4 | 0% | 0% | 0% | 0% |
| 5 | 0% | 0% | 0% | 0% |
Numbers from experiments/qwen35_4b_depth_wall_anatomy/runs/grid_analysis.json
Technical framing
Solve rate collapses ~30x in odds per composed op, regardless of op type — On collapse-free tasks the wall is at depth 2, not 3, and information-destroying ops barely matter — refuting the destruction hypothesis.
How to read
Six task types along the bottom (chain length two to four, clean or information-destroying steps). One bar per type is solving when steps are pre-separated; the other is accuracy telling the true pipeline from a decoy. Higher is better; half is a coin-flip.
Takeaway → Pre-separated solving slides from half toward zero as chains lengthen, and true-versus-decoy accuracy drops to a coin-flip at the deepest lossy cell. Even isolated pieces and simple checks break down.
Data table
| task cell (composition depth × destructive-op count) | segmented steps (pass@4) | 2AFC pipeline verification (accuracy) |
|---|---|---|
| depth 2, clean ops | 50% | 80% |
| depth 2, lossy ops | 15% | 70% |
| depth 3, clean ops | 25% | 75% |
| depth 3, lossy ops | 5% | 75% |
| depth 4, clean ops | 0% | 85% |
| depth 4, lossy ops | 0% | 50% |
Numbers from report table (reports/report.md, Phase 3 probes)
Technical framing
Even pre-segmented steps and yes/no verification decay with depth — The deficit is multi-step mental simulation: isolated depth-1 identification runs at 0.88, yet composed contexts drag it down and verification nears chance (0.5).
How to read
Two task groups along the bottom: tasks labeled three-steps-deep that secretly collapse to something shallower, versus genuinely three-step tasks. Bar height is how often an external split-and-search method solved them. Higher is better.
Takeaway → The search cleared every secretly-shallow task but only one in six genuinely deep ones, so nearly all the celebrated three-step success was a measurement artifact.
Data table
| task type | decompose+interpreter solve rate |
|---|---|
| collapsed 'depth-3' tasks (16/16) | 100% |
| true depth-3 tasks (4/24) | 16.7% |
Numbers from report table (reports/report.md, Phase 0)
Technical framing
The old depth-3 wall was mismeasured: decompose search only worked on collapsed tasks — 40% of nominal depth-3 tasks were behaviorally shallower; every recorded monolithic depth-3 solve rode on them — true depth-3 monolithic solves were 0.
In the author’s words from the Overview · “Results”
Three findings, each pre-registered: The wall was mismeasured (Phase 0): 40% of nominal depth-3 tasks are shallower-equivalent; true monolithic depth-3 was always 0; C12 retro-corrected. The destruction hypothesis died; a steeper law replaced it (Phase 1, P1/P2 refuted): on verified novel compositions, solve odds fall ~30× per composed op regardless of op type — identification beyond the first primitive runs at ~2× better than chance (63-op space), walling at depth 2. The wall is 100% identification, 0% execution (Phase 2, P7 strongest form): plan-given → 0.90–1.00 through depth 4; intermediates-shown barely helps (≤0.30) — the model can't segment chains into the depth-1 identifications it does at 0.88. … Read the full result →
Overview
Research Program
- Program:
structured_execution_and_compilers. Mission: understand the fixed Qwen3.5-4B's compositional frontier — insight-first: laws and mechanisms, not another lever. - Attacks the arc's center of mass: every C11/C12 headline routes through "the depth-3 wall." This experiment decomposes that wall into measurable parts, with pre-registered predictions (
reports/prereg.md) logged before each phase ran.
Question
What actually sets the compositional frontier of the fixed 4B on the contamination-free substrate? Candidate axes, cleanly separable:
- Substrate artifact — nominal depth-d compositions with behavioral min-depth < d (
sort_asc∘reverse ≡ sort_desc). Phase 0 audits ALL existing M1/M2/C12 tasks by exact BFS. - Serial depth — composition length per se (the implicit C11/C12 story).
- Information destruction — ops whose intermediates are unrecoverable from visible I/O (filters/dedup/mod/abs/...) making the composition un-invertible / unidentifiable.
- Identification vs execution — can the model execute a deep pipeline it is told, and does seeing intermediates rescue it (Phase 2 discriminator: bare vs plan-given vs intermediates-shown)?
Phase 0 results (CPU audit of existing data — both predictions confirmed)
- P0a: 40% of nominal depth-3 tasks (M1 and C12 sets) collapse to behavioral min-depth ≤2.
- P0b (strong form): monolithic true-depth-3 solves = 0 across the entire corpus; every recorded depth-3 solve rode on collapsed tasks. C12's decompose search: 16/16 collapsed vs 4/24 (17%) true. → C12's claim + report retro-corrected (commit f6c2ca7).
- Destruction signal survives the collapse control: true depth-2, k=0 → 6/8 solved; k≥1 → 0/8.
Design (Phases 1-2)
- Verified factorial grid: 17 cells (d 1–5 × k 0–3), n=25/cell, generator rejects collapsed compositions (exact BFS to depth min(d−1,3)). Monolithic thinking greedy@1 + pass@6, hidden-graded; per-candidate visible/hidden (false-pass slice); thinking length; first-op letter-logit rank (planner).
- Discriminator on the same tasks (d {2,3,4} × k {0,2}): bare vs plan-given vs intermediates-shown.
- Predictions P1–P9 in
reports/prereg.md, all logged in advance.
Run
../../.venv/bin/python scripts/min_depth_audit.py # Phase 0 (CPU)
../../.venv/bin/python scripts/run_grid.py --n-per-cell 25 --k-samples 6 # Phase 1
../../.venv/bin/python scripts/run_discriminator.py --per-cell 20 # Phase 2
../../.venv/bin/python analysis/analyze_grid.py # analysis vs predictionsResults
Full write-up in reports/report.md. Three findings, each pre-registered:
- The wall was mismeasured (Phase 0): 40% of nominal depth-3 tasks are shallower-equivalent; true monolithic depth-3 was always 0; C12 retro-corrected.
- The destruction hypothesis died; a steeper law replaced it (Phase 1, P1/P2 refuted): on verified novel compositions, solve odds fall ~30× per composed op regardless of op type — identification beyond the first primitive runs at ~2× better than chance (63-op space), walling at depth 2.
- The wall is 100% identification, 0% execution (Phase 2, P7 strongest form): plan-given → 0.90–1.00 through depth 4; intermediates-shown barely helps (≤0.30) — the model can't segment chains into the depth-1 identifications it does at 0.88.
One mechanism explains the arc: the fixed 4B is a reliable compiler starved of hypothesis search — why decompose+interpreter search works, why its guidance was only ~2× over brute force (same constant!), why banking is coverage-bounded, why execution feedback failed, why verify ≫ generate. Division of labor: tools identify, the model compiles.
Report
Rendered from reports/report.md
Summary
Insight-first anatomy of the fixed 4B's compositional wall (the "depth-3 frontier" every C11/C12 headline routes through), with pre-registered predictions (reports/prereg.md) logged before each phase. (Phase 0) A behavioral min-depth audit of ALL existing substrate tasks found the wall was mismeasured: 40% of nominal depth-3 tasks are shallower-equivalent, the frozen model has never solved a true full-depth-3 task monolithically, and C12's decompose search really solved 17% of true depth-3 (not 0.40) — retro-corrections landed in C12 (commit f6c2ca7). (Phase 1) On the verified factorial grid the pre-registered information-destruction hypothesis died (k=0 deep compositions are NOT solvable; op type barely matters) and a steeper law replaced it: solve odds fall ~30× per composed op — against a 63-op space, identification beyond the first primitive runs at only ~2× better than chance, walling at depth 2. (Phase 2, decisive) A three-condition discriminator shows the wall is 100% identification, 0% execution: told the pipeline, the model executes at 0.90–1.00 through depth 4; shown every intermediate state, it still cannot segment chains into the depth-1 identifications it performs at 0.88. One mechanism — the fixed 4B is a reliable compiler starved of hypothesis search — quantitatively retro-explains C10–C12 (decompose search works, its ~2× guidance efficiency, banking's coverage bound, feedback's failure, verify≫generate).
Research Program Fit
structured_execution_and_compilers + the insight-generation goal: convert the arc's central qualitative story ("the depth wall") into measured structure — what the wall is made of (artifact / serial depth / information destruction), and where it lives (hypothesis identification vs execution).
Method
- Phase 0 (CPU): exact BFS behavioral min-depth over all primitive pipelines vs ALL 18 examples, for every existing M1/M2/C12 task; restratify recorded solves. Predictions P0a/P0b.
- Phase 1 (verified factorial grid): 17 cells (depth d ∈ 1–5 × destructive-ops k ∈ 0–3), n=25/cell, 425 tasks, generator rejects collapsed compositions (BFS to depth min(d−1,3); d=5 may retain d4-equivalents — caveat). Destructive set (fixed in prereg): ops whose intermediates are unrecoverable from I/O (filters, dedup/unique, take/drop, chunk_sum, mod, abs, running_max). Monolithic thinking greedy@1 + pass@6 (hidden-graded), per-candidate visible/hidden pass, thinking length, and the first-op letter-logit rank (planner slice). Predictions P1–P6, P9.
- Phase 2 (discriminator): same verified tasks (d {2,3,4} × k {0,2}), three conditions — bare I/O (identify + execute), plan-given (pipeline stated; execution only), intermediates-shown (state chains visible; observability restored). Predictions P7–P8.
Results
Phase 0 — the wall was mismeasured (P0a, P0b confirmed)
| finding | value |
|---|---|
| nominal d3 tasks that are behaviorally ≤ d2 | 40% (M1 6/15; C12 16/40; M2 25%) |
| monolithic TRUE depth-3 solves, entire corpus | 0 (all recorded d3 solves were collapsed tasks) |
| C12 decompose on collapsed vs true d3 | 16/16 vs 4/24 (17%) |
| destruction signal after collapse control (M2 true-d2) | k=0: 6/8 solved; k≥1: 0/8 |
Phase 1 — verified factorial grid: the destruction hypothesis dies, a steeper law appears
pass@6 (n=25/cell, hidden-graded, verified-depth tasks):
| d\k | 0 | 1 | 2 | 3 |
|---|---|---|---|---|
| 1 | 0.88 | 0.72 | — | — |
| 2 | 0.16 | 0.04 | 0.08 | — |
| 3 | 0.00 | 0.04 | 0.00 | 0.00 |
| 4 | 0.00 | 0.00 | 0.00 | 0.00 |
| 5 | 0.00 | 0.00 | 0.00 | 0.00 |
- P1 REFUTED: transparent-only (k=0) compositions do NOT stay solvable at depth ≥3 (0.00 at d3–d5; predicted ≥0.4). P2 REFUTED: at fixed depth, destruction count barely matters (logistic coefficients per transparent vs destructive op: −3.24 vs −3.87 — statistically indistinguishable; depth-only AIC 112.7 vs two-parameter 111.9). The destruction signal that motivated the hypothesis (M2 true-d2 k=0: 6/8) was primitive-mix luck in an n=8 cell — the controlled n=25 grid eliminates it.
- What replaces it is a clean quantitative law: the odds of solving fall ~30× per composed op (logistic slope ≈ −3.5/op), uniform across op types. Anchored at d1 (0.88/0.72), this predicts d2 ≈ 0.17 and d3 ≈ 0.008 — matching the observed 0.16 and ~0.01. Since blind guessing over the 63-op vocabulary would cost ~63× per op, the model's identification power beyond the first primitive is only ~2× better than chance. On genuinely novel compositions the wall is at depth 2, not 3.
- P5: false-passes remain ~nil on verified tasks (3/156 visible-passers hidden-fail) — visible-test selection stays lossless; the C2 wall stays absent on this substrate.
- P9 partially refuted: first-op letter-logit rank is poor everywhere (median 7–13 of 23) with no clear destruction effect — the planner cannot identify the first op of any genuinely novel composition much better than chance, consistent with the ~2×-over-chance law.
- P6 (exploratory): thinking length is budget-saturated (mean 434→492 tokens from d1→d5 against the 512 cap; solved vs failed identical at matched depth) — uninformative at this budget; a budget sweep would be needed to test the serial-workspace account.
Phase 2 — identification vs execution: the wall is 100% identification
pass@4 by cell (n=20/cell, same verified tasks, three prompting conditions):
| cell | bare I/O | plan-given | intermediates-shown |
|---|---|---|---|
| d2k0 | 0.05 | 1.00 | 0.30 |
| d2k2 | 0.10 | 1.00 | 0.15 |
| d3k0 | 0.00 | 1.00 | 0.10 |
| d3k2 | 0.00 | 1.00 | 0.00 |
| d4k0 | 0.00 | 1.00 | 0.05 |
| d4k2 | 0.00 | 0.90 | 0.00 |
- P7 CONFIRMED (strongest form): told the pipeline, the model executes essentially perfectly at every depth and destruction level. The compositional wall contains zero execution deficit.
- P8 REFUTED: full observability of intermediate states barely rescues identification (≤0.30). Each adjacent state-hop is a depth-1 identification the model does at 0.88 — but it cannot segment a shown chain into those solvable pieces and re-compose. (Caveat: chains add prompt clutter; a segmented-presentation follow-up quantifies this.)
- Convergent constant: the grid's ~2×-over-chance-per-op identification law independently matches C12's finding that model-guided search beat brute-force enumeration by only ~2× in efficiency — two measurements, one constant.
Controls
Verified-depth generation (BFS rejection) removes the collapse artifact from the grid itself. The destructive/transparent classification was fixed in the prereg before data. The oracle solves 425/425. Phase-2's plan-given condition controls task content exactly (same tasks, information added).
Oracle Versus Deployable Evidence
All measures deployable (visible info only; hidden-graded). Reference oracle bounds everything at 1.0.
Phase 3 — probes: the deficit is inverse-inference itself, not just search
Same verified cells, two pre-registered probes (P10/P11 logged before running):
| cell | segmented pass@4 | 2AFC accuracy | (recall: bare / plan-given) |
|---|---|---|---|
| d2k0 | 0.50 | 0.80 | 0.05 / 1.00 |
| d2k2 | 0.15 | 0.70 | 0.10 / 1.00 |
| d3k0 | 0.25 | 0.75 | 0.00 / 1.00 |
| d3k2 | 0.05 | 0.75 | 0.00 / 1.00 |
| d4k0 | 0.00 | 0.85 | 0.00 / 1.00 |
| d4k2 | 0.00 | 0.50 | 0.00 / 0.90 |
- P10 partially refuted: pre-segmented per-step blocks (each an isolated depth-1 identification the model does at 0.88) help — but only to 0.50/0.25 at d2/d3, decaying to 0.00 at d4. If per-step identification stayed at its isolated rate, d3 would be ~0.68. So identification degrades per-step in composite context — an interference effect beyond segmentation/search.
- P11 mostly refuted: shown the true pipeline and a one-op decoy, no-think discrimination is only ~0.70–0.85 (chance at d4k2) — the model cannot reliably verify a stated hypothesis against I/O in a single forward pass, despite executing stated pipelines at ~1.0 when asked to produce code. (Caveat: no-think logit read; deliberate thinking-simulation may rescue discrimination.)
P12 (thinking-mode 2AFC, pre-registered, predicted ≥0.9): REFUTED — 0.50 overall, chance, and worse than the no-think logit read (0.73; Δ≈3.6 SE at n=120, same items/decoys). Asked to "work through the examples step by step," the model's mental execution of composed pipelines is systematically wrong and overwrites the weak-but-real surface signal the instant read used. (Parse rate 0.88; budget 512.)
Completed capability ladder (same tasks throughout): plan→code transcription 1.00 → no-think discrimination ~0.73 → segmented identification 0.50→0.00 → thinking discrimination 0.50 → bare identification ~0.0. The plan-given 1.00 is transcription, not semantic execution — the interpreter executed; the model only wrote code. What is intact: single-step recognition (0.88) and program→code transcription. What is broken: multi-step mental simulation — every task requiring internal state-tracking across composed ops (identify, segment, discriminate, simulate-and-compare) is weak and decays with length.
Interpretation
The fixed 4B is a reliable transcriber with a broken internal simulator. The wall is not search alone, not segmentation alone, not inverse inference alone — all of those reduce to one deficit: multi-step mental state-tracking. The model cannot reliably run a composed program in its head (P12: thinking-simulation performs at chance and below its own no-think surface heuristics), so every capability that requires it — identifying a composition from behavior, segmenting a shown chain, checking a hypothesis against examples, using execution feedback (M2) — collapses with composition length, while capabilities that don't (single-step recognition, program→code transcription) stay near ceiling. This also reframes C9: thinking helps when its content is coherent reasoning; here the required content is simulation, which the model generates incorrectly, so thinking hurts. The compositional wall decomposes as: (i) a large measurement artifact (40% shallower-equivalents; true monolithic depth-3 was always 0); (ii) zero execution deficit (plan-given ≈ 1.00 through depth 4); (iii) a hard identification wall — on genuinely novel compositions the model identifies each additional composed op at only ~2× better than chance (odds ∝ ~30⁻ᵈ against a 63-op space), collapsing at depth 2 — insensitive to op type (destruction hypothesis refuted) and barely helped by seeing intermediate states (it cannot segment chains into the depth-1 identifications it can do).
This one mechanism retro-explains the arc quantitatively:
- C12's decompose+interpreter search worked because it externalizes exactly what the model lacks — segmentation + hypothesis search — leaving only per-step ranking and execution (which are cheap for it).
- C12's guidance was only ~2× better than brute force — the same ~2×-over-chance constant, measured independently.
- C11's banking is coverage-bounded because SFT teaches production/execution patterns, but the wall is identification: you cannot bank hypotheses you cannot identify.
- M2's execution-feedback failure: feedback flags wrong outputs, but the binding constraint is identifying the right hypothesis — which not even explicit intermediate states unlock.
- C10's verify ≫ generate: verifying a given program is execution-shaped (the model's strength); generating requires identification (its weakness).
Deployment corollary: for a fixed small model, plan-conditioned execution is nearly free capability; the scarce resource is hypothesis search, which cheap external tools (enumeration + an interpreter) supply. Division of labor: tools identify, the model compiles.
Limitations
- One substrate family (list-of-int primitives), one model; the 63-op hypothesis space is enumerable (real-world identification may differ in structure). d5 verification capped at BFS depth 3 (d4-equivalents possible). Thinking budget 512 saturated (P6 untestable here). Intermediates-shown carries prompt-format burden (segmented-presentation follow-up pending).
Next Experiments
- Segmented-presentation probe: present each step's transition as separate mini-examples — if that rescues identification, the deficit localizes to segmentation specifically.
- Budget sweep at fixed depth (does thinking length become diagnostic once unsaturated — serial workspace?).
- Two-alternative identification (given two candidate pipelines, pick the consistent one) — measures identification as discrimination, isolating it from generation.
- Cross-substrate: does the ~30×/op identification decay constant transfer to other primitive families?
Artifact Manifest
See artifact_manifest.yaml.
Experiment log 8
Show the running log (8 entries)
Design
Insight-first: decompose the "depth-3 wall" (center of the C11/C12 arc) into measurable parts with pre-registered predictions (reports/prereg.md). Phase 0 = behavioral min-depth audit of ALL existing substrate tasks (CPU, exact BFS). Phase 1 = verified-depth factorial grid, depth d x destructive-ops k. Phase 2 = discriminator (bare vs plan-given vs intermediates-shown) separating hypothesis identification from execution.
Phase 0 results (predictions P0a/P0b CONFIRMED)
40% of nominal-d3 tasks behaviorally collapse to <=d2 (M1 6/15, C12 16/40). Monolithic TRUE depth-3 solves across the ENTIRE corpus: 0 -- every recorded d3 solve was a collapsed task. C12 decompose: 16/16 collapsed vs 4/24 (17%) true-d3. Destruction signal survives the collapse control (M2 true-d2: k=0 6/8, k>=1 0/8). Retro-corrections to C12 committed (f6c2ca7, CI green).
Phase 1
Grid generation: 425/425 verified tasks (17 cells, oracle 100%); generation cost ~35 min CPU (BFS rejection; transparent cells reject often, e.g. negate-negate collapses). GPU run in progress.
Phase 2
Pending (runs after grid; discriminator on grid task subset).
Phase 1 results (grid, 425 verified tasks -- P1/P2 REFUTED, steeper law found)
pass@6: d1 0.88(k0)/0.72(k1); d2 0.16/0.04/0.08; d3-d5 ~0.00 EVERYWHERE (2 solves in 275 tasks at d>=3). My pre-registered destruction hypothesis DIED: k=0 deep tasks are NOT solvable (P1 refuted -- predicted
=0.4 at d4, got 0.00) and destruction count barely matters at fixed depth (P2 refuted; logistic coefs
-3.24 transparent vs -3.87 destructive; depth-only AIC 112.7 vs two-param 111.9). The motivating M2 signal (k=0 true-d2 6/8) was primitive-mix luck at n=8.
REPLACEMENT LAW: odds of solving fall ~30x per composed op (~ -3.5 logits/op), uniform across op types; blind guessing over the 63-op space costs ~63x/op -> the model identifies each additional composed op at only ~2x better than chance. The wall on genuinely novel compositions is at depth TWO. False-passes ~nil (3/156). Planner first-op rank poor everywhere (median 7-13/23), no destruction effect.
Grid runtime note: sampled phase 6253s (2500 gens at budget 512) -- longer than projected (verified d>=3 tasks never terminate early + long thinking). Generation (BFS-verified) 35 min CPU.
Phase 2 results (discriminator -- P7 CONFIRMED strongest form, P8 REFUTED)
pass@4: plan_given 0.90-1.00 at EVERY cell (d2-d4, k0/k2) vs bare 0.00-0.10 vs intermediates-shown 0.00-0.30. The wall contains ZERO execution deficit -- it is 100% hypothesis identification, and even full observability of intermediate states barely rescues (the model cannot SEGMENT chains into the depth-1 identifications it does at 0.88). Convergent constant: the grid's ~2x-over-chance/op law equals C12's ~2x guided-vs-brute efficiency, measured independently. Headline: the fixed 4B is a reliable compiler starved of hypothesis search. Runtime: 2725s for 1440 gens.
Phase 3 results (probes -- P10 partial, P11 mostly refuted; ladder completed)
Segmented (pre-segmented per-step blocks): d2k0 0.50, d3k0 0.25, d4 0.00 -- helps vs intermediates-shown but does NOT recover the 0.88 isolated-d1 rate: per-step identification DEGRADES in composite context (interference beyond search/segmentation). 2AFC (true pipeline vs one-op decoy, no-think logit): 0.50-0.85 -- weak discrimination despite perfect plan-execution; the model cannot verify a stated hypothesis against I/O in one forward (caveat: no-think; thinking-simulation untested). COMPLETED LADDER on the same tasks: execution 1.00 > 2AFC ~0.73 > segmented 0.50->0.00 > bare ~0.0. Only the forward direction (program->behavior) is intact; inverse inference (behavior->program) is the wall, decaying with length. Ops note: first probe launch crashed (fla kernel, long segmented prompts at batch 48 -> batch 20 fix); a kill-preamble self-match killed the second launch (use clean launches).
P12 results (thinking 2AFC -- REFUTED at chance; the simulator is the wall)
Thinking-mode 2AFC (same items/decoys as no-think, greedy, budget 512): 0.50 overall (d2k0 .70 ... d4k2 .35), WORSE than the no-think logit read (0.73; ~3.6 SE, n=120). Deliberate simulate-and-compare fails -- the model's mental execution of composed pipelines is systematically wrong and destroys the surface signal. REFRAME: plan-given 1.00 = TRANSCRIPTION (interpreter executed; model wrote code). Intact: single-step recognition (0.88), program->code transcription (1.0). Broken: MULTI-STEP MENTAL SIMULATION -- the single deficit underlying identify/segment/discriminate/feedback-use. Also reframes C9: thinking helps when its content is coherent; here the required content is simulation, generated wrongly, so thinking HURTS. Caveats: budget 512, parse rate 0.88, one decoy family (one-op edits).
Figures 2
Reproduce
Run steps are documented inside the experiment folder (README and scripts).