Qwen3.5-4B Simulation Keystone Repair
The one idea you need
This model runs a recipe forward: given the steps and an input, it computes each result in turn. The reverse — naming the hidden steps from a finished dish — is a separate ability. Drilling forward recipes to perfection never touched it; the two sit in sealed compartments.
The question
If a small model keeps failing tasks because it cannot mentally trace a multi-step process, does training it to trace flawlessly finally fix those tasks?
What we found
No. Training made the model trace a multi-step process almost flawlessly — even on longer chains and steps it never studied, so it learned a genuine skill, not memorized answers. Yet every task that supposedly needs tracing stayed frozen: naming the hidden steps from examples crept from 8% to 9%. Oddly, unrelated training that merely matched a task's answer format helped three times more.
Why it matters
When a small model fails a task, don't train the underlying skill and hope the fix spreads. Fine-tuning buys only the input-to-output mapping you train, plus format look-alikes. To add a truly missing ability, give the model a tool instead.
On this page
Results at a glance 3
How to read
Horizontal axis is how many steps are chained (one to five); vertical axis is the share of traces the untrained model got exactly right. Two lines: answering after deliberate thinking, and answering immediately. Higher is better; far below the 80% pass mark counts as broken.
Takeaway → Both lines plunge to about a third correct by four steps — thinking from a near-perfect 96%, immediate answers from a lower 84% — so the untrained model's tracing genuinely breaks at length, qualifying it for repair.
Data table
| Pipeline depth (functions chained) | Thinking | No-think |
|---|---|---|
| 1 | 96% | 84% |
| 2 | 88% | 52% |
| 3 | 58% | 46% |
| 4 | 30% | 30% |
| 5 | 36% | 16% |
Numbers from experiment_log.md
Technical framing
Mental simulation breaks as pipeline depth grows (frozen model) — Exact-match falls from 0.96 at depth 1 to 0.30 at depth 4 — far below the 0.8 kill bar — so the keystone-repair experiment proceeds.
How to read
Horizontal axis is chain length (one to five); vertical axis is the share of exactly-correct traces. Four lines split deliberate thinking versus immediate answers, each with and without steps that throw away information. Higher is better.
Takeaway → Thinking lines mostly sit above their immediate-answer twins (the no-destructive pair briefly crosses at four steps), and information-losing steps hurt most without thinking — so slow deliberation is not the problem, chain length is.
Data table
| Pipeline depth (functions chained) | Thinking, no destructive ops | Thinking, 2 destructive ops | No-think, no destructive ops | No-think, 2 destructive ops |
|---|---|---|---|---|
| 1 | 96% | — | 84% | — |
| 2 | 92% | 84% | 68% | 36% |
| 3 | 52% | 64% | 48% | 44% |
| 4 | 28% | 32% | 36% | 24% |
| 5 | 20% | 52% | 8% | 24% |
Numbers from runs/simbench_run.log
Technical framing
Thinking helps at every depth, with and without destructive ops — Refutes prediction P-K0b: deliberate simulation is not globally wrong, just length-fragile; destructive ops hurt most without thinking.
How to read
Three bars, one per added step (one-to-two, two-to-three, three-to-four), from the deliberate-thinking runs. Each bar's height is the share of accuracy kept versus the shorter chain. Higher is better.
Takeaway → Retention falls from 92% to 66% to 52% — each step costs more than the last, so errors compound rather than leaking at one fixed rate.
Data table
| Added pipeline step | Retention |
|---|---|
| Step 1 to 2 | 92% |
| Step 2 to 3 | 66% |
| Step 3 to 4 | 52% |
Numbers from experiment_log.md
Technical framing
Decay accelerates: retention per added pipeline step (thinking arm) — Retention drops 0.92 to 0.66 to 0.52 per step — no clean single-rate geometric decay, so failure is not one constant per-step error.
In the author’s words from the Overview · “Results (verdict: KEYSTONE REFUTED — capability is format-local)”
The simulator was fully repaired: 0.80–0.84 through depth 5 (base 0.30–0.36), +54pp length- generalization beyond trained depths, held-out-primitive transfer 0.42→0.85 (a skill, not memorization). The inverse ladder did not move: bare 0.08→0.09, segmented 0.14→0.17 — with a working simulator. Transfer follows format adjacency: PROD (I/O→code SFT) tripled segmented identification (0.14→0.41) — same output format — while degrading transcription (0.93→0.72); both adapters crashed thinking-2AFC (~0.10) via verified format capture (answering A/B questions in their trained output format). Phase-0 bonus: thinking helps single-pipeline simulation — deliberate simulation is length-fragile, not globally wrong (refines C13/P12). … Read the full result →
Overview
Research Program
- Program:
structured_execution_and_compilers(+posttraining_and_adaptation). Insight-first, pre-registered (reports/prereg.md, predictions + decision rules locked before each phase). - The intervention test of C13's causal claim: C13 diagnosed one broken primitive — multi-step mental simulation — under every inverse capability (identification, segmentation, discrimination, feedback-use). This experiment repairs the primitive and watches whether the untrained capabilities move. Executes C13's next_tests #1 and #2.
Why this is the highest-stakes question in the arc
Unlike C11's banking (coverage-bounded by what the model can already solve), simulation training data is unlimited and teacher-free — the interpreter emits verified state-chain traces for any pipeline. If repairing the keystone transfers up the ladder, "train broken primitives, not end tasks" becomes the unearthing strategy, breaking the coverage bound. If simulation repairs but the ladder stays frozen, capabilities are separately represented in a fixed model — SFT is task-local, and mechanistic diagnoses do not license training-transfer predictions. If simulation cannot be trained at all, the wall is architectural serial compute. Every branch is a durable law.
Design
- Phase 0 (falsification gate): frozen-model simulator microbenchmark — given a STATED pipeline + one input, write the full state chain (no code). d 1–5 × k {0,2}, n=25/cell, no-think + thinking arms. Kill condition: simulation NOT broken in isolation (d4 ≥ 0.8).
- Phase 1: two QLoRA arms from base at matched training tokens (~230k): SIM (pipeline+input → state chain; depths 1–3 only; 3 primitives held out) vs PROD (I/O examples → reference code — direct end-task training, same unlimited-ground-truth regime; the only difference is the supervised content).
- Phase 2: all three models on (a) simulation — in-distribution, length-generalization d4–5, held-out primitives; (b) the full C13 ladder on fresh verified tasks — bare identification, plan-given transcription, segmented identification, no-think 2AFC, thinking 2AFC.
Phase 0 results (gate PASSED; P-K0b refuted — a C13 refinement)
| depth (better arm = thinking) | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| output exact-match | 0.96 | 0.88 | 0.58 | 0.30 | 0.36 |
- P-K0a confirmed: simulation is broken in isolation, decaying with length (d4 ~0.3 ≪ the 0.8 kill threshold) → keystone experiment proceeds.
- P-K0b refuted: thinking helps single-pipeline simulation at every depth (no-think d3: 0.46 vs think 0.58). Refines C13/P12: deliberate simulation is not globally wrong — structured single-pipeline simulation works at short lengths (0.88 at d2) and is length-fragile; P12's chance-level 2AFC failure reflects the double-simulation + comparison load.
Run
../../.venv/bin/python scripts/run_simbench.py --n-per-cell 25 # Phase 0
../../.venv/bin/python scripts/make_training_data.py 500 # Phase 1 data (CPU)
bash scripts/phase12_chain.sh # train arms + all retestsResults (verdict: KEYSTONE REFUTED — capability is format-local)
Full write-up in reports/report.md; figure analysis/keystone.png.
- The simulator was fully repaired: 0.80–0.84 through depth 5 (base 0.30–0.36), +54pp length- generalization beyond trained depths, held-out-primitive transfer 0.42→0.85 (a skill, not memorization).
- The inverse ladder did not move: bare 0.08→0.09, segmented 0.14→0.17 — with a working simulator.
- Transfer follows format adjacency: PROD (I/O→code SFT) tripled segmented identification (0.14→0.41) — same output format — while degrading transcription (0.93→0.72); both adapters crashed thinking-2AFC (~0.10) via verified format capture (answering A/B questions in their trained output format).
- Phase-0 bonus: thinking helps single-pipeline simulation — deliberate simulation is length-fragile, not globally wrong (refines C13/P12).
Insight (claim C14): capability in a fixed small model is organized by input→output format mappings, not shared internal primitives. Repairing the underlying skill doesn't propagate; SFT buys the trained mapping plus its format neighborhood; mechanism diagnoses don't license training-transfer predictions. The efficient strategy remains C13's: externalize missing primitives with tools.
Report
Rendered from reports/report.md
Summary
The intervention test of C13's causal claim, fully pre-registered (reports/prereg.md). C13 diagnosed one broken primitive — multi-step mental simulation — under every inverse capability. We repaired the primitive and watched whether the untrained capabilities moved. They did not. (Phase 0) The gate passed: simulation is broken in isolation (0.96→0.30 by depth 4), and — refining C13's P12 — thinking helps single-pipeline simulation (it is length-fragile, not globally wrong). (Phase 1–2) QLoRA-SFT on interpreter-generated state-chain traces fully repaired the simulator — 0.80–0.84 through depth 5, length-generalizing +54pp beyond trained depths, transferring to held-out primitives (0.42→0.85) — yet the inverse-capability ladder did not move: bare identification 0.08→0.09, segmented 0.14→0.17. The matched-token control (PROD: direct I/O→code training) moved segmented identification 3× (0.14→0.41 — format-adjacent transfer) while degrading transcription (0.93→0.72), and both adapters crashed thinking- 2AFC via format capture (answering in trained format instead of A/B; verified on raw generations). Verdict per the locked decision rules: KEYSTONE REFUTED — separable-representation branch. Capability in a fixed small model is organized by input→output format mappings, not shared internal primitives: repairing the "underlying" skill does not propagate, transfer follows format adjacency, and narrow-format SFT taxes unrelated instruction-following. Mechanistic diagnoses do not license training-transfer predictions.
Research Program Fit
structured_execution_and_compilers + posttraining_and_adaptation. Executes C13's next_tests #1–2. Every branch of the pre-registered outcome matrix was a durable law; the realized branch (separable representation + format locality) directly bounds the whole banking program (C11/C12) and the mission's "train broken primitives" hope.
Method
- Phase 0 (gate): frozen simulator microbenchmark — stated pipeline + one input → write the full state chain, no code. d 1–5 × k {0,2}, n=25/cell, no-think + thinking. Kill condition: d4 ≥ 0.8.
- Phase 1: matched-token QLoRA arms from base (~230k tokens each, identical hyperparams): SIM = pipeline+input→chain (depths 1–3 only; 3 primitives held out); PROD = I/O examples→reference code (direct end-task training; same unlimited generator ground truth — the only difference is supervised content).
- Phase 2: all three models on (a) simulation (in-distribution, length-gen d4–5, held-out primitives) and (b) the five-rung C13 ladder on fresh verified tasks (bare, plan-given, segmented, 2AFC no-think, 2AFC thinking). All predictions/decision rules locked in advance.
Results
Phase 0 — gate passed; P-K0b refuted (C13 refinement)
Output exact-match by depth — no-think: 0.84/0.52/0.46/0.30/0.16; thinking: 0.96/0.88/0.58/0.30/0.36. Simulation is broken in isolation (gate passes, P-K0a ✓), but thinking HELPS it at every depth (P-K0b ✗): deliberate single-pipeline simulation is length-fragile, not globally wrong — P12's chance-level 2AFC reflects the double-simulation + comparison load, not simulation per se.
Phase 2a — the simulator is repaired (P-K1 ~, P-K2 ✓✓, P-K6 ✓✓)
| simulation (thinking) | d1 | d2 | d3 | d4* | d5* | held-out prims |
|---|---|---|---|---|---|---|
| base | 0.96 | 0.88 | 0.58 | 0.30 | 0.36 | 0.42 |
| SIM | 0.92 | 0.82 | 0.80 | 0.84 | 0.76 | 0.85 |
| PROD | 0.88 | 0.98 | 0.88 | 0.60 | 0.36 | 0.59 |
(beyond SIM's trained depth.) P-K1's letter (+30pp at d3) just missed (+22pp) but the repair is massive and strongest beyond trained depths (+54pp at d4 — P-K2 confirmed); held-out-primitive transfer is nearly full (P-K6) — the model learned chain-simulation as a *skill. (PROD also lifts simulation in-distribution — substrate exposure — but not at length and much less on held-out primitives.)
Phase 2b — the ladder does not move (P-K3 ✗, P-K4 inverted)
| model | bare | segmented | 2AFC no-think | 2AFC thinking | plan-given |
|---|---|---|---|---|---|
| base | 0.08 | 0.14 | 0.75 | 0.47 | 0.93 |
| SIM | 0.09 | 0.17 | 0.78 | 0.10 | 0.90 |
| PROD | 0.13 | 0.41 | 0.82 | 0.15 | 0.72 |
- SIM (repaired simulator): every inverse rung flat. The primitive works; nothing downstream noticed.
- PROD moved segmented 3× (d3k0 0.20→0.65, d4k0 0.00→0.45): transfer exists but follows format adjacency (segmented shares PROD's I/O→code output format), not primitive dependency. Bare identification at depth stays dead for both.
- Both adapters crash thinking-2AFC (0.10–0.15, below chance) via format capture — raw generations show the SIM model answering the A/B question with
`python code blocks. Narrow-format SFT installs output-mode priors that override unrelated instructions (and PROD's transcription drop 0.93→0.72 is the same tax).
Controls
Matched training tokens (±2%); identical hyperparams; PROD as content-control (same ground-truth regime, different supervised content); held-out primitives excluded from both arms; fresh verified ladder tasks; format-capture verified on raw generations before interpreting the 2AFC crash; all predictions and decision rules pre-registered.
Oracle Versus Deployable Evidence
All behavioral; simulation graded by exact match against interpreter ground truth; ladder hidden-graded on fresh verified tasks.
Interpretation
Capability in this fixed small model is organized by input→output format mappings, not by shared internal primitives. Three mutually reinforcing observations: (1) repairing the hypothesized keystone — genuinely, with length generalization and skill-level transfer within the format — moves nothing outside its format; (2) the only cross-task transfer observed follows format adjacency (PROD→segmented); (3) SFT's strongest side effect is format capture — output-mode priors that damage unrelated instruction-following. Consequences: (a) C13's diagnosis stands, but its causal reading ("fix simulation → fix inverse tasks") is refuted — mechanism diagnoses do not license training-transfer predictions; (b) the banking program (C11/C12) is format-local: it teaches mappings, not components — which retroactively explains why banking never moved the planner (C12) and why production-SFT never improved verification; (c) the mission lesson — for a fixed small model, eliciting a "skill" via SFT buys exactly the trained mapping plus its format neighborhood, nothing more. The efficient strategy remains C13's: externalize the missing primitive with tools rather than trying to install it and hoping it propagates.
Limitations
One substrate family; QLoRA (r32) not full fine-tuning — conceivably full FT propagates differently; single training run per arm; 2AFC format capture means that rung measures instruction-robustness, not discrimination, for the adapters; P-K1's d3 letter missed by 8pp (d4/d5 vastly exceeded it).
Next Experiments
- Mixed-format SIM training (chains + A/B + code in one adapter) — does format diversity prevent capture and unlock cross-format use of the repaired simulator?
- Explicit composition: prompt the SIM model to use its repaired simulation inside identification ("simulate candidate pipelines, compare") — can prompting bridge what SFT does not?
- Full-FT vs QLoRA on the same design (is separability an adapter artifact?).
Artifact Manifest
See artifact_manifest.yaml. Adapters (~170MB ×2) regenerable via scripts/phase12_chain.sh; per-task results + prereg + figure in-repo.
Experiment log 5
Show the running log (5 entries)
Design
Intervention test of C13 (broken multi-step mental simulation as the keystone under the inverse-capability ladder). Pre-registered (reports/prereg.md). Phase 0 = frozen simulator microbenchmark (falsification gate). Phase 1 = matched-token QLoRA arms: SIM (state chains) vs PROD (reference code; direct end-task control). Phase 2 = simulation retest (in-dist / length-gen / held-out primitives) + full C13 ladder (bare / plan-given / segmented / 2AFC no-think / 2AFC thinking) on fresh verified tasks, all three models.
Phase 0 (gate PASSED; P-K0b refuted)
Frozen, output exact-match by depth (think arm): 0.96 / 0.88 / 0.58 / 0.30 / 0.36; no-think: 0.84 / 0.52 / 0.46 / 0.30 / 0.16. P-K0a CONFIRMED (broken in isolation, d4 ~0.3 << 0.8 kill threshold) -> proceed. P-K0b REFUTED: thinking HELPS single-pipeline simulation at every depth -- refines C13/P12: deliberate simulation is length-fragile, not globally wrong; P12's chance-level 2AFC = double-simulation + comparison load. No clean single-r geometric decay (retention 0.92 -> 0.66 -> 0.52 by step) -- decay accelerates.
Phase 1 data
SIM 1500 records / 227,938 tokens vs PROD 554 records / 232,447 tokens (matched within 2%). Held-out primitives excluded from both. PROD sharpened to reference-code supervision (prereg addendum, logged before Phase 1 ran).
Phases 1-2
Running (scripts/phase12_chain.sh): train SIM + PROD -> simbench x2 -> ladder x3.
Phase 1-2 results (VERDICT: keystone REFUTED -- separable/format-local branch)
Simulator REPAIRED: SIM 0.92/0.82/0.80/0.84/0.76 by depth (base 0.96/0.88/0.58/0.30/0.36); +54pp at untrained d4 (P-K2 ok); held-out prims 0.42->0.85 (P-K6 ok); P-K1's d3 letter missed by 8pp (+22 vs +30) but d4/d5 vastly exceed. LADDER FLAT for SIM: bare 0.08->0.09, segmented 0.14->0.17, afc_nothink 0.75->0.78 (P-K3 fail). P-K4 INVERTED: PROD tripled segmented (0.14->0.41; d3k0 0.20->0.65) -- format- adjacent transfer -- and degraded plan-given 0.93->0.72. Both adapters crashed afc_think to 0.10-0.15: verified FORMAT CAPTURE on raw generations (SIM answers the A/B question with `python blocks). Locked-rule verdict: REFUTED-separable. Insight: capability is FORMAT-LOCAL in the fixed 4B (claim C14). Chain runtime ~5.6h total (SIM train 2406s loss 0.021; PROD 898s loss 0.116; 3 ladders ~1-1.5h each).
Figures 1
Reproduce
Run steps are documented inside the experiment folder (README and scripts).