Qwen3.5-4B Latent Composition Probe: is the wall representational or expressive?
The one idea you need
The model sees input-output examples and must guess the hidden recipe of list operations behind them. Picture its internal activity as private scratch paper; a simple decoder reads whether the recipe's first step is written there, even when the model never says it aloud.
The question
When a small model can't produce the right chain of list operations, has it secretly figured out the answer inside, or does it genuinely not know it?
What we found
It depends on the number of steps. For a one-step recipe the first step is almost perfectly written on the internal scratch paper (99% readable), yet the model voices it only 44% of the time: it knows but stays silent. For the hardest three-step recipe the scratch paper is nearly blank: the readable signal (about 13%) barely clears the 14% noise floor, meaning the answer was never worked out.
Why it matters
Before betting on tricks that read or nudge a small model's internal state to fix failures, check the skill is actually in there. That only helps on shallow tasks; deep multi-step gaps need training or external tools to install the missing ability.
On this page
Results at a glance 2
How to read
Grouped bars for one-, two-, and three-step recipes; height is first-step accuracy, higher is better. Four bars each: the first step read from inside the model, the model naming it, the model generating the full recipe, and a shuffled-label noise floor for reference.
Takeaway → At one step the read-from-inside bar towers over what the model says or generates; by three steps all four bars collapse together near the noise floor, so hidden knowledge is present when shallow and gone when deep.
Data table
| condition | linear probe (representation) | model names 1st op (expression) | model generates it (ident@1) | shuffled-label floor |
|---|---|---|---|---|
| depth 1 | 99% | 44% | 68% | 5% |
| depth 2 | 42% | 13% | 7% | 9% |
| depth 3 | 27% | 13% | 1% | 14% |
Numbers from
Technical framing
Representation >> expression at shallow depth; both thin at the wall — The composition's first operation is linearly decoded from residual-stream activations at 0.99 at depth 1 while the model names it only 0.44 and generates it 0.68 = latent capability (present but unexpressed). The gap decays with depth: at depth 3 (the wall) the probe (0.27) barely clears the shuffled-label overfitting floor (0.14), so the real signal ~0.13 equals the model's behavioral naming -- the representation itself has thinned to a thread. The wall is an expression failure when shallow, a representation failure when deep.
How to read
Lines track how readable the first step is at each layer, from the input embedding on the left to the final layer on the right, one line per recipe depth. Higher is better; a flat band near 10% marks random guessing.
Takeaway → The one-step line climbs to near-perfect by the middle layers and holds; the two- and three-step lines stay low and flat, so deeper recipes are barely worked out anywhere inside.
Data table
| layer (0 = embedding) | depth 1 | depth 2 | depth 3 |
|---|---|---|---|
| 0 | 6.8% | 6.7% | 4% |
| 1 | 59.5% | 19.5% | 22% |
| 2 | 70.3% | 24.2% | 11.3% |
| 3 | 69.6% | 27.5% | 17.3% |
| 4 | 69.6% | 28.2% | 16.7% |
| 5 | 75% | 26.8% | 14.7% |
| 6 | 72.3% | 28.9% | 17.3% |
| 7 | 83.8% | 21.5% | 17.3% |
| 8 | 85.8% | 26.2% | 17.3% |
| 9 | 85.8% | 22.1% | 14.7% |
| 10 | 90.5% | 33.6% | 24% |
| 11 | 95.3% | 34.2% | 20.7% |
| 12 | 91.9% | 28.9% | 24% |
| 13 | 95.3% | 24.8% | 18% |
| 14 | 98.6% | 36.9% | 14% |
| 15 | 99.3% | 39.6% | 21.3% |
| 16 | 96.6% | 35.6% | 18% |
| 17 | 98.6% | 38.3% | 21.3% |
| 18 | 97.3% | 36.9% | 23.3% |
| 19 | 97.3% | 38.3% | 27.3% |
| 20 | 99.3% | 40.3% | 16.7% |
| 21 | 98% | 39.6% | 16.7% |
| 22 | 97.3% | 42.3% | 12.7% |
| 23 | 98% | 36.9% | 19.3% |
| 24 | 96.6% | 40.9% | 23.3% |
| 25 | 98% | 34.2% | 20.7% |
| 26 | 97.3% | 37.6% | 18% |
| 27 | 98.6% | 32.2% | 14% |
| 28 | 98% | 35.6% | 22% |
| 29 | 98% | 29.5% | 21.3% |
| 30 | 95.3% | 39.6% | 22% |
| 31 | 95.9% | 32.9% | 25.3% |
| 32 | 97.3% | 35.6% | 22% |
Numbers from
Technical framing
Where the first op is computed: layer profile of the probe — Depth-1 first-op rises from chance at the embedding to ~0.99 by layer 15 and plateaus (computed in early-mid layers). Deeper compositions are represented progressively less: depth 2 plateaus ~0.4, depth 3 hovers ~0.2 -- above the 0.10 chance line but a thread. The inverse-inference the model needs to propose is computed for shallow structure and largely not computed for deep structure.
In the author’s words from the Overview · “Results”
The wall's nature changes with depth. Depth 1: first-op probe 0.99 vs model naming 0.44 / generation 0.68 → representation ≫ expression (latent). Depth 2: probe 0.42 vs behavior ~0.13. Depth 3 (the wall): probe 0.27 but shuffled floor 0.14 → real signal ~0.13 ≈ behavior. So EXPRESSION failure when shallow, REPRESENTATION failure when deep. See reports/report.md, analysis/latent_probe.png, runs/probe_results.json.
Overview
Research Program
- Program:
structured_execution_and_compilers - Program question: when the model fails to identify a composition, is the answer linearly PRESENT in its activations (latent, an expression failure) or ABSENT (a representation/information gap)?
- Prior anchors: C13/C16 (the wall is generation), C17 (coverage not selection), C18 (banking installs it).
Question
The C13–C18 arc measured the generation wall only behaviorally. Look INSIDE: can a linear probe on residual-stream activations decode the composition the model cannot generate?
Hypothesis
Pre-registered (reports/prereg.md): depth-1 probe ≥ 0.80 (methodology); depth-3 probe ≥ 3× chance and ≥ behavioral naming + 0.15 (LATENT) or ≈ chance (ABSENT); monotonic decay with depth.
Setup
- Model: Qwen3.5-4B (only permitted model). Inference only (forward passes + linear probes).
- Tasks: fresh verified-depth, collapse-rejected
list, depths 1/2/3, 500 each, disjoint. - Capture: last identification-prompt-token residual stream at every layer (
gen_lib.activations,[1500, 33, 2560]). Probe: standardize → PCA(128) → L2 logistic, stratified 70/30 held-out; decode first-op. - Baselines: chance, shuffled-label (overfit floor), layer-0 (surface), behavioral first-op naming + identification pass@1 (150 tasks/depth).
Run
Smoke: python scripts/capture.py --smoke && python scripts/probe.py Full: python scripts/capture.py --n-per-depth 500 --n-behavioral 150 --depths 1 2 3 && python scripts/probe.py && python scripts/analyze.py
Results
The wall's nature changes with depth. Depth 1: first-op probe 0.99 vs model naming 0.44 / generation 0.68 → representation ≫ expression (latent). Depth 2: probe 0.42 vs behavior ~0.13. Depth 3 (the wall): probe 0.27 but shuffled floor 0.14 → real signal ~0.13 ≈ behavior. So EXPRESSION failure when shallow, REPRESENTATION failure when deep. See reports/report.md, analysis/latent_probe.png, runs/probe_results.json.
Interpretation
Activation steering has headroom at depth 1–2 (info present, unexpressed) but nothing to steer toward at the deep wall (info not computed). Explains why banking (C18) was necessary — it installs the representation the base lacks. Only proposal-installation, not readout, crosses the deep wall. Verdict: GRADIENT/crossover.
Knowledgebase Update
- Program evidence updated:
research_programs/structured_execution_and_compilers/evidence.md(C19) - Claim ledger updated: C19 added
Artifacts
scripts/capture.py(activations + behavioral),scripts/probe.py(linear probes),scripts/analyze.pydata/{present.npy, labels.json, tasks.jsonl};data/acts.npy(241MB) moved out of repo (regenerable)runs/probe_results.json,analysis/latent_probe.png,reports/prereg.md,reports/report.md
Report
Rendered from reports/report.md
Summary
The C13–C18 arc mapped the fixed 4B's compositional generation wall entirely from the OUTSIDE (behaviorally). This experiment looks INSIDE for the first time: it trains linear probes on residual-stream activations to ask whether, when the model fails to identify a composition, the answer is linearly present but unexpressed (latent capability) or absent (a genuine information gap). 1500 verified-depth list tasks (500 each at depths 1/2/3), probing the last-prompt-token activation at every layer for the composition's first operation (the confound-robust target — hardest to read off surface I/O).
Answer: the wall's nature CHANGES with depth. At shallow depth the composition is strongly encoded but under-expressed (an expression failure = latent capability); at the deep wall the representation itself thins out (a representation failure = information gap).
| depth | linear probe (first-op) | shuffled floor | real signal | model names 1st op | model generates (ident@1) |
|---|---|---|---|---|---|
| 1 | 0.99 @L15 | 0.05 | 0.94 | 0.44 | 0.68 |
| 2 | 0.42 @L22 | 0.09 | 0.34 | 0.13 | 0.07 |
| 3 | 0.27 @L19 | 0.14 | 0.13 | 0.13 | 0.01 |
(chance ≈ 0.06–0.10; layer-0/embedding probe stays at chance, so the signal is computed, not surface.)
- Depth 1: the first op is almost perfectly linearly decodable (0.99), rising from chance at the embedding to ~0.99 by layer 15 and plateauing — yet the model names it only 0.44 of the time. Representation ≫ expression.
- Depth 2: probe 0.42 (real signal 0.34 over the shuffled floor) vs behavior ~0.13 — the first op is encoded ~3× better than the model expresses it.
- Depth 3 (the wall): probe 0.27, but the shuffled-label floor is 0.14, so the real decodable signal is only ~0.13 — roughly equal to the model's (poor) behavioral naming. The representation-expression gap has nearly closed because the representation itself has thinned to a thread.
Research Program Fit
The first mechanistic (representation-level) probe of the C13–C18 generation wall. Directly tests the project's core thesis — "unearthed potential in the weights" — and adjudicates whether steering/readout could ever cross the wall.
Method
Fresh verified-depth, collapse-rejected list tasks (depths 1/2/3, 500 each, disjoint), 8 visible examples. Render the canonical no-menu identification prompt (enable_thinking=False) and capture the residual-stream vector at the last prompt token at every layer (gen_lib.activations → [1500, 33, 2560]). Probe = per layer standardize → PCA(128) → L2 logistic regression, stratified 70/30, held-out accuracy; sweep all 33 layers, report the best. Targets: first-op name (multiclass, ~16 primitives). Behavioral baselines on 150 tasks/depth: identification pass@1 (think, greedy) and first-op naming (think, greedy).
Controls
- Shuffled-label probe at the best layer — the leakage/overfitting floor. It is ~chance at depths 1–2 (0.05, 0.09) but rises to 0.14 at depth 3 (thin-n overfitting), so depth-3 claims use probe − shuffled (0.13), not raw probe.
- Layer-0 (embedding) probe stays at chance (0.07/0.07/0.04) at every depth → the decodable signal is the model's computation, not surface I/O token statistics.
- Depth-1 positive control (probe 0.99) validates the methodology.
- Confound-robust target: first-op (buried under later ops at depth ≥ 2) rather than easily-surface-detectable presence. (A presence probe was run but is uninformative — with ~19% base rate its raw accuracy tracks the majority-"absent" baseline; excluded from claims.)
Pre-registered verdicts
- P1 (methodology valid): HELD — depth-1 probe 0.99 ≥ 0.80; shuffled ≤ chance+0.05 at depths 1–2 (0.14 at depth 3 is the one exceedance, handled via probe−shuffled).
- P2 (the test at depth 3): MARGINAL — probe 0.27 is 2.7× chance (just below the pre-set 3×) and clears layer-0 by +0.23; not "absent" (probe−shuffled = 0.13 > 0), but a thin signal.
- P3 (representation ≫ expression = latent): HELD at depths 1–2 (gap +0.55, +0.30 over naming), NEARLY CLOSES at depth 3 (probe−shuffled 0.13 ≈ naming 0.13). The latent gap is real but concentrated at shallow depth.
- P4 (gradient): HELD — probe first-op decays monotonically 0.99 → 0.42 → 0.27 with depth, mirroring the behavioral wall.
Overall verdict: GRADIENT / crossover, not a clean LATENT or ABSENT.
Interpretation
- The generation wall is not one thing — it changes character with depth. For shallow structure the model has computed the inverse (the first op is strongly, linearly present in mid-network) but does not express it — a readout/routing failure, i.e. genuine latent capability. For deep compositions the model has not computed the inverse — the representation thins to near the overfitting floor — a real information/support gap.
- This adjudicates the steering hope honestly: activation steering (adding the probe direction at generation time) has real headroom at depth 1–2, where the info is present but unexpressed; but at the true wall (depth 3+) there is almost nothing to steer toward — the composition simply is not encoded. No clever readout conjures information the forward pass never computed.
- It explains WHY banking (C18) was necessary and worked. C18 showed banking expands the depth-2 coverage ceiling on held-out tasks. C19 shows the base model's depth-2 representation of the composition is weak (0.42) and its depth-3 representation is a thread (0.13). Banking installs the representation the base lacks — which is why only proposal-installation (banking / tools, C18 / C12), not test-time readout, crosses the deep wall. The two claims lock together: the wall is representational at depth, and banking is how you add the missing representation.
- Refines C13/C16 ("the model can't propose"): the mechanism is that inverse-inference is computed for shallow structure but not routed to output, and simply not computed for deep structure — a depth-graded mixture of expression and representation failure.
Honesty notes
- "Linearly decodable" shows the info is present, not that the model can route it into a correct program; the depth-1 gap is partly a naming-task artifact (the model does use the first op to generate at 0.68). The cleanest latent signal is the middle regime (depth 2): strongly encoded (0.42), barely expressed (0.13).
- Depth-3 shuffled floor at 0.14 caps confidence in the depth-3 residual; the honest statement is "thin but non-zero (probe−shuffled 0.13, ~4 SE above the floor)."
- Single substrate (list), single probe family (linear), last-token position. A steering experiment is the decisive follow-up (does adding the probe direction raise identification?).
Next Experiments
- Activation steering: add the depth-2 first-op probe direction to the residual stream at generation time; does identification / coverage rise? (Tests usability of the latent signal — the LATENT verdict's real test.)
- Full-composition probe: decode op-2 and op-3, not just op-1, to map how much of the whole pipeline is latent vs the first step only.
- Probe the banked model (C18): does banking raise the depth-2/3 first-op probe accuracy? (Direct test that banking installs the missing representation.)
Artifact Manifest
See reports/artifact_manifest.yaml. Key: scripts/capture.py, scripts/probe.py, scripts/analyze.py, data/{acts.npy, present.npy, labels.json, tasks.jsonl}, runs/probe_results.json, analysis/latent_probe.png. data/acts.npy (~200MB activations) is omitted from git; regenerate via scripts/capture.py.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/probe_results.json2.6 kB
Reproduce
Smoke test
python scripts/capture.py --smoke && python scripts/probe.pyFull run
python scripts/capture.py --n-per-depth 500 --n-behavioral 150 --depths 1 2 3 && python scripts/probe.py && python scripts/analyze.pyRun steps are documented inside the experiment folder (README and scripts).