Qwen3.5-4B: Is the Parameter Latent? (probe the full first op)
The one idea you need
A mechanic hears an engine and instantly diagnoses "needs an oil change" — real expertise, held in the head. But the exact oil grade? They read it off the cap. This model transforms lists of numbers, diagnosing each step's KIND internally yet reading its NUMBER off the examples.
The question
When a small model transforms a list of numbers step by step, does it hold each step's exact settings inside its weights, or just read them off the examples?
What we found
It splits. The model genuinely computes the KIND of operation inside itself — reading its internal activity names the kind far better than the examples alone do (41% versus 27%, against 6% for blind guessing). But the specific number is not stored anywhere special: a trivial calculator run over the examples recovers it just as well (49% versus 53%). The model reads it off, same as anyone could.
Why it matters
Before mining a model's internals for hidden knowledge, test whether the fact is simply readable off the input-output examples. Derive those parts cheaply outside the model — faster and more accurate — and probe internals only for what it truly computes.
On this page
Results at a glance 2
How to read
Three decode targets run along the bottom: the operation's kind, the full operation, and just its number. Bar height is decode accuracy — taller is better. One color reads the model's internal activity, the other a no-model classifier on the raw examples.
Takeaway → For the kind, the model's bar towers over the examples bar; for the number, the examples bar matches or beats it — the kind is computed inside, the number just read off.
Data table
| condition | model probe (4B residual) | external I/O classifier (no 4B) |
|---|---|---|
| op-TYPE | 41.3% | 27.2% |
| CONCRETE (op+param) | 25.8% | 16.3% |
| PARAMETER | type | 49.3% | 52.9% |
Numbers from
Technical framing
Is the parameter latent? The op-TYPE is model-computed (probe>surface); the PARAMETER is surface-readable (surface>=probe) — C30 found the deployable bottleneck is the concrete first op's PARAMETER, not the op-TYPE C19 decodes. Is the parameter model-LATENT or surface-readable? The real surface control is an external classifier on raw I/O features (list lengths, sums, min/max, elementwise diffs) with NO 4B forward pass (the layer-0 probe is degenerate: RoPE makes the fixed-template last-token embedding constant across tasks). FINDING: the op-TYPE is genuinely MODEL-LATENT -- the residual probe (0.41) beats the surface classifier (0.27). But the PARAMETER given the type is SURFACE-READABLE -- the trivial I/O classifier (0.53) decodes it as well as / better than the model residual (0.49), both above chance (0.30). So the forward pass COMPUTES the op-type (latent capability, elicitable via C30's externalization) but only READS the parameter off I/O magnitudes -- no privileged model knowledge to elicit. Retroactively strengthens C19/C30: the external-I/O baseline is the correct surface control and confirms the op-type is model-latent.
How to read
Six hint conditions run along the bottom: no hint, told only the kind, told the full operation, the model-decoded operation, the examples-decoded operation, and told a wrong number. Bar height is solve rate — one color a single best attempt, the other solved within six tries. Higher is better.
Takeaway → Only the full operation spikes; kind-only and wrong-number sit near zero — and the cheap examples-based hint outdoes the model's own decoded guess.
Data table
| condition | coverage@6 | greedy@1 (deployable) |
|---|---|---|
| no-hint | 0.7% | 0% |
| oracle_type | 3.4% | 0.7% |
| oracle_full | 16.9% | 9.5% |
| probe_full | 3.4% | 1.4% |
| surface_full | 5.4% | 2.7% |
| wrong_param | 0.7% | 0% |
Numbers from
Technical framing
Deployability on PARAM-first-op tasks: the full op deploys (oracle_full >> oracle_type), but the cheap surface pipeline >= the model probe — Externalize the decoded full first op as a prompt hint, on PARAM-first-op tasks (where the parameter matters). The parameter IS the deployable bottleneck: oracle_full (true op+param) 0.095 >> oracle_type (type only) 0.007 -- confirms C30, isolated to param tasks. But the model probe barely delivers (probe_full 0.014) and the CHEAP surface pipeline delivers MORE (surface_full 0.027): you do not need the 4B for the parameter. wrong_param (true type, wrong param) 0.000 -- content-causal. Two-term check textbook clean: probe_full deploys EXACTLY like the oracle on the tasks it decodes correctly (0.091=0.091) and like no-hint on those it gets wrong (0.0=0.0) -- a faithful readout bounded by 26% concrete accuracy. Sharp localization: the model computes the op-type but only reads the parameter off surface I/O.
In the author’s words from the Overview · “Results”
The op-TYPE is MODEL-LATENT (probe 0.41 > surface 0.27) but the PARAMETER is SURFACE-READABLE (probe 0.49 vs surface 0.53). Deployability: the param is the bottleneck (oracle_full 0.095 >> oracle_type 0.007) but the cheap surface pipeline (0.027) delivers more than the model probe (0.014). See reports/report.md, analysis/probe_the_parameter.png.
Overview
Research Program
- Program:
interpretability_and_diagnostics/structured_execution_and_compilers - Question (C30 follow-up): C30 found the deployable bottleneck is the concrete first op's PARAMETER, not the op-TYPE C19 decodes. Is the parameter model-LATENT (elicitable) or just surface-readable off the I/O?
Setup
- Fit 16-way op-TYPE + 32-way CONCRETE-op probes on residual activations (600/depth training).
- REAL surface control (the layer-0 probe is degenerate under RoPE): an external classifier on raw I/O features (lengths, sums, min/max, elementwise diffs) with NO 4B.
- Decodability on a large fsig-disjoint eval (activation-only); deployability arms (n=130/depth) split by param vs non-param first ops: no-hint, oracle_type, oracle_full, probe_full, surface_full, wrong_param.
Run
python scripts/fit_probe_full.py --n-per-depth 600 then decode_eval.py (decodability + surface baseline) then run_hints_full.py (deployability) then analyze.py.
Results
The op-TYPE is MODEL-LATENT (probe 0.41 > surface 0.27) but the PARAMETER is SURFACE-READABLE (probe 0.49 vs surface 0.53). Deployability: the param is the bottleneck (oracle_full 0.095 >> oracle_type 0.007) but the cheap surface pipeline (0.027) delivers more than the model probe (0.014). See reports/report.md, analysis/probe_the_parameter.png.
Interpretation
Sharp localization: the forward pass COMPUTES the op-type (latent, elicitable) but only READS the parameter off surface I/O -- no privileged model knowledge to elicit. The training-free latent-elicitation ceiling is the op-type.
Knowledgebase Update
- Claim ledger: C31
Artifacts
scripts/fit_probe_full.py,scripts/decode_eval.py(external-I/O surface baseline),scripts/run_hints_full.py,scripts/analyze.py,scripts/capture.pydata/{train_fsigs,concrete_vocab}.json,runs/{decode_results,full_results,verdict}.json,analysis/probe_the_parameter.png,reports/{report,design_review}.md- Probe pickle + activations moved out of repo.
Report
Rendered from reports/report.md
Motivation
C30 found the DEPLOYABLE bottleneck for depth-2 elicitation is the concrete first op's parameter (oracle-full lifts greedy@1 6×, oracle-type does not; the C19 type-only probe nets to zero). This asks the sharp question: is the parameter model-latent (elicitable, like the op-type C19 found) or merely surface-readable off the I/O examples? If latent → a full-op probe-hint delivers the lift training-free; if not → that pinpoints exactly what the forward pass computes (the type) vs reads off surface (the param).
Method
Fit two probes per depth on 600/depth training activations (last-prompt-token residual): a 16-way op-TYPE probe (C19) and a 32-way CONCRETE-op probe (op+param). The review caught a critical flaw: the layer-0 probe is a degenerate surface control — activations are at the fixed template tail, and RoPE makes the embedding-layer vector identical across tasks, so "layer-0 ≈ chance" is meaningless. The real control is an external classifier on raw I/O features (list lengths, sums, min/max, elementwise diffs, sortedness) with no 4B forward pass. Decodability on a large fsig-disjoint eval (600/depth, activation-only); deployability arms (n=130/depth) split by param vs non-param first ops.
(A) Decodability: model probe vs external-I/O surface baseline (depth-2, fsig-disjoint)
| target | model probe (residual) | external I/O (no 4B) | chance |
|---|---|---|---|
| op-TYPE | 0.413 | 0.272 | ~0.06 |
| CONCRETE (op+param) | 0.258 | 0.163 | ~0.03 |
| PARAM \| type | 0.493 | 0.529 | 0.303 |
- The op-TYPE is genuinely MODEL-LATENT — the residual probe (0.413) clearly beats the surface classifier (0.272). The model computes the first-op type (C19, now with a proper baseline).
- The PARAMETER given the type is SURFACE-READABLE — a trivial I/O classifier (0.529) decodes it as well as / better than the model residual (0.493). No privileged model knowledge: the param is just read off I/O magnitudes (e.g.
add_k(3): sum(out)−sum(in)=3·len). - Depth-3: op-type probe 0.193 ≈ surface 0.198 — at the wall the type is no longer model-latent either (the "thread", C19).
(B) Deployability on PARAM-first-op tasks (no-think, n=148)
| arm | greedy@1 | cov@6 |
|---|---|---|
| no-hint | 0.000 | 0.007 |
| oracle-type (type only) | 0.007 | 0.034 |
| oracle-full (op+param) | 0.095 | 0.169 |
| probe-full (32-way probe) | 0.014 | 0.034 |
| surface-full (external I/O) | 0.027 | 0.054 |
| wrong-param (true type, wrong param) | 0.000 | 0.007 |
- The parameter IS the deployable bottleneck (isolated to param tasks): oracle-full 0.095 ≫ oracle-type 0.007. Confirms C30.
- The model probe barely delivers, and the cheap surface pipeline delivers MORE: probe-full 0.014 < surface-full 0.027 — you do not need the 4B for the parameter. wrong-param 0.000 (content-causal).
- Two-term check (textbook clean): probe-full deploys exactly like the oracle on tasks it decodes correctly (0.091 = 0.091) and like no-hint on those it gets wrong (0.0 = 0.0) — a faithful readout that simply can't decode enough (26% concrete accuracy).
- (Non-param tasks: surface-full HURTS (0.009 < no-hint 0.045) — the surface classifier is bad at the structural op-types the model computes, consistent: surface is good at magnitudes/params, bad at model-latent types.)
Implication
Sharp localization of the wall and of C30: the forward pass genuinely computes the op-type (a real latent capability, elicitable training-free via C30's externalization) but has no privileged representation of the parameter — it just reads it off surface I/O, which any trivial classifier does equally. So the training-free latent-elicitation ceiling is the op-type; the parameter is not a model-latent thing to unearth. This closes the C30 loop: the "deployable bottleneck" (param) is real but is not a locked-in-the-weights capability — it's feature-engineering the model isn't special at.
Next
- Compose type-probe (model) + param-from-I/O (surface) into one hint pipeline — does the hybrid deliver oracle-full's lift? (both pieces are cheap/training-free.)
- Depth-3: nothing to elicit (type also surface-parity at the wall).
Artifact Manifest
See reports/artifact_manifest.yaml. Probe pickle + activations moved out of repo.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
python scripts/fit_probe_full.py --n-per-depth 60 && python scripts/decode_eval.py --n-per-depth 60Full run
fit_probe_full 600; decode_eval 600; run_hints_full 130; analyzeRun steps are documented inside the experiment folder (README and scripts).