Qwen3.5-4B Thinking Separability Probe

Results at a glance 3

Overview

Research Program

• Program: test_time_reasoning_budget
• Program question: is the native-thinking gain genuine reasoning, or compute/scaffold? And can the model's internal state predict its own correctness (attacking C2 from inside the model)?
• Prior anchors: qwen35_4b_thinking_budget_scaling (the shuffle control showed much of the gain is compute/scaffold, and said it could not isolate the coherent-reasoning contribution); qwen35_4b_thinking_budget_controller (bounded by visible-test false-passes — C2); the corpus's synthetic hidden-state probes (qwen_readable_candidate_verifier, qwen_candidate_conditioned_trace_verifier, qwen_prefix_state_process_verifier).

Question

Does native thinking make the model's own correctness more linearly decodable from its activations? Concretely: train a linear probe on the answer-token hidden state of Qwen3.5-4B's own generated MBPP solution (labels = execution pass) and compare its per-layer AUC across no_think, real think, and shuffled-think at matched budgets. "Does it know it's right better after reasoning?"

Hypothesis

The shuffle puzzle left two possibilities. If real thinking raises probe separability beyond shuffled-think (which preserves token-count/scaffold but destroys coherent order), that is the genuine-reasoning-content signal the scaling report could not isolate — reasoning reorganizes the model's internal state toward a more decodable correctness representation. If real ≈ shuffled, then thinking's effect on internal "knowing" is also mostly compute/scaffold, reinforcing C9.

Setup

• Model: Qwen3.5-4B (frozen, bf16, sdpa, fast path enabled). 32 layers → 33 hidden states.
• Dataset: MBPP sanitized test, first 100 tasks; k=8 sampled solutions per task per condition.
• Conditions: no_think, think_512, shuffle_512, think_1024, shuffle_1024 (s1-style budget forcing on </think>; shuffle = permute the model's own thinking tokens before the answer).
• Signal: the answer-token (last-token) hidden state of the model's own generated sequence (prompt + thinking + answer), extracted per layer with a clean right-padded forward pass (keeps the linear-attention recurrence uncorrupted).
• Probe: per-layer logistic regression (standardized), GroupKFold by task (no task-identity leakage), out-of-fold AUC for predicting full-test pass; bootstrap CI by resampling tasks.
• Controls: shuffled-label probe (must be ~0.5); think vs shuffle at matched budget; per-layer sweep.
• Deployable / oracle boundary: probe trained on hidden-test outcomes is a non-deployable diagnostic of decodability. The deployable angle is the false-pass test: among candidates that pass the visible test (what the controller commits), can the probe rank the true full-test passes above the C2 false-passes?

Run

Smoke (4 tasks, 2 conds): ../../.venv/bin/python scripts/run.py --smoke

Full:

HF_HUB_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  ../../.venv/bin/python scripts/run.py --tasks 100 --k 8      # generate + extract activations
../../.venv/bin/python scripts/verify_probe.py                 # torch-free execution labels
../../.venv/bin/python analysis/probe.py                       # per-layer probes + figure

Results

Full results in reports/report.md; figure analysis/auc_vs_layer.png. Best-layer probe AUC (predict full-test pass from the answer-token activation; n=100, 800/cond):

1. Correctness is moderately decodable from one answer-token activation (AUC 0.64–0.76 vs shuffled-label ~0.50) — "models know more than they say," novel on self-generated real code.
2. Thinking raises decodability across essentially every layer (no_think ~0.5–0.64 → thinking ~0.67–0.75): "knows it's right better after thinking."
3. But not via reasoning — hypothesis falsified. Shuffled thinking matches/exceeds real thinking at both budgets and across all layers. So the active ingredient is compute/scaffold/ token-presence, not coherent content — converging with the behavioral C9 shuffle finding at the representational level.
4. Deployable spinoff: among visible-passers (C2 false-pass regime), the probe has weak signal under thinking (~0.60–0.68) but ~chance under no-think.

Interpretation

The cleanest evidence yet that this 4B's thinking benefit is largely not coherent reasoning — now shown at two independent levels (behavioral accuracy C9 + internal representation). Thinking makes the model more internally aware of its own correctness, but scrambled thinking does so equally. A weak verifier-free signal (probe vs C2 false-passes) appears only under thinking. Caveats: moderate AUCs, n=100 single seed, real-vs-shuffle within overlapping CIs (robust claim is the across-depth ordering), last-token probe only.

Knowledgebase Update

• Program evidence updated: yes (research_programs/test_time_reasoning_budget/evidence.md).
• Claim ledger updated: C9 extended (reasoning-vs-compute confirmed at the representational level).

Artifacts

• src/probe_lib.py (generation + activation extraction), src/tasks.py (MBPP + sandbox verifier).
• scripts/run.py (extract), scripts/verify_probe.py (labels), analysis/probe.py (probes + figure).
• data/records.jsonl, data/labels.jsonl, data/tasks.json (small, in-repo).
• Activations (~0.7 GB) in large_artifacts/qwen35_4b_thinking_separability_probe/ (external, gitignored; regenerable by scripts/run.py) — see reports/artifact_manifest.yaml.

Report

Rendered from reports/report.md

Summary

We probe whether Qwen3.5-4B's answer-token hidden state linearly encodes whether its own generated MBPP solution is correct, and whether thinking makes that correctness more decodable. Three results. (1) Correctness IS moderately linearly decodable from a single answer-token activation (best-layer AUC ~0.64–0.76; shuffled-label control ~0.50) — a "models know more than they say" effect, here on a self-generated real-code answer token, which prior corpus probes never touched. (2) Thinking robustly raises decodability at essentially every layer (no_think ~0.5–0.64 → thinking ~0.67–0.75): the model "knows it's right better after thinking." (3) But that increase is not coherent reasoning — shuffled thinking matches or exceeds real thinking on separability at both budgets and across all layers, refuting our pre-registered hypothesis. This converges with the behavioral shuffle finding (C9) at the representational level: thinking's benefit, behaviorally and now internally, is largely compute/scaffold/token-presence, not logical content. A modest deployable spinoff: among visible-test passers (the C2 false-pass regime), the probe has weak signal under thinking (AUC ~0.60–0.68) but ~chance under no-think.

Research Program Fit

Third experiment of test_time_reasoning_budget. It turns C9's "reasoning vs compute" debate into a measurable interpretability quantity (probe AUC by layer, think vs shuffle vs no-think) and gives an internal-signal angle on the C2 selection bottleneck. The pre-registered hypothesis — that real thinking would raise separability above shuffled thinking and thereby isolate the genuine-reasoning contribution — is falsified, which is itself the durable lesson.

Method

• Model Qwen3.5-4B frozen (bf16, sdpa, fast path). MBPP sanitized test, 100 tasks, k=8 sampled solutions per task per condition: no_think, think_512, shuffle_512, think_1024, shuffle_1024.
• Signal: the last-token hidden state of the model's own generated sequence (prompt + thinking + answer), extracted per layer (33 states: embeddings + 32 layers) via a clean right-padded forward pass (so the linear-attention recurrence is uncorrupted by padding).
• Probe: per-layer standardized logistic regression, GroupKFold by task (no task-identity leakage), out-of-fold AUC predicting full-test execution pass; bootstrap CI by resampling tasks.
• Controls: shuffled-label probe (must ≈0.5); real vs shuffled thinking at matched budget.

Results

Best-layer probe AUC for predicting full-test pass (n=100 tasks, 800 samples/condition):

Per-layer AUC (every 4th layer) — the pattern is robust across depth, not a best-layer artifact:

Figure: analysis/auc_vs_layer.png.

Finding 1 — correctness is moderately decodable from one answer-token activation

All conditions beat the shuffled-label control (~0.50) with best-layer AUC 0.64–0.76. So the model's residual stream linearly encodes whether its own just-written solution is correct — a "models know more than they say" effect, novel on a self-generated real-code answer token (prior corpus probes read external candidates on synthetic tasks). Decodability is highest in early–mid layers (best layers 6–16 for thinking conditions), not the final layer.

Finding 2 — thinking robustly raises decodability

Every thinking condition sits above no_think at essentially every layer (thinking ~0.67–0.75 vs no_think ~0.49–0.64). The model's correctness becomes more internally decodable once it has produced thinking tokens — it "knows it's right better after thinking," consistent across depth (so not a best-layer fluke, even though individual best-layer CIs overlap no_think's).

Finding 3 — but it is NOT coherent reasoning (hypothesis falsified)

Shuffled thinking matches or exceeds real thinking on separability at both budgets and across all layers (shuffle_512 ≥ think_512; shuffle_1024 is the single highest curve). The pre-registered hypothesis — real > shuffled would isolate genuine reasoning — is refuted. Whatever makes correctness more decodable after thinking is the presence/length/compute of the thinking region, not its coherent content. This converges with the behavioral shuffle finding (C9: behavioral shuffled-thinking reproduced much of the accuracy gain) — now mirrored at the representational level.

Controls

Shuffled-label probes give AUC 0.46–0.52 (≈ chance) in every condition → the probes are not exploiting task-identity or pipeline leakage (GroupKFold splits by task). The real-vs-shuffled comparison is the mechanism control; the consistent shuffle ≥ real ordering across depth is the load-bearing result (individual best-layer CIs overlap, so we claim "real does not exceed shuffled", not "shuffled is significantly higher").

Oracle Versus Deployable Evidence

The probe is trained on hidden-test pass labels and is a non-deployable decodability diagnostic. The deployable angle is the visible-passer column: among candidates that pass the visible test (what the budget controller commits), can the probe rank true full-test passes above the C2 false-passes? Under thinking the probe has weak-moderate signal (AUC ~0.60–0.68); under no-think it is ~chance (0.518). So an internal probe can partially flag the C2 false-passes the visible test misses — but only once the model has thought, and not strongly enough to be a standalone selector.

Interpretation

This is the cleanest evidence yet that, for this 4B on this task, native thinking's benefit is largely not coherent reasoning — the conclusion now holds at two independent levels: behavioral accuracy (C9 shuffle control) and internal representation (this probe). Thinking does make the model more internally "aware" of its own correctness, but scrambled thinking does so equally, so the active ingredient is compute/scaffold/token-presence. The result also yields a weak verifier-free signal (internal probe vs C2 false-passes) that only appears under thinking.

Limitations

• AUCs are moderate (0.64–0.76), n=100 single seed, and best-layer real-vs-shuffle differences are within overlapping CIs; the robust claim is the across-depth ordering (shuffle ≥ real, thinking > no-think), not precise magnitudes.
• Probes read only the last token; other positions/poolings (e.g. mean over the answer, or a Kadavath P(IK) probe token) could differ.
• One model, one benchmark; MBPP is coverage-limited and likely partly in pretraining (see the contamination direction in the program backlog).

Next Experiments

• Foreign-task-thinking arm (remove token-presence, not just order): does it drop separability below shuffled? If even foreign thinking holds separability up, the active ingredient is pure compute.
• Probe-position sweep (post-</think>, mean-pooled answer, P(IK) probe token) and an MLP probe.
• Combine the internal probe with the visible test as a controller signal (does it lift the controller past the 0.91 deployable / 0.93 oracle wall?).
• Replicate on a contamination-controlled / harder substrate where the no-think baseline is weaker (more headroom for a reasoning-content effect to appear, if it exists).

Artifact Manifest

See artifact_manifest.yaml. Activations (~0.7 GB) are external/regenerable; small records, labels, probe results, and the figure are in-repo.

Experiment log 5

Figures 1

Data files 1

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

• previewruns/probe_results.json3.8 kB

Reproduce

Smoke test

../../.venv/bin/python scripts/run.py --smoke

Full run

HF_HUB_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True ../../.venv/bin/python scripts/run.py --tasks 100 --k 8

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Cited by claims

• ◑ Promising C9 · Native thinking's accuracy gain is coherent reasoning content at every budget (unused lever)

Related experiments

Nearest by shared name, tags, and programs.

• 2026-06-30 Qwen3.5-4B Thinking-Budget Scaling
• 2026-07-13 Qwen3.5-4B: The Compute-Optimal Confidence Policy (select + abstain + escalate)
• 2026-07-10 Qwen3.5-4B Verified Macro Invention Long-Context Rerun
• imported 2026-07-12 Qwen Register Trace Refiner
• imported 2026-07-12 Qwen Progressive Repair Compiler
• 2026-06-23 Qwen Prefix-State Process Verifier