Qwen3.5-4B Native-Thought Jacobian Value Transport
The one idea you need
The plan: pause a small model partway through its own written scratch-work, read a hidden gauge of whether it is headed for the right answer, then edit just that gauge to rescue a failing attempt. Like reading a runner's pace mid-race and nudging them — but only if they cross the finish line.
The question
Can we peek at a hidden "am I on track" signal partway through a small model's reasoning, then edit it to steer the final answer?
What we found
We could not even reach the test. The whole plan needs the model to finish reasoning and write an answer worth grading. But on all 48 attempts at simple two-step list puzzles, it rambled straight into its 160-token thinking limit and never concluded. Zero natural finishes. That is a setup-and-budget wall, not proof the hidden gauge is missing.
Why it matters
Before building any probe that reads or edits a small model's mid-reasoning confidence, first measure how often it actually finishes thinking and answers within your token budget. If it never stops on its own, nothing downstream is measurable.
On this page
Results at a glance 1
How to read
Bars show the share of all 48 reasoning attempts that closed thinking naturally, gave a readable answer, gave a correct answer, or hit the token cap. Higher is better for the first three; the cap bar should be low.
Takeaway → The first three bars sit flat at zero while the cap bar fills the full height — every attempt exhausted its thinking budget before finishing, so no signal could be measured.
Data table
| seam outcome | share of 48 traces |
|---|---|
| natural close | 0% |
| parseable answer | 0% |
| exact success | 0% |
| thought-cap contact | 100% |
Numbers from experiments/qwen35_4b_native_thought_jacobian_value_transport/analysis/seam_metrics.json
Technical framing
Natural-thinking seam gate — Sixteen fresh tasks, three traces each, under the frozen 160-token thought cap. All 48 stopped as think_cap_without_close; value and causal stages were canceled.
In the author’s words from the Overview · “Results”
Terminal frozen decision: NO_NATURAL_SEAM. The generator produced 16 seam, 32 value-fit, and 32 causal-confirmation tasks: 80/80 unique fingerprints, zero overlap with the direct Jacobian parent, balanced identifiable first-operation targets, and the exact frozen lens hash. The two-task model smoke validates revision, token IDs, one-token aliases, full 24-rank dictionaries, cache-free generation, and finite J coordinates without recording correctness. Both traces hit the frozen 160-token cap without natural close, and historical-token activations changed by up to 0.0625 across suffix lengths, so causal invariance currently fails. These are scientific seam/control risks. … Read the full result →
Overview
This experiment asks whether a task-general scalar continuation-value coordinate inside the replicated 24-token J space is causally consumed from a natural token inside Qwen3.5-4B's own <think> span.
Research Program
- Primary:
interpretability_and_diagnostics. - Conditional after an oracle causal pass:
test_time_reasoning_budgetandstructured_execution_and_compilers. - Direct parent:
qwen35_4b_jacobian_transport_control_replication, terminalREPLICATED_J_TRANSPORTon fresh lookup mappings with exact controls. - Closest canceled design:
qwen35_4b_jacobian_value_transport, whose native prefix stages were correctly forbidden when its late lens did not transport. - Other anchors:
qwen35_4b_activation_steering,qwen35_4b_probe_to_prompt,qwen35_4b_thinking_separability_probe, andqwen35_4b_prefix_value_guided_search.
Question
Can continuation success be decoded from J coordinates at natural thought prefixes and then changed by transferring only one learned certainty coordinate, rather than copying an answer identity or merely sampling again?
Fixed design
- Only
Qwen/Qwen3.5-4B, revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Exact replicated 24-concept lens SHA-256
e373b6e93956fdfc5cb446e9bee8249655707c8258a7868f0653d11f1ffd0213. - Frozen band
[4,5,6,7,8], natural thought checkpoints at 0.33/0.67, batch-one cache-free full-prefix recomputation. - Eighty fresh, parent-disjoint procedural depth-2 list tasks. Exhaustive CPU enumeration guarantees the visible I/O has one identifiable first-operation type. A fixed one-token alias mapping connects the 12 operation types to lens concepts; 11 types are valid targets.
- Prefix value is the fraction of three disjoint-seed natural continuations that answer the correct first-operation alias. Whole-trace labels are not assigned to tokens.
- Primary G1 signal is a held-out-by-task scalar readout of concatenated J coordinates. Primary G2 intervention transfers only that scalar coordinate.
- Two exact post-bf16 random arms, shuffled value axis, answer-identity J clamps, logit-lens, raw donor, J/non-J donor decomposition, ActAdd, and wrong-task donor controls are frozen.
The full gates are in preregistration.md, and the 24-threat pre-run review is in design_review.md.
Run
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m pytest \
experiments/qwen35_4b_native_thought_jacobian_value_transport/tests -q
.venv/bin/python experiments/qwen35_4b_native_thought_jacobian_value_transport/scripts/run.py --stage smoke
.venv/bin/python experiments/qwen35_4b_native_thought_jacobian_value_transport/scripts/run.py --stage model-smoke
.venv/bin/python experiments/qwen35_4b_native_thought_jacobian_value_transport/scripts/run.py --stage seam-calibration
.venv/bin/python experiments/qwen35_4b_native_thought_jacobian_value_transport/scripts/run.py --stage prefix-value
.venv/bin/python experiments/qwen35_4b_native_thought_jacobian_value_transport/scripts/run.py --stage control-calibration
.venv/bin/python experiments/qwen35_4b_native_thought_jacobian_value_transport/scripts/run.py --stage causal-confirmationEvery stage is fatal-gated. Model smoke and seam calibration are implemented; later stages still refuse placeholders until their audited implementations land.
Results
Terminal frozen decision: NO_NATURAL_SEAM. The generator produced 16 seam, 32 value-fit, and 32 causal-confirmation tasks: 80/80 unique fingerprints, zero overlap with the direct Jacobian parent, balanced identifiable first-operation targets, and the exact frozen lens hash.
The two-task model smoke validates revision, token IDs, one-token aliases, full 24-rank dictionaries, cache-free generation, and finite J coordinates without recording correctness. Both traces hit the frozen 160-token cap without natural close, and historical-token activations changed by up to 0.0625 across suffix lengths, so causal invariance currently fails. These are scientific seam/control risks.
The frozen 16-task seam confirmed the failure on all 48 traces:
- natural close: 0/48;
- parseable final alias: 0/48;
- exact success: 0/48;
- mixed correct/incorrect tasks: 0/16; and
- thought-cap contact: 48/48 at exactly 160 tokens (7,632 cache-free forwards).
The natural-close/parse/headroom gates fail, so prefix-value fitting, numeric controls, and causal confirmation are canceled. This is an interface-budget failure, not evidence that a J-space value coordinate is absent. The result is preserved rather than raising the budget in place.
Scope
Had the seam passed, ground-truth continuation outcomes would have fit the value readout and selected causal donors, making later evidence oracle-only. Those stages did not run. Any successor still requires a separate non-oracle controller to beat frozen and matched-compute sampling on new tasks.
Knowledgebase Update
- Idea intake and decision record connect this to the replicated mechanism.
- Program evidence changes only at a scientific gate.
- No claim number is reserved while the repository claim re-grade remains open.
Artifacts
All procedural data, receipts, frozen weights, control geometry, and reports are self-contained here. No adapter or external artifact is used at the design stage.
Report
Rendered from reports/report.md
Status
Terminal NO_NATURAL_SEAM; value and causal stages canceled.
Purpose
Transfer the independently replicated context-local J mechanism from a prompt-local lookup token to a natural native-thinking token, while separating a task-general scalar continuation-value coordinate from full answer identity.
Frozen method
- Fresh, first-operation-identifiable procedural list tasks only.
- Frozen replicated 24-token lens and layer band 4--8.
- Natural close, full-prefix batch-one Transformers recomputation without cache.
- Continuation-defined prefix value, held-out-by-task linear J-space readout, and a minimum-norm scalar coordinate clamp.
- Exact quantization-aware random controls and explicit identity, shuffled-axis, logit, raw, J/non-J, ActAdd, and wrong-donor arms.
CPU evidence
All 80 generated tasks are fingerprint-unique and disjoint from the direct Jacobian parent. Exhaustive enumeration certifies one visible-data-consistent first-operation type. The CPU adversarial pass rejected negate as a target because its two-step compositions are algebraically reorderable; it remains a prompt distractor/second operation. This prevents alternate valid explanations from being counted as model failures.
Current inference boundary
Revision, token, alias, lens rank, generation, capture, and J-coordinate plumbing pass. Both two-task smoke traces reached the frozen 160-token cap without natural close. Historical-token activations also changed by 0.0625 when evaluated under different suffix lengths, above the frozen causal-invariance threshold. The smoke itself was not a scientific sample and recorded no correctness.
Frozen seam result
The 16-task, three-trace-per-task scientific seam was opened once with unchanged sampling and the frozen 160-token thought cap.
| metric | result | gate |
|---|---|---|
| Natural close | 0/48 (0.0%) | >=80% |
| Parseable alias | 0/48 (0.0%) | >=90% |
| Exact alias success | 0/48 (0.0%) | 5%--95% |
| Mixed-value tasks | 0/16 | >=6 |
| Thought-cap contact | 48/48 | diagnostic |
Every row stopped as think_cap_without_close, with exactly 160 thought tokens. The full-recompute runner made 7,632 forwards in 389.9 seconds. There is no natural reachable prefix-to-answer seam at this budget and prompt grammar.
Frozen decision
Decision: NO_NATURAL_SEAM. Per preregistration, G1 value fitting, post-bf16 control calibration, donor selection, and causal confirmation are ineligible. No outcome from the untouched 32 causal tasks was opened.
Interpretation
The failure precedes the J-space hypothesis. The model continued reasoning until the cap on every trace, so assigning value at 0.33/0.67 and demanding a natural answer continuation would evaluate prefixes whose answer seam is never reached under the allowed budget. The 0.0625 historical-activation length sensitivity also warns that later patching needs per-length dynamic controls rather than a fixed raw-activation invariance assumption.
A separate successor may preregister a natural-close budget ladder on a selection split, freeze the smallest viable cap, and match control geometry at every live sequence length. Raising 160 or weakening invariance in this result-bearing experiment is forbidden.
Oracle boundary
Continuation labels and high/low donor selection are hidden-label oracle inputs. Even a terminal causal pass cannot support a capability claim or replace the required learned non-oracle matched-sampling experiment.
Experiment log 9
Show the running log (9 entries, 2026-07-12)
Scaffold
Created as a new experiment scaffold.
2026-07-12 — intake and adversarial design
- Routed through
make related; named five close prior lines and the canceled native-prefix stages ofqwen35_4b_jacobian_value_transportas the nearest duplicate. - Froze the replicated lens, band 4--8, natural prefix fractions, full-recompute backend, scalar value-coordinate formula, staged gates, and control family.
- CPU enumeration found
negate-first depth-2 tasks non-identifiable due to algebraic reorderings. Excluded it from target support before model calls while retaining it as a distractor/second operation. - Generated 80 unique fresh tasks with zero parent overlap and a one-type visible first-operation certificate.
- Completed the 24-threat adversarial review before implementation or any model call. Current state is design/CPU smoke only.
2026-07-12 — immutable design boundary
- Rebased onto current
origin/main, then froze design commitb87b67f28586687954c89ba653d22cafe93d6073. - Frozen README SHA-256:
221992f5ff29f74db16d29996d46a87e3162c49b41e03516812a3959aec692b1. - Frozen preregistration SHA-256:
62b975cae27bfdf842bdccc06ce395735469c3fbff3f596a5987aee803a56040. - No model call has occurred.
2026-07-12 — native generation and seam implementation
- Added pinned Transformers batch-one native-thinking generation with explicit temperature/top-p/top-k, natural-close-only stopping, full-prefix recomputation, and
use_cache=Falseon every token. - Added historical thought-token activation capture, frozen J coordinate reads, alias token/rank contracts, and exact causal suffix-invariance smoke checks.
- Implemented the 16-task/48-trace frozen seam gate. It stores natural close, parse, correctness, token, seed, and stopping receipts without forcing close.
- CPU suite passes four tests. No model call has occurred; implementation will be committed and pushed before model smoke.
2026-07-12 — model-smoke attempt 001
- The pinned model loaded, then the tokenizer contract stopped before generation: the user instruction literally contained
<think>...</think>, creating a second special-token pair in addition to the chat template's native opener. - No token was generated and no correctness outcome was observed or written.
- Removed literal delimiter strings from the instruction while retaining native thinking through the frozen chat template. No scientific setting changed.
2026-07-12 — model-smoke attempt 002
- The repaired one-opener token contract passed. The second smoke stopped before generation because an eight-example rendered prompt used 541 tokens, above an implementation-only 512-token context safety cap.
- Raised that cap to 768 so the frozen prompt plus 160-thought/8-answer allowance fits. Task data, output budget, sampling, checkpoints, gates, and controls are unchanged; no correctness outcome was observed or written.
2026-07-12 — model-smoke attempt 003
- Token IDs, prompt cardinality, model shape/revision, lens rank, J-coordinate finiteness, and generation plumbing passed on two tasks.
- Both traces hit the frozen 160-token thought cap without natural close; no final alias was parseable. Correctness was deliberately not computed.
- The same historical thought token differed by 0.0625 across short-prefix and full-suffix forwards, above the frozen 1e-3 causal-invariance bar. This is consistent with the Qwen hybrid sequence-length numerical sensitivity already seen for batch geometry.
- Preserved the failed receipt as
runs/model_smoke/attempt_003_failed.json. Plumbing pass is now reported separately from scientific causal invariance; the latter remains fatal before causal confirmation. Neither the 160-token cap nor invariance threshold is changed.
2026-07-12 — model plumbing pass
- Reran the identical two-task smoke with plumbing and scientific controls reported separately. Model/revision, alias/think token IDs, lens ranks, finite coordinates, full-recompute generation, and capture pass.
- The receipt remains explicit that causal invariance fails (max 0.0625) and both traces were cap-bound. Correctness outcomes remain absent.
- Model smoke now unlocks only the frozen seam calibration; it cannot unlock value fitting or causal work by itself.
2026-07-12 — terminal natural-seam gate
- Ran the 16 frozen seam tasks once, three traces each, after publishing the model-smoke boundary.
- All 48/48 rows contacted the 160-token cap without natural close; natural close, parse, and exact success were each 0/48, and mixed tasks were 0/16.
- The cache-free full-prefix run used 7,632 forwards over 389.9 seconds. Every stop reason was
think_cap_without_close. - Frozen decision:
NO_NATURAL_SEAM. Prefix-value, control-calibration, and causal-confirmation stages are canceled without opening their outcomes. - Preserve as an interface-budget negative. Next work requires a separate preregistered natural-close budget ladder and dynamic per-length patch/control geometry; do not rescue this experiment in place.
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m pytest experiments/qwen35_4b_native_thought_jacobian_value_transport/tests -q && .venv/bin/python experiments/qwen35_4b_native_thought_jacobian_value_transport/scripts/run.py --stage smokeFull run
.venv/bin/python experiments/qwen35_4b_native_thought_jacobian_value_transport/scripts/run.py --stage seam-calibrationRun steps are documented inside the experiment folder (README and scripts).