Qwen3.5-4B Jacobian Transport Control Replication

Finished2026-07-12Interpretability and Diagnostics GitHub ↗

The one idea you need

Picture a package on a sorting line with its destination written up front. Rewrite that early label and it not only drops into a different bin but gets a different zip code stamped on it downstream. A few early internal numbers are that label.

The question

Inside a small model, is there a compact spot that stores which concept it is handling, and does editing that spot steer the later steps too?

What we found

Yes. Editing a handful of internal numbers at the early moment the model names its concept made it answer as a completely different concept on all 48 fresh test items, and a separately computed digit that depends on that concept flipped to match too. Same-size random edits changed nothing, and knowing only the output direction changed nothing: the effect needs the real concept numbers.

Why it matters

To steer or debug a small model, edit the early spot where it names a concept, not the layers near the output, since changes there propagate downstream. Always include same-strength random edits, or you cannot separate real steering from noise.

Direct answer after overwriting the concept0 of 48 → 48 of 48fresh test items that output the swapped-in concept, up from none

Separate downstream digit after the same edit48 of 48a later computation that reads the concept flipped to match too

Same-size random edits0 of 48matched-strength random perturbations never moved the answer

Injecting the wrong concept48 of 48 its own answerproduced that concept's answer, never the target, proving specificity

On this page

Results at a glance
Overview
Report
Experiment log
Data files
Reproduce
Related

Results at a glance 2

How often each edit forced the swapped-in concept as the answer

How to read

Eight interventions across the bottom; bar height is the share of 48 items answering the swapped-in concept. Two bars per group: the model's direct answer and the separately computed digit. Taller is a stronger effect; controls should sit at zero.

Takeaway → The three genuine concept edits reach 100% (the paired edit 96% on the digit), while the wrong-concept, output-direction, and both random controls sit flat at zero, a specific effect, not any-edit-works.

Data table

intervention	direct key target rate	mapped digit target rate
baseline	0%	0%
full donor	100%	100%
all-24 J	100%	100%
pair J	100%	95.8%
wrong-donor J	0%	0%
logit lens	0%	0%
random A	0%	0%
random B	0%	0%

Numbers from experiments/qwen35_4b_jacobian_transport_control_replication/runs/confirmation.json

Technical framing

Target-answer rate on fresh exact-control confirmation — Forty-eight untouched mappings at frozen band 4-8. Wrong-donor J produced its own key and digit on 48/48, while both independent random arms passed all post-bf16 geometry constraints.

How strongly the model preferred the new concept over the old

How to read

Five interventions across the bottom; bar height is the model's average preference for the new concept minus the original, shown for the direct answer and the downstream digit. Above zero favors the new concept; below zero favors the original.

Takeaway → Only the real concept edit crosses above zero, reversing a strong preference for the original into a strong preference for the new concept; the output-direction and both random controls stay firmly negative.

Data table

intervention	direct key margin	mapped digit margin
baseline	-11.8	-9.42
all-24 J	11.2	8.26
logit lens	-11.7	-9.41
random A	-11.8	-9.44
random B	-11.8	-9.44

Numbers from experiments/qwen35_4b_jacobian_transport_control_replication/runs/confirmation.json

Technical framing

Mean target-minus-source logit margin — Positive values favor the target; negative values favor the source. The J clamp reverses both preferences while the logit-lens and both exact controls remain baseline-like.

In the author’s words from the Overview · “Results”

The outcome-blind model smoke passes all 20/20 random layer deltas. Maximum post-bf16 relative norm error is 9.0113e-6 and maximum realized J-span projection fraction is 0.0098674, both inside the frozen gates. Exact lattice repair was needed for four layer-8 rows (three one-pair repairs and one two-pair repair). Model, lens, token, position, length, and causal-suffix contracts pass. Three preceding failed smoke receipts are preserved. The implementation history and post-smoke adversarial audit explain the geometry-only repair. The frozen numeric firewall subsequently passed all 480/480 calibration rows. … Read the full result →

Overview

This experiment independently replicates the perfect-but-invalid result from qwen35_4b_context_local_jacobian_clamp. It freezes that experiment's lens and band, generates fresh mappings, and repairs the only failed measurement: random controls must satisfy both perturbation norm and J-span orthogonality after bf16 application.

Research Program

Primary: interpretability_and_diagnostics
Conditional secondary: structured_execution_and_compilers and test_time_reasoning_budget only after a valid replication.
Closest near-duplicate: qwen35_4b_context_local_jacobian_clamp, terminal INVALID_CONTROL despite 48/48 J transport because one of 96 random rows had norm error 1.155e-5 >1e-5.

Question

Does the frozen early context-local J clamp again change a separately computed lookup consequence on fresh mappings when every random control is valid after quantization?

Fixed mechanism

Only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
Frozen prior 24-concept lens, SHA-256 e373b6e93956fdfc5cb446e9bee8249655707c8258a7868f0653d11f1ffd0213.
Frozen band [4,5,6,7,8], alpha one, selected-key token position.
Fresh 24-item numeric-control calibration and 48-item untouched confirmation.
Two independent random controls per item and prompt kind.
Every random layer delta must have post-bf16 relative norm error <=1e-5 and post-bf16 J-span projection fraction <=0.01. Requested vectors are projected orthogonal before quantization.
Same-subspace specificity: wrong-donor J must produce the wrong donor's own digit, not the registered target.
No digit direction or consequence gradient can construct any intervention.

The frozen rules are in reports/preregistration.md and the pre-run adversarial review is in reports/design_review.md.

Run

.venv/bin/python -m pytest experiments/qwen35_4b_jacobian_transport_control_replication/tests -q
.venv/bin/python experiments/qwen35_4b_jacobian_transport_control_replication/scripts/run.py --stage smoke
.venv/bin/python experiments/qwen35_4b_jacobian_transport_control_replication/scripts/run.py --stage model-smoke
.venv/bin/python experiments/qwen35_4b_jacobian_transport_control_replication/scripts/run.py --stage confirmation

Confirmation consumes the exact committed calibration artifacts; do not rerun and overwrite calibration in this result-bearing checkout. The calibration command is retained as a stage for independent clean-checkout reproduction.

Results

The outcome-blind model smoke passes all 20/20 random layer deltas. Maximum post-bf16 relative norm error is 9.0113e-6 and maximum realized J-span projection fraction is 0.0098674, both inside the frozen gates. Exact lattice repair was needed for four layer-8 rows (three one-pair repairs and one two-pair repair). Model, lens, token, position, length, and causal-suffix contracts pass.

Three preceding failed smoke receipts are preserved. The implementation history and post-smoke adversarial audit explain the geometry-only repair.

The frozen numeric firewall subsequently passed all 480/480 calibration rows. Maximum relative norm error was 9.8216e-6, maximum realized J-span projection fraction was 0.00999293, and exact causal-suffix difference remained zero. Thirty-seven rows used lattice repair (34 at layer 8), with at most three pairs. The calibration artifacts contain only numeric geometry and explicitly record that logits/outcomes were not written. These are still plumbing results, not causal evidence. The hash-locked eight-arm confirmation runner and a second pre-run adversarial implementation audit were completed before the single run.

Untouched confirmation returns REPLICATED_J_TRANSPORT:

all-24 J: 48/48 target keys and 48/48 target mapped digits;
full target donor: 48/48 and 48/48;
source/target pair J: 48/48 and 46/48;
wrong-donor J: 48/48 its own key and digit, 0/48 registered target;
concept logit lens, random_a, and random_b: 0/48 target in both prompt kinds;
both paired-bootstrap J-minus-random 95% intervals: [1.0, 1.0]; and
960/960 realized confirmation control-layer rows valid, with maximum relative norm error 9.9709e-6 and maximum J-span projection 0.0099970.

This independently confirms that the early context-local J coordinates carry a causally consumed concept state, not merely an imminent output direction.

Scope

Even a valid replication is oracle causal-mechanism evidence. The target concept and donor coordinates are supplied. It would license, but not itself constitute, a new experiment on native thinking and a learned non-oracle controller. It is not a capability gain and has not been compared with matched-compute sampling.

Knowledgebase Update

Remain unclaimed while the repository claim re-grade is open.
Update program evidence only after the frozen terminal decision.

Artifacts

All small data, the frozen lens copy, failed and passing smoke receipts, calibration/confirmation controls, outcome rows, metrics, and reports are committed. No training or adapter is used.

Report

Rendered from reports/report.md

Status

REPLICATED_J_TRANSPORT on the single untouched confirmation run.

Frozen purpose

Replicate the parent context-local semantic-transport effect on fresh mappings while repairing its post-bf16 random-control failure. See preregistration.md and design_review.md.

The fourth, repaired smoke passes all 20 random layer deltas across direct and consequence prompts, two independent arms, and layers 4--8:

maximum relative norm error: 9.0113e-6 (gate 1e-5);
maximum realized J-span projection fraction: 0.0098674 (gate 0.01);
exact causal-suffix activation difference: 0.0;
four layer-8 rows used exact neighboring-bf16 pair repair, with at most two pairs; and
no outcomes or logits are present in the smoke receipt.

Failed attempts 001--003 are preserved beside the passing receipt. They are engineering evidence about sequential bf16 quantization, not scientific model outcomes. The post-smoke section of design_review.md audits the implementation repair before calibration.

Numeric calibration firewall

The frozen 24-item calibration passed all 480 expected rows:

maximum relative norm error: 9.8216e-6;
maximum realized J-span projection fraction: 0.00999293;
37 rows used exact lattice repair, including 34/96 layer-8 rows;
maximum lattice repair: three neighboring-bf16 coordinate pairs;
both prompt kinds and both random arms contribute exactly 240 rows each; and
exact causal-suffix difference: 0.0.

The row schema contains only item/prompt/arm/layer identity, candidate and iteration indices, delta norms, the two numeric errors, lattice pair count, and pass state. The summary records logits_recorded=false and outcomes_recorded=false.

Current inference boundary

The calibration boundary was committed and pushed before the hash-locked runner opened confirmation. The runner itself was then committed, pushed, and green in CI before the one result-bearing run.

Untouched confirmation

Intervention	Direct target	Mapped target	Own wrong target
Baseline	0/48	0/48	0/48
Full target donor	48/48	48/48	0/48
All-24 J target clamp	48/48	48/48	0/48
Source/target pair J	48/48	46/48	0/48
Wrong-donor J	0/48	0/48	48/48
Concept logit lens	0/48	0/48	0/48
Random A	0/48	0/48	0/48
Random B	0/48	0/48	0/48

All arms parsed on 48/48 items. Mean target-minus-source margin moved from -11.8079 to +11.1797 direct and from -9.4245 to +8.2604 on the mapped consequence. Both random arms stayed at baseline-like negative margins. Both paired 10,000-resample J-minus-random 95% intervals were [1.0, 1.0].

Every one of 960 confirmation random-control layer deltas passed after bf16 application. Maximum norm error was 9.9709e-6 and maximum J-span projection fraction was 0.0099970; 47 rows used exact lattice repair and no row required more than three coordinate pairs. Exact causal-suffix activation difference was zero.

Frozen decision

Every clean, donor, direct-shift, consequence-shift, worse-random, two-bootstrap, wrong-donor specificity, parse, causal, and numeric gate passes. The frozen terminal label is REPLICATED_J_TRANSPORT.

Interpretation and limits

The late answer-position Jacobian in the grandparent was writable but did not transport. In contrast, the early selected-token clamp controls both the concept and a later computation that consumes it, survives a fresh replication, and is specific to donor identity. This is strong evidence for a compact causally consumed concept state in this prompt-local task.

It is still an oracle mechanism. The target concept and clean donor coordinates are supplied; the task is a procedural lookup; there is no learned controller, native <think> prefix, installed capability, or comparison to matched-compute sampling. The result licenses those experiments but cannot substitute for them. No repository claim ID is allocated while the ledger re-grade remains open.

Experiment log 11

Show the running log (11 entries, 2026-07-12)

Scaffold

Created as a new experiment scaffold.

2026-07-12 — intake and adversarial review

Named the invalid 48/48 context-local clamp result as the direct parent.
Froze its exact lens, band 4–8, alpha one, prompt grammar, and model revision.
Registered fresh calibration/confirmation mappings and two independent random controls per item.
Added post-bf16 gates for realized norm (1e-5 relative) and J-span projection fraction (0.01), plus wrong-donor same-span specificity.
Completed the adversarial review before implementation or any model call.

2026-07-12 — immutable design boundary

Pushed design commit 27b9da2a0973dbddbdfd2b6f7acddbfc7f4f736f before model inference.
Recorded exact frozen README/preregistration hashes and the byte-identical parent lens hash.

2026-07-12 — quantization-aware control implementation

Copied the parent's cache-free batch-one Qwen patching and coordinate code, then added a numeric-only post-bf16 control optimizer.
For 32 fixed random candidates per layer, the hook alternates realized-span removal/renormalization with 64-step scale search and chooses the first candidate meeting both frozen constraints. Candidate selection cannot access logits or labels.
Implemented a model smoke and the 480-layer numeric calibration gate. The calibration writer rejects outcome-like fields and discards every forward's logits before serialization.
CPU suite passes 24 tests plus 24 subtests; no model call has occurred in this replication.

2026-07-12 — model-smoke attempt 001

Ran the first outcome-blind model smoke after pushing the implementation.
Model/lens/token/position/causal contracts passed and 17/20 realized random layer deltas met both frozen constraints.
Three layer-8 deltas had valid norm matching but realized J-span projection fractions 0.01116, 0.01214, and 0.01429, above the frozen 0.01 ceiling.
Preserved the failed receipt as runs/model_smoke/attempt_001_failed.json. This is an engineering smoke failure, not a calibration or scientific result; calibration remains unopened and no outcome logits were recorded.
The correction path had only retained scale-searched states at 16-iteration checkpoints. The implementation now retains each fixed candidate's lowest- projection intermediate quantization cell for a final geometry-only scale search, without changing seeds, candidate count, damping, iteration budget, binary-search budget, or either frozen threshold.

2026-07-12 — model-smoke attempt 002

The retained-cell correction improved one failed layer-8 projection from 0.01429 to 0.01135, but the smoke again passed only 17/20 rows; the other two failed projections remained 0.01116 and 0.01214.
Preserved this outcome-blind receipt as runs/model_smoke/attempt_002_failed.json; calibration remains unopened.
Audit found that scale search selected only the closest-norm state and could discard another visited bf16 plateau that better satisfies the joint frozen norm/projection gate. Scale search now ranks every visited state by the two preregistered constraints jointly. This repairs the implementation of the frozen gate; it changes no optimizer budget, seed, threshold, or outcome rule.

2026-07-12 — model-smoke attempt 003 and exact-lattice audit

Joint scale selection exposed four layer-8 failures at projection fractions 0.01037, 0.01054, 0.01135, and 0.01214; preserved the receipt as runs/model_smoke/attempt_003_failed.json.
An outcome-blind diagnostic separated direction supply from live-lattice feasibility. On the unperturbed layer-8 state, 8,044--8,091 of 8,320 tested fixed-seed mixture states passed. After upstream random patches changed the live layer-8 bf16 state, zero mixture states passed; this localized the issue to sequential quantization, not favorable random-direction selection.
For each failed live delta, exhaustively scored pairs of neighboring bf16 coordinate moves using the exact norm and orthogonal-projector identities. One pair repaired three rows and two pairs repaired the fourth, producing projection 0.00852--0.00966 while keeping norm error <=9.02e-6.
Integrated this deterministic pair repair only after the frozen continuous optimizer fails. It uses current residual, dictionary, target norm, and the same 512-step bound; it cannot inspect logits, labels, or answers and changes no seed, random draw, intervention, threshold, or endpoint.

2026-07-12 — model-smoke pass

Fresh rerun with the exact-lattice implementation passed 20/20 numeric rows.
Maximum post-bf16 relative norm error was 9.0113e-6 and maximum realized J-span projection fraction was 0.0098674.
Four layer-8 rows required lattice repair: three used one pair and one used two pairs. All other rows passed without lattice moves.
Exact model revision, lens rank/hash, token, position, equal-length, and causal-suffix contracts passed. The receipt records no outcomes.
Calibration and confirmation remain unopened pending commit/push of this complete smoke history and adversarial implementation audit.

2026-07-12 — numeric calibration pass

After the passing smoke boundary was committed, pushed, and green in CI, ran the frozen 24-item outcome-blind calibration once.
All 480/480 rows passed. Maximum relative norm error was 9.8216e-6; maximum realized J-span projection fraction was 0.00999293; causal difference was 0.
Thirty-seven rows used lattice repair: one each at layers 5, 6, and 7, plus 34/96 at layer 8. Maximum repair was three coordinate pairs.
Both prompt kinds and both random arms contain exactly 240 rows. No logits or outcome fields occur in the row artifact, and the summary records both as absent.
Decision: CONTROL_CALIBRATION_PASS. Confirmation remains unopened pending a separate calibration commit/push.

2026-07-12 — confirmation implementation and adversarial audit

Locked confirmation to calibration commit 2bd6376c28283546a111e54ec2dd2e92a0fd6a64 and exact summary/row SHA-256 hashes. The unlock reconstructs all 480 numeric identities and maxima.
Implemented exactly the eight frozen arms, 768-row completeness contract, and a separate 960-row realized-control artifact.
Implemented the frozen endpoint against the worse of two random target rates, paired bootstrap lower bounds against both arms, wrong-donor own-value specificity, parse, clean, donor, causal, and numeric gates.
Added pure tests for committed calibration integrity, complete arm pairing, both bootstrap comparisons, and failure of nonspecific wrong-donor behavior.
Completed and saved the pre-confirmation adversarial implementation audit. No confirmation outcome has been opened; implementation must be committed and pushed before the one run.

2026-07-12 — untouched confirmation

Ran confirmation exactly once after implementation commit e1de55cc passed both GitHub validation and site publication.
All-24 J selected the target on 48/48 direct prompts and 48/48 separately computed mapped consequences. Baseline, random_a, random_b, and the concept logit-lens selected the target on 0/48 for both prompt kinds.
Full target donors were 48/48 for both. Pair-only J was 48/48 direct and 46/48 consequence. Wrong-donor J selected its own key and mapped digit on 48/48 and the registered target on 0/48.
All 960/960 confirmation control-layer rows passed. Maximum realized norm error was 9.9709e-6; maximum J-span projection was 0.0099970. Forty-seven rows used exact lattice repair, with at most three pairs.
Both paired 10,000-resample J-minus-random 95% intervals were exactly [1.0, 1.0]; every clean, donor, specificity, parse, causal, and numeric gate passed. Frozen decision: REPLICATED_J_TRANSPORT.
Scope remains oracle mechanism evidence: target donor identity is supplied, the substrate is a fresh prompt-local lookup, and no model capability has been installed or shown to beat matched-compute sampling.

Data files 1

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

runs/model_smoke/result.json11 kB

Reproduce

Smoke test

.venv/bin/python -m pytest experiments/qwen35_4b_jacobian_transport_control_replication/tests -q && .venv/bin/python experiments/qwen35_4b_jacobian_transport_control_replication/scripts/run.py --stage smoke

Full run

.venv/bin/python experiments/qwen35_4b_jacobian_transport_control_replication/scripts/run.py --stage confirmation

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗