Qwen3.5-4B Jacobian Counterfactual Branching

Finished2026-07-13Interpretability and Diagnostics Posttraining and Adaptation Process Control and Tool Use GitHub ↗

The one idea you need

Imagine pausing one scratchpad halfway through, then making twelve balanced copies. Each copy gets a tiny internal nudge toward a different possible answer and is allowed to finish reasoning. Because every answer gets one nudge, no answer is secretly favored.

The question

Can balanced internal nudges make one half-finished reasoning path explore better futures than spending the same effort on ordinary independent attempts?

What we found

No. Across all three allowed strengths, the meaningful nudge made its assigned answer win only 4 of 48 times—exactly the one-in-twelve chance rate and identical to a generic nudge. The probabilities barely moved even at the strongest setting. Because the write itself failed, no expensive continuation or correctness run was opened.

Why it matters

A direction that works when copied from a clean semantic example is not automatically a push-button control inside arbitrary free-form reasoning. The next test must restore an explicit semantic anchor and copy full coordinates, rather than merely pushing the last scratchpad token harder.

Assigned answer selected4 / 488.3%, exactly chance, at every tested strength

Largest average probability lift+0.006versus +0.15 required for a usable write

Numeric control rows validallthe null is behavioral, not a broken control

On this page

Results at a glance
Overview
Report
Experiment log
Reproduce
Related

Results at a glance 2

Every nudge strength stays at chance

How to read

For each of three nudge strengths, compare the meaningful and generic bars with the tall required bar. Height is how often the branch's assigned answer became the model's answer.

Takeaway → Meaningful and generic nudges overlap at one-in-twelve while the required bar is 60%, so this internal write method is unusable.

Data table

norm-anchored branch amplitude	J target selection	J-orthogonal control
alpha 0.5	8.3%	8.3%
alpha 1.0	8.3%	8.3%
alpha 2.0	8.3%	8.3%

Numbers from experiments/qwen35_4b_jacobian_counterfactual_branching/runs/mechanics.json

Technical framing

Native J branches do not write their supplied alias — Each alpha contains 48 label-free supplied-target interventions: 12 aliases on each of four 512-token native thoughts. J and generic branches both remain at the 1-in-12 chance rate while all numeric controls pass. No alpha qualifies, so continuations and correctness remain unopened.

Meaningful and generic nudges were physically matched

How to read

At each of five internal layers, neighboring bars show the physical size of the two nudges. Matching bars rule out a weaker generic control.

Takeaway → The control matched the intervention's strength, confirming that the chance result is not a simple size mismatch.

Data table

frozen intervention layer	J-space branch RMS norm	J-orthogonal control RMS norm
layer 4	2.1	2.1
layer 5	0.0945	0.0945
layer 6	0.131	0.131
layer 7	0.147	0.147
layer 8	0.062	0.062

Numbers from experiments/qwen35_4b_jacobian_counterfactual_branching/runs/smoke/cpu.json

Technical framing

Balanced semantic and generic branch norms match by layer — CPU geometry at alpha 1.0. The generic control is an orthogonal rotation preserving the complete semantic branch Gram matrix, not merely average norm. These valid controls later accompanied the terminal chance-level mechanics result.

In the author’s words from the Report · “Summary”

Design/CPU smoke only. No model or correctness outcome has run.

Overview

This experiment tests whether a balanced bank of early J-space edits can shift the proposal distribution of native reasoning, rather than trying to value a finished thought.

Research Program

Primary: interpretability_and_diagnostics.
Secondary: test_time_reasoning_budget, evidence_conditioned_selection, and structured_execution_and_compilers.
Direct parents: qwen35_4b_jacobian_transport_control_replication, qwen35_4b_commit_slot_semantic_power_replication, and qwen35_4b_counterfactual_order_support_selector.

Question

Starting from one shared 512-token native thought, can 12 zero-sum semantic J branches create more useful 512-token continuations than clean or generic branches and beat fully independent sample-more at matched model compute?

Hypothesis

The replicated early J directions are causally consumed concept state, while terminal J/value readouts and probability attribution fail. A balanced branch bank uses each of the 12 public alias directions once, with deltas centered to sum exactly to zero. Averaging final alias probabilities cancels direct branch bias, but systematic hypothesis exploration may uncover a correct continuation that repeated stochastic decoding misses.

Setup

Only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
Frozen lens SHA-256 e373b6e93956fdfc5cb446e9bee8249655707c8258a7868f0653d11f1ffd0213, layers 4--8, first 12 public aliases.
Layer norm anchors are the median replicated donor-clamp norms from the independent transport confirmation: 2.09603, 0.09454, 0.13078, 0.14686, and 0.06200.
Fresh exact-depth-two procedural tasks: four label-sealed mechanics, 24 qualification, and 48 untouched confirmation.
One shared prefix runs to 512 tokens. Cache is forked before its final token; each arm consumes 12 continuations to the fixed 1,024 cap.
J deltas are the centered 12-alias dictionary, scaled by one common multiplier per layer so branch deltas sum exactly to zero.
Non-J deltas are built by an orthogonal rotation preserving the complete 12 x 12 J branch Gram matrix at every layer, then repaired/audited after bf16.
Primary output is argmax mean final constrained alias probability across the 12 branches. No branch target, score, alpha, or seed uses the correct alias.

Arms and matched compute

j_balanced: 12 centered J branches from the shared midpoint.
clean_shared: 12 stochastic branches from the identical midpoint/cache.
non_j_gram: 12 Gram/norm-matched J-orthogonal branches.
full_sample_more: a frozen master pool of 12 independent 1,024-token thoughts, reported at sampled-token- and total-forward-token-matched prefixes plus the conservative full K=12 overmatch.

All arms use Transformers bf16 SDPA because activation intervention requires internals. Backend mixing is forbidden. Every prompt/decode/prefill/cache/slot token and attention-shape proxy is counted.

Gates

Before any correctness-scored continuation, four mechanics tasks select the smallest frozen multiplier from [0.5, 1.0, 2.0] that reaches at least 60% immediate target selection, at least +0.15 mean target-probability lift, at least +35pp target-selection specificity over non-J, exact zero-sum branch geometry, and 100% post-bf16 numeric controls. No multiplier passing is terminal NO_NATIVE_J_BRANCH_CONTROL.

On qualification, J must beat every clean/non-J/compute-matched deployable selector by at least 10pp with one-sided paired-task lower bound above zero, preserve or improve oracle answer coverage by 5pp, span at least eight predicted and six successful aliases, and pass all resource/backend/numeric contracts. Only then may identical untouched confirmation open. Stages never pool.

Run

The label-free mechanics stage completed all four public task units at all three preregistered amplitudes. It selected no amplitude and automatically stopped before continuation or correctness scoring. See runs/mechanics.json and the post-mechanics adversarial audit for the full receipt and non-rescues.

Status

Terminal NO_NATIVE_J_BRANCH_CONTROL. All live controls pass, but supplied- target selection is exactly 4/48 (8.33%) for J and non-J at alpha 0.5, 1.0, and 2.0. Mean J target-probability lift reaches only 0.00566 at alpha 2 versus 0.15 required. No alpha is selected; continuation, qualification, confirmation, and correctness remain unopened.

Artifacts

idea_intake.md: novelty and routing.
reports/preregistration.md: immutable scientific rules.
reports/design_review.md: adversarial review before implementation/GPU use.
reports/pre_model_implementation_audit.md: 30 assertions authorizing only outcome-blind live-bf16 model smoke after a pushed hash boundary.
reports/post_model_smoke_001_audit.md: invalid zero-delta receipt, root cause, repaired guards, and explicit non-authorization.
reports/post_model_smoke_002_audit.md: valid naïve-bf16 control failure and geometry-only live repair boundary.
reports/post_model_smoke_003_audit.md: 51/60 iterative-repair near miss and exact neighboring-bf16 lattice boundary.
reports/post_model_smoke_004_audit.md: layer-8 repair success and one independently detected layer-4 boundary miss requiring a stricter safety objective.
reports/post_model_smoke_005_audit.md: 60/60 live-control pass and the mechanics authorization boundary.
reports/pre_mechanics_implementation_audit.md: 35 firewall, intervention, metric, gate, artifact, and fail-closed assertions.
reports/post_mechanics_adversarial_audit.md: nine non-rescues and routing to explicit semantic anchors plus donor-coordinate replacement.
assets/context_lens.pt: byte-identical causal lens anchor.
src/: frozen tasks, branch geometry, cache-fork model operations, and pure statistics.
reports/artifact_manifest.yaml: external/omitted artifact policy.

Report

Rendered from reports/report.md

Summary

Design/CPU smoke only. No model or correctness outcome has run.

Research Program Fit

Method

The preregistration freezes zero-sum centered semantic branches, exact-Gram J-orthogonal controls, cache-fork generation, full-path sample-more resource matching, and a label-free mechanics gate. See the 22-point design review.

Results

CPU/data smoke only:

76 fresh exact-depth-two task fingerprints are unique and disjoint from 634 direct-ancestor fingerprints;
lens hash is exact and layers 4--8 retain rank 24;
every alpha/layer J branch bank is width 12, rank 11, and zero-sum within 5.22e-7 maximum coordinate residue;
non-J branch Gram relative error is at most 1.14e-6;
float non-J projection into the complete J span is at most 3.05e-7; and
no model, outcome, correct alias, or confirmation stage was opened.

The fifth outcome-blind live smoke subsequently passes all 60 post-bf16 non-J controls. Maximum paired norm error is 9.3881e-6, maximum complete-J-span projection 0.00912094, and lattice repair uses at most five pairs. It records no behavioral or target-selection metric.

Label-free mechanics is terminal NO_NATIVE_J_BRANCH_CONTROL:

alpha	J target selected	non-J	mean J target-probability lift	numeric
0.5	4/48	4/48	+0.000471	pass
1.0	4/48	4/48	+0.001498	pass
2.0	4/48	4/48	+0.005664	pass

Every write is finite and every live control passes, but target selection is exactly the 1/12 chance rate with zero J specificity. No alpha meets the 60%, +0.15 lift, or +35pp J-minus-non-J gates.

Controls

Oracle Versus Deployable Evidence

Interpretation

The causal lens does not transfer through centered additive directions at an arbitrary last-thought token. This does not contradict donor-coordinate transport at an explicit semantic token. It isolates the next mechanism test to semantic anchoring and coordinate replacement; continuation branching here is cancelled.

Next Experiments

Implement and adversarially audit cache forking, one-shot layer hooks, live-bf16 control repair, fixed-cap sampler, and resource matcher. Only then may a pushed boundary authorize label-free model mechanics.

Superseded by the terminal mechanics gate. Create a distinct successor for an explicit hypothesis anchor and donor-coordinate replacement; do not implement cache-fork continuations in this experiment.

Artifact Manifest

See artifact_manifest.yaml.

Experiment log 9

Show the running log (9 entries, 2026-07-13)

2026-07-13 — Design and data preflight

Completed idea intake, preregistration, and a 22-point adversarial review before model implementation or load.
First proposed split seed 2026071301 was rejected before writing results: all 24 qualification fingerprints collided with a direct ancestor stream.
Changed only the pre-outcome split seed to 2026072301; collision checking remains fatal. No model or correctness metric had run.
Regeneration produced 76 unique new fingerprints with zero overlap against 634 direct-ancestor fingerprints: 4 mechanics, 24 qualification, 48 sealed confirmation.
Four geometry tests and CPU smoke pass. Across layers 4--8 and alpha 0.5/1/2, every J/non-J branch bank has width 12, rank 11, near-zero vector sum, Gram relative error at most 1.14e-6, and float non-J projection at most 3.05e-7. Live bf16 controls remain for model smoke.
Model stages remain fatal-unavailable pending cache-fork implementation, implementation audit, commit, and push.

2026-07-13 — Model-smoke implementation boundary

Six implementation tests pass; pending-boundary invocation fails before model load.
Pushed cache-free one-shot branch readout implementation at 9100395f and froze exact runner/model/geometry/test hashes in config.
This boundary authorizes outcome-blind model smoke only. It does not authorize alpha mechanics or continuation generation.

2026-07-13 — Invalid first live-bf16 receipt

The first outcome-blind model smoke loaded the exact model/lens and applied every hook once, but its numeric receipt was invalid.
Root cause: current was a view into the cloned activation; assigning the changed activation mutated the supposed before-state, so all realized deltas were falsely recorded as zero.
A second guard omission reported J requested-norm error 1.0 but did not include it in the conjunctive numeric pass, producing an impossible false pass.
No branch probabilities, choices, correct alias, outcome, qualification, or confirmation data were recorded. The receipt is preserved as model_001_invalid_receipt.json and cannot authorize mechanics.
Fixed by cloning the float before-state before assignment, computing realized deltas against it, and making J requested-norm error a mandatory gate. A new pushed implementation hash is required before rerun.
Corrected implementation was pushed at 802cf1a5; config now anchors its updated runner/model/test hashes. Smoke 002 is authorized only after this re-anchor commit is pushed.

2026-07-13 — Valid smoke 002 requires quantization-aware control repair

Corrected deltas are nonzero and the receipt is valid/outcome-blind.
Naïve bf16 non-J controls missed paired J norm by up to 3.39% and leaked up to 2.96% into J, failing the frozen 1e-5/1% live gates.
Requested J fidelity (4.16%), realized Gram (5.83%), and zero-sum residue (0.015625) remain diagnostics; exact Gram/rank/zero-sum is the pre-bf16 CPU construction gate, while paired norm/span are the explicitly frozen live gates.
Added vectorized outcome-blind repair of each fixed non-J request toward its paired realized J norm outside the complete J span, with 512 iterations and 0.5 damping. Code must be pushed/re-anchored before smoke 003.
Repair implementation pushed at 934f4d59; exact updated hashes are anchored for smoke 003.

2026-07-13 — Smoke 003 reaches 51/60, exact lattice repair next

Iterative outcome-blind repair validates all 12 rows at layers 5--7 and reaches the paired norm/span boundary on layer 4; layer 8 retains nine fails.
Maximum remaining norm error is 1.3206e-5 and span leakage 2.096%; no tolerance changed.
Added the independent transport replication's exact one-ULP pair search for only failing rows, bounded at 32 pairs. Code must be pushed/re-anchored before smoke 004.
Lattice implementation pushed at dc298278; exact runner/model/test hashes are anchored for smoke 004.

2026-07-13 — Smoke 004 fixes layer 8, misses one layer-4 receipt boundary

Exact lattice search validates all layer-8 rows with at most five pairs; all independent span projections pass.
Independent layer-4 paired norm is 1.033804e-5 versus 1e-5, although the device-local repair check stopped inside the boundary.
Added a stricter 0.95 geometry repair objective so near-boundary rows cannot stop until they have 5% guard band. Scientific tolerances remain unchanged.
Code must be pushed/re-anchored before smoke 005.
Safety-margin implementation pushed at 6660bd94; exact hashes are anchored for smoke 005.

2026-07-13 — Smoke 005 passes all live controls

All 60 non-J rows pass: maximum paired norm error 9.3881e-6 and maximum full-J projection 0.00912094, within frozen 1e-5/0.01 gates.
Every hook applied once; all branches/logits finite; peak allocation 9,115,230,720 bytes.
At most five lattice pairs were required. Branch probabilities, choices, supplied-target metrics, correct aliases, and outcomes remained unrecorded.
Mechanics implementation may begin, but no model mechanics or continuation is yet authorized.

2026-07-13 — Mechanics implementation complete, pending anchor

Added an experiment-local public mechanics file containing only task ID and visible examples; its hash is frozen in the regenerated disjointness manifest.
Implemented one 512-token prefix per four tasks, all three frozen alphas, full baseline/J/non-J probabilities, live controls, automatic smallest-alpha gates, and trace/result artifacts without any correct alias field.
Eight tests and Python syntax pass. Pending mechanics boundary fails before model load; qualification/confirmation remain unavailable.
Completed a 35-point implementation audit. Commit/push and smoke-hash anchor are required before the single mechanics run.
Mechanics implementation pushed at 84c12e9b; exact runner/model/geometry/ test and smoke-005 hashes are now anchored. Mechanics may run only after this anchor commit is pushed.

2026-07-13 — Terminal native branch-control negative

All four 512-token prefixes completed; no mechanics gold/outcome field was loaded.
Automatic NO_NATIVE_J_BRANCH_CONTROL: J target selection was 4/48 at every alpha, exactly equal to non-J and the 1/12 chance rate.
Mean J target-probability lift was +0.000471, +0.001498, and +0.005664 for alpha 0.5/1/2, versus +0.15 required.
All numeric controls passed at every task/alpha. Global maximum paired non-J norm error was 9.60e-6 and J-span projection 0.00950.
No alpha selected. Continuations, qualification, confirmation, and all correctness data remain unopened.
Completed a nine-point post-mechanics adversarial audit. The next warranted experiment must restore the positive control's explicit semantic token and donor-coordinate replacement rather than increase alpha post hoc.

Reproduce

Smoke test

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python experiments/qwen35_4b_jacobian_counterfactual_branching/scripts/run.py --stage smoke

Full run

unavailable before implementation and mechanics boundaries

Runnable scripts exist in the experiment folder, but the exact invocation was not written down.

Browse the experiment folder on GitHub ↗