Qwen3.5-4B Commit-Slot Jacobian Value Transport
The one idea you need
Picture a multiple-choice bubble sheet handed to the model right after it thinks: one fixed answer line, twelve permitted words. It now fills a bubble almost every time instead of leaving the line blank. But filling a bubble is not the same as filling the correct one.
The question
After a small model thinks, if you force it to pick one of twelve allowed answer words, does the thinking it just did actually help it choose correctly?
What we found
Barely. Forcing the format fixed one problem outright: an allowed word was the model's top choice 85% of the time, versus 4% when it answered freely. But real step-by-step thinking beat the very same thought words scrambled into random order by only about eight points, small enough to be noise, and the edge piled onto just one or two easy words.
Why it matters
When you box a small model into a forced-choice answer, don't assume the reasoning earned the result. Scramble the model's own thought words and re-measure: if shuffled thinking ties real thinking, the format is doing the work, not the thinking.
On this page
Results at a glance 2
How to read
Bars grouped by thinking budget (256, 512, 1,024 tokens). Each group has three bars: real ordered thinking, no thinking at all (a flat 25% across budgets), and the same thought words shuffled. Taller means more exactly-correct answers.
Takeaway → Only at the largest budget does real thinking clearly top both controls; at shorter budgets scrambled thinking ties or beats it, revealing a fragile, budget-dependent edge, not a general one.
Data table
| paired thought cap | real ordered thought | no generated thought | same thought tokens shuffled |
|---|---|---|---|
| 256 cap | 20.8% | 25% | 22.9% |
| 512 cap | 18.8% | 25% | 31.2% |
| 1024 cap | 31.2% | 25% | 22.9% |
Numbers from experiments/qwen35_4b_commit_slot_jacobian_value_transport/runs/seam_selection.json
Technical framing
Commit-slot accuracy across paired thought caps — Each cap contains 48 paired real/shuffled trace rows; no-thought is one deterministic row per each of 16 tasks. The 1,024 real arm passed accuracy-gap gates but had only five mixed tasks versus six required, so no cap was selected.
How to read
Three rates at the longest budget: how often an allowed word was already the model's top choice (85%), the average probability weight sitting on allowed words (68%), and how often free-form answers without the slot produced a usable answer (4%). Higher is better.
Takeaway → The two forced-slot bars tower over the free-form bar: the syntax alone cured the answering problem, even though picking the correct word stayed unreliable.
Data table
| interface diagnostic | share or mean mass |
|---|---|
| unmasked top is alias | 85.4% |
| total alias probability mass | 68.5% |
| close-only free-form parses | 4.2% |
Numbers from experiments/qwen35_4b_commit_slot_jacobian_value_transport/runs/seam_selection.json
Technical framing
Answer-mode diagnostics at the 1,024 cap — The first two values use the fixed `First:` slot before alias masking; the last uses matched close-only free-form output. Syntax repaired answer mode even though the semantic selection gate failed.
In the author’s words from the Report · “Interpretation”
The honest next step is a fresh, powered, fixed-1,024 replication with more balanced task units and a task-bootstrap gate—not confirmation of this failed selector, not a relaxed mixed-task threshold, and not a larger cap yet.
Overview
This study tests whether a syntax-only forced answer slot reveals a usable alias decision and, if so, whether a scalar Jacobian certainty coordinate can improve that decision.
Research Program
- Primary:
interpretability_and_diagnostics. - Secondary:
test_time_reasoning_budgetandstructured_execution_and_compilers(the fixed latent answer interface). - Direct parent:
qwen35_4b_forced_commit_jacobian_value_transport. - Earlier seam parents:
qwen35_4b_native_thought_jacobian_value_transportandqwen35_4b_native_thought_seam_budget_ladder. - Mechanism anchor:
qwen35_4b_jacobian_transport_control_replication.
Question
After a fixed thought budget, can the controller append only </think>\n\nFirst: and read a correct next choice from the 12 public alias tokens often enough to support value measurement? If yes, is correct-alias probability held-out-by-task decodable and causally writable as one task-general J-space coordinate rather than answer identity or ordinary slot confidence?
Hypothesis
Close-only forced answers failed because the model restarted analysis instead of entering the requested output grammar. Supplying the grammar but not the alias should isolate the semantic choice: the constrained next-token argmax may recover information already present in the thought prefix. Real thought must beat an immediate no-thought slot and retain correct/incorrect variation. A learned scalar value coordinate must then beat direct correct-alias activity, ordinary slot margin, shuffled labels, exact random, direct-output-gradient, and non-J controls.
Setup
- Only
Qwen/Qwen3.5-4B, revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Exact replicated 24-coordinate lens, SHA-256
e373b6e93956fdfc5cb446e9bee8249655707c8258a7868f0653d11f1ffd0213, frozen layers 4--8. - Transformers bf16 SDPA, unpadded batch one; cached native trace generation and exact full-prefill slot logits.
- 96 new exact-depth-two procedural tasks: 16 slot selection, 16 untouched slot confirmation, 32 value fit, 32 causal confirmation.
- Every visible set has one identifiable first-operation type and no depth-one solution; all fingerprints are unique and disjoint from four parents.
- Three trace samples/task, temperature 0.6, top-p 0.95, top-k 20.
- Paired caps 256/512/1024. At each cap, append close plus fixed slot text and take argmax over the 12 public one-token aliases.
- Controls: immediate no-thought slot, an exact-length deterministic permutation of the same thought-token multiset, unconstrained full-vocabulary logits, and close-only free-form output from the exact same real trace prefix.
The controller supplies syntax and a closed public answer vocabulary, never the correct alias. This is a constrained deployment interface, not natural output and not directly comparable to free-form accuracy.
Frozen Gates
Selection requires constrained slot accuracy in 20%--80%, at least six tasks with both correct and incorrect traces, 100% finite rows, at least +5pp over the task-level no-thought slot, and +3pp over exact-length shuffled thought. Freeze the smallest passing cap. Untouched confirmation repeats the range/count/finite gates and requires +3pp over no-thought and +2pp over shuffled thought. Close-only free-form and unmasked alias mass are diagnostics, not rescue arms.
Only COMMIT_SLOT_SEAM_REPLICATED opens value fitting. Deterministic correct-alias probability at 0.5 and 1.0 of the cap is the oracle value label. J value at the causally earlier final-thought position must reach held-out-by-task pairwise AUC 0.65, beat correct-alias coordinate activity by 0.03 and ordinary slot margin by 0.02, and leave a within-task shuffled null near chance. Only that pass may open exact per-prefix bf16 controls and causal tests.
Run
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python \
experiments/qwen35_4b_commit_slot_jacobian_value_transport/scripts/run.py \
--stage smokeAfter the design boundary is anchored:
.venv/bin/python experiments/qwen35_4b_commit_slot_jacobian_value_transport/scripts/run.py --stage model-smoke
.venv/bin/python experiments/qwen35_4b_commit_slot_jacobian_value_transport/scripts/run.py --stage seam-selection
.venv/bin/python experiments/qwen35_4b_commit_slot_jacobian_value_transport/scripts/run.py --stage seam-confirmationValue/control/causal stages fail closed until their audited implementations are committed after a replicated slot seam.
Status
Terminal COMMIT_SLOT_SEAM_FAIL; confirmation, value, control, and causal stages are ineligible. All 48 traces remained open through 1,024. The fixed slot did repair answer mode—an alias was already the unmasked top token on 41/48 1,024 rows—but did not clear the semantic-variation gate:
| cap | real slot | no-thought | shuffled thought | mixed tasks | gate |
|---|---|---|---|---|---|
| 256 | 10/48 | 4/16 | 11/48 | 2 | fail |
| 512 | 9/48 | 4/16 | 15/48 | 2 | fail |
| 1024 | 15/48 | 4/16 | 11/48 | 5 | fail |
At 1,024, real thought met the 20%--80% headroom and both frozen accuracy-gap gates (+6.25pp over no-thought, +8.33pp over shuffled), but only five tasks had both correct and incorrect traces versus six required. The task-bootstrap intervals for both gains crossed zero. The +4 net real-over-shuffled successes were alias/task concentrated, not explained by correct-alias verbalization, and three exploratory label-free logit-residual policies did not improve 15/48. The near miss cannot select a cap or reopen J value.
Scope
The slot and alias mask are explicit deployment scaffolds. They may elicit a choice, but do not install a general free-form capability. Correct-label value and donor selection are oracle. Capability requires a later non-oracle method to beat frozen, close-only/free-form, no-thought slot, and matched sampling on untouched contamination-free tasks.
Knowledgebase Update
- Update all three program ledgers and synthesis at each terminal gate.
- No claim ID while the repository claim re-grade remains open.
Artifacts
assets/context_lens.pt: byte-identical replicated lens.data/procedural/: four frozen splits and manifest.runs/smoke/: CPU/gate receipts.reports/preregistration.mdandreports/design_review.md: immutable rules; both prose and the semantic config payload are design-commit hash-anchored.reports/pre_selection_implementation_audit.md: outcome-blind code audit completed before the selection run.runs/seam_selection*.json*: complete hash-locked raw rows and terminal selection summary.analysis/selection_diagnostics.json: task/alias, verbalization, unmasked, paired, bootstrap, and explicitly non-rescuing logit-residual diagnostics.
Report
Rendered from reports/report.md
Status
Terminal COMMIT_SLOT_SEAM_FAIL; all later stages are ineligible.
Purpose
Natural close and close-only free-form output both failed before J value. This experiment supplies fixed answer syntax but not identity, constrains the next choice to public aliases, requires real thought to beat both an immediate slot and the same thought tokens in shuffled order, and then tests whether a scalar J value coordinate improves the semantic choice.
Plumbing
CPU smoke only: 96/96 unique fresh exact-depth tasks, zero overlap with four parents, no visible depth-one fits, exact lens hash, and reachable gates. The CPU check caught an initial generator-seed collision before design freeze; the entire seed block was replaced and the final manifest has zero overlap.
After the immutable design commit, model smoke passed the pinned 32-layer, 2,560-wide revision; five rank-24 lens matrices; 12 distinct leading-space alias tokens; fixed slot tokenization; finite constrained logits; and native/free-form cache contracts. Peak allocated GPU memory was 8,515,461,632 bytes. The receipt contains neither task correctness nor the chosen alias.
The pre-selection implementation audit then verified exact real/shuffled token multisets, runtime data/config hashes, complete-row buffering, unmasked-logit diagnostics, control finiteness, and fail-closed later stages. No scientific outcome informed the audit.
Selection results
The scientific run completed 48/48 native paths, 144 real slot rows, 144 exact-length shuffled-thought slot rows, 144 matched close-only free-form rows, and 16 task-level no-thought slots. Every path contacted 1,024; there were no natural closes or malformed rows. All cache, finite, exact-multiset, cardinality, and observed-gate-feasibility contracts passed.
| cap | real slot | no-thought | shuffled | real−none | real−shuffle | mixed | pass |
|---|---|---|---|---|---|---|---|
| 256 | 10/48 (20.8%) | 4/16 (25.0%) | 11/48 (22.9%) | −4.17pp | −2.08pp | 2 | no |
| 512 | 9/48 (18.8%) | 4/16 (25.0%) | 15/48 (31.2%) | −6.25pp | −12.5pp | 2 | no |
| 1024 | 15/48 (31.2%) | 4/16 (25.0%) | 11/48 (22.9%) | +6.25pp | +8.33pp | 5 | no |
The 1,024 rung passed the frozen 20%--80% accuracy range, +5pp no-thought gap, +3pp shuffled gap, and 100% finite-row gate. It failed only the requirement for six tasks with both correct and incorrect traces: observed five. Because the gate was frozen and mathematically reachable, the one-task miss is terminal; it does not authorize confirmation or a J-space value fit.
The slot itself repaired most of the prior emission wall. At 1,024, an alias was already the unmasked full-vocabulary top token on 41/48 rows (85.4%), total alias probability mass averaged 68.5%, and only 2/15 correct constrained choices needed the mask because a non-alias was the unmasked top. By contrast, matched close-only free-form parsing was 2/48 (4.2%), exact success 1/48, and 43/48 answers exhausted 16 tokens. Fixed syntax—not the close token—created answer mode, but answer mode alone did not make the semantic effect stable enough.
The run sampled 49,152 thought tokens in 1,678.391 seconds. Peak allocated GPU memory was 8,705,302,016 bytes. Both real and shuffled slot prefills processed 148,395 tokens; the free-form controls generated 2,171 answer tokens.
Frozen decision
No cap passes, so selected_cap=null and the automatic decision is COMMIT_SLOT_SEAM_FAIL. The untouched seam-confirmation, value-fit, numeric control, and causal-confirmation task sets remain unopened. No J-space observation or intervention result exists.
Post-decision diagnostics
Diagnostics cannot rescue the gate. At 1,024, real thought beat shuffled on seven paired traces and lost on three, a net four. Across 16 task units, the real-minus-shuffled mean was +8.33pp with a 10,000-resample 95% interval [-6.25pp, +27.08pp]; real-minus-no-thought was +6.25pp with [-14.58pp, +27.08pp]. Both include zero.
The signal was concentrated. Real thought produced four correct cat rows versus zero shuffled and two correct ocean rows versus zero, while all six tiger rows were correct under both real and shuffled thought. Eight tasks were 0/3 real, four were 1/3, one was 2/3, and three were 3/3. The slot chose only six of twelve aliases and selected tiger on 25/48 rows; shuffled thought selected it on 31/48. This concentration explains why a modest pooled gain did not earn the task-variation gate.
Direct verbalization does not explain the hint. Prefixes containing the correct alias token were 2/10 correct (20%); prefixes without it were 13/38 (34.2%). Rows with no alias mention at all were 10/24 correct. Ordered-content benefit, where present, is not simply copying the written correct alias into the slot.
Three label-free but explicitly post-hoc residual decoders were checked only to choose a fresh successor. At 1,024, subtracting the same-task no-thought logits scored 13/48, subtracting exact shuffled-thought logits scored 8/48, and subtracting a global no-thought log-bias scored 14/48, all below the registered 15/48 slot. Bias subtraction is therefore not the next branch.
Interpretation
The fixed slot successfully separates emission syntax from semantic resolution. The earlier close-only failure was mostly an answer-mode problem; First: makes an alias naturally competitive even without masking. At the semantic level, 1,024 ordered tokens show a small coherent-content hint, but it is clustered and task-level uncertainty remains wide. The honest next step is a fresh, powered, fixed-1,024 replication with more balanced task units and a task-bootstrap gate—not confirmation of this failed selector, not a relaxed mixed-task threshold, and not a larger cap yet.
Boundary
The slot and alias mask are deployment scaffolds. Any positive is constrained choice evidence, not natural/free-form capability. This result does not license a value label, J coordinate, donor, controller, or capability claim.
Artifact Manifest
See artifact_manifest.yaml.
Experiment log 3
Show the running log (3 entries, 2026-07-12)
2026-07-12 — Intake and design
- Created as a distinct answer-interface change after terminal close-only
FORCED_COMMIT_SEAM_FAIL. - Fixed action: append close plus
First:; restrict only the next choice to the 12 public aliases; never supply answer identity. - Retained no-thought slot, unmasked logits, and matched close-only free-form controls; adversarial review added an exact-length token-multiset shuffle as the load-bearing coherent-content control before design freeze.
- Tightened seam headroom to 20%--80% and required real-over-shuffled gains in both selection and untouched confirmation.
- First CPU generation attempt caught an exact seed collision with the prior causal split. Replaced the entire new seed block before model calls.
- Final CPU smoke: 96/96 unique fresh exact-depth tasks, zero overlap with four parents, exact lens hash, reachable gates.
- Completed adversarial review before any model call; outcomes unopened.
2026-07-12 — Outcome-blind model smoke
- Anchored README, preregistration, 58-point adversarial review, semantic config, lens, and all four procedural splits to design commit
73deac8a. - Passed exact revision/architecture, tokenizer, rank-24 lens, one-token alias, fixed-slot, finite-logit, and cached forward contracts.
- Peak allocated GPU memory: 8,515,461,632 bytes. Correctness, trace text, and the chosen alias were not recorded; no scientific result opened.
- A second outcome-blind implementation audit added runtime exact-multiset shuffle hashes, full-vocabulary alias mass, row cardinality, control-finiteness, and observed-baseline feasibility receipts before selection.
2026-07-12 — Terminal seam selection
- Completed 48 traces and all 448 slot/control rows with exact cardinality, cache, finite, shuffle-multiset, and hash contracts.
- Caps 256/512/1024 yielded real accuracy 10/48, 9/48, and 15/48. The 1,024 rung beat no-thought by +6.25pp and shuffled thought by +8.33pp but had five mixed tasks versus the frozen six-task gate.
- Honored terminal
COMMIT_SLOT_SEAM_FAIL; no confirmation, value, control, or causal stage opened. - Post-decision task bootstrap intervals crossed zero; alias/task concentration was strong; correct-alias mentions did not explain success; three label-free logit-residual diagnostics all underperformed the registered 15/48 slot.
- Next branch: a new powered fixed-1,024 seam replication with fresh tasks and a task-level uncertainty gate. Do not relax this experiment or raise its cap.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python experiments/qwen35_4b_commit_slot_jacobian_value_transport/scripts/run.py --stage smokeFull run
.venv/bin/python experiments/qwen35_4b_commit_slot_jacobian_value_transport/scripts/run.py --stage seam-selectionRun steps are documented inside the experiment folder (README and scripts).