Qwen3.5-4B Forced-Commit Jacobian Value Transport
The one idea you need
A reasoning model works in two phases: a private scratchpad where it thinks, then a final answer, split by a hidden "done thinking" marker. This test jammed that marker in early to cut thinking short, like telling a colleague mid-sentence "time's up" without handing them an answer form.
The question
If a small model keeps thinking past its budget and never wraps up on its own, can you just insert a stop-thinking marker to make it commit to an answer?
What we found
No. Inserting the model's own "done thinking" marker cut the reasoning off but almost never flipped it into answer mode. Across three thinking budgets, only 13% to 19% of forced stops produced anything readable, and just 1 of 48 was correct, versus the 90% readability the test demanded. Usually the model simply restarted its analysis instead of committing.
Why it matters
If you cap a small model's thinking to save tokens, don't trust the stop signal alone to produce an answer. Give it an explicit answer slot to fill, and verify output is readable, not just that thinking stopped.
On this page
Results at a glance 1
How to read
Bars grouped by the thinking budget before the forced stop (256, 512, 1024 tokens); height is the share of 48 traces. Three bars per group: readable-answer rate, correct-answer rate, and still-rambling-at-the-cap rate. Higher readable and correct is better; lower rambling is better.
Takeaway → Readable bars sit near one-fifth, correct bars hug zero, while rambling bars tower above 85% at every budget, so the stop signal never produced a clean answer.
Data table
| forced-commit thought cap | parse rate | exact success rate | answer-cap contact rate |
|---|---|---|---|
| 256 cap | 12.5% | 2.1% | 91.7% |
| 512 cap | 16.7% | 2.1% | 85.4% |
| 1024 cap | 18.8% | 2.1% | 95.8% |
Numbers from experiments/qwen35_4b_forced_commit_jacobian_value_transport/analysis/seam_metrics.json
Technical framing
Forced-close answer usability across thought caps — All rows used the explicit counterfactual close action. No cap met 90% parse, 5%--95% success, six mixed tasks, or <=5% answer-cap contact; confirmation and all J-space stages were canceled.
In the author’s words from the Overview · “Results”
All 48 traces required forced close at every cap. The run sampled 49,152 thought tokens plus 2,225 answer tokens in 1,640.539 seconds. Typical post-close outputs restarted analysis or emitted free-form reasoning instead of the requested slot. A post-decision regex diagnostic tolerated aliases attached directly to special EOS tokens. It raised parse counts only to 7/11/10 and correct counts to 1/2/2 at 256/512/1024. Those remain far below every frozen parse, success, mixed-task, and answer-termination gate, so the parser edge case does not affect the decision.
Overview
This study first validates an explicit fixed-budget commit action, then asks whether a scalar Jacobian continuation-value coordinate changes fresh answers under that same action.
Research Program
- Primary:
interpretability_and_diagnostics. - Secondary:
test_time_reasoning_budget. - Direct parents:
qwen35_4b_native_thought_jacobian_value_transportandqwen35_4b_native_thought_seam_budget_ladder. - Mechanism anchor:
qwen35_4b_jacobian_transport_control_replication. - Forced-interface warning:
qwen35_4b_answer_potential_trace_sft(C51).
Question
When Qwen3.5-4B does not naturally finish by a fixed thought budget, can an explicit deployed “commit now” action expose a parseable, variable-quality answer seam—and is continuation value at the last thought token both decodable and causally writable in the replicated 24-coordinate J space?
Hypothesis
The failed natural-close ladder showed active re-analysis through 1,024 tokens, not exact looping. A budget controller that injects the standard </think> token may therefore turn partially resolved thoughts into usable answers. If a task-general value signal is present, fresh forced-policy continuations should be rankable from J coordinates at the live prefix endpoint, and raising only that scalar coordinate should improve new continuations beyond shuffled-axis, exact random, answer-identity, raw, ActAdd, and non-J controls.
The policy is deliberately counterfactual to autonomous close. It is legitimate only because calibration, value labels, causal evaluation, and any eventual deployment use the exact same commit action. Nothing here may be described as a natural seam.
Setup
- Model: only
Qwen/Qwen3.5-4B, revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Lens: exact replicated 24-coordinate context lens, SHA-256
e373b6e93956fdfc5cb446e9bee8249655707c8258a7868f0653d11f1ffd0213, layers 4--8. - Backend: Transformers bf16 SDPA, unpadded batch one, audited KV cache.
- Fresh data: 16 seam-selection, 16 seam-confirmation, 32 value-fit, and 32 causal-confirmation tasks; 96/96 unique and zero overlap with three parents.
- Task hardening: every visible set identifies one first-operation type and has no matching depth-one operation, so “exactly two” is behaviorally true.
- Sampling: temperature 0.6, top-p 0.95, top-k 20, three traces per task.
- Candidate caps: 256, 512, 1024. Selection uses paired trace prefixes; untouched confirmation opens only the smallest passing cap.
- Policy: if the model naturally closes before the cap, keep that answer; otherwise append one
</think>token and generate at most 16 answer tokens.
Frozen Gates
At selection and confirmation, policy parse and forced-only parse must each be at least 90%, at least half the rows must actually require the forced action, policy success must lie in 5%--95%, at least six tasks must mix correct and incorrect policy outcomes, and answer-cap contact must be at most 5%.
Only FORCED_COMMIT_SEAM_REPLICATED opens value fitting. Prefixes at 0.5 and 1.0 of the selected cap receive three disjoint forced-policy continuations. The held-out-by-task J readout must reach task-macro pairwise AUC 0.65, beat correct alias activity by 0.03, and leave the within-task shuffled null near chance.
Only a value pass opens exact post-bf16 control calibration and one untouched causal confirmation. The primary scalar clamp must improve paired success by at least 0.10 with a positive bootstrap lower bound and beat exact random, shuffled-axis, and matched non-J controls by frozen margins. Identity/full-donor arms cannot rescue it.
Run
CPU smoke:
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python \
experiments/qwen35_4b_forced_commit_jacobian_value_transport/scripts/run.py \
--stage smokeAfter the immutable design commit is anchored:
.venv/bin/python experiments/qwen35_4b_forced_commit_jacobian_value_transport/scripts/run.py --stage model-smoke
.venv/bin/python experiments/qwen35_4b_forced_commit_jacobian_value_transport/scripts/run.py --stage seam-selection
.venv/bin/python experiments/qwen35_4b_forced_commit_jacobian_value_transport/scripts/run.py --stage seam-confirmationLater stages remain fatal placeholders until their audited implementations are committed; the runner refuses to emit placeholder value or causal results.
Status
Terminal FORCED_COMMIT_SEAM_FAIL. Design and the 46-threat adversarial review were frozen before model calls; CPU and outcome-blind model smokes passed. The complete 48-trace/144-policy-row selection found that appending close alone did not reliably switch the model into answer mode. No cap was selected, and seam confirmation, value fitting, controls, and causal outcomes remain sealed.
Results
| cap | forced parse | exact success | mixed tasks | answer-cap contact | gate |
|---|---|---|---|---|---|
| 256 | 6/48 (12.5%) | 1/48 (2.1%) | 1 | 44/48 (91.7%) | fail |
| 512 | 8/48 (16.7%) | 1/48 (2.1%) | 1 | 41/48 (85.4%) | fail |
| 1024 | 9/48 (18.8%) | 1/48 (2.1%) | 1 | 46/48 (95.8%) | fail |
All 48 traces required forced close at every cap. The run sampled 49,152 thought tokens plus 2,225 answer tokens in 1,640.539 seconds. Typical post-close outputs restarted analysis or emitted free-form reasoning instead of the requested slot.
A post-decision regex diagnostic tolerated aliases attached directly to special EOS tokens. It raised parse counts only to 7/11/10 and correct counts to 1/2/2 at 256/512/1024. Those remain far below every frozen parse, success, mixed-task, and answer-termination gate, so the parser edge case does not affect the decision.
Scope
No value coordinate was fit and no activation was patched. The failure precedes J space: a lone close token is not a usable answer-emission interface. The next distinct experiment may supply syntax but not identity—append close plus the fixed First: slot and read only the alias choice—while retaining free-form forced answer as a control. It must use fresh tasks and a new adversarial review.
If a later value/causal stage becomes eligible, ground-truth continuations and donors remain oracle. Capability still requires a non-oracle controller that beats frozen Qwen, strongest controls, and matched-compute sampling.
Knowledgebase Update
- Program ledgers: record terminal close-only interface failure.
- Shared synthesis: distinguishes close-token injection from an answer slot.
- Claim ledger: no claim ID while the repository claim re-grade is open.
Artifacts
assets/context_lens.pt: byte-identical replicated lens.data/procedural/: four frozen fresh splits and manifest.runs/smoke/: CPU/gate receipts.reports/preregistration.md: immutable decision rules.reports/design_review.md: adversarial review before model work.reports/artifact_manifest.yaml: reproduction/omission contract.
Report
Rendered from reports/report.md
Status
Terminal FORCED_COMMIT_SEAM_FAIL; all later stages ineligible.
Purpose
Natural closure failed on 48/48 traces through 1,024 tokens. This successor treats close injection as the explicit deployed budget-controller action, first validates that interface on fresh data, and only then asks whether forced-policy continuation value is readable and causal in J space.
Method
See preregistration.md. Selection tests 256/512/1024 paired prefixes and freezes the smallest forced-commit policy meeting parse, headroom, mixed-task, and answer termination gates. Untouched confirmation opens only that cap. Later value and causal stages are strictly gated and currently refuse placeholders.
Results
CPU smoke produced 96/96 unique fresh exact-depth-two tasks, zero overlap with all three scientific parents, no visible depth-one fits, exact replicated lens hash, and reachable seam gates.
The non-result-bearing model smoke verified the exact pinned model, special and alias token IDs, and rank 24 at lens layers 4--8. Its native trace forward input lengths were [375, 1, 1, 1, 1, 1, 1, 1]; forced replay appended eight thought tokens plus close and used [384, 1]. Both cache audits passed. Close injection was explicitly marked counterfactual and no correctness was computed or stored. The scientific seam selection then completed 48 traces and 144 paired policy rows. All 48 traces contacted 1,024, so every cap used forced close.
| cap | forced parse | exact success | mixed tasks | answer-cap contact |
|---|---|---|---|---|
| 256 | 6/48 (0.125) | 1/48 (0.0208) | 1 | 44/48 (0.9167) |
| 512 | 8/48 (0.1667) | 1/48 (0.0208) | 1 | 41/48 (0.8542) |
| 1024 | 9/48 (0.1875) | 1/48 (0.0208) | 1 | 46/48 (0.9583) |
None approached the frozen 90% overall/forced parse gates, 5% success floor, six mixed tasks, or <=5% answer-cap contact. The run sampled 49,152 thought and 2,225 answer tokens in 1,640.539 seconds. All trace and replay cache audits passed. Trace SHA-256 is af5ed5ef6df892d98760c51c4881a1e29da911fb74549bebfe1b17be0cf6fbd8; policy rows SHA-256 is 432668875b5183d1a32f649e41686951362bf3e88c6c374ee7045903b2c2cfe6.
Post-decision parser diagnostic
Some correctly shaped answers placed the special EOS token directly after the alias, which the frozen whitespace parser rejected. A regex diagnostic accepting that suffix raised parse counts from 6/8/9 to 7/11/10 and correct counts from 1/1/1 to 1/2/2. Robust parse was still only 14.6%/22.9%/20.8%, answer-cap contact was unchanged, and every gate remained far out of reach. The registered result therefore remains valid; parser relaxation cannot rescue it.
Decoded rows inspected only after the automatic decision show the primary failure: after </think>, the model frequently began a new analysis, explained a free-form operation sequence, or refused to commit rather than entering the requested First: slot.
Frozen decision
No cap passes, so selection returns no cap and terminal FORCED_COMMIT_SEAM_FAIL. The untouched seam-confirmation, value-fit, and causal-confirmation task sets remain unopened. No J-space conclusion is licensed.
Interpretation
Close-token injection is not equivalent to an answer-emission seam for this model/workload. This independently echoes C51: a counterfactual post-thinking state can exist yet fail deployment-shaped expression. The failed dimension is now localized more sharply than “forced close”: the controller supplied a mode delimiter but not an output slot, and the model often re-entered analysis.
The next experiment must be distinct and fresh. A fixed commit-slot controller may append </think> plus syntax First: and then measure the next alias choice, with the close-only free-form policy retained as a control. Supplying syntax does not supply answer identity. Constrained/slot behavior must be reported as its own deployment interface, not natural reasoning or a capability gain. Only after that interface preserves correctness headroom may J value reopen.
Interpretation Boundary
Injected close remained counterfactual and unusable. The terminal result is an interface negative, not evidence about value, J-space causality, or installed capability.
Artifact Manifest
See artifact_manifest.yaml.
Experiment log 3
Show the running log (3 entries, 2026-07-12)
2026-07-12 — Intake and design
- Created as the distinct successor to terminal
NO_BUDGET_SELECTED, not an in-place weakening of the natural-close gate. - Ran related-work search; named the two seam parents, replicated J mechanism, and C51 forced-interface warning.
- Defined close injection as the actual deployed policy and labeled every such state counterfactual to autonomous termination.
- Froze selection/confirmation before value and causal stages.
- Completed the adversarial review before any model call.
- CPU smoke generated 96/96 unique fresh tasks, zero parent overlap, no visible depth-one shortcut, exact lens hash, and reachable seam gates.
- Scientific outcomes remain unopened.
2026-07-12 — Outcome-blind model smoke
- Ran only after the reviewed design and hash anchor were pushed.
- Verified pinned Qwen3.5-4B, exact special/alias tokens, and rank-24 lens matrices at all five frozen layers.
- Native trace cache audit:
[375, 1, 1, 1, 1, 1, 1, 1]. - Explicit forced replay audit:
[384, 1]after appending eight thought tokens and exactly one close token. - Recorded the action as counterfactual; no answer correctness was computed.
- Scientific seam selection remains unopened.
2026-07-12 — Forced-commit seam terminal
- Opened the 48-trace selection once and evaluated all three paired caps, producing 144 complete policy rows.
- Every trace required forced close. Forced parse was 6/48, 8/48, and 9/48 at 256/512/1024; exact success was 1/48 at every cap; only one task mixed outcomes.
- Answer-cap contact was 44/48, 41/48, and 46/48, far above the 5% ceiling.
- Terminal decision:
FORCED_COMMIT_SEAM_FAIL; no cap selected and all later task sets remained sealed. - Sampled 49,152 thought plus 2,225 answer tokens in 1,640.539 seconds.
- A post-decision EOS-tolerant regex raised parse to 7/11/10 and correct to 1/2/2, still decisively below every gate. No parser repair can rescue the run.
- Next branch: a fresh fixed answer-slot interface that supplies
First:syntax but never the alias identity; close-only free-form remains its control.
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python experiments/qwen35_4b_forced_commit_jacobian_value_transport/scripts/run.py --stage smokeFull run
.venv/bin/python experiments/qwen35_4b_forced_commit_jacobian_value_transport/scripts/run.py --stage seam-selectionRun steps are documented inside the experiment folder (README and scripts).