Qwen3.5-4B Native-Thought Seam Budget Ladder
The one idea you need
This 4-billion-parameter model reasons on a private scratchpad, then is supposed to write an end-of-thinking mark and its answer. It behaved like a worker told to show all work who keeps filling pages rechecking, and never writes the final line.
The question
If you let a small reasoning model think freely on simple list puzzles and wait for it to finish and hand you an answer, will it ever stop?
What we found
No. Across 48 tries on simple list-transformation puzzles, and at every thinking budget up to 1,024 tokens, the model closed its reasoning and produced an answer exactly zero times. It always burned the whole budget still analyzing. To be usable it needed to stop on its own at least 80 percent of the time, and it never once did. This was not a stuck repeating loop; it kept genuinely reworking the problem, just never committing.
Why it matters
Do not trust a small thinking model to end its own reasoning. If your pipeline waits for a natural stop, you may get nothing back. Cap the thinking, force a commit at the cap, and measure your real self-stop rate before relying on it.
On this page
Results at a glance 1
How to read
Grouped bars for three thinking-budget caps (256, 512, 1,024 tokens) along the bottom; height is the share of 48 traces. One color is the natural-close rate (higher is better), the other is the share that ran into the ceiling.
Takeaway → Every natural-close bar sits flat at zero while every ceiling bar is full at 100 percent, so the model exhausted its room and never once stopped to answer.
Data table
| frozen selection rung | natural close rate | cap-contact rate |
|---|---|---|
| 256 cap | 0% | 100% |
| 512 cap | 0% | 100% |
| 1024 cap | 0% | 100% |
Numbers from experiments/qwen35_4b_native_thought_seam_budget_ladder/analysis/selection_metrics.json
Technical framing
Natural close versus cap contact across the frozen ladder — All 48 paired traces reached the 1,024 ceiling without emitting the natural close token; lower rungs are exact right-censored views of those same paths. No cap was selected and confirmation was ineligible.
In the author’s words from the Overview · “Results”
The cached runner made 49,152 audited one-step generation forwards and sampled 49,152 thought tokens in 1,618.1 seconds. All 48 rows passed the cache contract. Post-decision token diagnostics found no exact short-period repetition across the final 256 tokens of any trace; absence of an exact loop does not prove semantic progress. Per the frozen smallest-cap rule, no cap is selected. The 24 confirmation tasks are never loaded into a model stage. Raising the ceiling or adding a rung here is prohibited.
Overview
This study selects the smallest naturally closing Qwen3.5-4B thought cap from 256/512/1024 on fresh list tasks, then repeats that exact cap on untouched tasks before any Jacobian value test.
Research Program
- Primary:
interpretability_and_diagnostics. - Secondary:
test_time_reasoning_budget. - Cross-cutting catalog assignment:
benchmark_generalization, because the repository'sladdertag routes frozen selection/confirmation ladders there. - Program question: what exact natural reasoning interface is valid before a thought-prefix state can be assigned continuation value or causally edited?
- Direct parent:
qwen35_4b_native_thought_jacobian_value_transport. - Other anchors:
qwen35_4b_thinking_budget_scalingandqwen35_4b_answer_potential_trace_sft.
Question
On fresh, first-operation-identifiable list tasks, what is the smallest cap in [256, 512, 1024] at which Qwen3.5-4B naturally emits </think> and a parseable answer often enough to support a later thought-prefix value experiment—and does that cap pass unchanged on an untouched confirmation split?
Hypothesis
The failed 160-token parent exposed an interface-budget problem, not a J-space result. Prior native-thinking evidence suggests that 512--1024 tokens is the model's ordinary reasoning scale. At least one frozen rung should therefore pass natural close, parse, headroom, and usable-prefix gates, then replicate on fresh tasks. The smallest passing cap is selected; 1024 is expected to be the most likely winner but receives no preferential rule.
Setup
- Model: only
Qwen/Qwen3.5-4B, revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Backend: Transformers bf16 SDPA, unpadded batch one, KV cache enabled. This is the backend required by the immediate activation-intervention successor.
- Task source: 40 newly generated procedural depth-two list transformations; 16 budget-selection and 24 untouched confirmation tasks.
- Freshness: 40/40 unique fingerprints, zero overlap with both Jacobian parents; visible examples exhaustively identify one first-operation type.
- Sampling: temperature 0.6, top-p 0.95, top-k 20, three traces per task.
- Ladder: paired right-censoring at 256, 512, and 1024 thought-generation steps.
- Answer allowance: 16 naturally generated tokens after a natural close.
- Prohibited: injected close tokens, force-close answer generation, fallback to a different cap after confirmation, benchmark content, or correctness tuning.
The selection run generates each trace once to the 1024 ceiling and classifies whether its natural close was reachable at each smaller cap. This is paired right-censoring, not three independent samples. The untouched confirmation run opens only the smallest selected cap.
Frozen Decisions
Selection requires at least 80% natural close, 90% parsing conditional on close, 32 usable traces of at least 16 thought tokens, 5%--95% usable success, and six mixed-success tasks. Confirmation scales the usable/mixed counts to 48/eight, repeats the rates unchanged, and additionally requires the 95% Wilson lower bound on natural close to be at least 0.70.
Terminal labels are NO_BUDGET_SELECTED, SEAM_NOT_REPLICATED, or NATURAL_SEAM_REPLICATED. Only the last label licenses a separate value/Jacobian experiment at the frozen selected cap. It is setup evidence, not a capability gain or a J-space mechanism result.
Run
CPU smoke:
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python \
experiments/qwen35_4b_native_thought_seam_budget_ladder/scripts/run.py \
--stage smokeAfter the immutable design boundary is anchored:
.venv/bin/python experiments/qwen35_4b_native_thought_seam_budget_ladder/scripts/run.py \
--stage model-smoke
.venv/bin/python experiments/qwen35_4b_native_thought_seam_budget_ladder/scripts/run.py \
--stage budget-selection
.venv/bin/python experiments/qwen35_4b_native_thought_seam_budget_ladder/scripts/run.py \
--stage seam-confirmationStatus
Terminal NO_BUDGET_SELECTED. Design and the 32-threat adversarial review were frozen before model calls. CPU and outcome-blind model smokes passed, including the exact cached-forward contract. The complete selection then found zero natural closes at every rung: all 48 traces consumed all 1,024 allowed thought steps. Confirmation is ineligible and remains unopened.
Results
| cap | natural close | parseable | usable | cap contact | gate |
|---|---|---|---|---|---|
| 256 | 0/48 | 0/48 | 0/48 | 48/48 | fail |
| 512 | 0/48 | 0/48 | 0/48 | 48/48 | fail |
| 1024 | 0/48 | 0/48 | 0/48 | 48/48 | fail |
The cached runner made 49,152 audited one-step generation forwards and sampled 49,152 thought tokens in 1,618.1 seconds. All 48 rows passed the cache contract. Post-decision token diagnostics found no exact short-period repetition across the final 256 tokens of any trace; absence of an exact loop does not prove semantic progress.
Per the frozen smallest-cap rule, no cap is selected. The 24 confirmation tasks are never loaded into a model stage. Raising the ceiling or adding a rung here is prohibited.
Scope
This experiment selected no natural interface budget. It did not fit a value coordinate, patch an activation, train a controller, compare capability against sampling, or license a claim. The result does not count against J-space value: the close-and-answer event remained unreachable.
The warranted branch is a new experiment that makes forced commit an explicit, deployable protocol rather than mislabeling it natural. It must first pass fresh force-close parse/headroom controls, preserve C51's counterfactual-state warning, and use exact-prefix replay plus post-bf16 controls at every live sequence length before any causal value conclusion.
Knowledgebase Update
- Program evidence: records terminal
NO_BUDGET_SELECTED. - Program backlog: pivots from natural close to an explicit forced-commit policy.
- Claim ledger: no claim ID; the repository claim re-grade remains open.
Artifacts
data/procedural/: frozen fresh splits and manifest.runs/smoke/: CPU and gate-reachability receipts.runs/model_smoke/: backend/token/cache plumbing only.runs/budget_selection*: paired ladder rows and frozen selection.runs/seam_confirmation*: untouched single-cap confirmation.reports/preregistration.mdandreports/design_review.md: immutable rules.reports/artifact_manifest.yaml: omission and reproduction contract.
Report
Rendered from reports/report.md
Status
Terminal NO_BUDGET_SELECTED; confirmation ineligible.
Purpose
The direct parent stopped because all 48 native-thinking traces were still inside <think> at 160 tokens. This separate experiment selects and confirms a natural-close cap before any continuation-value or causal J-space measurement.
Method
See preregistration.md. The selection split uses paired right-censoring at 256/512/1024; the confirmation split opens only the smallest selected cap. No close token is injected and no cap-bound trace receives an answer continuation.
Results
The non-result-bearing smoke loaded the exact pinned 32-layer, 2560-wide model, verified all special/alias token IDs, rendered a 472-token prompt, and sampled eight tokens. Its forward-input lengths were [472, 1, 1, 1, 1, 1, 1, 1], so the prefill-plus-KV-cache contract passed. Correctness was not computed or stored.
The complete 16-task, three-trace selection stage then produced:
| cap | close | parse/all | usable | mixed tasks | cap contact | pass |
|---|---|---|---|---|---|---|
| 256 | 0/48 | 0/48 | 0/48 | 0 | 48/48 | no |
| 512 | 0/48 | 0/48 | 0/48 | 0 | 48/48 | no |
| 1024 | 0/48 | 0/48 | 0/48 | 0 | 48/48 | no |
Every trace stopped as think_cap_without_close after exactly 1,024 thought tokens. The stage sampled 49,152 tokens through 49,152 audited cached forwards in 1,618.080 seconds. Every scientific row passed the prefill/one-token cache contract. The row file SHA-256 is 17e3b107154079ecd857af45544c92c2e11b13cd495edfeb6eb24dcf97f5d39c.
After the automatic terminal decision, token diagnostics found 0/48 exact periodic tails over the final 256 tokens for periods 1--32. Maximum trigram reuse ranged from 10 to 34 (median 17.5). Sampled tails showed continued task analysis and rechecking, but qualitative content is diagnostic only and cannot rescue the failed close gate.
Frozen decision
No rung satisfies even the first natural-close requirement, so the smallest-cap selector returns no cap and writes terminal NO_BUDGET_SELECTED. The untouched 24-task confirmation split is ineligible. It was not generated or scored.
Interpretation
The natural seam is absent through 1,024 steps for this prompt/workload under the audited cached backend. This repeats the parent's interface failure at a 6.4x larger ceiling and on fresh tasks, but it still precedes every J-space question. No evidence about value decodability, certainty, causal transport, or capability was produced.
The preregistration forbids appending 2,048 or opening confirmation at a larger cap. The next scientifically distinct route is an explicit forced-commit controller: inject close as the deployed policy, calibrate autonomous answer parse/headroom on new tasks, and define prefix value under that exact policy. Such a state is counterfactual relative to natural closure (C51), so reports must say so; its legitimacy comes only from testing the same protocol at deployment. If that seam passes, a later causal J edit must replay the live prefix and build exact post-bf16 controls per length.
Interpretation Boundary
This negative is setup evidence only. It establishes that natural closure is not the usable interface through 1,024 on this workload; it does not lower the evidence bar for any forced-commit, value, or capability result.
Artifact Manifest
See artifact_manifest.yaml.
Experiment log 3
Show the running log (3 entries, 2026-07-12)
2026-07-12 — Intake and design
- Created as the required separate successor to the parent's terminal
NO_NATURAL_SEAMat 160 thought tokens. - Ran related-work search and named the closest duplicate plus three anchors.
- Froze a paired 256/512/1024 selection ladder and untouched single-cap confirmation; no forced close or failed-confirmation fallback is allowed.
- Completed the adversarial review before any model call.
- CPU smoke generated 40/40 unique fresh task fingerprints with zero overlap against both scientific parents and proved all terminal gates reachable.
- No scientific outcome has been opened.
2026-07-12 — Outcome-blind model smoke
- Ran only after the reviewed design and executable hash anchor were pushed.
- Loaded pinned Qwen3.5-4B (32 layers, width 2560) under Transformers 5.13.0, torch 2.11.0+cu129, bf16 SDPA on an RTX 6000 Ada.
- Verified exact think/open/EOS IDs and 12 unique one-token aliases.
- Audited forward input lengths
[472, 1, 1, 1, 1, 1, 1, 1]; cached decoding is active rather than a silently repeated full-prefix path. - Eight diagnostic tokens were sampled; no answer correctness was computed or recorded. Scientific selection remains unopened.
2026-07-12 — Budget selection terminal
- Opened the 48-trace selection once under the frozen 1,024 maximum rung.
- Completed all 48 rows under one model instance; all passed the audited cache contract and all stopped as
think_cap_without_closeat exactly 1,024 tokens. - Nested metrics were zero natural close, parse, and usable traces at 256, 512, and 1,024. Decision:
NO_BUDGET_SELECTED. - Sampled 49,152 tokens/forwards in 1,618.080 seconds. Rows hash:
17e3b107154079ecd857af45544c92c2e11b13cd495edfeb6eb24dcf97f5d39c. - Confirmation is ineligible and was not opened; no cap or threshold changed.
- Post-decision diagnostics found no exact 1--32-token periodicity across any 256-token tail. This does not turn cap-bound reasoning into a natural seam.
- Next branch: a separate forced-commit protocol whose injected close is an explicit deployed action, not evidence of autonomous termination.
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python experiments/qwen35_4b_native_thought_seam_budget_ladder/scripts/run.py --stage smokeFull run
.venv/bin/python experiments/qwen35_4b_native_thought_seam_budget_ladder/scripts/run.py --stage budget-selectionRun steps are documented inside the experiment folder (README and scripts).