Research log Small Model Experimentation
GitHub

Test-Time Reasoning Budget

Study the native thinking-token budget as a first-class controllable test-time-compute axis for Qwen3.5-4B, which the corpus has universally disabled.

What we have learned

Seed Experiments

  • qwen_python_shaped_silent_executor: the only corpus experiment that ever enabled native thinking. Its CoT baseline reached 62.5% on length-4 programs (746 emitted tokens) but collapsed to 0% on length-24/32 at a fixed ~768-token thinking budget that was never swept. It was framed as a foil for "silent latent compute" (which was itself a controlled negative). This is the contrast the program exists to revisit: was the collapse a reasoning limit or a budget limit?

Corpus-Wide Fact (verified)

  • Across all 155 experiments, native thinking is disabled (enable_thinking=False ×48, True ×0 besides the one seed); <think> blocks are stripped as boilerplate. "Budget" always means evidence/probe/tool/program/sample budget, never reasoning tokens. So the reasoning-budget axis is genuine, verified white space.

Anchor Experiments

Confirmed Claims

  • Native thinking is a deployable win the corpus disabled. Greedy pass@1 0.76 → 0.91 (+15pp); the deployable line moves more than the oracle ceiling (pass@8 0.91 → 0.96) and the oracle−deployable gap narrows — the opposite of the C2-based prior. Paired-robust (17 fail→pass vs 2 pass→fail at think_1024 vs no_think; McNemar p≈0.001). So C2 (coverage ≫ deployable selection) does not hold for the thinking axis on MBPP.
  • More thinking is not monotonically better. Broad optimum ~512–1024 tokens then decline; unbudgeted (greedy 0.84) is worse than a cap (0.91). Shape corroborated by greedy and pass@1.
  • A visible-test budget controller is an efficiency win, not an accuracy win. A draft→escalate rule Pareto-dominates every fixed budget except the peak (matches ~0.88 at 113 mean thinking tokens vs fixed think_256/512 at 246–404), but cannot beat the best fixed budget (think_1024, 0.91). Its deployable gap to the oracle ceiling (0.93) is bounded by visible-test false-passes (~8–11%) — the C2 effect made concrete on the thinking axis.

Negative / Cautionary Findings

  • Much of the "thinking" benefit is not coherent reasoning. A shuffled-thinking control (scramble the model's own thinking tokens, keep count/scaffold) reproduces most of the gain; at 2048 shuffled = real. Evidence that coherent reasoning order adds beyond compute + scaffold

    • token-presence is weak and budget-dependent. Needs a stronger control (substitute a different

    task's thinking) before claiming the gain is "reasoning."

  • The exact optimum (1024) and the never-solved-bucket effect (3/9 tasks) rest on small n / single-seed; treat as suggestive, not pinned.
  • Reasoning budgets do not automatically transfer across workload classes within one substrate. In the verified-macro follow-up, a train-only plan-given calibration selected think@16,384 and a disjoint plan-given interface passed 16/16 records with zero unresolved caps. The fresh induction base at the identical budget then contacted the cap in 144/144 samples; only 13 were exact loops, leaving 131/144 unresolved and 60 answer-limit contacts. Doubling the allowance did not clear the workload: at think@32,768 all 144 samples again contacted the boundary, with 81 exact loops, 63 unresolved contacts, and 37 answer-limit contacts. Both rungs were excluded before decoding or scoring. A max-seqs-64 K=4 probe at think@49,152 later force-closed all 48 samples (34 loops, 14 unresolved, 13 answer-limit contacts) while generating 2,366,620 tokens in 4,035.356 seconds (586.47 tokens/s), but Amendment 12 had already made it diagnostic-only before its receipt: block-rounded demand was 2,433,024 tokens against 995,328 of cache. No decoded or scored content was inspected. The independent capacity-fit follow-up then completed a fresh K=4 probe with max-seqs 19; its live audit fit 963,072 tokens into a 997,888-token cache, leaving 34,816. All 48 samples still contacted the boundary (37 exact token-ID loops, 11 unresolved, 9 answer-limit), so 49k was rejected before decoding or scoring. It sampled 2,364,643 tokens in 5,012.451 seconds (471.754 tokens/s), 19.6% slower than the max-seqs-64 diagnostic's 586.471 tokens/s. The subsequent 61k attempt was stopped before a receipt after an audit found that its implicit CUDA-graph list covered only through width 8 rather than max-seqs 15, leaving no reusable rows. A separate exact-capture follow-up then passed both live-KV and exact-graph gates at 49k: 963,072 required tokens fit into 996,864 live tokens, and [1, 2, 4, 8, 16, 19] resolved exactly. Termination still failed with 38/48 periodic loops, 10/48 unresolved contacts, and 6/48 answer-limit contacts. That probe generated 2,363,163 tokens in 4,809.081 seconds (491.396 tokens/s), descriptively 4.16% faster than the implicit-capture capacity-fit probe. The terminal exact-capture 61k probe passed the same runtime gates: 950,400 required tokens fit into 997,888 with 47,488 headroom, and FULL decode graphs resolved exactly at [1, 2, 4, 8, 15]. Termination nevertheless failed with 40/48 periodic loops, 8/48 unresolved contacts, and 4/48 answer-limit contacts; 2,951,995 sampled tokens took 7,422.886 seconds (397.688 tokens/s). The selector ended pass=false with no selected budget, authorizing no K=12 arm or semantic analysis. Cache-safe concurrency and active-width graph coverage are both required for a clean inference envelope, but neither guarantees useful termination. This is setup evidence, not a task result: calibrate on the actual workload class, gate termination before correctness, stop increasing context once the registered ladder is exhausted, and never interpret a cap-bound score. No decoded or scored content informed these decisions.
  • Answer-conditioned trace scores can validate an artificial post-thinking state. In C51, 99.37% of 2,048 thoughts hit the 512-token cap. Canonical-answer gain after an injected close contained real trace information, but fresh answers parsed only 13.2% and the scorer missed its actionable G0 bars. Natural closure and autonomous commit must be launch gates when thinking traces feed selection or SFT.
  • Natural termination is absent through the deployed 1,024 scale on the fresh list-composition workload. The frozen paired selector observed 0/48 closes at 256, 512, and 1,024; all rows hit the largest cap and the untouched confirmation remained sealed. No exact 1--32-token periodicity occupied any final 256-token tail, so this does not join the 16k+ exact-loop line. Do not keep raising a natural cap. An external commit action is now a distinct deployable policy: valid only if the same forced interface clears parse and headroom gates at calibration and deployment, and always labeled counterfactual relative to autonomous close.
  • A close token alone does not establish answer mode. On a new exact-depth workload, forced-only parse stayed 12.5%--18.8%, success was 1/48 at every cap, and 85%--96% of post-close outputs exhausted 16 answer tokens. Decoded only after the automatic failure, many rows restarted analysis. A tolerant parser remained <=22.9%, so formatting edge cases do not explain the stop. The next controller may supply a fixed answer slot but must treat that syntax as part of deployment and retain close-only output as a control.
  • Entropy/varentropy localize interesting forks but do not make weight edits local. C52 round 2 selected 155 low-entropy, non-degenerate-varentropy confident wrong turns and compared demotion, positive-only uplift, and shuffled uplift. Pull-up was safer and true labels separated from shuffled locally, but every LoRA arm exceeded the 0.10 non-target-logit-drift ceiling and held-out coding stayed below base (39/72 vs 43/72). Higher varentropy was not monotonically safer; the lowest quartile was cleanest. Treat uncertainty as a routing/diagnostic variable, not a correctness label or pressure scale.
  • The model uses thinking as CONTENT (separability probe + foreign control). Linear probes show correctness is moderately decodable from the answer-token activation (AUC 0.64–0.76). The foreign-task-thinking ladder is decisive: splicing a different task's thinking collapses accuracy to ~4% (the model follows it to the wrong problem) — so thinking is not a content-free compute/scaffold crutch. Weak deployable spinoff: the probe partially flags C2 false-passes (visible-passer AUC ~0.60–0.68) only under thinking.

Correction

  • "Thinking is mostly compute/scaffold, not reasoning" was wrong for the efficient budget. The full content ladder (pure-compute filler arm included) gives, at budget 512: no_think 0.749, filler 0.744, shuffle 0.739, real 0.861, foreign 0.040. Attribution: pure compute (filler − no_think) −0.005; relevance (shuffle − filler) −0.005; coherent content (real − shuffle) +0.122; misleading content (foreign) −0.709. So pure compute buys ~0 and the efficient-budget gain is 100% coherent reasoning content, which the model uses (foreign → wrong problem). A budget sweep (512/1024/2048) then showed the coherence advantage (real − shuffle) does not shrink but grows (+0.105 → +0.108 → +0.150) — so this holds at every budget, and the scaling run's "2048 shuffle ≈ real" was a shuffle-protocol artifact. The "mostly compute" read only appeared through a greedy-metric lens (the representational slice is separate and noisy). See claim C9 (corrected).

Current Read

Turning thinking on is a real, cheap deployable lever the corpus left unused — but it is a budget to be controlled, and the controller experiment shows the budget knob is mostly an efficiency lever (near-iso-accuracy at much lower cost), not a new accuracy frontier: a near-optimal fixed budget already sits close to the oracle, and the deployable controller is capped by C2 false-passes. The foreign control then refined the nature of the gain: at the behavioral gain is coherent reasoning over relevant content at every budget (the model uses thinking as content — irrelevant thinking is catastrophic; pure-compute filler ≈ no-think; and the coherence advantage grows with budget). It is not clearly reflected in internal correctness-decodability (separability noisy). Honest read: use thinking, cap it (greedy overthinking optimum ~1024), and a cheap controller buys back most of the cost; the accuracy gain is genuine reasoning content the model uses at all budgets — don't dismiss it as compute; behavioral ≠ representational. The filler arm (pure compute ≈ 0) and the budget sweep (coherence grows) are now done. Priority follow-ups: a learned controller with richer visible signals to chase the C2 wall; and a contamination-controlled / harder substrate — does coherent reasoning still carry the whole gain when the no-think baseline is weaker and memorization is defeated? (This is the most load-bearing open question given MBPP is basic and likely partly contaminated.) C51 adds that a thinking budget is not merely a token count: when almost every trace is force-closed, a teacher-forced answer score can describe a counterfactual state rather than deployable reasoning. Calibrate termination on the actual workload and include the close/commit event in any trace-value measurement. C52 adds that editing one thought token in the loss is not the same as making a context-local model edit: future steering must pass an exact-logit locality preflight before a larger harvest. This does not weaken the separate long-context loop-control mandate, whose pathology begins only at 16k+.

Pareto Policy Integration Qualification (2026-07-12)

The proposed short/deep policy consolidation did not reach distillation. Across two fresh blocks, C54's blend checkpoint was not a better quick teacher than apex (-0.00693, -0.03789; pooled -0.02241), while apex had a replicated deep advantage (+0.04563) but missed retention. Thus an external thinking-budget tier split does not by itself define a same-prefix teacher split. A future policy-space test should estimate verified continuation advantage at the actual state, then freeze and replicate the route before training; the visible two-checkpoint tier router remains only an inference upper reference.

Replicated fixed-cap semantic commit (2026-07-12)

qwen35_4b_commit_slot_semantic_power_replication resolves the earlier 16-task near miss without raising the budget. At fixed cap 1,024, 113-task qualification and 113-task untouched confirmation independently passed every frozen semantic, task-bootstrap, breadth, and unrestricted-interface gate. Ordered versus exact- token-shuffle accuracy was 27.14% versus 13.57%, then 28.91% versus 13.86%; no-thought was 9.73% then 7.08%. One-sided task lower bounds were +8.85pp and +9.44pp. Every path still required an external cap commit, so this is evidence for coherent thought content at a fixed counterfactual state, not autonomous termination. Hold the budget fixed and test whether task-held-out prefix value can route or causally improve this deployed state. That test is now negative for the frozen shared J readout: overall AUC was 0.5021, below slot margin 0.5448 and non-J residual features 0.5292. Midpoint prospective AUC was 0.6083 but endpoint reversed to 0.3958, so no causal/controller stage opened. The allowed phase-specific audit found midpoint-only J AUC 0.5375 (lower 0.4417), below equal-width non-J 0.6000 and tied slot margin 0.5396. The apparent midpoint lead is retired; keep the replicated semantic-content seam, not its failed scalar readout, and do not infer a larger token budget.

The label-free counterfactual selector follow-up is also negative. Across the same 113 qualification tasks, choosing the largest mean alias probability gain from ordered over exact-shuffled thought reached 43/113, above majority 33/113 but only two tasks above minimum entropy and three above max confidence; both uncertainty bounds crossed zero. A task-mismatched oracle-balanced shuffle control reached 44/113. The confirmation and K=3-versus-K=6 compute comparison stayed sealed. Coherent content is a real lever, but scoring its terminal logit effect is not yet a cost-justified controller.

Moving the edit earlier did not rescue it. In qwen35_4b_jacobian_counterfactual_branching, all 144 supplied-target J writes across three amplitudes were label-free and numerically valid, but hard target selection stayed 1/12 and mean probability lift stayed <=0.00566. Thus no 512-token continuation branches ran. The next budget experiment cannot assume an arbitrary final-prefix token is a semantic control site; it needs an explicit anchor or should leave J branching behind.

Mid-stack layer looping crosses the induction wall without tokens (2026-07-25)

qwen35_4b_depth_recurrence_probe reopens the question C59 appeared to close. C59's law -- serial compute helps only through reasoning CONTENT -- was established against two mechanisms: feeding the last hidden state back as an INPUT EMBEDDING (a slot the model was never trained to consume) and content-free filler tokens. Neither is weight-shared depth.

Running layers 12:16 of the frozen model a second time, in one forward pass with ZERO tokens emitted, takes forced-answer accuracy on held-out shift induction from 0.105 to 0.245 at n=400, and on out-of-family affine from 0.217 to 0.278.

CORRECTION 2026-07-26: this entry originally said the looped forward pass "matches what C59 measured for full chain-of-thought generation (0.235)". THAT COMPARISON IS WITHDRAWN. C59's real_cot was measured with a 768-token cap; measured here at 3072 with stop-on-commit, base CoT reaches 0.685 with a mean generation of 1781 tokens (2.3x C59's whole budget) and 29.5% of episodes STILL uncommitted, and it is correct 97.2% of the time among episodes that do commit. So looping reaches about a THIRD of what generated reasoning buys, not parity, and C59's affine real-CoT 0.020 is unusable for the same reason. C59's qualitative ordering strengthens (CoT beats forced/latent/filler by ~8x, not ~3x) while every CoT-anchored NUMBER in it is a budget artifact. The forced-read effect and all four of its controls stand; only its significance shrinks -- a 2.8x lift on a probe that denies the model tokens.

Four adversarial controls survived on both substrates: balanced accuracy (a constant predictor scores exactly 0.10) rises 0.112 -> 0.251, killing label-prior exploitation; a sweep of every 4-layer block shows only 12:16 and 16:20 help, with 20:24 onward at or below baseline; inserting a copy of a DIFFERENT 4-layer block at the same position -- identical added depth and parameters -- collapses to 0.000/0.007, so the specific layers are load-bearing rather than depth; and disjoint halves replicate (0.240/0.250 and 0.265/0.290). Damping gives a monotone dose-response (0.240/0.195/0.155/0.095 as the second pass is admitted at 100/50/25/10%), and prose next-token logprob moves only -0.021 nats, so the looped stack is still a fluent LM. The winning block ENDS at layer 15, exactly where C19/C31 found op-type maximally decodable -- looping past it hurts.

Two harness bugs manufactured fake results first and are now guarded. The Qwen3.5 decoder iterates self.layers[: config.num_hidden_layers], so a lengthened ModuleList is silently truncated: the "deeper" model ran FEWER layers, producing a spurious +0.125 and a 6-nat coherence collapse, with byte-identical k=2/k=3 numbers as the tell. Every arm now asserts its own executed depth by counting layer invocations and refuses to report otherwise. Damping hooks also leaked hidden state across batches until reset per forward pass.

What this does NOT establish: the gain is on a forced single-pass read chosen to isolate forward-pass computation, on two digit-induction substrates. Generation-mode behaviour (does looping STACK with CoT or merely substitute for it?), any deployable task (MBPP, pi-coding-agent), and general capability beyond a prose-logprob check are all untested. Those three, in that order, are what a claim needs.

Scorecard

  • Program: charter
  • Current read: native thinking is a real coherent-content lever, but its value geometry and write sites are workload/phase-specific. Fixed First: exposes a replicated semantic state, shared/midpoint value and last-token additive branching fail, and early concrete text routes one-operation execution without reaching a valid full-program controller. The answer seam qualifies no-think tokenizer EOS at 48/48 in both prefix cells against 0/48 HF EOS, while thinking is worse at 38/48 structured and 30/48 freeform. Replay-hardened mechanics then passed transport and ABI but produced 0/24 oracle proposal coverage for materialized and every matched comparator. The short no-think seam controls emission; it does not unlock residual composition.
  • Best next experiment: retain the separate 16k+ loop-control line. For J-space, measure whether any within-<think> coordinate predicts eventual correctness across held-out tasks beyond margin/position/equal-width non-J controls, and only then test a same-prefix causal branch intervention against matched sampling.
  • Strong anchors: qwen35_4b_thinking_content_vs_compute, qwen35_4b_overthinking_content_ladder, qwen35_4b_answer_potential_trace_sft, qwen35_4b_native_thought_seam_budget_ladder, qwen35_4b_forced_commit_jacobian_value_transport, qwen35_4b_commit_slot_jacobian_value_transport, qwen35_4b_commit_slot_semantic_power_replication, qwen35_4b_think_ftpo_round2.
  • Avoid repeating: thinking-budget wins without content controls, calibration on a different workload class, cap-bound score interpretation, larger-N harvesting before termination/locality works, or treating high varentropy as a monotone “push harder” signal.
  • Evidence that advances the program: a controller or distillation that Pareto-beats fixed budgets, and a content control that isolates genuine reasoning from compute + scaffold + token-presence.

Charter

Show charter.md

Purpose

Study the native thinking-token (reasoning) budget as a first-class, controllable test-time-compute axis for Qwen3.5-4B. The imported 155-experiment corpus universally disabled this axis: enable_thinking=False appears 48 times in the code and True zero times, except a single fixed-768-token CoT baseline that was never swept. Every "budget" the corpus studies is an external one — evidence probes, tool calls, program portfolios, sample counts. None is the model's internal reasoning-token budget.

This program asks what happens when the 4B is allowed to think, and treats the amount of thinking as a dial to be measured, controlled, and ultimately learned.

Why This Is A Program (Not A Variant)

  • Distinct compute axis. process_control_and_tool_use budgets observations of the world (probes, tools); this program budgets the model's own latent reasoning. They trade off against each other but are mechanistically different and scale differently.
  • It hosts many experiments. Scaling curves; oracle-ceiling vs deployable-line decomposition; a STOP/MORE controller over thinking tokens; distillation of long thinking into short/no-think; thinking-as-verifier vs thinking-as-generator; whether more thinking rescues the corpus's length-generalization failures.
  • It produces positive and negative knowledge. A clean negative ("thinking only raises cost, not the deployable line") is as valuable as a positive, and directly pressure-tests the corpus's implicit "silent latent compute beats CoT" thesis (qwen_python_shaped_silent_executor).
  • Existing programs would hide the uncertainty. Folded into posttraining or process control, the reasoning-budget question disappears; it deserves its own evidence ledger.

Progress Signals

The line is advancing when we can answer, with controls:

  • Does more thinking raise the oracle ceiling (pass@k), the deployable line (greedy / visible-selected pass@1), or only cost? (The corpus's oracle-vs-deployable framing, applied to thinking for the first time.)
  • Does the content of thinking matter, or only the <think> scaffold + extra compute? (Shuffled/truncated-thinking controls.)
  • Is there a budget sweet spot, and does it depend on task difficulty / length?
  • Can a deployable controller allocate thinking tokens better than a fixed budget?
  • Does thinking transfer the corpus's confirmed bottleneck (C2: coverage ≫ deployable selection) — i.e., does thinking widen or close the selection gap?

Boundaries

  • Not about external evidence/tool budgets (that is process_control_and_tool_use).
  • Not about new model architectures; the substrate model is fixed at Qwen3.5-4B.
  • Training-based methods (controller, distillation) belong here only insofar as they manage the reasoning budget; generic posttraining belongs in posttraining_and_adaptation.

Anchors

Backlog

Show backlog.md

Done

  • qwen35_4b_thinking_budget_scaling: MBPP scaling curve; oracle-vs-deployable decomposition; shuffled-thinking control. (+15pp greedy, greedy overthinking optimum ~1024. NOTE: its "gain is mostly compute/scaffold" and "2048 shuffle ≈ real" readings were later corrected — see the content ladders.)
  • qwen35_4b_thinking_budget_controller: fixed-rule visible-test escalation controller — an efficiency win (Pareto-dominates fixed budgets except the peak), bounded by C2 false-passes.
  • qwen35_4b_thinking_separability_probe: per-layer probes on answer-token activations. Correctness is moderately decodable (AUC 0.64–0.76); thinking raises decodability; shuffled ≈ real in decodability (representational side is noisy across experiments). Weak probe signal on C2 false-passes.
  • qwen35_4b_thinking_content_vs_compute: full content ladder (no_think / filler / shuffle / real / foreign). Complete attribution at budget 512: pure compute (filler) ≈ 0, token-presence (shuffle) ≈ 0, coherent content = the entire +0.122, misleading content (foreign) −0.71 (the model follows it to the wrong problem). Conclusively corrected the earlier "mostly compute/scaffold" claim — pure compute buys nothing; the efficient-budget gain is 100% coherent reasoning content.
  • qwen35_4b_overthinking_content_ladder: the content ladder across budgets {512,1024,2048}. The coherence advantage (real − shuffle) grows with budget (+0.105 → +0.108 → +0.150), refuting the overthinking-washout hypothesis; pure compute (filler) ≈ no-think and foreign catastrophic at every budget. So coherent reasoning is the entire gain at ALL budgets; the scaling run's "2048 shuffle ≈ real" was a shuffle-protocol artifact.
  • qwen35_4b_answer_potential_trace_sft: terminal SCORER_NEGATIVE. Answer gain had modest real within-task signal but missed its actionability bars; 99.37% cap contact and 13.2% answer parsing exposed a forced-close deployment-seam mismatch. No N=128 harvest or SFT ran.

Next Experiments

  • Completed negative, cross-program seam calibration: qwen35_4b_native_thought_seam_budget_ladder ended NO_BUDGET_SELECTED: 48/48 fresh traces were still reasoning at 1,024, so natural close was 0% at 256/512/1024 and confirmation stayed sealed. Exact short-period tail loops were 0/48, so this is unresolved reasoning rather than a registered loop-control target. Stop raising the natural cap on this task.
  • Completed negative, explicit close-only commit: qwen35_4b_forced_commit_jacobian_value_transport reproduced C51's interface wall on fresh exact-depth tasks: forced parse 12.5%--18.8%, success 2.1%, and answer-cap contact 85%--96% across 256/512/1024. Close alone often restarted reasoning; no cap or downstream value stage opened.
  • Completed near-miss, fixed answer slot: qwen35_4b_commit_slot_jacobian_value_transport made the unmasked next token an alias on 41/48 long traces and real 1,024-token thought scored 15/48 versus the equivalent 12/48 no-thought and 11/48 shuffled. Five mixed tasks missed the frozen six-task gate, and task-level intervals included zero; no cap or J stage opened.
  • Completed replicated seam / value stop: qwen35_4b_commit_slot_semantic_power_replication held the budget at 1,024 and assigns 113 fresh tasks to each of qualification and untouched confirmation. It requires a positive one-sided task-bootstrap lower bound over exact-length shuffle, +3pp over no-thought, broad alias support, and no pooled rescue. Qualification passed all gates with +13.57pp over shuffle and +17.40pp over no-thought; confirmation independently repeated +15.04pp and +21.83pp with task lower +9.44pp. This validates fixed-cap coherent-content transport into the deployed slot, not natural termination. Holding the cap fixed, the task-held-out shared J-value readout then failed at chance (0.502) despite midpoint 0.608 because endpoint reversed to 0.396; slot margin and generic non-J state were stronger. Do not increase the cap or open causal work. Post-decision midpoint-only J fell to 0.538, below non-J 0.600 and tied margin 0.540, so do not open the proposed phase-specific successor. Test label-free ways of using coherent-thought diversity against matched- compute sampling instead.
  • The first such selector is terminal NO_ORDER_SUPPORT_SELECTOR: exact-shuffle probability deltas beat hard voting but not confidence/entropy robustly and failed relevance control. Do not retune terminal logits. A new successor must use fixed cap 1,024 to change the continuation/proposal distribution and beat matched-forward-token sample-more.
  • The upstream additive-J attempt stopped at 4/48 supplied targets, and the follow-on late semantic anchor is terminal invalid: direct aliases are conditionally writable but consequence parse is 0/440, consequence choices stay near chance, and the composed label map was fixed. Do not spend another native-J budget sweep. A fresh early textual-hypothesis fork must count every branch token and beat matched-compute sampling before receiving capability credit.
  • Completed cross-program qualification negative: qwen35_4b_pareto_policy_integration found that C54's short/deep tier labels did not become a clean procedural teacher crossover. blend lost both quick blocks; apex won deep capability but missed retention. The visible two-checkpoint tier router remains an external upper reference, not a one-checkpoint installation result. Any future consolidation must first demonstrate state-local continuation advantage and retain that visible router as a control.
  • Close/commit potential follow-up (new experiment only): compare joint probability of autonomously closing and emitting the canonical answer against answer-only potential, after a frozen natural-close and parseability calibration gate. Do not reuse C51's observed tasks to tune thresholds.
  • Symmetric loop-control follow-up (next; new experiment): the exact-capture verified-macro ladder is terminal pass=false, with no selected budget and no authorized K=12/semantic stage. Its clean 61k envelope still produced 40/48 exact loops, 8/48 unresolved contacts, and 4/48 answer-limit contacts at 397.688 sampled tokens/s; 49k produced 38/10/6 at 491.396 tokens/s. Stop increasing context. Preregister one loop-control intervention symmetrically across all arms and matched-compute baselines, keep the unresolved-contact and answer-limit gates unchanged, and use fresh artifacts. Loop detection is a termination mechanism, not evidence of correctness.
  • Learned thinking-budget controller with richer visible signals (token entropy/logprob, self-consistency): can it close the deployable→oracle gap (0.89→0.93)? (Queue: thinking_budget_controller.)
  • Thinking-budget sweep on the silent_executor substrate (modular-program execution): its CoT collapsed to 0% at len-24 at a fixed 768-token budget — does a larger thinking budget rescue length generalization? Direct pressure-test of the silent-compute thesis.
  • Matched-compute scratchpad compression at inference time: periodically ask the same Qwen3.5-4B call protocol to replace a long intermediate state with a short verified checkpoint, then spend the saved context on continued reasoning. Compare against ordinary sampling at equal model-forward tokens; no teacher, distillation, or cross-model trace source is permitted.
  • Thinking as verifier vs generator under matched token budget: spend the budget on a thinking generator vs a thinking verifier/selector over no-think candidates — which buys more deployable accuracy? (Couples to evidence_conditioned_selection / C2.)
  • Replicate the sweep + controller on a harder substrate (full MBPP / LiveCodeBench / math): does the optimum move, does the C2 false-pass rate grow?

Required Controls

  • Baseline: no_think (enable_thinking=False) — the corpus's universal setting.
  • Mechanism-falsifying control: shuffled-thinking (scramble the model's own thinking tokens, then force the answer) and truncated-thinking — isolates thinking content from the <think> scaffold + extra compute.
  • Matched-compute control: compare conditions at equal total forward tokens (the corpus's matched-compute methodology), so gains are not just "more tokens".
  • Shift check: difficulty / length slices (does the optimal budget move with task length?).
  • For trace selection, report natural-close rate and autonomous answer parse rate before scaling the pool.

Stop Conditions

  • Retire/demote if, across substrates and at matched compute, thinking never raises the deployable line beyond shuffled-thinking — i.e., only the scaffold + extra compute help.
  • Branch a new program if a thinking-budget controller becomes a rich line of its own.

  • Stop answer-only trace-potential scaling when the thought pool is cap-bound; changing N does not repair an injected close/commit seam.

  • Experiment: long-context loop-FTPO (the loop-control mandate's intervention branch): loops are ~0.1% at deployed budgets but 56–83% of cap contacts at 32k–61k (C52 census + verified-macro ladder). Mine loop-initiation rows at 16k+ budgets, train with contexts ≤6k (short-to-long transfer is the open question), preregister the symmetric-arms rule from the model playbook.

Experiments 25

  • 2026-07-25 Depth Recurrence Probe

    Partly, and the honest version is smaller than the first claim. Re-running layers 12-15 once more takes a single-pass answer from 0.105 (essentially guessing) to 0.245, on frozen weights with no tokens written, and that

  • 2026-07-13 Qwen3.5-4B Materialized Residual Sibling Search Fresh Replication

    The generation test cannot answer the question because every reasoning trace reached its length cap and the materialized arm parsed only 12 of 52 outputs, with zero successes. The separate one-token ranking test is a cle

  • 2026-07-13 Qwen3.5-4B Materialized Residual Sibling Search

    Not yet. The scientific design and every model-free construction check pass, but the model has not run. The frozen test contains 264 fresh functions, and all 38,596 planned prompt renderings fit their assigned context li

  • 2026-07-13 Qwen3.5-4B Early Text Hypothesis Forking

    The experiment design and model-free checks now pass, but the model has not run. Review expanded the first draft from twelve operation names to twenty-four fully specified operations, added two fair late-hint comparisons

  • 2026-07-13 Qwen3.5-4B Counterfactual Order-Support Selector

    Not reliably. The meaningful-order rule got 43 of 113 puzzles right, clearly better than first attempt or majority vote. But it was only two puzzles ahead of simply choosing the most decisive attempt, and a deliberately

  • 2026-07-13 Qwen3.5-4B: The Compute-Optimal Confidence Policy (select + abstain + escalate)

    Partly. Picking the answer with the highest single-token P(True) confidence is the best grader-free selector at every compute level, and abstaining below a confidence threshold trades coverage for accuracy cleanly — both

  • 2026-07-12 Payload-capable recovery agent harness

    No. It led on the first unseen task block at 71%, but confirmed at 69%, exactly tied with the action-only model and short of the required lead. The external benchmark stayed sealed.

  • 2026-07-12 Qwen3.5-4B Native-Thought Seam Budget Ladder

    No. Across 48 tries on simple list-transformation puzzles, and at every thinking budget up to 1,024 tokens, the model closed its reasoning and produced an answer exactly zero times. It always burned the whole budget stil

  • 2026-07-12 Qwen3.5-4B Jacobian Value Transport

    No. Editing one internal direction at a late layer flipped the concept the model said aloud on 18 of 24 tries, up from never, and far beating an ordinary readout-style edit that worked only 1 in 5 times. But when that co

  • 2026-07-12 Qwen3.5-4B Forced-Commit Jacobian Value Transport

    No. Inserting the model's own "done thinking" marker cut the reasoning off but almost never flipped it into answer mode. Across three thinking budgets, only 13% to 19% of forced stops produced anything readable, and just

  • 2026-07-12 Qwen3.5-4B Commit-Slot Semantic Power Replication

    The reasoning result is real, but the proposed internal value meter failed. Ordered scratch work beat the same words shuffled in two independent stages (about 29% versus 14% on confirmation). Yet a task-held-out model of

  • 2026-07-12 Qwen3.5-4B Commit-Slot Jacobian Value Transport

    Barely. Forcing the format fixed one problem outright: an allowed word was the model's top choice 85% of the time, versus 4% when it answered freely. But real step-by-step thinking beat the very same thought words scramb

  • 2026-07-12 Qwen3.5-4B Balanced-Core Answer-Potential SFT

    Not run yet — the reasoning bank is fully built (360 tasks, six competing selection rules staged) but no model has been trained or scored. The built test will fine-tune six copies, each fed reasoning chosen a different w

  • 2026-07-11 Entropy-routed think-pivot optimization round 2

    No. Gently pulling the better word up at 155 hand-picked wrong-turn moments was cleaner than shoving the bad word down, and it carried real signal, beating a scrambled-label control by about 14 points on fresh coding rep

  • 2026-07-10 → 11 Think-block FTPO round 1: outcome-conditioned pivot steering as an agentic install recipe

    No. Nudging at those forks made the model worse, not better — success on fresh tasks fell about 4 to 8 percent instead of clearing the 5-point gain hoped for. The tell: feeding it deliberately scrambled labels did nearly

  • 2026-07-10 Qwen3.5-4B Verified Macro Invention Long-Context Rerun

    No. The model was not short on thinking space, it was stuck repeating itself, and more space only bought longer loops. Handed the blueprint, all 16 tasks finished cleanly. Asked to invent the program from examples alone,

  • 2026-07-10 Qwen3.5-4B verified-macro exact CUDA-graph vLLM rerun

    No, it just loops harder. At both budgets, all 48 reasoning attempts ran out of thinking room and had to be force-stopped, none finishing on its own. Widening the budget from 49,152 to 61,440 tokens pushed the share stuc

  • 2026-07-10 Qwen3.5-4B verified-macro capacity-fit vLLM rerun

    No. Two hidden limits bit first. Configuring the server for 64 simultaneous long prompts demanded about 2.4 million tokens of fast memory from a pool holding roughly 1 million, forcing constant re-reading. Cutting to 19

  • 2026-07-10 Qwen3.5-4B Long-Horizon Answer-Potential SFT

    Barely. That confidence signal does sort stronger reasoning above weaker about 60% of the time — better than a coin flip's 50%, and enough to lift a picked chain's success from 16% to 22%. But two dumb rules beat it: jus

  • 2026-07-10 Qwen3.5-4B Answer-Potential Trace SFT

    No. The score carried genuine signal — it tracked what the reasoning actually said, not its length or wording — but its top pick raised fresh-answer success only from about 13% to 20%, short of the useful margin needed t

  • 2026-06-30 Qwen3.5-4B Thinking Content vs Compute

    It genuinely reasons; the boost is content, not compute. Blank filler of the same length, and the real thinking scrambled into nonsense, both scored like skipping thinking entirely, around 74 to 75 percent. Only coherent

  • 2026-06-30 Qwen3.5-4B Thinking-Budget Scaling

    Yes, but with a sweet spot. Turning on the model's built-in reasoning lifted the accuracy of the answer you would actually ship from 76% to 91% on basic Python tasks — a genuine capability gain, not just a wider pool of

  • ~2026-06-30 Qwen3.5-4B Thinking-Budget Controller

    For cost, yes; for accuracy, no. Drafting cheaply and thinking harder only when the draft fails the one visible example matches a generous fixed budget's accuracy (88%) using about a fifth of the thinking (113 versus 507

  • 2026-06-30 Qwen3.5-4B Overthinking Content Ladder

    No, the opposite. Giving this small coding model more room to reason made genuine, ordered reasoning matter more, not less. Scrambling its own reasoning steps into random order cost about 10 points of solved tasks at the

  • 2026-06-29 → 30 Qwen3.5-4B Thinking Separability Probe

    Yes, but not for the reason you would expect. From one snapshot of internal activity, whether the model's own code is correct is readable well above a coin-flip, about 64 to 76 percent of the time. Thinking first sharpen

Claims

Queued proposals 2