Qwen3.5-4B: Do Banking and Thinking Stack?
The one idea you need
The model must pick the right next move toward a goal several moves off, like hunting a forced checkmate. Two separate upgrades: drilling the winning pattern into its training, and letting it reason longer before each move. Does stacking both beat either alone?
The question
If extra training and extra thinking each help a small model pick better moves toward a goal, does combining both give you the sum of their gains?
What we found
It depends on how far the goal is. One move away, the two boosts stack almost perfectly: a plain model picks the right move 27.5% of the time, extra training lifts that to 52.5%, and adding thinking reaches 85% — the exact sum, no wasted overlap. But three moves out, extra thinking adds nothing (about 17% right, with or without it); only training moved planning. Blind guessing scores about 3%.
Why it matters
Need multi-step planning from a small model? Earn it in training. A bigger thinking budget at answer time won't manufacture foresight the weights never learned — it only sharpens moves already near the goal. And score accuracy by distance-to-goal, not one blended number.
Results at a glance 1
How to read
Four conditions across the bottom: plain model, plain plus thinking, trained, trained plus thinking. Two bars each — how often the correct next move is ranked first with the goal one move away (easy) versus three moves away (hard). Taller is better; chance sits near 3%.
Takeaway → The easy-move bars climb with each boost, peaking at 85% when both combine; the hard-move bars stay low and barely rise once thinking is added — planning resists extra thinking.
Data table
| condition | step-1 next-op top-1 (PLANNING, goal 3 ops away) | step-3 next-op top-1 (RECOGNITION, goal 1 op away) |
|---|---|---|
| base + no-think | 2.5% | 27.5% |
| base + think | 7.5% | 60% |
| banked + no-think | 17.5% | 52.5% |
| banked + think | 15% | 85% |
Numbers from
Technical framing
Banking + thinking stack ADDITIVELY on recognition, not at all on planning — 2x2 {base,banked_1280} x {no-think,think} on per-step next-op ranking (channel-matched think->RANK), n=40 true-depth-3 held-out. RECOGNITION (step-3, goal 1 op away): the two levers STACK almost EXACTLY additively -- 0.275 -> +banking 0.525 -> +thinking 0.850 = additive prediction, interaction ~0.00. PLANNING (step-1, goal 3 ops away, the clean test): banking lifts (0.025->0.175) but test-time thinking adds ~0 to base OR banked -- NO stacking (banking owns the planning axis). CRITICAL SCOPE: the banked model was trained NO-THINK, so 'test-time thinking adds no planning' is about test-time thinking on a model never trained to reason about this task -- NOT that thinking is useless for planning. This is a BASELINE motivating the bank-the-thoughts experiment (SFT on prompt->reasoning-trace->code).
In the author’s words from the Overview · “Results”
RECOGNITION (step-3): additive stacking -- 0.275 -> 0.525 (banking) -> 0.850 (banking+thinking), interaction ~0.00. PLANNING (step-1): no stacking -- banking lifts 0.025->0.175, test-time thinking adds ~0 (base 0.075, banked 0.150). Banked thinking channel intact. See reports/report.md, analysis/stack.png.
Overview
Research Program
- Program:
structured_execution_and_compilers/test_time_reasoning_budget - Question: do banking (C25, installs lookahead-distance ranking) and TEST-TIME thinking (C26, amplifies recognition) compose?
- STATUS: honest NARROW BASELINE + scope caveat (banked model trained no-think) -> motivates bank-the-thoughts.
Setup
- 2x2 {base, banked_1280} x {no-think, think} on per-step next-op RANKING (think->RANK, channel-matched), n=40 true-depth-3 held-out. Banked in-run; base inherited from C26 (identical slice/harness).
Run
python scripts/run_thinking.py --tag banked1280 --adapter <C24 banked_1280> --budgets 0 1024 2048 --steps 1 2 3 then python scripts/analyze.py
Results
RECOGNITION (step-3): additive stacking -- 0.275 -> 0.525 (banking) -> 0.850 (banking+thinking), interaction ~0.00. PLANNING (step-1): no stacking -- banking lifts 0.025->0.175, test-time thinking adds ~0 (base 0.075, banked 0.150). Banked thinking channel intact. See reports/report.md, analysis/stack.png.
Scope (important)
The banked adapter was trained NO-THINK, so 'test-time thinking adds no planning' is about test-time thinking on a model never trained to reason about this task -- NOT that thinking is useless for planning. Clean test = bank-the-thoughts.
Knowledgebase Update
Artifacts
scripts/run_thinking.py(--adapter),scripts/analyze.py,runs/results_banked1280.json,runs/traces_banked1280_B*.json,runs/verdict.json,analysis/stack.png,reports/{prereg,report}.md. Reuses C24 banked_1280 (out of repo).
Report
Rendered from reports/report.md
Summary (honest scope up front)
This experiment asks whether the two capability levers — banking (C25: no-think SFT that lifts step-wise next-op ranking at lookahead distance) and test-time thinking (C26: amplifies recognition) — compose. It is a narrow baseline, not a bold "stacking" result, and it carries an important scope caveat surfaced by the user: the banked adapter (C24) was trained no-think (prompt→code, no reasoning traces), so any statement that "test-time thinking adds no planning even to the banked model" is about test-time thinking on a no-think-trained model — not evidence that thinking is fundamentally useless for planning. That clean question is the motivation for the bank-the-thoughts follow-up.
Result: 2×2 {base, banked} × {no-think, think}, per-step next-op top-1 (n=40, chance 0.031)
| step | base+no-think | base+think(2048) | banked+no-think | banked+think(2048) |
|---|---|---|---|---|
| step 1 (planning, goal 3 away) | 0.025 | 0.075 | 0.175 | 0.150 |
| step 2 (2 away, state given) | 0.000 | 0.325 | 0.150 | 0.250 |
| step 3 (recognition, goal 1 away) | 0.275 | 0.600 | 0.525 | 0.850 |
- Recognition (step-3): the levers STACK almost exactly additively. Banking adds +0.25, thinking adds +0.325, and banked+think = 0.850 = the additive prediction (0.275 + 0.25 + 0.325); interaction ≈ 0.00. The two levers act on the recognition axis independently and compose.
- Planning (step-1): no stacking. Banking lifts (0.025 → 0.175) but test-time thinking adds ~nothing to base (0.025 → 0.075) or banked (0.175 → 0.150). The thinking effect on planning is ~0 (and not significant at n=40; the step-1 CIs are wide and overlapping). So banking owns the planning lift; test-time thinking does not contribute to it.
- The banked model's thinking channel is intact (traces coherent, median ~2500 chars, even showing backward-from-goal reasoning) — so the banked+think cell is a fair thinking condition, not a broken one.
Interpretation (scoped)
Test-time thinking and answer-banking are orthogonal on recognition (additive) and non-interacting on planning (thinking contributes 0 to planning regardless of banking). This is consistent with the arc: banking installs the lookahead-distance ranking lift (C25); test-time thinking amplifies recognition (C26) and stacks there. But it says nothing about whether training the model to reason (banking successful thinking traces) would install planning-via-thinking — because this model was only ever trained to emit answers. That is the open question, and the reason this experiment is a baseline, not a conclusion.
Method
2×2 on per-step next-op RANKING (think→RANK, channel-matched to C25/C26, parse-immune). banked_1280 cells measured in-run at budgets {0,1024,2048}; base cells inherited from C26 (identical n=40 first-40 slice, same think_rank harness, same frozen held-out — verified byte-identical). scripts/run_thinking.py (with --adapter), scripts/analyze.py.
Honest limits
n=40, one seed/budget — step-1 CIs are wide (the "no stacking on planning" is "thinking effect ≈ 0 and not significant", not a tight null). Base cells inherited from C26 (same slice/harness, so valid, but not re-run here). Closed-set ranking is easier than free generation. The end-to-end thinking-guided search was NOT run: the per-step 2×2 already shows step-1 planning is unmoved by thinking, so a search gated by the first move has no mechanism to beat banked-no-think — a pre-registered null (per the design review).
Next: bank-the-thoughts (the experiment this motivates)
Rejection-sample verified-correct depth-3 thinking traces from the banked model, SFT on prompt → ⟨planning trace⟩ → code, and test whether that installs planning the model can then use (deployable depth / coverage that stacks with multi-sampling). This is the clean separation of "weights can't plan" vs "never taught to think-to-plan" that the present test cannot make.
Artifact Manifest
See reports/artifact_manifest.yaml. Reuses C24 banked_1280 adapter (out of repo); no training here.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
python scripts/run_thinking.py --tag banked1280 --adapter <banked_1280> --n 8 --budgets 0 1024 --steps 1Full run
python scripts/run_thinking.py --tag banked1280 --adapter <banked_1280> --budgets 0 1024 2048 --steps 1 2 3 && python scripts/analyze.pyRun steps are documented inside the experiment folder (README and scripts).