Qwen3.5-4B: Thinking vs the Lookahead Wall
The one idea you need
Picture a chess puzzle. One move from checkmate, extra thinking time reliably helps you spot the winning move. But asked to name the first move of a forced three-move mate, with no board to picture the middle positions, thinking out loud simply doesn't help.
The question
If you let a small model think out loud at length before answering, can it learn to plan several steps ahead toward a goal on its own?
What we found
No. Asked to name the first of three moves toward a goal, the model stayed stuck at pure guessing — right about 1 time in 32 — no matter how long it thought, even with 2,048 tokens to think first. Yet recognizing a goal just one move away leapt from roughly 28% to 60% as thinking grew. So extra thinking amplifies spotting a near answer, not planning a distant one.
Why it matters
If your system needs genuine multi-step planning, a longer chain of thought won't manufacture it — here only training installed that ability. Spend thinking budget on recognition-style subtasks where the intermediate state is visible, and measure planning and recognition separately.
On this page
Results at a glance 2
How to read
Three groups along the bottom mark how far the goal sits: three moves away (real planning), two away, and one away (recognition). Within each, bars are no thinking, medium, and large scratchpad. Height is next-move accuracy; taller is better, about 3% is chance.
Takeaway → The one-away group climbs steeply toward 60% as thinking grows, while the three-away planning group stays flat near chance — thinking boosts recognition, not planning.
Data table
| condition | no-think (B=0) | think B=1024 | think B=2048 |
|---|---|---|---|
| step 1 (goal 3 ops away = lookahead) | 2.5% | 5% | 7.5% |
| step 2 (2 away, state given) | 0% | 12.5% | 32.5% |
| step 3 (1 away = recognition) | 27.5% | 60% | 60% |
Numbers from
Technical framing
Thinking amplifies RECOGNITION, not PLANNING: next-op accuracy by step, across thinking budgets — Can THINKING breach the lookahead wall (C25)? Channel-matched test: think B tokens, close </think>, then the SAME 32-way likelihood ranking C25 used (parse-immune). STEP 1 is the only clean lookahead test (goal 3 ops away, no intermediate state) -- and it STAYS AT CHANCE across budgets (0.025->0.050->0.075, CIs overlap). THINKING DOES NOT BREACH THE WALL. The benefit scales INVERSELY with lookahead distance: huge for step-3 recognition (goal 1 op away, 0.275->0.600), moderate at step-2 (0->0.325), ~zero at step-1 (real planning). So thinking amplifies RECOGNITION, not PLANNING -- and only where the interpreter materializes the true intermediate state (steps 2/3). Internal-brute-force refuted: if the model could simulate the depth-3 path in its scratchpad, step-1 would rise; it doesn't.
How to read
The horizontal axis is scratchpad size (0, 1,024, 2,048 tokens); the vertical axis is accuracy at picking the correct first move. One line tracks the model; a flat reference line marks the 1-in-32 chance rate. Rising well above it would be better.
Takeaway → The line barely lifts from about 3% to 8% and hugs the chance line at every budget — more thinking buys essentially no planning ability.
Data table
| thinking budget (tokens) | step-1 next-op top-1 (think->rank) | chance (1/32) |
|---|---|---|
| 0 | 2.5% | 3.1% |
| 1024 | 5% | 3.1% |
| 2048 | 7.5% | 3.1% |
Numbers from
Technical framing
Step-1 lookahead stays at chance no matter the thinking budget — The clean lookahead test (step 1: current==input, goal 3 ops away, NO intermediate state materialized). Across 0/1024/2048 thinking tokens, step-1 next-op ranking accuracy hovers at chance (0.031) with a whisper of upward creep (0.025->0.075) that is not significant (Wilson CIs all overlap). Contrast with C25, where BANKING lifted this same step-1 metric 0.013->0.138 dose-dependently. So for the multi-step PLANNING/lookahead gap, training (banking) is the lever; test-time serial compute (thinking) is not.
In the author’s words from the Overview · “Results”
Thinking does NOT breach the lookahead wall. Step-1 stays at chance (0.025->0.050->0.075 at B=0/1024/2048; CIs overlap). But thinking amplifies RECOGNITION: step-3 (1 away) 0.275->0.600, step-2 (2 away) 0->0.325 -- the lift scales inversely with lookahead distance. Internal-brute-force refuted (step-1 flat; traces are meta-reasoning not enumerate-and-test). See reports/report.md, analysis/thinking_lookahead.png.
Overview
Research Program
- Program:
structured_execution_and_compilers/test_time_reasoning_budget - Question: does test-time THINKING breach the lookahead wall (C25) with no training? (reactivates the C9 lever)
- Anchors: C25 (lookahead wall + banking lifts it), C9 (thinking is an unused lever), C23 (base think depth-3=0).
Setup
- Model: Qwen3.5-4B only. list 16-op DSL (32 op/param combos). 80 min-depth-verified true-depth-3 held-out (reuse C25; used 40).
- PRIMARY metric = think->RANK vs no-think->RANK (channel-matched to C25, parse-immune). HEADLINE = STEP 1 (goal 3 ops away, no intermediate state = the only clean lookahead test).
- Budgets B in {0, 1024, 2048}. No training (test-time only).
Run
python scripts/run_thinking.py --n 40 --budgets 0 1024 2048 --steps 1 2 3 then python scripts/analyze.py
Results
Thinking does NOT breach the lookahead wall. Step-1 stays at chance (0.025->0.050->0.075 at B=0/1024/2048; CIs overlap). But thinking amplifies RECOGNITION: step-3 (1 away) 0.275->0.600, step-2 (2 away) 0->0.325 -- the lift scales inversely with lookahead distance. Internal-brute-force refuted (step-1 flat; traces are meta-reasoning not enumerate-and-test). See reports/report.md, analysis/thinking_lookahead.png.
Interpretation
For the multi-step PLANNING/lookahead gap, TRAINING (banking, C25: 0.013->0.138) is required; test-time thinking (C26) only amplifies RECOGNITION. Reconciles with C23 (base think single-shot depth-3 = 0).
Knowledgebase Update
- Program evidence:
research_programs/structured_execution_and_compilers/evidence.md(C26) - Claim ledger: C26 added
Artifacts
scripts/think_rank.py(think->rank, chunked op-scoring for long thinking prefixes),scripts/run_thinking.py(batched thinking generation),scripts/analyze.pydata/eval_frozen_d3.jsonl(reused from C25),runs/results.json,runs/traces_B*.json,runs/verdict.json,analysis/thinking_lookahead.png,reports/{prereg,report,design_review}.md
Report
Rendered from reports/report.md
Summary
C25 found the fixed 4B has a lookahead wall: in a single forward pass it can't plan the first of 3 ops (step-1 next-op likelihood-ranking ≈ chance) though it recognizes a 1-step transform (step-3 0.275). Thinking is serial test-time compute — the natural lookahead mechanism, and the dormant C9 lever. Does a thinking budget breach the wall with no training? Design hardened by an adversarial workflow review (verdict sound_with_fixes): the primary metric is think→RANK vs no-think→RANK (think B tokens, close </think>, then the SAME 32-way likelihood ranking as C25 — channel-matched, immune to parse/truncation), and the headline is STEP 1 (goal 3 ops away, no intermediate state materialized — the only clean lookahead test; steps 2/3 are handed the true intermediate list, so a lift there is state-materialization, not planning).
Result (n=40, chance top-1 = 0.031)
| step | B=0 | B=1024 | B=2048 |
|---|---|---|---|
| step 1 (3 away, no state — real lookahead) | 0.025 | 0.050 | 0.075 |
| step 2 (2 away, true state given) | 0.000 | 0.125 | 0.325 |
| step 3 (1 away, recognition) | 0.275 | 0.600 | 0.600 |
- Thinking does NOT breach the lookahead wall. Step-1 stays at chance across budgets (0.025 → 0.075; Wilson CIs all overlapping: B=0 [0.004, 0.129], B=2048 [0.026, 0.199]). Even 2048 thinking tokens do not let the model plan the first of 3 ops.
- Thinking's benefit scales INVERSELY with lookahead distance: huge for recognition (step-3, goal 1 op away: 0.275 → 0.600), moderate at 2 away (0.000 → 0.325), essentially zero at 3 away (real planning). So thinking amplifies RECOGNITION, not PLANNING — and only where the interpreter materializes the true intermediate state.
- Internal brute-force refuted. If thinking let the model simulate the depth-3 path in its scratchpad (be its own interpreter), step-1 would rise — it doesn't. The step-1 traces show confused meta-reasoning about the prompt, not systematic enumerate-and-test simulation.
Research Program Fit
The killer juxtaposition with C25: banking lifted step-1 lookahead (0.013 → 0.138, dose-dependent) while thinking does not. So the two capability levers are qualitatively different — for the planning/lookahead gap, TRAINING (banking) is required; test-time compute (thinking) alone cannot elicit it. For recognition, thinking is a powerful amplifier. This reconciles with C23 (base think single-shot depth-3 coverage = 0): thinking can't do the whole composition precisely because it can't plan the first steps. It also sharpens the mission read: "elicit latent capability without training / beat sample-more" works for RECOGNITION (thinking helps) but NOT for multi-step PLANNING (thinking fails; banking is needed).
Method
List 16-op DSL (32 op/param combos), 80 min-depth-verified true-depth-3 held-out (reuse C25's; used first 40). think→rank: gen_sequences(think=True, budget=B) produces a ≤B-token thinking trace, forces </think>, then score_ops_prefix ranks the 32 ops after prompt+thinking (chunked to fit the long prefix in memory). Batched generation across tasks. scripts/think_rank.py, scripts/run_thinking.py, scripts/analyze.py.
Pre-registered verdicts
- P1 (does thinking breach the wall?): NO — step-1 stays ≈ chance (0.025 → 0.075), not ≥ 0.10, CIs overlap. The wall is a planning gap, not a forward-pass compute limit. (Clean refutation because the ranking channel is parse-immune.)
- P2 (contamination check): HELD — steps 2/3 (materialized state) lift far more than step-1, confirming step-1 is the clean test and that thinking's gains come from recognition given the scaffold.
- P3 (planning vs internal enumeration): internal simulation refuted — step-1 flat and traces are not enumerate-and-test.
Honest limits
Ranking a closed 32-op set is easier than free generation. Single frozen held-out (n=40), one seed per budget (thinking generation is expensive even batched). Budgets ≤ 2048; a much larger budget is untested (but the flat trend and overlapping CIs make a sudden breach unlikely). The step-1 point estimate does creep (0.025 → 0.075) — a whisper of signal, not significant.
Next Experiments
- Much larger thinking budgets (8k–16k) to confirm the step-1 flatline is asymptotic, not just under-budgeted.
- Does a BANKED model + thinking stack (banking installs lookahead; does thinking then amplify it)?
- Free-generation channel with robust re-prompting, to check the ranking result survives generation.
Artifact Manifest
See reports/artifact_manifest.yaml. Reuses C25's frozen depth-3 held-out; no training (test-time only).
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/results.json1.1 kB
Reproduce
Smoke test
python scripts/run_thinking.py --n 8 --budgets 0 1024 --steps 1Full run
python scripts/run_thinking.py --n 40 --budgets 0 1024 2048 --steps 1 2 3 && python scripts/analyze.pyRun steps are documented inside the experiment folder (README and scripts).