Qwen3.5-4B Overthinking Content Ladder
The one idea you need
Picture the model writing itself a recipe, then cooking from it. Blank filler pages help nothing; a different dish's recipe ruins the meal; and the same steps in scrambled order spoil it too. The longer the recipe, the more that scrambling hurts.
The question
When you give a small model more space to reason before it answers, does that extra room dilute the value of thinking in a coherent order?
What we found
No, the opposite. Giving this small coding model more room to reason made genuine, ordered reasoning matter more, not less. Scrambling its own reasoning steps into random order cost about 10 points of solved tasks at the short budget but 15 points at the longest. Blank filler of the same length bought essentially nothing over not thinking at all, so raw extra compute was not the driver.
Why it matters
Don't buy accuracy by padding prompts or counting thinking tokens; empty length adds nothing. A longer thinking budget pays off only when it holds genuine coherent reasoning. Measure that by scrambling the steps, not by counting them.
On this page
Results at a glance 2
How to read
The horizontal axis is thinking budget in tokens (512, 1024, 2048). The vertical axis is the accuracy gap between real ordered reasoning and the exact same reasoning shuffled into random order. Higher means coherent order matters more.
Takeaway → The single line climbs from about 11 to 15 points, so a longer scrambled chain hurts more; extra thinking room strengthens rather than dilutes coherent reasoning.
Data table
| thinking budget (tokens) | real − shuffle |
|---|---|
| 512 | 10.5% |
| 1024 | 10.8% |
| 2048 | 15% |
Numbers from report table (experiments/qwen35_4b_overthinking_content_ladder/reports/report.md)
Technical framing
Coherence advantage of real thinking grows with budget — The overthinking hypothesis is refuted: the value of coherent reasoning order grows from +10.5pp at 512 to +15pp at 2048.
How to read
The horizontal axis is thinking budget in tokens; the vertical axis is the share of coding tasks solved. Five lines: real thinking on top, then shuffled tokens, filler dots and no thinking bunched together, and a foreign task's thinking near the floor. Higher is better.
Takeaway → Real thinking sits alone near 85%, while filler, shuffle and no-thinking cluster around 72-76% and foreign reasoning craters near 3% -- only coherent, relevant reasoning lifts accuracy.
Data table
| thinking budget (tokens) | real thinking | shuffled tokens | filler dots | no thinking | foreign task's thinking |
|---|---|---|---|---|---|
| 512 | 85.6% | 75.1% | 72.4% | 75.7% | 4.1% |
| 1024 | 84.1% | 73.4% | 73.8% | 75.7% | 3% |
| 2048 | 87.1% | 72.1% | 74.5% | 75.7% | 3.3% |
Numbers from report table (experiments/qwen35_4b_overthinking_content_ladder/reports/report.md)
Technical framing
Content ladder pass rates across thinking budgets — At every budget filler tracks no-think (pure compute is worth ~0), foreign thinking is catastrophic, and coherent real thinking is the entire gain.
In the author’s words from the Overview · “Results”
Coherence advantage (real − shuffle) vs budget (table on the experiment page). The coherence advantage grows with budget (+0.105 → +0.150), refuting the "overthinking washes out coherence" hypothesis — scrambling a longer thinking region hurts more (shuffle drops) while real holds. At every budget: filler ≈ no_think (pure compute ≈ 0), foreign catastrophic, real is the entire gain.
Overview
Research Program
- Program:
test_time_reasoning_budget - Program question: does the coherent-content advantage of thinking shrink as the thinking budget grows (overthinking)? i.e. is the residual "compute/scaffold, not reasoning" reading purely the high-budget regime?
- Prior anchors:
qwen35_4b_thinking_content_vs_compute(at budget 512 the gain is 100% coherent content: filler ≈ shuffle ≈ no_think, real +12pp, foreign collapses) andqwen35_4b_thinking_budget_scaling(behavioral overthinking optimum ~1024; its 2048 shuffle ≈ real hinted coherence stops mattering).
Question
The content ladder at budget 512 showed coherent reasoning is the entire thinking gain. The scaling experiment showed accuracy peaks ~1024 then declines, and that at 2048 shuffled ≈ real. So: does the coherence advantage real − shuffle fall toward 0 as the budget grows? If yes, the "thinking ≈ compute" reading is confirmed as the overthinking regime, and coherent reasoning is the efficient-budget story — closing the program's central question across the budget axis.
Hypothesis
real − shuffle is large at the efficient budget (~+0.12 at 512) and shrinks toward 0 at 2048. filler ≈ no_think and foreign collapses at every budget (the relevance/compute facts are budget-independent).
Setup
- Model Qwen3.5-4B frozen (bf16, fast path). MBPP sanitized
test, 100 tasks, k=8. Behavioral-only (no activations — the separability side was noisy and isn't the question here). - Ladder at budgets {512, 1024, 2048}: real thinking generated once per budget (capturing tokens); filler (contentless
.tokens, matched to real length), shuffle (permuted real tokens), foreign (a different task's tokens) regenerate only the answer. no_think once (budget-independent). - Metrics: full-test pass per (budget, condition); the headline curve is
real − shufflevs budget, plusreal − foreign(content-is-used) and filler ≈ no_think (compute ≈ 0) at each budget.
Run
HF_HUB_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
../../.venv/bin/python scripts/run.py --tasks 100 --k 8 --budgets 512,1024,2048
../../.venv/bin/python scripts/verify.py
../../.venv/bin/python analysis/curve.pyResults
Full results in reports/report.md. Coherence advantage (real − shuffle) vs budget:
| budget | no_think | filler | shuffle | foreign | real | coherence |
|---|---|---|---|---|---|---|
| 512 | 0.757 | 0.724 | 0.751 | 0.041 | 0.856 | +0.105 |
| 1024 | 0.757 | 0.738 | 0.734 | 0.030 | 0.841 | +0.108 |
| 2048 | 0.757 | 0.745 | 0.721 | 0.033 | 0.871 | +0.150 |
The coherence advantage grows with budget (+0.105 → +0.150), refuting the "overthinking washes out coherence" hypothesis — scrambling a longer thinking region hurts more (shuffle drops) while real holds. At every budget: filler ≈ no_think (pure compute ≈ 0), foreign catastrophic, real is the entire gain.
Interpretation
Coherent reasoning is the entire thinking gain at every budget, and matters more as the budget grows. This retires the "thinking ≈ compute/scaffold" reading (which had survived as a high-budget caveat) and shows the scaling experiment's "2048 shuffle ≈ real" was a shuffle-protocol artifact. See the corrected claim C9.
Artifacts
src/ladder_lib.py,src/tasks.py;scripts/run.py(multi-budget, behavioral-only),scripts/verify.py;analysis/curve.py.data/records.jsonl,data/labels.jsonl,data/tasks.json(small, in-repo).- No external artifacts (behavioral-only; no activations cached).
Report
Rendered from reports/report.md
Summary
We ran the content ladder (no_think / filler / shuffle / foreign / real) across thinking budgets {512, 1024, 2048} to test whether the coherent-content advantage shrinks as the budget grows (overthinking). It does not — the coherence advantage real − shuffle grows: +0.105 (512) → +0.108 (1024) → +0.150 (2048). Scrambling a longer thinking region hurts more (shuffle falls 0.751 → 0.734 → 0.721) while coherent thinking holds up (real ≈ 0.84–0.87), so the gap widens. At every budget: filler ≈ no_think (pure compute ≈ 0), foreign is catastrophic (0.03–0.04; the model follows the misleading content to the wrong problem), and real is the entire gain. This refutes two earlier readings: (1) the hypothesis that overthinking washes out coherence, and (2) the caveat that the "thinking ≈ compute/scaffold, not reasoning" reading survives at high budgets. The scaling experiment's "2048 shuffle ≈ real" was a protocol artifact (it shuffled fresh thinking; this tighter run reuses real's exact tokens). Corrected conclusion: coherent reasoning is the entire thinking gain at every budget, and matters more as the budget grows.
Research Program Fit
Fifth experiment of test_time_reasoning_budget; closes the budget-axis question. Combined with the content ladder at 512, it makes the program's central claim uniform across budgets and corrects C9 (which had preserved a "compute reading holds at high budgets" caveat now shown false).
Method
- Model Qwen3.5-4B frozen (bf16, fast path). MBPP sanitized
test, 100 tasks, k=8. Behavioral-only. - Ladder at budgets {512, 1024, 2048}: real thinking generated once per budget (capturing thinking tokens); filler (contentless
.matched to real length), shuffle (permuted real tokens), foreign (a cyclically-shifted other task's tokens) regenerate only the answer. no_think once (budget-independent). - Metric: full-test pass per (budget, condition); headline curve
real − shufflevs budget.
Results
| budget | no_think | filler | shuffle | foreign | real | coherence (real−shuffle) | content-used (real−foreign) |
|---|---|---|---|---|---|---|---|
| 512 | 0.757 | 0.724 | 0.751 | 0.041 | 0.856 | +0.105 | +0.815 |
| 1024 | 0.757 | 0.738 | 0.734 | 0.030 | 0.841 | +0.108 | +0.811 |
| 2048 | 0.757 | 0.745 | 0.721 | 0.033 | 0.871 | +0.150 | +0.839 |
Figures: analysis/coherence_vs_budget.png (the headline), analysis/ladder_vs_budget.png.
Finding 1 — the coherence advantage grows with budget (hypothesis refuted)
real − shuffle rises +0.105 → +0.108 → +0.150. The prediction that overthinking would erase the value of coherent order is wrong; the opposite holds — a longer thinking region, scrambled, is more disruptive (shuffle drops), while coherent thinking stays high.
Finding 2 — pure compute ≈ 0 at every budget
filler (contentless . matched to real length) tracks no_think at all budgets (0.724 / 0.738 / 0.745 vs 0.757). No "dot-by-dot" extra-compute benefit appears at any budget.
Finding 3 — the model uses content at every budget
foreign stays catastrophic (0.041 / 0.030 / 0.033): a different task's thinking sends the model to the wrong problem regardless of budget. real − foreign ≈ +0.82 throughout.
Finding 4 — real accuracy is roughly flat across budgets (sampled full-pass)
real ≈ 0.84–0.87 across 512/1024/2048. The greedy "overthinking optimum then decline" seen in the scaling experiment is a greedy/single-sample phenomenon, not a drop in sampled full-pass or in the value of reasoning.
Controls
The ladder is the control structure; filler isolates pure compute, foreign isolates misleading content, shuffle isolates relevance-without-order, real adds coherent order — replicated at three budgets. The key artifact check: this experiment's shuffle reuses real's exact thinking tokens (permuted), unlike the scaling experiment's shuffle (which shuffled a fresh thinking sample); that difference explains why the scaling run saw 2048 shuffle ≈ real while this run sees real ≫ shuffle.
Oracle Versus Deployable Evidence
All behavioral full-pass (deployable). No probe / oracle metrics here.
Interpretation
The program's central question is now settled across the budget axis: the native-thinking gain is coherent reasoning content at every budget — not compute (filler ≈ baseline everywhere), not scaffold, not token-presence (shuffle ≈ baseline everywhere) — and the model genuinely uses that content (foreign catastrophic everywhere). If anything the reasoning contribution grows with budget. The "thinking ≈ compute" reading, which earlier survived as a high-budget caveat, is fully retired; it only ever appeared through a greedy-metric lens or the scaling run's shuffle-protocol artifact.
Limitations
- MBPP (basic, likely partly contaminated), n=100, single seed, sampled full-pass. Minor protocol asymmetry (real's answer is its original generation; others are regenerated from the modified prefix). The 2048 arm was regenerated in a slow recovery run after a mid-run CUDA "device not ready" in the fla kernel (answer-regen batch too large over ~2000-token prefixes); fixed by budget-scaled batch sizes.
Next Experiments
- Contamination-controlled / harder substrate: does coherent reasoning still carry the whole gain when the no-think baseline is weaker and memorization is defeated?
- Does the growing-with-budget coherence advantage hold on non-code reasoning (math)?
Artifact Manifest
See artifact_manifest.yaml. Behavioral-only; small records/labels + table + figures in-repo; no external activation artifacts.
Experiment log 5
Show the running log (5 entries)
Scaffold
Fifth experiment of test_time_reasoning_budget. Runs the content ladder (no_think/filler/shuffle/ foreign/real) across thinking budgets {512,1024,2048} to test whether the coherent-content advantage (real - shuffle) shrinks as the budget grows (overthinking). Behavioral-only (no activations).
Method
Real thinking generated once per budget (gen_real captures thinking tokens); filler/shuffle/foreign reuse those tokens / matched length and regenerate only the answer (gen_answer). no_think once. Headline curve: real - shuffle vs budget. Reuses ladder_lib + tasks from the content_vs_compute experiment.
Smoke
4 tasks x k=2, budget 512: ladder generated + verified end-to-end.
Run notes
- The full sweep crashed mid-2048 (
CUDA device not readyin the fla kernel — answer-regen at batch 48 over ~2000-token thinking prefixes). Fixed with budget-scaled batch sizes; the 2048 arm was recovered viascripts/add_2048.py. Lesson: batch ~24-32 clears the memory error; batch 12 (my over-correction) turned it into a slow ~2.4h recovery — pick the batch just below the failure point, not far below.
Results (see reports/report.md)
Coherence advantage (real − shuffle) grows with budget: +0.105 (512) → +0.108 (1024) → +0.150 (2048). At every budget filler ≈ no_think (0.72–0.75 vs 0.757; pure compute ≈ 0), foreign catastrophic (0.03–0.04; the model follows misleading content to the wrong problem), real is the entire gain (0.84–0.87). REFUTES the "overthinking washes out coherence" hypothesis and the "compute reading holds at high budgets" caveat; the scaling experiment's 2048 shuffle ≈ real was a shuffle-protocol artifact. Corrected conclusion: coherent reasoning is the entire thinking gain at every budget.
Figures 2
Reproduce
Smoke test
../../.venv/bin/python scripts/run.py --smokeFull run
HF_HUB_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True ../../.venv/bin/python scripts/run.py --tasks 100 --k 8 --budgets 512,1024,2048Run steps are documented inside the experiment folder (README and scripts).