Qwen3.5-4B Thinking-Budget Controller
The one idea you need
Like a mechanic who tries a quick fix, takes one test drive, and only opens up the engine if the car still won't start. Here the quick fix is a fast draft answer, the test drive is the single example check shown in the task, and opening the engine is extra thinking.
The question
When a model can choose how long to think on each task, can a simple rule spend that effort more wisely than always thinking a fixed amount?
What we found
For cost, yes; for accuracy, no. Drafting cheaply and thinking harder only when the draft fails the one visible example matches a generous fixed budget's accuracy (88%) using about a fifth of the thinking (113 versus 507 tokens per task). But it never beats the best fixed budget (91%), because that quick check passes a wrong answer about 1 in 10 times.
Why it matters
Treat the thinking budget as an efficiency dial, not an accuracy dial: draft cheaply and spend more thinking only when a quick check fails. To actually raise accuracy, you need a commit signal better than a single example test.
On this page
Results at a glance 3
How to read
The horizontal axis is average thinking tokens per task (further left is cheaper); the vertical axis is the share of tasks fully solved (higher is better). One line traces fixed budgets, another the recheck rule, and a lone dot marks the best-possible ceiling. Up-and-left wins.
Takeaway → The recheck line sits far left of the fixed line at the same heights, so equal accuracy for a fraction of the thinking, but stops just below the fixed peak; the ceiling dot floats highest.
Data table
| mean thinking tokens per task | fixed thinking budget | escalation controller (visible-test signal) | oracle ceiling (non-deployable) |
|---|---|---|---|
| 0 | 76% | — | — |
| 56.4 | — | 87% | — |
| 81.2 | — | 87% | — |
| 113 | — | 88% | — |
| 132 | — | — | 93% |
| 246 | 87% | — | — |
| 277 | — | 89% | — |
| 298 | — | 89% | — |
| 317 | — | 89% | — |
| 359 | — | 89% | — |
| 404 | 87% | — | — |
| 473 | 84% | — | — |
| 507 | 91% | — | — |
| 596 | 86% | — | — |
Numbers from experiments/qwen35_4b_thinking_budget_controller/runs/summary.json
Technical framing
Accuracy vs mean thinking tokens: escalation controller vs fixed budgets — Draft-then-escalate matches fixed-budget accuracy at a fraction of the thinking cost, but never beats the best fixed budget (0.91).
How to read
Each bar is one strategy's share of tasks fully solved, taller is better, running left to right from no thinking through two fixed budgets and the two recheck rules to the non-deployable ideal ceiling.
Takeaway → The recheck rules land at 88 to 89%, wedged between cheaper fixed budgets and the 93% ceiling; only the best fixed budget, at 91%, tops them.
Data table
| strategy | accuracy |
|---|---|
| no thinking | 76% |
| fixed 256 | 87% |
| fixed 1024 (best fixed) | 91% |
| esc no-think to 1024 (2-tier) | 88% |
| esc 256 to 1024 (3-tier) | 89% |
| oracle ceiling | 93% |
Numbers from experiments/qwen35_4b_thinking_budget_controller/runs/summary.json
Technical framing
Accuracy by strategy: controllers trade ~2pp for big cost cuts — Controllers land at 0.88-0.89, between fixed budgets and the 0.93 oracle; only fixed 1024 is more accurate.
How to read
Each bar shows the share of tasks a strategy commits to because the visible example passed, yet which fail the hidden tests; lower is better. Bars run from the best fixed budget to leaner, cheaper recheck rules.
Takeaway → The rate climbs from 6% for the fixed budget to 11% for the leanest rule, so trusting thinner drafts leaks more errors, and this gap caps accuracy below the ceiling.
Data table
| strategy | false-visible-commit |
|---|---|
| fixed 1024 | 6% |
| esc 256 to 1024 (3-tier) | 8% |
| esc no-think to 1024 (2-tier) | 10% |
| esc no-think to 1024 (4-tier) | 11% |
Numbers from experiments/qwen35_4b_thinking_budget_controller/runs/summary.json
Technical framing
False-visible-commit rate: the signal that bounds the controller — 8-11% of controller commits pass the visible test but fail hidden tests - this imperfect signal caps accuracy below the oracle.
In the author’s words from the Overview · “Results”
Headline Efficiency win: the visible-test escalation controller Pareto-dominates every fixed budget except the peak — it matches think_256/512 accuracy (~0.88) at ¼–½ the thinking cost (113–317 vs 246–404 tokens). Not an accuracy win: it does not beat the best fixed budget (think_1024, 0.91); it trades ~2pp accuracy for a large cost cut. If only peak accuracy matters, fixed ~1024 wins. Bounded by C2: the gap to the oracle (0.93) is set by visible-test false-passes (false-visible-commit ~8–11%) — the visible test is a decent but imperfect signal.
Overview
Research Program
- Program:
test_time_reasoning_budget - Program question: can a deployable controller allocate the thinking-token budget better than a fixed budget, given thinking has an overthinking cost and uneven per-task value?
- Prior anchors:
qwen35_4b_thinking_budget_scaling(the sweep that found the +15pp deployable gain, the ~1024 optimum, and the overthinking decline);qwen35_4b_adaptive_evidence_budget_policy(the STOP/MORE analog over evidence probes rather than thinking tokens).
Question
Can a visible-signal controller that decides how much to think per task beat fixed thinking budgets on the deployable accuracy-vs-cost (mean thinking tokens) Pareto?
Hypothesis
The sweep showed thinking's value is uneven (easy tasks barely benefit; hard tasks need the full budget) and that overthinking hurts. So a controller that thinks little by default and escalates only when a cheap visible signal says the answer is wrong should match a fixed budget's accuracy at much lower mean cost. The natural deployable signal is whether a draft answer passes the visible test (the one assert shown in the prompt).
Setup
- Model: Qwen3.5-4B (results reused; this experiment is offline, no new generation).
- Dataset/task source: MBPP sanitized
testsplit, 100 tasks — the same greedy generations at each thinking budget produced byqwen35_4b_thinking_budget_scaling, copied intodata/. - Train/eval split: none — strategies are fixed rules (no learned parameters); evaluated on all 100.
- Baselines: every fixed budget (no_think, 256, 512, 1024, 2048, unbudgeted) as Pareto points.
- Controls: cumulative-cost (re-generate at each rung) vs continue-cost accounting; random/oracle.
- Primary metric (deployable): full-test accuracy vs mean thinking tokens (Pareto), using only the visible test as the escalation signal.
- Oracle-only metric: per-task cheapest full-passing budget → a non-deployable accuracy/cost ceiling.
- Hidden-label boundary: controllers may read only the visible test (first assert); full-test pass and the oracle ceiling use hidden asserts and are reported separately.
false-visible-commit= fraction of tasks committed on a visible pass that actually fail the full test (the C2 risk).
Run
Smoke (re-verify visible tests + load):
../../.venv/bin/python scripts/run.py --smokeFull (offline; ~1–2 min to re-verify 600 stored answers, then simulate):
../../.venv/bin/python scripts/run.py # re-verifies visible tests
../../.venv/bin/python scripts/run.py --no-reverify # reuse cached visible-test resultsResults
Full table in reports/report.md; figure analysis/pareto.png. Headline (deployable full-test accuracy @ mean thinking tokens):
| strategy | acc | think tok |
|---|---|---|
| fixed think_256 | 0.870 | 246 |
| fixed think_512 | 0.870 | 404 |
| fixed think_1024 (best fixed) | 0.910 | 507 |
| fixed think_2048 | 0.860 | 596 |
| esc[no_think→1024] (2-tier) | 0.880 | 113 |
| esc[256→512→1024] | 0.890 | 317 |
| oracle ceiling (non-deployable) | 0.930 | 132 |
- Efficiency win: the visible-test escalation controller Pareto-dominates every fixed budget except the peak — it matches think_256/512 accuracy (~0.88) at ¼–½ the thinking cost (113–317 vs 246–404 tokens).
- Not an accuracy win: it does not beat the best fixed budget (think_1024, 0.91); it trades ~2pp accuracy for a large cost cut. If only peak accuracy matters, fixed ~1024 wins.
- Bounded by C2: the gap to the oracle (0.93) is set by visible-test false-passes (false-visible-commit ~8–11%) — the visible test is a decent but imperfect signal.
Interpretation
The reasoning budget is a worthwhile efficiency knob: a trivial draft-then-escalate rule gets most of thinking's deployable benefit far more cheaply than any fixed budget, but it cannot exceed the best fixed budget's accuracy because the only deployable signal (the visible test) sometimes passes on wrong answers (C2). The headroom to the oracle (0.91→0.93) is small, so the lever here is cost, not peak accuracy; a learned controller's job would be to push toward the oracle by reading richer visible signals (token entropy, self-consistency) than a single visible test.
Knowledgebase Update
- Program evidence updated: yes (
research_programs/test_time_reasoning_budget/evidence.md). - Program backlog updated: yes (learned controller with richer visible signals).
- Claim ledger updated: C9 extended (controller is an efficiency win, not an accuracy win; bounded by C2).
Artifacts
src/controller.pysimulation + visible-test verifier;scripts/run.pyrunner.data/greedy_records.jsonl,data/tasks.json(copied from the sibling sweep; self-contained),data/greedy_with_visible.jsonl(cached visible-test results).runs/summary.json;analysis/pareto_table.md,analysis/pareto.png.reports/report.md,reports/artifact_manifest.yaml.
Report
Rendered from reports/report.md
Summary
Following the thinking-budget sweep (which found native thinking lifts deployable MBPP greedy pass@1 +15pp, with an overthinking optimum ~1024 and uneven per-task value), we ask whether a deployable controller can allocate the thinking budget better than a fixed one. Offline, over the sweep's stored greedy answers at every budget, we simulate visible-test escalation controllers (draft → if the answer fails the one visible assert, escalate to more thinking) and plot deployable full-test accuracy vs mean thinking-token cost. Result: the controller Pareto-dominates every fixed budget except the peak — it matches think_256/512 accuracy (~0.88) at ¼–½ the cost (113–317 vs 246–404 mean thinking tokens) — but it does not beat the best fixed budget (think_1024, 0.91); it trades ~2pp accuracy for a large cost cut. The gap to the non-deployable oracle (0.93) is bounded by visible-test false-passes (~8–11%), a concrete instance of C2.
Research Program Fit
Second experiment of test_time_reasoning_budget. The sweep established that the thinking budget is a deployable lever with an overthinking cost; this experiment asks the deployment question — how to spend that budget — and grounds the program's controller line with a measured Pareto and a clear ceiling (the oracle and the C2-bounded false-pass rate).
Method
- Offline / reuse: no new generation. We copy the greedy generations (one per task per budget) from
qwen35_4b_thinking_budget_scalingintodata/; each already has its full-test pass. - New signal: we re-verify the visible test (the first assert, shown in the prompt) for each stored greedy answer in a sandboxed subprocess — the only signal a deployable controller may read.
- Strategies: fixed budgets (points); visible-test escalation ladders (ascending budgets; commit at the first rung whose answer passes the visible test, else the last rung); a 2-tier
no_think → 1024variant; cumulative-cost (re-generate each rung) vs continue-cost accounting; and a non-deployable oracle ceiling (per task, cheapest budget that full-passes). - Metrics: deployable full-test accuracy; mean thinking tokens (greedy);
false-visible-commit(committed on a visible pass but full-fails). n=100 MBPP test tasks.
Results
| strategy | deployable acc | mean think tok | false-visible-commit |
|---|---|---|---|
| fixed: no_think | 0.760 | 0 | 0.08 |
| fixed: think_256 | 0.870 | 246 | 0.07 |
| fixed: think_512 | 0.870 | 404 | 0.07 |
| fixed: think_1024 | 0.910 | 507 | 0.06 |
| fixed: think_2048 | 0.860 | 596 | 0.05 |
| fixed: think_unbudgeted | 0.840 | 473 | 0.05 |
| esc[256→512→1024] | 0.890 | 317 | 0.08 |
| esc[256→512→1024] +continue | 0.890 | 277 | 0.08 |
| esc[256→512→1024→2048] | 0.890 | 359 | 0.08 |
| esc[no_think→256→512→1024] | 0.870 | 81 | 0.11 |
| esc[no_think→1024] (2-tier) | 0.880 | 113 | 0.10 |
| ORACLE ceiling (non-deployable) | 0.930 | 132 | – |
Figure: analysis/pareto.png. (Mean-thinking-token values use the greedy generations and differ slightly from the sweep report's sampled means.)
- Efficiency Pareto win. The deployable frontier is {2-tier 0.88@113, esc[256→512→1024] 0.89@317, fixed think_1024 0.91@507}. The escalation controllers dominate fixed think_256, 512, 2048, and unbudgeted (same-or-better accuracy at lower cost). The 2-tier rule reaches 0.88 at 113 mean thinking tokens — ~22% of fixed think_1024's cost — losing only ~3pp accuracy.
- No peak-accuracy win. No deployable controller exceeds fixed think_1024 (0.91). Notably, think_1024 alone is already close to the oracle (0.91 vs 0.93), so a fixed near-optimal budget captures most of the achievable accuracy; the controller's lever is cost.
- Escalation is cheap because most tasks stop early. Cumulative vs continue cost differs little (e.g. 317 vs 277), so few tasks pay for multiple rungs.
Controls
Cumulative-cost (re-generate each escalated rung — faithful to budget forcing) vs continue-cost (only the committed rung — an optimistic "keep thinking" bound) bracket the true deployment cost; both leave the controllers Pareto-dominant on the cheap frontier. The oracle ceiling controls for "how much is achievable by any per-task budget choice."
Oracle Versus Deployable Evidence
Deployable strategies read only the visible assert; full-test accuracy and the oracle ceiling use hidden asserts and are labelled non-deployable. The decisive deployable-vs-oracle quantity is false-visible-commit (~8–11%): tasks the controller commits because the visible test passes but which fail the hidden tests. This caps deployable accuracy ~2–4pp below the oracle and is the reasoning-budget instance of the corpus's C2 bottleneck (a visible pass is not a correct answer).
Interpretation
The thinking budget is a real efficiency knob: a trivial draft-then-escalate rule gets most of thinking's deployable benefit far more cheaply than any fixed budget. But it cannot beat the best fixed budget's accuracy, because its only deployable signal sometimes passes on wrong answers (C2), and because a near-optimal fixed budget is already close to the oracle. So the controller's value is cost reduction at near-iso-accuracy, not a new accuracy frontier.
Next Experiments
- Learned controller with richer visible signals (token entropy/logprob, self-consistency across 2 cheap samples) — can it close the 0.89→0.93 oracle gap that the single visible test leaves?
- A latency/wall-clock cost axis (not just token count), connecting to the on-device program.
- The same controller on a harder substrate where the optimum and the C2 false-pass rate are larger.
Artifact Manifest
See artifact_manifest.yaml. Offline experiment; inputs reused from the sibling sweep (copied into data/); no external model call at run time.
Experiment log 4
Show the running log (4 entries)
Scaffold
New experiment under test_time_reasoning_budget, following qwen35_4b_thinking_budget_scaling.
Design
Offline experiment — no new generation. The sibling sweep already produced, per task and per thinking budget, the greedy answer and its full-test pass. We copy those greedy records + the MBPP test metadata into data/ (self-contained), re-verify the visible test (first assert) for each stored answer, and simulate visible-test escalation controllers vs fixed budgets vs an oracle ceiling on a deployable accuracy-vs-mean-thinking-token Pareto.
Result
The escalation controller Pareto-dominates every fixed budget except the peak: the 2-tier no_think → 1024 rule reaches 0.88 deployable full-test accuracy at 113 mean thinking tokens (vs fixed think_256 0.87@246, think_512 0.87@404). It does not beat the best fixed budget (think_1024 0.91@507) — an efficiency win, not an accuracy win. The deployable gap to the oracle ceiling (0.93@132) is bounded by visible-test false-passes (false-visible-commit ~8–11%), the C2 effect. See reports/report.md.
Notes
- Mean-thinking-token figures here use the greedy generation's
n_thinkand differ slightly from the sweep report's sampled means (e.g. think_1024 507 vs 530). - Visible-test re-verification reuses the sandbox approach from the sibling experiment (fork subprocess, rlimits, 10s timeout + retry); cached to
data/greedy_with_visible.jsonl.
Figures 1
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/summary.json3.1 kB
Reproduce
Smoke test
../../.venv/bin/python scripts/run.py --smokeFull run
../../.venv/bin/python scripts/run.pyRun steps are documented inside the experiment folder (README and scripts).