Qwen3.5-4B Verifier vs Visible Selector Showdown
The one idea you need
Picking the best of eight code drafts is like hiring: run each past one cheap screening test, and the handful that survive are so close in quality that a quick gut-feel judgment ranks them as well as slow, careful deliberation.
The question
When a small model writes eight candidate programs and you must pick one to ship, is it worth having it slowly reason through which one is correct?
What we found
No. When you can run even a single example test on each candidate, that filter alone lifts the share of shipped programs that fully work from 77% to 85%. Adding a free, instant self-confidence rating reaches 87% — matching a careful step-by-step self-check that costs five times the compute. Slow deliberation only earns its keep when no test can be run.
Why it matters
When your task ships an example test, spend compute running it first, then break ties among survivors with the model's free instant confidence score. Reserve slow step-by-step self-verification for settings where you cannot execute the code at all.
On this page
Results at a glance 3
How to read
Each bar is one method for picking one of eight programs; height is the share of picks that pass every hidden test (taller is better), running from random pick at 77% up to a non-deployable perfect picker at 89%.
Takeaway → The test-plus-free-confidence bar reaches 87%, level with the five-times-costlier deliberate check and just below the 89% perfect ceiling; cheap methods capture almost all the gain.
Data table
| selector | selection accuracy |
|---|---|
| pass@1 | 77.1% |
| no-think verifier | 80% |
| visible test only | 85% |
| thinking verifier | 86% |
| visible + no-think verifier | 87% |
| visible + thinking verifier | 87% |
| oracle pass@8 | 89% |
Numbers from experiments/qwen35_4b_verifier_selector_showdown/runs/summary.json
Technical framing
Which selector picks the right candidate? (100 MBPP tasks, k=8 pool) — Visible test + a free no-think verifier hits 0.870, tying the 5x-costlier thinking verifier and closing 83% of the pass@1-to-oracle gap.
How to read
Each bar is estimated compute per task for one method; shorter is cheaper. The two deliberate self-check bars tower near 4,960 while the test and quick-confidence options sit down at 960.
Takeaway → The deliberate check costs about five times the cheap combo yet, from the accuracy chart, buys no extra accuracy — the visual signature of wasted spend.
Data table
| selector | estimated tokens per task |
|---|---|
| pass@1 | 120 |
| visible test only | 960 |
| no-think verifier | 960 |
| visible + no-think verifier | 960 |
| thinking verifier | 4960 |
| visible + thinking verifier | 4960 |
Numbers from experiments/qwen35_4b_verifier_selector_showdown/runs/summary.json
Technical framing
Token cost per task: the thinking verifier's edge is expensive — Thinking-verification costs ~5x the visible+no-think combo yet delivers no extra accuracy, so it is Pareto-dominated when a visible test exists.
How to read
Two bars show how reliably each self-check sorts a truly-correct survivor above a secretly-broken one; higher is better, and a coin flip would score 50%. Deliberate scores 76%, quick 70%.
Takeaway → The free quick check trails the costly one only slightly, because after the test filters candidates few broken survivors remain to sort apart.
Data table
| verifier mode | AUROC (true-pass vs false-pass, among visible-passers) |
|---|---|
| thinking verifier | 0.758 |
| no-think verifier | 0.701 |
Numbers from experiments/qwen35_4b_verifier_selector_showdown/runs/summary.json
Technical framing
Ranking survivors of the visible test: no-think verification nearly suffices — Once the visible test filters candidates, only 6.6% are false-passes and the cheap no-think verifier ranks them almost as well as thinking.
In the author’s words from the Overview · “Results”
The thinking verifier is Pareto-dominated: standalone barely beats visible (0.860 vs 0.850) at ~5x cost, and in combination the no-think verifier ties it (both 0.870). Best deployable = visible + no-think verifier (0.870, ~free), closing 83% of the pass@1->oracle gap. C2 false-pass rate here is only 6.6%.
Overview
Research Program
- Program:
evidence_conditioned_selection - Program question: is the model's own thinking-verifier (C10) worth its cost as a deployable selector, and does combining it with the visible test break the C2 false-pass wall?
- Prior anchors:
qwen35_4b_generator_verifier_gap(C10: thinking-verifier is strong standalone),qwen35_4b_thinking_budget_controller(visible-test selector, bounded by false-passes).
Question
On one k=8 candidate pool, compare deployable selectors head-to-head at matched cost: pass@1, visible-only, no-think verifier, thinking verifier, visible+verifier, oracle. Does self-verification beat / complement the visible test, and is thinking-verification worth its ~5x token cost?
Setup
- Reuse the generator-verifier pool (100 MBPP tasks x k=8 no-think candidates, with full-test labels + the model's thinking/no-think black-box P(correct) already computed). Add a visible-test (first-assert) label per candidate. Fully offline (no new generation).
- Selectors pick one candidate per task -> its true full-test pass. Cost = estimated tokens/task (stated assumptions). See
analysis/analyze.py.
Run
../../.venv/bin/python scripts/build_pool.py # add visible-test labels (torch-free)
../../.venv/bin/python analysis/analyze.py # selectors + false-pass breakdown + figureResults
Full results in reports/report.md.
| selector (deployable) | accuracy | ~tok/task |
|---|---|---|
| pass@1 | 0.771 | 120 |
| visible-only | 0.850 | 960 |
| thinking verifier | 0.860 | ~4960 |
| visible + no-think verifier | 0.870 | 960 |
| visible + thinking verifier | 0.870 | ~4960 |
| oracle pass@8 | 0.890 | — |
The thinking verifier is Pareto-dominated: standalone barely beats visible (0.860 vs 0.850) at ~5x cost, and in combination the no-think verifier ties it (both 0.870). Best deployable = visible + no-think verifier (0.870, ~free), closing 83% of the pass@1->oracle gap. C2 false-pass rate here is only 6.6%.
Interpretation
Refines C10: cheap self-verification (no-think) + the cheap visible test is the deployable sweet spot; thinking-verification's expensive edge only matters in verifier-only settings (no cheap execution signal). When a visible test exists, spend tokens on it + a free no-think verifier, not on thinking-verification.
Artifacts
src/tasks.py;scripts/build_pool.py;analysis/analyze.py.data/pool.jsonl+ copied gv_records/labels;runs/summary.json,analysis/selectors.png. No external artifacts.
Report
Rendered from reports/report.md
Summary
Head-to-head deployable selection signals on one k=8 MBPP candidate pool, at matched cost: does the model's own thinking-verifier (claim C10) beat / break the visible-test false-pass wall the earlier thinking controller was bounded by? The matched-cost lens tempers C10. The visible test alone is already a strong, nearly-free selector (0.850, from pass@1 0.771 toward oracle 0.890). The thinking verifier is Pareto-dominated: standalone it reaches 0.860 (barely above visible's 0.850) at ~5× the token cost, and in combination the no-think verifier ties it (both 0.870) — once the cheap visible test does the coarse filtering, ranking the survivors is easy enough that no-think verification suffices. Best deployable = visible + no-think verifier: 0.870 at ~free cost, closing 83% of the pass@1→oracle gap by catching false-passes. So thinking-verification's expensive standalone edge (the big win in the generator-verifier experiment) evaporates whenever a cheap visible signal exists — its value is confined to verifier-only settings.
Research Program Fit
Follows up qwen35_4b_generator_verifier_gap (C10) and qwen35_4b_thinking_budget_controller under evidence_conditioned_selection. It supplies the matched-cost, deployable comparison the C10 result begged for, and answers whether self-verification is worth its token cost as a selector.
Method
- Reuse the generator-verifier candidate pool (100 MBPP tasks × k=8 no-think candidates, with full-test execution labels + the model's thinking/no-think black-box verifier P(correct) already computed). Add a fresh visible-test label per candidate (passes the first assert — the deployable signal the controller used). All offline.
- Deployable selectors (per task pick one candidate → its true full-test pass): pass@1 (random single); visible-only (first visible-passer); no-think / thinking verifier (max P(A)); visible+verifier (among visible-passers, max P(A)); oracle pass@k (non-deployable ceiling).
- Cost: estimated tokens/task (no-think candidate ~120; thinking-verification ~500; visible test / no-think verification ≈ 0 extra generation) — stated assumptions, for the Pareto picture, not measured per-item.
Results
| selector (deployable) | accuracy | ~tokens/task | gap closed (pass@1→oracle) |
|---|---|---|---|
| pass@1 (random single) | 0.771 | 120 | — |
| visible-only (first visible-pass) | 0.850 | 960 | 66% |
| no-think verifier (max P_A) | 0.800 | 960 | 24% |
| thinking verifier (max P_A) | 0.860 | ~4960 | 75% |
| visible + no-think verifier | 0.870 | 960 | 83% |
| visible + thinking verifier | 0.870 | ~4960 | 83% |
| ORACLE pass@8 (non-deployable) | 0.890 | 960 | 100% |
C2 false-pass wall: visible-pass rate 0.818; 6.6% of visible-passers full-fail (43 false-passes). Among visible-passers, the verifier ranks true>false at AUROC 0.758 (think) / 0.701 (no-think). Figure: analysis/selectors.png.
Finding 1 — the visible test is already a strong, nearly-free selector
Visible-only reaches 0.850 (66% of the oracle gap) at ~0 extra tokens. The C2 false-pass rate on this pool is modest (6.6% of visible-passers), so there is limited headroom for any verifier to add.
Finding 2 — the thinking verifier is Pareto-dominated
Standalone thinking-verifier 0.860 barely beats visible-only 0.850, at ~5× the token cost. Combined, visible+thinking (0.870) equals visible+no-think (0.870): the thinking verifier's large standalone edge over no-think (0.860 vs 0.800) disappears once the visible test pre-filters, because ranking visible-passers is an easier discrimination that no-think verification handles nearly as well (AUROC 0.701 vs 0.758).
Finding 3 — best deployable selector = visible + no-think verifier (0.870, ~free)
It beats visible-only by +2pp (catching false-passes the first-visible-passer rule commits) at the same cost, closing 83% of the pass@1→oracle gap. Thinking-verification is not worth its 5× token cost here.
Controls
Same pool for every selector (paired). The +2pp of visible+verifier over visible-only is small (~2 tasks at n=100, within per-condition noise) but paired and mechanistic (it reranks visible-passers to avoid false-passes). The oracle is the non-deployable ceiling.
Oracle Versus Deployable Evidence
All selectors except the oracle read only visible info (visible test + the model's own black-box judgment), so their accuracies are deployable. pass@8 = 0.890 is the non-deployable ceiling.
Interpretation
This refines C10. Self-verification does help selection, but cheap self-verification (no-think) combined with the cheap visible test is the deployable sweet spot — thinking-verification's expensive edge only matters in verifier-only settings (no cheap execution-based signal), which is exactly the regime the generator-verifier experiment measured. The durable lesson: when a cheap ground-truth-ish signal (a visible test) exists, spend tokens on it plus a free no-think verifier, not on expensive thinking-verification; the matched-cost lens flips the standalone ranking.
Limitations
- MBPP (basic, likely contaminated), n=100, single seed; the modest C2 false-pass rate (6.6%) caps verifier headroom on this pool — a harder pool with more false-passes could favor the (stronger) thinking verifier.
- Token costs are estimates (stated assumptions), not measured per-item; the accuracy ranking is exact.
- Not run: the matched-cost-vs-more-sampling arm (spend the ~4000 verification tokens on more candidates to raise coverage) — but visible+no-think-verifier already Pareto-dominates the thinking verifier at ~5× less cost.
Next Experiments
- Harder / contamination-controlled pool with a higher false-pass rate — does the thinking verifier's edge survive when the visible test is weaker?
- Matched-cost vs more-sampling: is any verifier worth its tokens vs just raising k?
Artifact Manifest
See artifact_manifest.yaml. Offline; small pool/summary + figure in-repo; candidate pool reused from qwen35_4b_generator_verifier_gap.
Experiment log 2
Show the running log (2 entries)
Scaffold
Matched-cost, deployable follow-up to the generator-verifier gap (C10): is the thinking-verifier worth its cost as a selector vs the cheap visible test? Fully offline -- reuses the generator-verifier candidate pool (k=8, P(A) signals + full_pass) and adds a visible-test (first-assert) label per candidate.
Results (see reports/report.md)
Deployable selectors on the k=8 pool: pass@1 0.771, visible-only 0.850, no-think verifier 0.800, thinking verifier 0.860, visible+no-think verifier 0.870, visible+thinking verifier 0.870, oracle 0.890. C2 false-pass rate 6.6%. The thinking verifier is Pareto-dominated (0.860 at ~5x cost); no-think ties it in the combination. Best deployable = visible + no-think verifier (0.870, ~free), closing 83% of the pass@1->oracle gap. Tempers C10: thinking-verification's edge only matters in verifier-only settings; when a visible test exists, use it
- a free no-think verifier.
Figures 1
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/summary.json1.0 kB
Reproduce
Smoke test
../../.venv/bin/python scripts/build_pool.pyFull run
../../.venv/bin/python scripts/build_pool.py && ../../.venv/bin/python analysis/analyze.pyRun steps are documented inside the experiment folder (README and scripts).