Qwen3.5-4B Generator-Verifier Gap
The one idea you need
Picture a coder who dashes off eight rough versions of a function. Asked to grade them on sight, they nod along and mark almost everything correct. Given a minute to trace each line, they turn sharp — reliably catching the broken ones.
The question
If a small model writes eight tries at a coding problem, can it reliably tell which ones actually work — without running the code?
What we found
Only after it thinks. Judging on sight, the model rubber-stamps 91% of its tries as correct while just 77% truly pass — barely a check, mostly agreeing with itself. Given room to reason first, it becomes a real critic: letting it pick its best of eight lifts success from 77% to 86%, closing three-quarters of the gap to a flawless picker's 89% ceiling.
Why it matters
Before training a separate model to rank candidate answers, first have the model reason through and grade its own — zero extra training, it beats blind picking, and it works from the task text alone with no hidden tests.
On this page
Results at a glance 3
How to read
Four accuracy bars for the coding tasks, left to right: picking one of eight tries at random, the snap-judge's pick, the thinking-judge's pick, and a flawless picker that peeks at hidden tests. Taller is better; the last bar is the ceiling.
Takeaway → The thinking-judge bar nearly reaches the flawless ceiling and towers over random and snap-judge picks — reasoning before judging recovers most of the achievable gain.
Data table
| condition | MBPP accuracy |
|---|---|
| pass@1 (random pick) | 77.1% |
| no-think verifier pick | 80% |
| thinking verifier pick | 86% |
| oracle pass@8 | 89% |
Numbers from experiments/qwen35_4b_generator_verifier_gap/runs/summary.json
Technical framing
Best-of-8 accuracy: the model's own verifier closes most of the oracle gap — A zero-training thinking self-verifier picks best-of-8 at 86%, closing 75% of the pass@1-to-oracle gap (24% without thinking).
How to read
Two skill scores — grading accuracy that corrects for guessing, and how reliably a correct answer outranks a wrong one — each shown for snap-judging (lighter) versus thinking (darker). Higher is better; 50% is pure guessing.
Takeaway → Both scores jump sharply with thinking — grading from 63% to 83% — turning a near-coin-flip judge into a dependable critic of its own code.
Data table
| condition | no-think | thinking |
|---|---|---|
| balanced accuracy | 62.7% | 82.7% |
| AUROC | 77.3% | 92.6% |
Numbers from experiments/qwen35_4b_generator_verifier_gap/runs/summary.json
Technical framing
Intrinsic self-verification skill: thinking turns the model into a real critic — Thinking adds +0.20 balanced accuracy and +0.15 AUROC when judging the model's own candidates correct vs incorrect.
How to read
Three bars: the share of candidates the snap-judge calls correct, the share the thinking-judge calls correct, and the true pass rate as the honest target. Landing closer to the true rate is better; far above it means rubber-stamping.
Takeaway → The snap-judge sits well above the truth at 91% against a real 77%; thinking pulls its approvals down toward the actual rate — a sign of genuine judgment.
Data table
| condition | rate judged correct (say-A) |
|---|---|
| no-think verifier | 91.2% |
| thinking verifier | 82.8% |
| actual correct rate | 77.1% |
Numbers from experiments/qwen35_4b_generator_verifier_gap/runs/summary.json
Technical framing
Yes-bias control: without thinking the verifier mostly agrees with the generator — No-think says 'correct' 91% of the time vs a true 77% pass rate; thinking lowers the yes-bias toward the truth.
In the author’s words from the Overview · “Results”
Checking is easier than doing — but only with thinking. No-think self-verification is weak/yes-biased (AUROC 0.77, says "correct" 91%); thinking makes it a real critic (AUROC 0.93). C2 is plumbing, not capability. The model's own black-box, training-free, deployable thinking-verifier closes 75% of the pass@1→oracle gap. The selection program has real headroom. C9 inversion: thinking helps verification (+0.20 balanced-acc) at least as much as generation — its deepest value may be helping the model know which answer is right.
Overview
Research Program
- Program:
evidence_conditioned_selection(bridges totest_time_reasoning_budgetvia the thinking axis). - Program question: is the C2 selection wall (coverage ≫ deployable selection) a capability limit (the model can't tell right from wrong any better than it can produce right) or a plumbing limit (verification is good, but current selectors waste it)?
- Prior anchors: claim C2 (
qwen35_4b_retrieval_adapt_verify_scale,qwen35_4b_real_sample_verify_commit); the whole selection program builds trained selectors and reports SELECTED accuracy but never isolates the upstream black-box primitive of self-verification. Claim C9 (thinking helps generation content).
Question
Is checking easier than doing for a frozen 4B? Measure its intrinsic verification skill (given a candidate solution, judge correct/incorrect as a black box — no execution, no hidden tests) against its generation skill (pass@1) on the same MBPP items, at no-think and thinking-on. And can the model's own verifier close the pass@1 → oracle-pass@k gap — i.e. is C2 fixable? Does thinking help verification asymmetrically more than it helped generation (checking is often the easier reasoning problem)?
Hypothesis
If verification balanced-accuracy ≫ generation pass@1 and verifier-selected accuracy approaches oracle pass@k, the selection program has real headroom (C2 is plumbing). If verification ≈ generation and verifier-selection ≈ pass@1, C2 is a capability limit and the portfolio should pivot to coverage.
Setup
- Model Qwen3.5-4B frozen (bf16, fast path). MBPP sanitized
test, 100 tasks, k=8 no-think candidates, execution-labeled. - Verification judge: present the task spec + one example assert + the candidate code; read the A/B logit at a forced answer position → P(correct) (A=32 correct, B=33 incorrect, single tokens). No-think: one forward. Thinking: generate up to a 1024-token budget, force
</think>, then read the A/B logit. - Metrics: generation pass@1 / oracle pass@k; verification balanced-accuracy + AUROC (discriminate the model's OWN correct vs incorrect candidates) at no-think and think; verifier-selected best-of-k accuracy vs pass@1 vs oracle (the fraction of the pass@1→oracle gap the verifier closes); the thinking asymmetry.
- Controls: foreign-solution judge (a different task's candidate → should be rejected, P(A) low) tests that the verifier actually reads the task; say-A rate + balanced accuracy control the yes-bias.
Run
HF_HUB_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
../../.venv/bin/python scripts/run.py --tasks 100 --k 8 --budget 1024
../../.venv/bin/python scripts/verify.py # execution labels
../../.venv/bin/python analysis/analyze.py # metrics + figureResults
Full results in reports/report.md.
| quantity | no-think | thinking |
|---|---|---|
| verification balanced accuracy | 0.627 | 0.827 |
| verification AUROC | 0.773 | 0.926 |
| verifier-selected best-of-8 (deployable) | 0.800 | 0.860 |
| pass@1(0.771)→oracle(0.890) gap closed | +24% | +75% |
| foreign reject rate | 1.00 | 1.00 |
- Checking is easier than doing — but only with thinking. No-think self-verification is weak/yes-biased (AUROC 0.77, says "correct" 91%); thinking makes it a real critic (AUROC 0.93).
- C2 is plumbing, not capability. The model's own black-box, training-free, deployable thinking-verifier closes 75% of the pass@1→oracle gap. The selection program has real headroom.
- C9 inversion: thinking helps verification (+0.20 balanced-acc) at least as much as generation — its deepest value may be helping the model know which answer is right.
Interpretation
The selection bottleneck isn't that the model can't tell right from wrong — it can, well, once it thinks. The highest-leverage "plumbing" for C2 is thinking-augmented self-verification (cheaper and stronger than the trained selectors the corpus favored). Thinking (C9) and selection (C2) meet in thinking-verification. See claim C10.
Artifacts
src/judge_lib.py(generation + A/B-logit verification judge),src/tasks.py.scripts/run.py,scripts/verify.py,analysis/analyze.py.data/records.jsonl,data/labels.jsonl,data/tasks.json(small, in-repo). No external artifacts (behavioral + logit-read; no activation caching).
Report
Rendered from reports/report.md
Summary
Is the C2 selection wall (coverage ≫ deployable selection) a capability limit or a plumbing limit? We measure a frozen Qwen3.5-4B's intrinsic verification skill (black-box: judge a candidate correct/incorrect from the A/B logit, no execution, no hidden tests) against its generation skill (pass@1) on the same MBPP items, at no-think and thinking-on. Result: checking is easier than doing — but only with thinking. No-think self-verification is weak and heavily yes-biased (balanced-acc 0.627, AUROC 0.773, says "correct" 91% of the time — essentially the generator agreeing with itself). Thinking turns it into a real critic (balanced-acc 0.827, AUROC 0.926). And the model's own thinking-verifier — zero training, fully deployable — selects best-of-8 at 0.860, closing 75% of the pass@1→oracle gap (0.771 → 0.890), vs 24% for no-think. So C2 is plumbing, not capability: the selection program has real headroom, and the lever is thinking-augmented self-verification. This also inverts C9: thinking helps verification (+0.20 balanced-acc) at least as much as it helped generation — its deepest value may be helping the model know which answer is right, not just produce right ones.
Research Program Fit
Bridges evidence_conditioned_selection (C2) and test_time_reasoning_budget (C9). C2 has been treated as an empirical wall across ~160 experiments, all of which build trained selectors and report selected accuracy; none isolated the upstream black-box primitive of self-verification, or asked whether the wall is verification capability vs evidence plumbing. This is that one measurement.
Method
- Model Qwen3.5-4B frozen (bf16, fast path). MBPP sanitized
test, 100 tasks, k=8 no-think candidates, execution-labeled (pass rate 0.771). - Black-box verification judge: present the task spec + one example assert + the candidate code, read P(correct) from the A(=correct, tok 32) vs B(=incorrect, tok 33) logit at a forced answer position — no execution, no hidden tests. No-think = one forward; thinking = generate up to 1024 thinking tokens, force
</think>, then read the A/B logit at "Answer: ". - Metrics: generation pass@1 / oracle pass@k; verification balanced-accuracy + AUROC (discriminate the model's own correct vs incorrect candidates); verifier-selected best-of-k vs pass@1 vs oracle (gap closed); the thinking asymmetry. Controls: foreign-solution judge (a different task's candidate → should be rejected); say-A rate + balanced accuracy control the yes-bias.
Results
| quantity | no-think | thinking |
|---|---|---|
| verification balanced accuracy | 0.627 | 0.827 |
| verification AUROC | 0.773 | 0.926 |
| say-A (correct) rate | 0.91 | 0.83 |
| verifier-selected best-of-8 | 0.800 | 0.860 |
| pass@1→oracle gap closed | +24% | +75% |
| foreign reject rate | 1.00 | 1.00 |
Generation: pass@1 0.771, oracle pass@8 0.890. Figure: analysis/gen_verify.png.
Finding 1 — checking is easier than doing, but only with thinking
No-think self-verification is weak (balanced-acc 0.627, AUROC 0.773) with a strong yes-bias (say-A 0.91 > base pass 0.771) — the no-think "verifier" mostly re-agrees with the generator. Thinking makes it a genuine critic: balanced-acc 0.827, AUROC 0.926, and a lower yes-bias (0.83). So the model can tell its own good solutions from its bad ones — when it thinks.
Finding 2 — C2 is plumbing, not a capability wall
Using the model's own (black-box, training-free, deployable) verifier to pick best-of-8 lifts pass@1 0.771 → 0.860 with thinking (0.800 no-think), against an oracle of 0.890 — closing 75% of the achievable gap (24% no-think). The selection program the corpus never stopped building has real headroom; the missing piece was thinking-augmented self-verification, not a better trained selector.
Finding 3 — thinking helps verification (the C9 inversion)
Thinking raised verification balanced-acc +0.20 (AUROC +0.15), comparable to or exceeding its effect on generation (C9: greedy +15pp). Thinking's value is not only in producing correct answers but in recognizing them — which is exactly the primitive selection needs.
Controls
Foreign-solution judgments (a different task's candidate spliced in) are rejected at rate 1.00 in both modes → the verifier genuinely reads the task, not a length/format heuristic. Balanced accuracy and the say-A rate control for the yes-bias (a constant "A" predictor scores 0.5 balanced-acc; no-think's 0.627 shows weak-but-real discrimination, thinking's 0.827 shows strong discrimination).
Oracle Versus Deployable Evidence
The verification judge uses only the task spec + candidate (no execution, no hidden tests), so verifier-selected accuracy (0.860 thinking) is deployable. pass@8 = 0.890 is the non-deployable oracle ceiling. So a deployable, zero-training thinking-verifier recovers 75% of the oracle headroom — notably, without the visible-test signal the earlier thinking controller relied on (which was bounded by visible-test false-passes at ~0.91 deployable / 0.93 oracle on a different pool).
Interpretation
For a small model on this benchmark, the selection bottleneck is not that the model can't tell right from wrong — it can, quite well, once it thinks (AUROC 0.93). The corpus's C2 wall is an evidence-plumbing problem, and the highest-leverage plumbing is the model's own thinking-verifier. This unifies the corpus's two strongest recent threads: thinking (C9) and selection (C2) meet in thinking-augmented verification, which is cheaper and stronger than the trained selectors the selection program has favored, and a natural next controller signal.
Limitations
- MBPP (basic, likely partly contaminated), n=100, single seed; candidates are no-think generations (a think-generated pool may be harder to verify). The verifier sees one example assert (as the generator did). The "checking > doing" comparison is across different scales (discrimination AUROC vs generation pass@1); the deployable selection result (verifier-selected vs pass@1 vs oracle) is the apples-to-apples headline.
Next Experiments
- Wire the thinking-verifier as the controller signal (vs / combined with the visible test) and measure the deployable accuracy-vs-token Pareto — does self-verification beat the visible-test C2 wall at matched cost?
- Verify a think-generated candidate pool (harder negatives) and on a contamination-controlled substrate.
- Iterated generate→self-verify→revise loop: does the strong thinking-verifier drive self-correction?
Artifact Manifest
See artifact_manifest.yaml. Small records/labels + summary + figure in-repo; no external artifacts.
Experiment log 4
Show the running log (4 entries)
Scaffold
Attacks the C2 selection bottleneck at its root: is the wall a verification-CAPABILITY limit or a selection-PLUMBING limit? Measures the frozen 4B's intrinsic black-box verification skill vs its generation skill on the same MBPP items, at no-think and thinking-on. Also inverts C9 (thinking helps generation content; does it help verification asymmetrically?).
Method
- k=8 no-think candidates per task (execution-labeled). Verification judge: task spec + candidate code -> read A/B logit -> P(correct); A=32/B=33 single tokens. No-think = one forward; thinking = generate up to 1024 thinking tokens, force </think>, then read the A/B logit at "Answer: ".
- Metrics: generation pass@1/pass@k; verification balanced-accuracy + AUROC; verifier-selected best-of-k vs pass@1 vs oracle (gap closed); thinking asymmetry. Controls: foreign-solution judge, say-A rate.
Smoke
4 tasks x k=4: pipeline validated. P(A) ~0.65-0.88 on own (correct) candidates, ~0.01 on FOREIGN candidates (verifier correctly rejects a different task's solution) -- the judge reads the task. Thinking raised P(A) on correct candidates (0.7 -> 0.9+); need incorrect candidates (full run) to see discrimination.
Results (see reports/report.md)
Generation pass@1 0.771, oracle pass@8 0.890. Verification balanced-acc/AUROC: no_think 0.627/0.773 (say-A 0.91, heavy yes-bias), think 0.827/0.926 (say-A 0.83). Verifier-selected best-of-8: no_think 0.800 (+24% of the pass@1->oracle gap), think 0.860 (+75%). Foreign reject 1.00 both. Findings: checking is easier than doing but only WITH thinking; C2 is plumbing not capability (deployable thinking-verifier closes 75% of the oracle gap, no training/execution); thinking helps verification (+0.20 bal-acc) >= generation (C9 inversion). Claim C10.
Note: judge_think initially generated thinking for the whole 800-item list in one _gen call (fine for the 16-item smoke, would OOM at 800) -- fixed with chunked OOM-resilient processing before the full run.
Figures 1
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/summary.json771 B
Reproduce
Smoke test
../../.venv/bin/python scripts/run.py --smokeFull run
HF_HUB_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True ../../.venv/bin/python scripts/run.py --tasks 100 --k 8 --budget 1024Run steps are documented inside the experiment folder (README and scripts).