Qwen3.5-4B: Does the Confidence Toolkit Survive on Real Code?
The one idea you need
A model that just wrote a program can gauge its own confidence two ways. Average its word-by-word certainty across hundreds of code tokens, and the signal washes out. Or make it write the code, then answer one yes/no question about it and read that single verdict token — concentrated and sharp.
The question
When a small model writes eight versions of a program, can it tell which one is actually correct just by reading its own confidence, without running any tests?
What we found
Yes, but not the obvious way. Averaging the model's certainty across every token of a program barely beats a plain majority vote among the tries. The real winner: make the model write the code, then answer one yes/no question about it and read that single verdict token. It picks the correct program 76% of the time versus 70% by chance, no tests run — though one real test still beats it, at 82%.
Why it matters
To pick the best of several small-model code drafts with no test harness, have the model self-judge each draft in one extra forward pass and take the highest score — don't rank by average token confidence. If even one real test exists, run it first.
On this page
Results at a glance 3
How to read
Each bar is one way to pick a single program from eight samples; height is how often that pick passes the hidden tests, higher is better. Left to right: random guess, majority vote, average writing confidence, one-token self-judge, running the visible test, and the perfect pick.
Takeaway → The one-token self-judge bar clears majority vote and averaged confidence, which barely beat random; running an actual test towers over every no-test signal, nearly reaching the perfect ceiling.
Data table
| condition | selection accuracy (k=8, 244 MBPP problems) |
|---|---|
| random pick | 69.6% |
| self-consistency (majority) | 72.1% |
| mean-logprob (implicit) | 73% |
| P(True) self-judge (1 token) | 76.2% |
| run the visible test | 81.6% |
| oracle pass@8 | 84.4% |
Numbers from
Technical framing
Picking 1 of 8 code samples: the single-token P(True) self-judgment beats majority voting, verification-free; running a test still wins — The C41 generalization test on real code. Per MBPP problem: 8 temperature samples; each selector picks ONE; scored by hidden asserts. Verification-free P(True) (ask the model 'is this correct, A or B?' and read P(A) -- one forward pass, no execution) beats self-consistency (public-output majority, strictly deployable) by +0.041 (paired bootstrap CI 0.004-0.078, p=0.014) and random by +0.066. The toy substrate's star signal, sequence mean-logprob, does NOT significantly beat majority voting (p=0.37) -- averaging logprob over hundreds of program tokens dilutes the calibrated signal. Honest hierarchy: executing even the single visible test (0.816) crushes every verification-free signal. Confidence is the lever for the verifier-free regime, where it recovers ~45% of the random-to-oracle headroom at zero execution cost.
How to read
Three bars show how reliably each signal ranks a correct program above a wrong one for the same problem — half is a coin flip, higher is better: average writing confidence, the one-token self-judge, and code length as the decoy to beat.
Takeaway → Both confidence signals sit well above the code-length decoy, so it is genuine self-knowledge, not verbosity; the concentrated self-judge edges out the averaged writing confidence.
Data table
| condition | within-problem AUROC (78 mixed problems) |
|---|---|
| mean-logprob (implicit) | 0.693 |
| P(True) self-judge (explicit-framed) | 0.738 |
| code length (surface baseline) | 0.548 |
Numbers from
Technical framing
Is it confidence or just verbosity? Within-problem AUROC vs a code-length surface baseline — The C40 analog, review-hardened: AUROC computed WITHIN each problem that has both passing and failing samples (pooled AUROC is inflated by problem difficulty), against the natural confound on real code -- verbose solutions could fake 'confidence'. Paired per-problem difference vs the length baseline: mean-logprob +0.242 (CI 0.145-0.338), P(True) +0.287 (CI 0.183-0.386) -- the signal is genuine self-knowledge, not verbosity. Both are far below the toy's 0.95: real-code confidence is real but attenuated, and the toy hierarchy INVERTS -- the concentrated single-token readout beats the sequence average.
How to read
The horizontal axis is the fraction of problems kept, most-confident first; the vertical axis is accuracy on those kept — up and to the left is better. One line ranks by the self-judge, one by average writing confidence, and a flat line marks keeping everything.
Takeaway → The self-judge line climbs to about 95% when only the most-trusted third is kept and stays above averaged confidence across nearly the entire range — the model reliably knows which answers to trust.
Data table
| coverage (fraction of greedy attempts kept) | ranked by P(True) | ranked by mean-logprob | no abstention |
|---|---|---|---|
| 0.04 | 90% | 80% | 70.1% |
| 0.08 | 95% | 90% | — |
| 0.16 | 95% | 85% | — |
| 0.2 | 96% | — | — |
| 0.29 | 95.7% | 85.7% | — |
| 0.41 | 92% | 85% | — |
| 0.53 | 90% | 85.4% | — |
| 0.66 | 85.6% | 83.1% | — |
| 0.78 | 81.1% | 81.6% | — |
| 0.9 | 77.3% | 75% | — |
| 1 | 70.1% | 70.1% | 70.1% |
Numbers from
Technical framing
Abstention: keep only the problems the model is confident on and accuracy jumps from 0.70 to ~0.95 — Deployment calibration on the model's single greedy attempt per problem: rank all 244 problems by the confidence signal, keep only the top fraction (x-axis), measure accuracy on what was kept. P(True) solvability AUROC is 0.837 -- essentially identical to the toy substrate's 0.83 (C41) -- and dominates mean-logprob (0.760) everywhere. The abstention half of the C41 recipe transfers to real code almost exactly: a coder that knows which third of its answers to trust at ~95%.
In the author’s words from the Overview · “Results”
The toolkit TRANSFERS — but the hierarchy INVERTS: the single-token P(True) readout, not sequence mean-logprob, is the program-level confidence. C40 analog (within-problem, 78/244 mixed problems): mean-logprob AUROC 0.693 (CI 0.631–0.751), P(True)-no-think 0.738, length baseline 0.548. Paired diff vs length: +0.242 (CI 0.145–0.338) and +0.287 (CI 0.183–0.386) — the signal is confidence, not verbosity. C41 analog (selection at k=8): random 0.696 | mean-logprob 0.730 | P(True) 0.762 | self-consistency (public-output majority) 0.721 | visible-test execution 0.816 | oracle pass@k 0.844. … Read the full result →
Overview
Research Program
- Program:
benchmark_generalization - Program question: do the toy-substrate confidence laws (C40 implicit metacognition, C41 confidence-guided compute) transfer to real program synthesis, where the "answer" is a multi-hundred-token program rather than a single digit?
- Prior anchors: C40 (
qwen35_4b_implicit_metacognition— answer-token P(answer) predicts correctness AUROC 0.95), C41 (qwen35_4b_confidence_guided_compute— confidence-select beats self-consistency; max-confidence predicts solvability AUROC 0.83), C10/C17 (P(True) judge, execution-consensus selection), C42 (step-resolved confidence).
Question
C41's verdict-free selection win was demonstrated on a TOY substrate where confidence = one digit's P(answer). Real code has no single answer token. Which program-level confidence — sequence mean-logprob (implicit), single-token P(True) judge readout (explicit-framed logit), execution-cluster majority (self-consistency) — carries the calibrated signal on MBPP, and does verification-free selection still beat self-consistency?
Hypothesis
The calibrated uncertainty C40 found in the logits is a property of the model, not the task, so SOME logit readout should discriminate correct from incorrect programs better than surface features (length). Expected: mean-logprob inherits the C40 role. (Falsified in part: the hierarchy inverts — see Results.)
Setup
- Model: Qwen3.5-4B, no-think mode for generation and judging (code-mode; dict-format prompts).
- Dataset/task source: MBPP sanitized test split (offline HF cache), 244 usable records; 1 visible test, hidden asserts held out.
- Train/eval split: no training anywhere — pure elicitation. Greedy + k=8 temperature samples (T=0.7, top_p=0.8, top_k=20) per problem = 2,196 candidates.
- Baseline: random-pick expectation, self-consistency (execution-cluster majority via behavior signatures), visible-test execution, oracle pass@k.
- Controls: code-length surface baseline (both signs) for the AUROC claims; within-problem AUROC on mixed problems only (pooled AUROC is inflated by problem difficulty); paired bootstrap over problems for every selection delta.
- Primary metric: (1) within-problem AUROC of each confidence signal vs the length baseline; (2) selection accuracy at k=8; (3) abstention curve on greedy.
- Oracle-only metrics:
full_pass(hidden-assert execution) is ground truth for scoring; never visible to any selection method exceptvisible-test execution(which sees only the public test) andoracle pass@k. - Hidden-label boundary: mean-logprob and P(True) read ONLY the model's own logits; behavior signatures execute candidates on probe inputs without correctness labels.
Run
Smoke:
python scripts/run.py --smokeFull (~35 min generation + logprob + judge on an RTX 4090):
python scripts/run.py # eval_code_conf.py --n 260 --k 8, then analyze.pyOr the stages directly: python scripts/eval_code_conf.py --n 260 --k 8 then python scripts/analyze.py.
Results
The toolkit TRANSFERS — but the hierarchy INVERTS: the single-token P(True) readout, not sequence mean-logprob, is the program-level confidence.
- C40 analog (within-problem, 78/244 mixed problems): mean-logprob AUROC 0.693 (CI 0.631–0.751), P(True)-no-think 0.738, length baseline 0.548. Paired diff vs length: +0.242 (CI 0.145–0.338) and +0.287 (CI 0.183–0.386) — the signal is confidence, not verbosity.
- C41 analog (selection at k=8): random 0.696 | mean-logprob 0.730 | P(True) 0.762 | self-consistency (public-output majority) 0.721 | visible-test execution 0.816 | oracle pass@k 0.844. Paired bootstrap: P(True) beats self-consistency +0.041 (CI 0.004–0.078, p=0.014) and beats mean-logprob +0.033 (p=0.034); mean-logprob does NOT significantly beat self-consistency (p=0.37).
- Abstention (deployable): greedy problem-level AUROC — P(True) 0.837 (≈ the toy's 0.83), mean-logprob 0.760, length 0.688. Keeping the top third of greedy attempts by P(True) yields ~0.94–0.96 accuracy vs 0.701 unfiltered.
- Honest hierarchy: when even ONE visible test exists, executing it (0.816) crushes every verification-free signal and nearly closes the oracle gap. Verification-free confidence recovers ~45% of the selection headroom (P(True): +0.066 of the 0.148 random→oracle gap) with zero execution. Duplicate rate 0.775 on public outputs — most samples agree on the visible test, which is why majority vote adds little (+0.025, echoing C41's flat self-consistency).
Interpretation
C40's law survives contact with real code in refined form: what carries calibrated signal is a CONCENTRATED SINGLE-TOKEN logit readout — P(answer) on the toy, P(A=correct) under a judge prompt on code. Averaging logprob over a few hundred program tokens DILUTES the signal (0.69 vs 0.74 within-problem, 0.76 vs 0.84 on greedy, n.s. vs self-consistency at selection). "Read logits, not self-report" still holds — P(True) here is a logit read of a judgment token, not a sampled verbalization — but the C40 toy hierarchy (implicit >> explicit) does not transfer as stated: the explicit-FRAMED single-token readout wins on long-form outputs. C41's deployable recipe updates to: sample k, judge each with self-P(True) (one no-think forward pass per candidate, batchable), argmax; abstain on low max-P(True); if visible tests exist, execute them first.
Knowledgebase Update
- Program evidence updated:
benchmark_generalization(C40/C41 generalization test). - Program backlog updated: HumanEval replication moved to standalone experiment
qwen35_4b_humaneval_code_confidence; remaining follow-ups are thinking-judge P(True), select+abstain+route matched-compute policy, and step-resolved P(True) repair (C42 bridge). - Claim ledger updated: C46.
Artifacts
src/— generation/judging library (gen_lib), MBPP execution utilities (coverage_utils, code_env)scripts/—eval_code_conf.py(run),analyze.py(verdict + figure),run.py(orchestrator)configs/runs/—code_conf.json(per-candidate records),verdict.json, logsanalysis/—code_confidence.pngreports/—report.mdreports/artifact_manifest.yaml
Report
Rendered from reports/report.md
Summary
Yes — with an inversion. On 244 MBPP problems (greedy + 8 samples each, 2,196 candidates), the model's own logits discriminate correct from incorrect programs well above a length/verbosity surface baseline, and verification-free confidence-selection beats self-consistency (public-output majority), generalizing C40/C41 off the toy substrate. But the WINNING signal is not the toy's implicit mean-logprob — it is the single-token P(True) judge readout (P of the "A = correct" token under a self-judge prompt, no-think, one forward pass per candidate). Sequence mean-logprob is diluted over hundreds of tokens and does not significantly beat majority vote. Abstention transfers almost exactly: greedy-solvability AUROC 0.837 on code vs 0.83 on the toy; keeping the top third by P(True) gives ~0.95 accuracy vs 0.70 unfiltered.
Research Program Fit
benchmark_generalization: the owed generalization test written into C41's caveats ("real code needs a PROGRAM-level confidence — untested, owed"). Verification-free selection is the mission's purest lever: no training, no tools, no external verifier — beat sample-more by reading signal already in the fixed weights.
Method
- MBPP sanitized test (offline cache), 244 records with executable hidden asserts; 1 test visible in the prompt.
- Per problem: greedy + k=8 no-think samples (T=0.7, top_p=0.8, top_k=20), answer budget 420 tokens.
- Per candidate:
full_passground truth by hidden-assert execution;visible_all_pass(public test only);behavior_signature(outputs on probe inputs — execution-clustering);mean_logprob(teacher-forced mean token logprob of the completion, bf16 logits, chunked float32 log-softmax);p_true(P(A) after "Answer: " under a strict-reviewer A/B judge prompt, no-think — the C10 readout);code_len. - Analysis (review-hardened): within-problem AUROC on mixed problems only (pooled AUROC is difficulty-inflated); length baseline both signs; cluster bootstrap over problems; paired bootstrap for every selection delta; selection compared against random, self-consistency (behavior-cluster majority), visible-test execution, and oracle pass@k.
Results
| signal | within-problem AUROC (78 mixed) | greedy problem AUROC | selection at k=8 |
|---|---|---|---|
| random / length | 0.548 (length) | 0.688 (length) | 0.696 |
| mean-logprob (implicit) | 0.693 (CI 0.631–0.751) | 0.760 | 0.730 |
| P(True) no-think | 0.738 (CI 0.665–0.808) | 0.837 | 0.762 |
| self-consistency (public-output majority) | — | — | 0.721 |
| visible-test execution | — | — | 0.816 |
| oracle pass@k | — | — | 0.844 |
Significance (paired bootstrap over 244 problems): P(True) > self-consistency +0.041 (CI 0.004–0.078, p=0.014); P(True) > mean-logprob +0.033 (p=0.034); mean-logprob vs self-consistency n.s. (p=0.37). Within-problem paired diff vs length: logprob +0.242 (CI 0.145–0.338), P(True) +0.287 (CI 0.183–0.386).
Abstention on greedy ranked by P(True): coverage 0.33 → accuracy 0.94; coverage 0.20 → 0.96; unfiltered 0.701. Duplicate rate among samples (public outputs) 0.775. Note: self-consistency clusters on PUBLIC outputs only (visible-test behavior) so the baseline is strictly deployable — hidden-test outputs never inform any selector.
Controls
- Length/verbosity confound: the central threat on code (verbose solutions could inflate mean-logprob or judge scores). Within-problem AUROC vs length in both directions: confidence wins by +0.24/+0.29 with CIs far from zero. Confidence is not verbosity.
- Difficulty confound: headline AUROC is within-problem (mixed problems only); pooled numbers reported separately.
- Selection deltas: all paired over problems, bootstrap CIs; the honest negative (mean-logprob n.s. vs self-consistency) is reported alongside the positive.
Oracle Versus Deployable Evidence
Deployable (no hidden labels): mean-logprob, P(True), behavior-signature majority, visible-test execution (public test only). Oracle-only (scoring): full_pass, pass@k. The abstention curve uses only P(True) ranks; its y-axis is oracle-scored.
Interpretation
The C40 law refines rather than breaks: calibrated uncertainty lives in CONCENTRATED SINGLE-TOKEN logit readouts — the answer digit's P on the toy, the judgment token's P on code. Sequence-averaged logprob dilutes it. C40's "read logits, not self-report" survives — P(True) IS a logit read, not a sampled verbalization — but the toy's implicit-beats-explicit hierarchy inverts on long-form outputs where no single answer token exists. C41's recipe updates for code: sample k, self-judge each candidate (one batchable no-think forward pass), pick argmax P(True), abstain on low max; execution still king when any test exists (0.816 vs 0.762) — confidence is for the verifier-free regime, where it recovers ~45% of the selection headroom at zero execution cost.
Next Experiments
- HumanEval replication, completed separately in
qwen35_4b_humaneval_code_confidence(second real-code substrate; MBPP problems are short). - Thinking-judge P(True) (judge_think exists in gen_lib; does deliberation improve the readout enough to pay for its tokens?).
- Select + abstain + route policy vs matched-compute sample-more (the C41 "owed" compute-optimal curve).
- Step-resolved P(True) on code for targeted repair (C42 bridge): resample only low-confidence regions.
Artifact Manifest
reports/artifact_manifest.yaml — no external artifacts; everything regenerates from scripts/run.py (offline HF cache required for MBPP).
Experiment log 5
Show the running log (5 entries, 2026-07-07)
Scaffold
Created as a new experiment scaffold; src/ seeded with the shared generation/judging library (gen_lib, coverage_utils, code_env) from the coverage-line experiments.
2026-07-07 smoke
eval_code_conf.py --n 4 --k 2 end-to-end on 4 MBPP records: generation, hidden-assert execution, mean-logprob, P(True) all populate. Offline HF cache serves the SANITIZED config ('prompt' field, test_list as JSON string) — adapted in load_sanitized_test.
2026-07-07 full run, attempt 1: CUDA OOM
--n 260 --k 8 crashed in mean_logprobs: full-vocab float32 log-softmax over whole sequences OOMs at batch 8 x ~800 tokens (24 GB card). Fix: keep logits bf16, log-softmax in float32 over 128-token sequence chunks, batch 4, empty_cache() per batch. Lesson consistent with the repo's WSL2 GPU notes: do NOT launch competing GPU jobs while recovering.
2026-07-07 full run, attempt 2: clean
PID 428223, ~35 min: 244 problems x 9 candidates = 2,196 generations, all executed against hidden asserts (overall pass rate 0.70), checkpoint saved before logprob phase, mean_logprob for 2,196/2,196, p_true for 2,163 (33 unparseable completions skipped by design). Output runs/code_conf.json.
2026-07-07 analysis + hardening
analyze.py initial verdict; then hardened in-place: paired bootstrap significance for every selection delta and for within-problem AUROC vs the length baseline; P(True)-ranked abstention curve added alongside mean-logprob. Key numbers: within-problem AUROC logprob 0.693 / P(True) 0.738 / length 0.548; selection at k=8 P(True) 0.762 > self-consistency 0.717 (p=0.005) > random 0.696; visible-test execution 0.816; oracle 0.844; greedy solvability AUROC P(True) 0.837. Verdict: toolkit transfers, hierarchy inverts (single-token P(True) readout beats sequence mean-logprob). Claim C46.
Figures 1
Reproduce
Smoke test
python scripts/run.py --smokeFull run
python scripts/run.pyRun steps are documented inside the experiment folder (README and scripts).