Research log Small Model Experimentation
GitHub

Qwen3.5-4B: Does Code Confidence Replicate on HumanEval?

Finished2026-07-08Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and DiagnosticsReliability and SafetyGitHub ↗
One yes-or-no check beats averaging the whole answer

The one idea you need

A model writes a function one token at a time. You could gauge its confidence from how smoothly it "typed" hundreds of characters, but doubt in one spot blurs into the average. Better to ask point-blank, "Is this right?", and read how firmly it answers that one word.

The question

When a small model writes several candidate programs and no tests are available to run them, how can it tell which one is actually correct?

What we found

Ask it directly. Averaging the model's confidence across every code token barely helps, nudging correct picks from 77% (blind guessing) to just 79%. But handing the model its own code and reading its confidence in one yes-or-no verdict jumps that to 84% — near the 87% ceiling of a flawless picker. How you ask matters more than whether you ask.

Why it matters

When no tests exist to run generated code, don't rank candidates by average per-token confidence — long answers dilute the signal. Feed each solution back, ask the model to judge it yes-or-no, and pick by its confidence in that verdict word.

Picking the right program79% → 84%averaging confidence over every code token vs. reading one yes-or-no self-verdict
Blind guessing baseline77%expected accuracy from picking a candidate at random
Best possible picker87%ceiling if a correct candidate were always chosen when one existed
Ranking a correct solution above a wrong one78% vs 67%the self-verdict vs token-averaging, where chance is 50%
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Next Experiments
    9. Artifact Manifest
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 1

How well each method picks correct code when no tests can run

How to read

Four bars, taller is better, showing how often each method's pick passes the hidden tests across all 164 coding tasks. Left to right: blind random pick (77%), averaging confidence over every code token (79%), one yes-or-no self-verdict (84%), and a flawless picker (87%).

0%25%50%75%100%random pickrandom pick76.6%mean-logprob (implicit)mean-logprob (implicit)78.7%P(True) self-judge (1 token)P(True) self-judge (1 token)83.5%oracle pass@8oracle pass@887.2%

Takeaway → The self-verdict bar clears both averaging and guessing and nearly reaches the flawless-picker ceiling, showing one concentrated judgment carries more correctness signal than the diluted token-by-token average.

Data table
conditionselection accuracy (k=8, 164 HumanEval tasks)
random pick76.6%
mean-logprob (implicit)78.7%
P(True) self-judge (1 token)83.5%
oracle pass@887.2%

Numbers from

Technical framing

HumanEval with no public probes: P(True) beats mean-logprob and random in the strict verifier-free setting — Standalone HumanEval replication of C46. All 164 tasks are included with no public probes, so public-output majority and visible-test execution are deliberately unavailable. P(True) beats mean-logprob by +0.049 (paired bootstrap CI 0.012-0.091, p=0.011) and random by +0.069. Greedy solvability AUROC is 0.862 for P(True) vs 0.734 for mean-logprob.

In the author’s words from the Overview · “Results”

Main all-task no-probe run: Selection at k=8: random 0.766, mean-logprob 0.787, P(True) 0.835, oracle pass@8 0.872. P(True) beats mean-logprob by +0.049 (paired bootstrap CI 0.012 to 0.091, p=0.011) and random by +0.069 (p<0.001). Within-problem AUROC on 51 mixed problems: length 0.573, mean-logprob 0.672, P(True) 0.779. P(True) beats the length baseline by +0.351 (CI 0.229 to 0.475). Greedy solvability AUROC: P(True) 0.862 vs mean-logprob 0.734 and length 0.612. The public-probe diagnostic is ceiling-limited but useful for comparison with MBPP's visible-test setting: 68 tasks with one parseable doctest public example. … Read the full result →

Overview

Standalone HumanEval replication of the C46 code-confidence result: test whether the P(True) judgment-token readout beats verifier-free sample-more baselines on HumanEval.

Research Program

  • Programs: benchmark_generalization, evidence_conditioned_selection
  • Program question: does the real-code confidence result from C46 survive a second program-synthesis substrate when public probes are removed?
  • Prior anchors: C40 (qwen35_4b_implicit_metacognition) found calibrated single-answer-token confidence on a toy substrate; C41 (qwen35_4b_confidence_guided_compute) used that confidence to beat self-consistency; C46 (qwen35_4b_code_confidence) transferred the idea to MBPP and found that program-level confidence is P(True), not sequence mean-logprob.

Question

On HumanEval, with no public tests available to execute or cluster on, can a single-token P(True) self-judge readout pick better samples than random selection and sequence mean-logprob?

Hypothesis

If C46's refined law is real, calibrated uncertainty should live in a concentrated judgment-token logit readout. P(True) should beat sequence mean-logprob, because averaging logprob over hundreds of code tokens dilutes the correctness signal.

Setup

  • Model: Qwen/Qwen3.5-4B, no-think generation and no-think judging.
  • Dataset/task source: all 164 HumanEval tasks, using hidden tests only for scoring. The main run uses --visible-tests 0, so no method can execute or cluster on public probes.
  • Train/eval split: no training. Each problem gets one greedy candidate plus 8 sampled candidates.
  • Baselines: random-pick expectation and sequence mean-logprob; oracle pass@8 brackets remaining headroom.
  • Controls: code-length surface baseline, within-problem AUROC on mixed problems only, paired bootstrap over tasks for selection deltas.
  • Primary metrics: selection accuracy at k=8, within-problem AUROC of confidence vs correctness, and greedy solvability AUROC.
  • Oracle-only metrics: full_pass and pass@8 use hidden tests for scoring only.
  • Hidden-label boundary: P(True) and mean-logprob read only model logits; no selector sees hidden-test results.

Run

Smoke:

python scripts/run.py --smoke

Full HumanEval no-public-probe replication:

python scripts/run.py

Public-probe diagnostic, limited to the 68 HumanEval tasks with one parseable doctest example:

python scripts/run.py --visible-tests 1 --out-name humaneval_code_conf --title "HumanEval public-probe subset" --judge-batch-size 1

Results

Main all-task no-probe run:

  • Selection at k=8: random 0.766, mean-logprob 0.787, P(True) 0.835, oracle pass@8 0.872.
  • P(True) beats mean-logprob by +0.049 (paired bootstrap CI 0.012 to 0.091, p=0.011) and random by +0.069 (p<0.001).
  • Within-problem AUROC on 51 mixed problems: length 0.573, mean-logprob 0.672, P(True) 0.779. P(True) beats the length baseline by +0.351 (CI 0.229 to 0.475).
  • Greedy solvability AUROC: P(True) 0.862 vs mean-logprob 0.734 and length 0.612.

HumanEval no-public-probe figure

The public-probe diagnostic is ceiling-limited but useful for comparison with MBPP's visible-test setting:

  • 68 tasks with one parseable doctest public example.
  • Selection: random 0.904, mean-logprob 0.926, public-output majority 0.926, P(True) 0.941, visible-test execution 0.941, oracle 0.971.
  • P(True) beats random (+0.037, p=0.020) but does not significantly beat public-output majority or mean-logprob on this easy subset.

HumanEval public-probe diagnostic

Interpretation

This independently supports the C46 law on a second code benchmark: verification-free confidence should be read from a concentrated P(True) judgment-token logit, not from sequence-averaged completion likelihood. The result is not just verbosity: the P(True) within-problem AUROC is far above the length baseline. It also sharpens the deployment boundary: when public tests exist, executing them remains the stronger method; when no verifier is available, P(True) is the selector and abstention signal.

Knowledgebase Update

  • Program evidence updated: benchmark_generalization and evidence_conditioned_selection.
  • Claim ledger updated: C46 now cites this standalone HumanEval replication alongside the MBPP experiment.
  • Shared synthesis updated: C46 is a two-experiment result, not a single extended experiment.

Artifacts

  • src/ and scripts/ contain the standalone HumanEval-capable harness copied into this experiment.
  • configs/default.yaml records the HumanEval no-probe setup.
  • runs/humaneval_code_conf_novis*.json are the full no-public-probe run.
  • runs/humaneval_code_conf*.json are the public-probe diagnostic subset.
  • analysis/ contains the no-probe and public-probe figures.
  • reports/report.md, reports/design_review.md, and reports/artifact_manifest.yaml document the result and reproduction path.

Report

Rendered from reports/report.md

Summary

The C46 code-confidence result replicates on HumanEval when evaluated as a standalone, verifier-free experiment. On all 164 HumanEval tasks with no public probes, P(True) selection reaches 0.835 at k=8, beating sequence mean-logprob 0.787 and random selection 0.766. Oracle pass@8 is 0.872, so P(True) recovers most of the available selection headroom without executing tests.

The replication preserves the important inversion from MBPP: the useful program-level confidence signal is not average completion likelihood. It is a single-token judgment readout, P(A = correct), under a strict A/B self-judge prompt.

Research Program Fit

benchmark_generalization: tests whether the MBPP real-code confidence result survives a second benchmark.

evidence_conditioned_selection: tests a deployable selector for the no-verifier regime, where public tests or execution signals are unavailable.

Method

  • Dataset: all 164 HumanEval tasks.
  • Main condition: --visible-tests 0, so public-output majority and visible-test execution are intentionally unavailable.
  • Candidates: one greedy completion plus 8 no-think samples per task.
  • Generation: temperature 0.7, top_p 0.8, top_k 20, 420-token answer budget.
  • Ground truth: hidden tests score full_pass only after selection.
  • Signals: mean_logprob, p_true, and code_len.
  • Analysis: within-problem AUROC on mixed problems only; paired bootstrap over tasks for selection deltas.

Results

All HumanEval Tasks, No Public Probes

selectorselection accuracy at k=8
random pick expectation0.766
mean-logprob0.787
P(True) self-judge token0.835
oracle pass@80.872

Paired bootstrap:

  • P(True) > mean-logprob: +0.049, CI 0.012 to 0.091, p=0.011.
  • P(True) > random: +0.069, CI 0.040 to 0.100, p<0.001.

Within-problem AUROC on 51 mixed problems:

signalAUROC
length baseline0.573
mean-logprob0.672
P(True)0.779

P(True) beats the length baseline by +0.351 (CI 0.229 to 0.475). Greedy solvability AUROC is 0.862 for P(True), 0.734 for mean-logprob, and 0.612 for length.

HumanEval no-public-probe figure

Public-Probe Diagnostic

The public-probe run uses the 68 HumanEval tasks with one parseable doctest example. This is not the main endpoint because the subset is much easier:

selectorselection accuracy at k=8
random pick expectation0.904
mean-logprob0.926
public-output majority0.926
P(True)0.941
visible-test execution0.941
oracle pass@80.971

P(True) beats random (+0.037, p=0.020), but does not significantly beat mean-logprob or public-output majority on this ceiling-limited subset.

Controls

  • No public-probe leakage: the main run has zero public cases, so self-consistency over public behavior and visible execution are undefined rather than weak.
  • Verbosity control: P(True) is compared against code length within each mixed problem; its +0.351 AUROC advantage rules out a pure length surface.
  • Difficulty control: AUROC is reported within mixed problems only.
  • Selection uncertainty: paired bootstrap over tasks is reported for the P(True) deltas.
  • Compute/memory control: the long HumanEval judge prompts require --judge-batch-size 1 on the 24 GB RTX 4090 used here; logprob artifacts are persisted before the judge phase so an OOM does not discard generation work.

Oracle Versus Deployable Evidence

Deployable, no hidden labels: mean-logprob, P(True), and code length. Oracle-only: full_pass and pass@8. The public-probe diagnostic adds deployable visible-test execution and public-output majority, but only on the 68-task subset with a parseable doctest example.

Interpretation

HumanEval supports the refined C46 law: calibrated uncertainty for long-form code is best read from a concentrated judgment-token logit. Sequence mean-logprob still carries signal, but it is weaker because correctness evidence is diluted across many tokens. This makes P(True) the current default confidence signal for verifier-free selection and abstention on code.

The result does not imply confidence should replace execution. Where public tests exist, visible execution remains the stronger deployable selector. The scope is the no-verifier regime: when execution evidence is absent or too expensive, P(True) beats sample-more baselines that only count or average candidates.

Next Experiments

  1. HumanEval/MBPP select + abstain + route policy versus matched-compute sample-more.
  2. Thinking-judge P(True): test whether deliberative judging improves enough to justify its token cost.
  3. Step-resolved P(True) repair for code, connecting this line to C42.

Artifact Manifest

reports/artifact_manifest.yaml records that all artifacts are checked in. The full reproduction command is python scripts/run.py.

Experiment log 4

Show the running log (4 entries, 2026-07-08)

2026-07-08 scaffold split

Created as a standalone experiment after the HumanEval replication had been incorrectly added inside qwen35_4b_code_confidence. The harness, HumanEval run artifacts, and figures were copied into this experiment so C46's MBPP experiment can remain self-contained and this replication has its own README, report, log, manifest, site brief, and chart.

2026-07-08 smoke

python scripts/run.py --dataset humaneval --visible-tests 0 --n 4 --k 2 --out-name humaneval_smoke --judge-batch-size 1 completed generation, execution, mean-logprob, and P(True) population on a small HumanEval subset.

2026-07-08 public-probe diagnostic

Command:

python scripts/run.py --dataset humaneval --visible-tests 1 --n 164 --k 8

Only 68 HumanEval tasks had one parseable doctest example usable as a public probe. The subset is ceiling-limited: random 0.904, mean-logprob 0.926, public-output majority 0.926, P(True) 0.941, visible-test execution 0.941, oracle 0.971. P(True) beats random but not public-output majority or mean-logprob significantly.

2026-07-08 all-task no-public-probe run

First all-task attempt reached generation, hidden-test scoring, and mean-logprob but OOMed during the P(True) judge at batch size 16. The harness now persists runs/humaneval_code_conf_novis_logprob.json before judging and exposes --judge-batch-size.

Successful command:

python scripts/run.py --dataset humaneval --visible-tests 0 --n 164 --k 8 \
  --out-name humaneval_code_conf_novis \
  --title "HumanEval all tasks, no public probes" \
  --judge-batch-size 1

Result: random 0.766, mean-logprob 0.787, P(True) 0.835, oracle pass@8 0.872. Within-problem AUROC on 51 mixed problems: length 0.573, mean-logprob 0.672, P(True) 0.779. P(True) beats mean-logprob by +0.049 (p=0.011) and beats random by +0.069 (p<0.001).

Figures 2

humaneval code conf novis
humaneval code conf novis · analysis/
humaneval code confidence
humaneval code confidence · analysis/

Data files 2

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

Smoke test

python scripts/run.py --smoke

Full run

python scripts/run.py

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗