Qwen3.5-4B: Does the Model Know When It Will Fail?
The one idea you need
The model answers with one digit, but underneath it spreads a fixed pile of betting chips across all ten possible digits. When it knows the answer, nearly every chip lands on one; when guessing, they scatter thin. That hidden pile is its true confidence — not anything it says aloud.
The question
When a small model is about to get an answer wrong, does it have any inner sense that it's just guessing — and can it actually tell you so?
What we found
Yes, but only in its numbers, never its words. The probability it quietly places on the digit it writes sorts right answers from wrong ones almost perfectly — 95 out of 100, versus 50 for a coin flip — and beats guessing from surface prompt features alone (61). Yet ask it outright "is your answer correct?" and it scores 46, no better than chance.
Why it matters
For a confidence or abstain gate on a small model, read the probability it assigns to its own answer — one forward pass. Never trust a "how confident are you?" prompt or a self-check; both are noise here.
On this page
Results at a glance 2
How to read
Four task types run left to right, easiest to near-impossible. Each shows three bars: how often the model is actually right, the probability it placed on its own answer, and its yes/no self-check. A confidence bar that hugs the accuracy bar is good.
Takeaway → The answer-probability bars rise and fall in near-lockstep with accuracy, from 100% down to about 15%; the self-check bars stay pinned near 40%, blind to difficulty.
Data table
| condition | actual accuracy | IMPLICIT: mean P(answer) | EXPLICIT: mean P(True) self-verification |
|---|---|---|---|
| familiar_execute | 100% | 100% | 42% |
| familiar_induce | 40% | 44% | 35% |
| reversal_induce | 19% | 29% | 38% |
| novel_induce | 10% | 15% | 38% |
Numbers from
Technical framing
Does the model know when it will fail? YES implicitly (answer-token probability tracks accuracy), NO explicitly (self-verification is flat) — Uses C39's VERIFIED competence boundary as crisp ground truth (rare in calibration research). Task: 'advance k in a cyclic order', format-equalized, across a competence spectrum. Two non-degenerate logit confidence signals (verbalized 0-100 is a useless constant 100): IMPLICIT P(answer) = the model's own probability on the digit it emits (softmax over the 10 digit tokens, one forward pass); EXPLICIT P(True) = Kadavath-style self-verification ('is your answer correct? A/B'). RESULT: implicit P(answer) tracks accuracy almost perfectly (1.00/0.44/0.29/0.15 vs acc 1.00/0.40/0.19/0.10), while explicit P(True) is FLAT (~0.4) and even underconfident on the perfect execute cell. The model's uncertainty lives in its output distribution, not its self-report.
How to read
Within one medium-difficulty task, five signals ranked by how reliably each separates items the model got right from ones it got wrong. Higher is better; 50 out of 100 is a coin flip. Includes the answer-odds, their margin and spread, the self-check, and prompt features.
Takeaway → Answer-odds tower near 95; prompt features reach only 61 and the spoken self-check sits at 46 — a coin flip. Real self-knowledge lives in the odds, not the words.
Data table
| condition | AUROC predicting per-item correctness (within familiar_induce, acc 0.40) |
|---|---|
| P(answer) implicit | 0.95 |
| margin | 0.9 |
| -entropy | 0.9 |
| P(True) explicit | 0.46 |
| surface baseline | 0.61 |
Numbers from
Technical framing
Item-level self-knowledge WITHIN the surface-matched familiar-induce cell: implicit P(answer) beats surface features and explicit self-report — The clean self-knowledge test (review-mandated): WITHIN the surface-matched intermediate cell (familiar_induce, acc 0.40, where both correct and incorrect items exist), can a signal predict WHICH items the model gets right? The implicit P(answer) achieves AUROC 0.95 (95% CI 0.90-0.99) -- it knows its own per-item correctness. It CRUSHES the external surface-feature baseline (0.61 -- a logistic classifier on prompt features like k, gap-to-seen, so this is genuine self-knowledge, not surface-reading) and the explicit P(True) self-verification (0.46 = chance). So the model has real item-level self-knowledge in its logits but cannot verbalize or self-verify it. Deployable: selective prediction by low P(answer) lifts accuracy-on-attempted from 0.23 to ~1.0. AUROC 0.5 = chance.
In the author’s words from the Overview · “Results”
Implicit P(answer) tracks accuracy (1.00/0.44/0.29/0.15 vs acc 1.00/0.40/0.19/0.10); explicit P(True) flat (~0.4). Within familiar_induce: P(answer) AUROC 0.95 (CI 0.90-0.99) >> surface 0.61 >> P(True) 0.46 (chance). Selective prediction lifts accuracy-on-attempted 0.23 -> ~1.0. See reports/report.md, analysis/metacognitive_boundary.png.
Overview
Research Program
- Program:
benchmark_generalization(first metacognition/calibration claim) - Question: the arc pinned a VERIFIED competence boundary (C39). Does the model's own confidence/uncertainty track it -- does it know when it is guessing?
Setup
- Format-equalized single-value task ('advance k in a cyclic order') across a verified competence spectrum. Two non-degenerate logit confidence signals (verbalized 0-100 is degenerate = constant 100): IMPLICIT P(answer) = softmax over the 10 digit tokens at 'Answer: ' (one forward pass); EXPLICIT P(True) = Kadavath self-verification. Clean test = WITHIN-condition item-level AUROC in the surface-matched familiar_induce cell vs an external surface-feature baseline.
Run
python scripts/eval_metacog.py --n 150 then python scripts/analyze.py.
Results
Implicit P(answer) tracks accuracy (1.00/0.44/0.29/0.15 vs acc 1.00/0.40/0.19/0.10); explicit P(True) flat (~0.4). Within familiar_induce: P(answer) AUROC 0.95 (CI 0.90-0.99) >> surface 0.61 >> P(True) 0.46 (chance). Selective prediction lifts accuracy-on-attempted 0.23 -> ~1.0. See reports/report.md, analysis/metacognitive_boundary.png.
Interpretation
The model knows when it will fail -- but only in its OUTPUT DISTRIBUTION, not in anything it can SAY. Deployable: read answer-token probability for a confidence/abstain signal; never trust explicit self-assessment.
Knowledgebase Update
- Claim ledger: C40
Artifacts
scripts/succ_family.py(format-equalized substrate),scripts/eval_metacog.py(logit confidence signals),scripts/analyze.pyruns/metacog_records.json,runs/verdict.json,analysis/metacognitive_boundary.png,reports/{report,design_review}.md
Report
Rendered from reports/report.md
Motivation
The arc pinned a verified competence boundary (C39): on "advance k in a cyclic order" the model executes near-perfectly, induces a familiar rule partly (~0.45), and induces a novel rule at chance (~0.10). Unlike normal calibration work, we know exactly which tasks the model gets right and why. So: does the model's own confidence/uncertainty track that boundary — does it know when it's guessing?
Method (review-hardened)
Format-equalized single-value task (every condition shows an order block, so block-presence isn't a cue). Conditions: familiar_execute (anchor ~1.0), familiar_induce (headline — intermediate acc, surface-matched), reversal_induce (intended dissociation), novel_induce (chance). Verbalized 0–100 confidence is a degenerate constant 100, so we use two non-degenerate logit signals:
- Implicit — P(answer): the model's probability on the digit it emits (softmax over the 10 digit tokens at the
Answer:position, one forward pass) + entropy + top-2 margin. - Explicit — P(True): Kadavath-style self-verification ("is your answer correct? A/B"), read P(A).
The clean self-knowledge test is within-condition item-level AUROC in familiar_induce (surface matched, both classes present), compared against an external surface-feature baseline (logistic regression on {k, gap-to-seen, n-distinct-seen, query}) — a signal is self-knowledge only if it beats surface.
Results (n=150/condition)
Condition-level calibration:
| condition | acc | mean P(answer) | mean P(True) |
|---|---|---|---|
| familiar_execute | 1.00 | 1.00 | 0.42 |
| familiar_induce | 0.40 | 0.44 | 0.35 |
| reversal_induce | 0.19 | 0.29 | 0.38 |
| novel_induce | 0.10 | 0.15 | 0.38 |
Implicit P(answer) tracks accuracy almost perfectly; explicit P(True) is flat (~0.4) and even underconfident on the perfect execute cell.
Headline — within familiar_induce (surface-matched, acc 0.40), AUROC predicting per-item correctness:
| signal | AUROC |
|---|---|
| P(answer) (implicit) | 0.95 (95% CI 0.90–0.99) |
| margin / −entropy | ~0.90 |
| external surface baseline | 0.61 |
| P(True) (explicit) | 0.46 (= chance) |
The implicit signal predicts which specific items the model gets right, far beyond surface features — genuine item-level self-knowledge. The explicit signal is at chance.
Deployable: selective prediction by low P(answer) lifts accuracy on attempted from 0.23 to ~1.0.
Mechanism: on novel_induce only 0.11 of wrong answers are the natural-successor intrusion → failures are high-entropy scatter (the model is uncertain there), not a confident consistent wrong-rule.
Conclusion
The model knows when it will fail — but only in its output distribution, not in anything it can say. Implicit metacognition (answer-token probability) is excellent and beats both surface and explicit self-report; explicit self-assessment (P(True), verbalized confidence) is broken. For deployment: read the answer-token probability as a confidence/abstain signal; never trust the model's explicit self-assessment. This is a latent capability unearthed (a usable self-knowledge signal exists in the fixed weights) with a sharp caveat on where to read it.
Honest caveats
- P(True) at chance may partly reflect elicitation, but the independently-degenerate verbalized-100 confirms explicit metacognition is broken regardless.
- reversal_induce turned out genuinely hard (0.19), not the intended "scrambled-looking-but-easy" dissociation, so the surface-vs-competence contrast rests on the external surface-baseline comparison (0.61 ≪ 0.95), not reversal.
- Single seed; no-think answer channel (think triggers code-mode — C39).
Artifact Manifest
See reports/artifact_manifest.yaml.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Reproduce
Smoke test
python scripts/eval_metacog.py --n 20Full run
eval_metacog.py --n 150; analyze.pyRun steps are documented inside the experiment folder (README and scripts).