Qwen3.5-4B: Can Confidence Replace the Verifier in the Banking Flywheel?

Results at a glance 4

Overview

Research Program

• Program: posttraining_and_adaptation (× evidence_conditioned_selection)
• Program question: the banking arc (C11–C24) requires an executable verifier at every rung; the confidence arc (C40–C46) built a calibrated verification-free judge. Can the second power the first — extending self-training to domains with no interpreter?
• Prior anchors: C18 (qwen35_4b_coverage_banking — the harness and effect this replicates), C46 (qwen35_4b_code_confidence + HumanEval — P(True) is the program-level confidence), C11 (banking law), C29 (read-only verifier 2AFC 0.81), C24 (banking gain is diversity-driven).

Question

Does banking the model's own high-P(True) solutions — with NO execution anywhere in the training pipeline — recover the capability gain of banking execution-verified solutions? And does the calibration signal itself SURVIVE being trained on (flywheel viability), or does self-training inflate P(True) / collapse the judge?

Hypothesis

C46 says top-P(True) selection approaches visible-test execution per-candidate, so a top-fraction filter over a large harvest should yield a training set pure enough (~0.8+) for C18-style banking, whose gain is diversity- not purity-driven (C24). Falsifiers, each a distinct autopsy: (a) banking is less noise-tolerant than selection — confident-but-WRONG pairs teach the confidently-wrong modes; (b) the filter biases toward easy tasks, collapsing the diversity that drives banking; (c) training on own confident outputs inflates P(True) and destroys calibration (feedback collapse — kills round 2 even if round 1 works).

Setup

• Model: Qwen3.5-4B, QLoRA r32/α64 (C18-identical: epochs 3, lr 2e-4, no-think prompt→code pairs).
• Dataset/task source: contamination-free procedural list-DSL identification tasks (families.py), 90 train tasks at depths 1/2/3 (C18 schedule), K=40 think-mode samples each.
• Train/eval split: frozen held-out eval set (eval_ladder.py), behavioral func-sig + op-composition dedup vs ALL train tasks, 0 leakage, paired across arms.
• Baseline: base model (no training) and rand (same-size, draw-frequency-weighted sample of the same pool — what "bank with no filter" actually means; uniform-over-unique would deflate the floor).
• Arms (all filter the SAME candidate pool; matched size is enforced HARD = matched optimizer steps; identical training recipe; the ONLY variable is the keep-test): exec (execution-verified, C18-identical — the ceiling), conf_strat (PRIMARY: depth-stratified top think-P(True), per-depth quotas ∝ pool judge-score MASS — attempt-2 pivot: candidate-count quotas provably allocate slots where wrong candidates explode; verifier-free, isolates the keep-test from cross-depth slot allocation), conf_global (ablation: naive global top-P(True), the fully-deployable policy; its expected depth-collapse is reported, not conflated), rand.
• Gate (pre-registered, run.py hard-stops otherwise): pool P(True)-vs-full_pass AUROC ≥ 0.65 AND conf_strat−rand purity gap ≥ 0.10 AND matched n ≥ 60. Below the gate the finding is "the judge does not transfer to this substrate" and training is not burned. (The gate reads oracle purity — an experiment-level run/stop decision, documented like the matched-n scalar.)
• Primary metric & decision rule (pre-registered): per cell (mode × metric × depth, incl. pooled) where exec−base ≥ 0.10, classify by paired-bootstrap CIs into a TRICHOTOMY — conf~rand (conf_strat-vs-rand CI ≤ 0: filter adds nothing), intermediate (beats rand, below exec), conf~exec (beats rand, CI vs exec includes 0: verifier replaceable). The joint-bootstrap recovery ratio (conf−base)/(exec−base) is reported with CI as a magnitude, not a cutoff (n=25/depth cannot resolve 0.8 vs 1.0).
• Secondary: calibration survival — (a) fixed judge set (base's no-think eval-task samples; every model judges the SAME candidates → judge-change isolated), inflation headline = P(True)-on-INCORRECT drift (overall mean drift confounds inflation with real ability); (b) self-distribution pass (each model judges its OWN think-mode eval-task candidates — the actual round-2 flywheel number).
• Oracle-only metrics: full_pass grades all evals, builds the exec arm, and gates the run; in the conf arms' selection it never appears (post-hoc purity reporting only).
• Hidden-label boundary: conf-arm selection reads nothing but the model's own P(True) logit (no-think A/B judge, P(A) after "Answer: ") + generator metadata (task depth, draw frequency for tiebreaks). Matching size to the exec arm leaks one scalar (documented design constant).
• Known limits (pre-registered): single LoRA seed per arm (~30 optimizer steps — arm deltas include seed noise); no depth-4 cell (C18 comparability cell dropped for time); conf_global gets no think-mode eval (ablation, nothink only).

Run

Smoke (~10 min end-to-end, all five stages):

python scripts/run.py --smoke

Full (~12 h on the RTX 4090: harvest ~2 h, pool think-judge ~50 min, 4 trains ~12 min, 9 pipeline evals ~5 h + 1 held-back conf_global think eval post-pipeline, 10 calib passes ~4 h — the think-judge and self-distribution passes dominate):

python scripts/run.py            # idempotent; safe to re-run after interruption
python scripts/analyze.py

Results

Full narrative in reports/report.md; stats in runs/verdict.json; saga in experiment_log.md.

1. Gate stop first (kept): the NO-think P(True) judge is at chance within-task on this substrate (0.471; pooled 0.749 = task difficulty). C46's within-problem law is substrate-scoped: it holds where correctness is semantically readable, not where it must be computed.
2. Serial compute rescues the judge: CoT judging (judge_think, budget 512, 99% forced-close) lifts within-task AUROC to 0.845 (pooled 0.961), P(True) correct/incorrect 0.61/0.08 full-pool — C44's law governs the judge seat. The pivot re-ranked the conf arms on think-P(True); gate passed (purity gap 0.400).
3. Trichotomy = conf~rand. No pre-registered cell cleared exec−base ≥ 0.10. In the post-hoc cell where exec moved (d2 think greedy 0.08 → 0.24, CI [+0.04,+0.32], p=0.011), conf_strat +0.04 = rand +0.04 exactly, despite ~15× purity (0.429 vs 0.029). Recovery 0.25, CI [−1,+1]. Banking is far less noise-tolerant than selection.
4. The exec ceiling is dose-limited: first matched-dose C18 re-run under the strict frozen paired eval → d2 think coverage flat (0.28 → 0.28). With the lineage audit (C18: p=0.082, 3/20 d2 eval tasks leaked under the canonical probe-set func-sig; C22/C23 replications at 3.5× dose; C24 dose law), C18's headline was a low-dose overestimate, not a false effect.
5. Calibration survives banking as a RANKER: fixed-set think-judge within-AUROC 0.872 → 0.873 (exec) / 0.883 (conf_strat). But P(True) on own INCORRECT candidates doubles on the self-distribution (0.091 → 0.204). Rank filters survive round 2; fixed thresholds silently degrade. Own pass-rate 0.120 → 0.093 (no capability gain, consistent with 3).
6. No-think SFT reallocates the no-think proposal prior; CoT shields think: d1 no-think cov@16 collapses 0.72 → 0.32–0.40 for exec/conf_global/rand (conserved correct mass crammed onto the banked op-family; dedup-family tasks 9/10 → 0/10) but conf_strat's d3-heavy quota is sub-threshold (0.76); think-mode diversity untouched. Registered prediction for the held-back conf_global think eval: d1 cov_any ≈ 0.80–0.88, losses on dedup tasks — result: 0.88, uniq 8.6, all 3 residual losses in the dedup/unique family (3/3 confirmed).

Interpretation

The verifier is load-bearing at the TRAINING seat, not the JUDGING seat. As a selector (C41/C46), P(True) approaches execution; as a training filter it fails — not because the judge ranks badly (0.845) but because banking at this dose tolerates almost no confidently-wrong data, and the wrongs a confidence filter admits are precisely the plausible ones. The judge itself is robust to self-training (ranks survive, scores inflate on the self-distribution), so the flywheel's binding constraint is not feedback collapse but dose (C24): scale the harvest so the top-rank slice is pure AND diverse, or keep the executable verifier. Deployment rules: think-judge (never no-think) on computational substrates; filter by rank within depth strata (score-mass quotas), never by a fixed threshold.

Knowledgebase Update

• Program evidence updated: posttraining_and_adaptation × evidence_conditioned_selection
• Claim ledger updated: C47 (this experiment); C18 annotated with the eval audit + dose-law framing

Artifacts

• src/ — shared banking substrate (families list-DSL, gen_lib with the P(True) judge, code_env)
• scripts/ — harvest_pool.py (shared pool + oracle + P(True) annotations, AUROC canary), build_arms.py (matched-size exec/conf/rand sets), train_lora.py (C18-identical), eval_ladder.py (frozen paired eval, C24 version), calib_eval.py (fixed-judge-set calibration survival), run.py, analyze.py
• data/ — train_tasks, pool, per-arm training sets, frozen eval, judge set
• runs/ — adapters (moved external before commit), eval/calib JSONs
• reports/ — design_review.md, report.md, artifact_manifest.yaml

Report

Rendered from reports/report.md

Summary

Split verdict, and the split is the finding. At the capability seat: no — in the one eval cell where execution-verified banking produced a real gain (depth-2 think greedy 0.08 → 0.24, CI [+0.04, +0.32]; a post-hoc cell — no pre-registered cell cleared the bar), the pre-registered trichotomy RULE classifies confidence-filtered banking exactly on the no-filter floor (both +0.04): conf~rand, despite the confidence arm being ~15× purer than random (0.43 vs 0.03). At the judge seat: yes, with a caveat — think-judge discrimination fully survives being trained on its own approvals (fixed-set within-task AUROC 0.872 → 0.883), but P(True) scores inflate on the model's own post-banking distribution (P(True) on own incorrect candidates 0.091 → 0.204, doubled). A flywheel that filters by rank keeps its filter at round 2; one that filters by a fixed threshold silently degrades. Two upstream findings condition everything: the no-think P(True) judge is at chance within-task on this substrate (C46's law is substrate-scoped), and CoT judging rescues it (0.47 → 0.85 within-task) — verification of computational correctness is itself a serial computation (C44's law reaches the judge seat).

Research Program Fit

posttraining_and_adaptation × evidence_conditioned_selection: first experiment to connect the banking arc (C11–C24: self-training needs an executable verifier at every rung) to the confidence arc (C40–C46: a calibrated verification-free judge exists). The connection fails at the training seat and succeeds at the judging seat, which cleanly relocates where the verifier is load-bearing.

Method

Shared candidate pool: 90 procedural list-DSL identification tasks (depths 1/2/3 = 20/35/35), K=40 think-mode samples each → 2,130 unique candidates, oracle-graded (unique-candidate pass rate 0.033) and judged twice: no-think P(True) at harvest, think-P(True) (judge_pool_think.py, budget 512) after the gate-forced pivot. Four matched-size arms (n=70 pairs, hard trim; identical C18 QLoRA recipe r32/α64, 3 epochs, lr 2e-4, no-think prompt→code; the ONLY variable is the keep-test): exec (execution-verified, C18-identical keep rule — the ceiling, purity 1.0, depth mix 43/20/7), conf_strat (PRIMARY verifier-free arm: depth-stratified top think-P(True), per-depth quotas ∝ judge-score MASS, purity 0.429, mix 15/30/25), conf_global (ablation: naive global top-score, purity 0.614, mix 36/29/5), rand (draw-frequency-weighted floor, purity 0.029). Frozen paired eval (75 held-out tasks, behavioral func-sig + op-composition dedup, 0 leakage), 25/depth, K=16, greedy@1 + coverage. Pre-registered run/stop gate and trichotomy decision rule; paired-bootstrap CIs on every delta; calibration survival on a fixed judge set (777 identical base-generated candidates judged by every arm) plus a self-distribution pass (each model think-judges its OWN K=8 eval-task candidates — the round-2 flywheel condition).

Results

1. The no-think judge reads difficulty, not correctness, on this substrate (gate stop). Pooled no-think P(True) AUROC 0.749 but within-task 0.471 = chance (mean P(True) by depth 0.38/0.25/0.20 merely tracks solvability). C46's within-problem discrimination (0.74 MBPP, 0.78 HumanEval) does not transfer to a substrate where correctness is only computable (mentally execute the candidate on 8 examples), not semantically readable from a docstring.

2. Serial compute rescues the judge. judge_think (CoT before the same A/B verdict token) lifts within-task AUROC 0.471 → 0.845 on the full pool (pooled 0.961; diagnostic subset 0.49 → 0.802), P(True) correct/incorrect 0.61/0.08 full-pool (0.58/0.10 on the diagnostic subset) — with thinking budget-truncated at 512 tokens in 99% of judgments. C44's serial-compute law governs the judge seat: verifying a computational candidate IS a serial computation. This is what made the experiment runnable verifier-free: the conf arms rank on think-P(True); the gate passed (pool AUROC 0.961, purity gap 0.400).

3. MAIN — trichotomy verdict: conf~rand; purity is not the binding constraint. No pre-registered cell cleared the exec-gain ≥ 0.10 bar (see Result 4). The only significant pre-registered exec effect is depth-2 think cov_frac +0.035 (CI [+0.003, +0.072]) — concentration of sample mass on already-covered tasks, not expansion. In the post-hoc cell (labeled .POSTHOC in runs/verdict.json; added after seeing raw evals) exec moved hard — depth-2 think greedy 0.08 → 0.24 (CI [+0.04, +0.32], p=0.011):

Recovery ratio (conf−base)/(exec−base) = 0.25, joint-bootstrap CI [−1.0, +1.0] (uninformative at n=25). The ~15× purity advantage bought zero deployable gain — only the 100%-pure arm moved. Banking is far less noise-tolerant than selection (C41/C46's seat): 40% confidently-wrong pairs — plausible near-misses, by construction of the filter — erase what 100%-pure data teaches. Suggestive (multiple-comparison caveat): conf_strat is the only arm that did NOT collapse depth-1 no-think coverage (Result 6) and shows small d1 think cov_frac wins over both rand (+0.110, CI [+0.018, +0.207]) and exec (+0.090, CI [+0.005, +0.185]).

4. The exec ceiling itself is dose-limited: C18's headline was a low-dose overestimate. The exec arm is the first matched-dose re-run of the C18 recipe (20 d2 pairs / 8 distinct d2 functions vs C18's 24 / 9) under the strict C24-standard eval — and C18's depth-2 coverage expansion (0.15 → 0.45) is flat here (0.28 → 0.28). Lineage audit: (a) C18's headline was 3/20 vs 9/20 (Fisher p=0.082) on a leak-permissive eval — exact-ops dedup only; 3/20 of its d2 eval tasks were train-identical under the canonical probe-set func-sig (the same criterion the frozen eval dedups on), leak-excluded 2/17 vs 7/17 (p=0.118); (b) the expansion effect is real but was re-established at 3.5× the d2 dose — C22 0.225 → 0.425 (n=40), C23 0.175 → 0.50 (n=40, p=0.004, 83 pairs / 26 distinct functions); (c) under C24's diversity dose law, 8–9 distinct d2 functions sit at the bottom of the curve where a null at n=25 is expected. Consequence: at matched dose the exec ceiling is intrinsically small, so the trichotomy had little headroom — conf~rand holds in the one cell that cleared the bar, with n=25 power caveats.

5. Calibration survives banking as a RANKER; scores inflate on the self-distribution. Fixed judge set (777 identical candidates, 47 correct; think judge = the flywheel's filter):

Self-distribution (own think-mode K=8 eval-task candidates — the round-2 harvest):

Discrimination is untouched — even for the arm trained on its own judge approvals; the self-reinforcement collapse hypothesis (falsifier c) is dead. But P(True) on own incorrect candidates doubles (fixed-set drift only +0.043 → the inflation is distribution-specific).

6. Banking reallocates the no-think proposal distribution; CoT shields the think one. Banked arms collapse depth-1 no-think coverage@16 (base 0.72 → exec 0.36 / conf_global 0.32 / rand 0.40; conf_strat 0.76 does NOT collapse) while greedy stays flat. Per-task paired forensics: zero-sum mode reallocation, not capability loss — correct d1 samples per 400 are conserved across arms (126/115/130/106/108) while covered-task count crashes 18 → 9/8/10; the same correct mass gets crammed onto the banked program family. All four banks' d1 pairs come exclusively from the arithmetic-map/reorder/slice families and 0% from dedup_adjacent/unique_stable; exactly that family goes from 9/10 tasks covered (66 correct samples) at base to 0/10 (0) under exec/conf_global. Dose-dependent (exec banks 43 d1 pairs → collapse; conf_strat only 15 via its d3-heavy score-mass quotas → sub-threshold, 7/10 dedup tasks survive) and correctness-dependent (rand collapses by the other route: mass shifts toward wrong programs, diversity retained). Training pairs are NO-THINK, so the sharpening lives in the immediate-answer prior; CoT re-derives the answer and shields the think distribution (think-mode unique-programs unchanged, 7.7 → 7.8–9.0). Registered prediction (logged before running): the missing conf_global think eval should show d1 cov_any ≈ 0.80–0.88 with residual losses on dedup tasks, vs its 0.32 no-think collapse. Test result: 3/3 — d1 cov_any 0.88 (top of the predicted band), uniq 8.6 (predicted 7–9), and all three residual lost-vs-base tasks are exactly the dedup/unique family (unique_stable, dedup_adjacent, unique_stable). The out-of-sample test confirms the mechanism.

Controls

• rand is draw-frequency-weighted (what "bank with no filter" means); uniform-over-unique would deflate the floor. Matched size is a hard trim (C23 count-confound), matched optimizer steps.
• Score-mass quotas for conf_strat exist because candidate-count quotas provably allocate slots where wrong candidates explode (d2: 915 wrong / 20 correct; d3: 1066 / 7) — the attempt-2 lesson.
• The fixed judge set isolates judge-change from candidate-change (every arm judges the identical 777 candidates); the inflation headline is P(True)-on-INCORRECT drift, since overall mean drift confounds inflation with real ability gain.
• No-think judge secondary control: its within-task AUROC improves after banking (0.679 → 0.71–0.82: exec 0.815, rand 0.800, conf_strat 0.792, conf_global 0.713) in every arm including rand (97% wrong pairs) — a generic task-domain-SFT effect on the readout, not learning of correctness. (It also shows the no-think judge is not uniformly at chance: on the cruder no-think error distribution it is partially informative, 0.679; on the plausible think-harvest near-misses it is chance, 0.471.)

Oracle Versus Deployable Evidence

Oracle (full_pass) appears ONLY in: exec's keep-test (the ceiling arm by design), the experiment-level gate, and post-hoc purity/calibration REPORTING. The conf arms' selection reads nothing but the model's own think-P(True) logit + generator metadata (depth, draw frequency); matching size to exec leaks one scalar (documented design constant). The deployable recipe this experiment licenses: think-judge rank filtering (no oracle) — but for training data, its 0.43 purity was not enough to matter at this dose.

Interpretation

The verifier is load-bearing at the TRAINING seat, not the JUDGING seat. Confidence transfers across seats asymmetrically: as a selector (C41/C46) P(True) approaches execution; as a training filter it fails not because the judge ranks badly (0.845 within-task) but because banking tolerates almost no confidently-wrong data at this dose — and the wrongs a confidence filter admits are precisely the plausible ones. Meanwhile the judge itself is robust to self-training (ranks survive, scores inflate), so the flywheel's failure mode is not feedback collapse but simple dose: at 70 pairs the exec ceiling is already at the bottom of C24's diversity dose curve, and a 0.43-pure filter recovers none of it. Three laws sharpen: C46 is substrate-scoped (single-token readouts work where verification is shallow; computational correctness needs serial compute — C44 governs the judge seat too); C24's dose law now has a matched-dose null confirming its bottom end (and retro-explains C18 as a low-dose overestimate on a leak-permissive eval); and the banking arc gains a distribution rule — no-think SFT reallocates the no-think proposal prior toward banked families while CoT shields the think distribution, so flywheel harvests (think-mode) survive banking that would look catastrophic no-think.

Next Experiments

1. Scale the harvest, not the filter: K=40 → K=160+ on more tasks, then take the top-rank slice at conf_strat's quota — does purity×diversity at larger dose cross the banking threshold verifier-free? (The direct test of "dose is the binding constraint".)
2. Threshold-vs-rank flywheel round 2 in anger: run one actual round-2 harvest+train with rank filtering (predicted fine) vs fixed-threshold filtering (predicted purity decay via the +0.113 inflation).
3. Semantically-readable substrate (natural-language task descriptions, C37-style): does the NO-think judge's within-task discrimination return, making the cheap filter viable there?
4. Judge-budget dose-response: think-judge scores fell by depth (0.36/0.10/0.07 mean by depth) at budget 512, 99% forced-close — does budget 1024/2048 lift deep-candidate ranking?

Artifact Manifest

runs/verdict.json (full stats incl. .POSTHOC labels), runs/arms_summary.json (gate + arm composition), runs/judge_think_diag.json (Result 2 diagnostic), runs/calib*.json (Result 5: fixed / think / self-think), runs/eval_*.json (10 arm×mode evals, 25 tasks/depth), analysis/ verifier_free_banking.png, experiment_log.md (full saga incl. two failed attempts and the session crash), reports/design_review.md (pre-registration). Adapters (5 × 182 MB) moved out of the repo — external paths + regenerate commands in reports/artifact_manifest.yaml.

Experiment log 10

Figures 1

Data files 14

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

• previewruns/arms_summary.json1.5 kB
• previewruns/arms_summary_smoke.json1.1 kB
• previewruns/eval_base_nothink.json9.2 kB
• previewruns/eval_base_think.json9.1 kB
• previewruns/eval_conf_global_nothink.json9.7 kB
• previewruns/eval_conf_global_think.json9.6 kB
• previewruns/eval_conf_strat_nothink.json9.6 kB
• previewruns/eval_conf_strat_think.json9.5 kB
• previewruns/eval_exec_nothink.json9.2 kB
• previewruns/eval_exec_think.json9.1 kB
• previewruns/eval_rand_nothink.json9.2 kB
• previewruns/eval_rand_think.json9.1 kB
• previewruns/eval_smbase_nothink.json755 B
• previewruns/eval_smconf_nothink.json755 B

Reproduce

Smoke test

python scripts/run.py --smoke

Full run

python scripts/run.py && python scripts/eval_ladder.py --tag conf_global_think --K 16 --n-per-depth 25 --depths 1 2 3 --adapter runs/lora_conf_global --think && python scripts/analyze.py

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Cited by claims

• ◑ Promising C47 · Confidence cannot replace the execution verifier at the TRAINING seat (conf-filtered banking ~ random banking) -- but the judge itself SURVIVES self-training: rank-filter flywheels stay viable while fixed-threshold filters silently degrade

Related experiments

Nearest by shared name, tags, and programs.

• ~2026-06-25 Qwen3.5-4B Oracle-Distilled Semantic Verifier
• 2026-06-22 Qwen Verifier-Guided Slot Repair
• imported 2026-07-12 Qwen Learned Repair Verifier
• 2026-06-23 Qwen Prefix-State Process Verifier
• 2026-06-23 Qwen Mixed-Domain Trace Verifier
• 2026-07-12 Verifier-conditioned recovery banking curriculum