Evidence-Conditioned Selection

What we have learned

Seed Experiments

• qwen35_4b_retrieval_adapt_verify_scale
• qwen35_4b_foofah_selective_program_fallback
• qwen35_4b_foofah_program_ensemble_consensus
• qwen35_4b_independent_retrieval_consensus

Key Result

• qwen35_4b_partial_structure_search adds a hard negative boundary to confidence-based control. On 7,200 unfinished type-prefix children, thinking P(viable) had pooled AUROC 0.557 but task-macro AUROC 0.506 and recall@4 0.251; task-shuffled visible evidence was no worse. No-think was modestly stronger (0.556 AUROC, 0.303 recall) but still below the frozen actionability threshold. Thus C10/C46-style confidence selection does not transfer automatically from completed candidates to existential partial reachability. Search controllers must be evaluated within task and at the actual retained beam, and should expose residual evidence rather than only symbolic prefix names.

• qwen35_4b_generator_verifier_gap (claim C10): the C2 wall is plumbing, not capability. A frozen 4B's black-box self-verification is weak/yes-biased with no-think (AUROC 0.77) but strong with thinking (AUROC 0.93); its own zero-training, deployable thinking-verifier selects best-of-8 to close 75% of the pass@1(0.771)→oracle(0.890) gap (no-think 24%). Foreign-solution reject rate 1.00. So the model CAN tell its own good solutions from bad — once it thinks — and selection has real headroom.

• qwen35_4b_verifier_selector_showdown (matched-cost follow-up): on one k=8 pool, the thinking-verifier is Pareto-dominated when a cheap visible test exists — standalone 0.860 ≈ visible-only 0.850 at ~5× cost; the deployable sweet spot is visible + free no-think verifier (0.870), 83% of the pass@1→oracle gap. Expensive thinking-verification only earns its cost in verifier-only settings. So the C2 wall is fixable with cheap plumbing.

• qwen35_4b_code_confidence (claim C46, MBPP leg): in the verifier-free code regime, single-token P(True) is the selector to beat. MBPP P(True) selection 0.762 beats public-output majority 0.721 and random 0.696. When a visible test exists, execute it first (MBPP visible-test execution 0.816), then use confidence for abstain/route or no-test settings.

• qwen35_4b_humaneval_code_confidence (claim C46, HumanEval replication): in the strict no-public-probe setting, P(True) selection 0.835 beats mean-logprob 0.787 and random 0.766, with oracle pass@8 0.872. This is the clean no-verifier selection replication.

• qwen35_4b_answer_potential_trace_sft (claim C51): same-model canonical-answer likelihood after a sampled thought is real but not actionable under the tested forced-close protocol. Within-task AUROC was 0.617 (gate 0.65), and top-one success beat random/shortest by +0.073/+0.058 but missed the frozen +0.10 bar. Real thoughts beat token-shuffled and foreign controls and format ranks were stable, yet 99.37% of thoughts hit the cap and autonomous answers parsed only 13.2%. G0 failed and correctly stopped before SFT.

• qwen35_4b_balanced_core_answer_potential_sft is an operational continuation, not yet a capability result. Removing the 512-token cap yielded 22,681 naturally closed candidates across 360 balanced tasks, and official answer-potential selections cover all 360 tasks with thoughts as long as 14,240 tokens. But the nominal success-RFT control has only 97 unique successful traces from 58 tasks in four of nine cells; matching 720 rows repeats each source seven or eight times. The six-arm two-epoch dose is 34,446,994 forward tokens, so training is paused. Selector studies must audit control support and token dose before treating equal row counts as a matched comparison.

• qwen35_4b_same_prefix_advantage_routing adds a training-time selection boundary. Three policies' absolute continuation estimates replicated well across disjoint four-branch halves (r=0.79--0.86), but conditioning on the statewise maximum was unstable: only 6/26 block-1 quick selections remained quick winners on audit, and its apparent +0.319 margin over the student became -0.019. Thresholds of +0.10 and +0.25 did not repair the sign. Selection quality must be measured after conditioning, on disjoint outcomes, rather than inferred from component-score reliability.

• qwen35_4b_counterfactual_order_support_selector adds a forward-counterfactual boundary. The label-free mean per-alias ordered- minus-exact-shuffle probability reached 43/113 (0.381), beating first trace 31/113 and majority 33/113 with positive paired lower bounds. But it was only +2 tasks over minimum entropy and +3 over max confidence, with lower bounds -0.035/-0.027, and an oracle-balanced task-mismatched shuffle reached 44/113. Thus the replicated coherent-content group effect contains weak selection information but raw subtraction is not a robust, task-specific selector. Confirmation and the K=3-versus-K=6 matched-compute successor stayed sealed.

Current Read

The biggest strategic gap is selection under deployable evidence — and C10/C46 say that gap is fixable with cheap plumbing: execute visible evidence when it exists, then use a free no-think P(True) readout for verifier-free selection, abstention, or routing. Thinking-verification is reserved for verifier-only settings where its added cost beats the no-think readout. Future selection work should (a) benchmark against the thinking-verifier before building trained selectors, (b) treat native thinking as a verification lever (not only generation), and (c) still report oracle coverage only as a diagnostic with the deployable decision rule as the main object. Top follow-up: wire the thinking-verifier into a controller (vs/with the visible test) and measure the deployable accuracy-vs-token Pareto.

The partial-structure result narrows that optimism: confidence is useful when correctness is readable from a completed candidate, but an existential unfinished-state judgment can collapse to task difficulty. Any controller follow-up must report task-macro discrimination, sibling recall at the deployed beam, a task- shuffled evidence canary, and prefill-inclusive compute—not just pooled AUROC.

C51 adds a second boundary: a score may contain trace-specific information and still condition on an unreachable deployment state. Oracle-side trace selectors must predict fresh autonomous outcomes within task, clear a practical top-choice effect-size gate, and include termination/parseability in validation. Do not scale a dense score because its corruption controls pass; first prove the scored seam is deployable.

The uncapped balanced continuation clears C51's termination pathology but exposes two pre-training gates. Outcome-rejection controls need per-cell task support, not global oversampling, and full-thought SFT must be budgeted in forward tokens rather than examples. Its selected datasets are banked, but no selector is promoted until a prospective, affordable training comparison produces fresh held-out behavior.

The same-prefix result adds winner conditioning to that checklist. Reliable component scores do not imply a reliable argmax label when differences are small and the selected tail is reused as the curriculum. For policy routing, estimate direct advantages with cross-fitting, expose abstention and per-route support, and retain independent block signs; a positive pooled router average cannot certify every named teacher.

The order-support negative adds a group-to-instance warning. A perturbation can causally improve correctness on average and its signed probability vector can beat hard voting, yet still fail against cheap confidence/entropy or a relevance control. Before spending matched compute, require the counterfactual readout to add task-specific information beyond ordinary probability geometry.

The subsequent J-branch mechanics negative reinforces the proposal/value split: moving before commit is not enough when the intervention cannot write its own supplied hypothesis. All numeric controls passed, but additive J target selection was exactly chance. Proposal-shifting systems need a label-free write/coverage gate before selector or matched-compute evaluation.

Scorecard

• Program: charter
• Current read: confidence can rank completed candidates, but value signals do not automatically become controllers. The coherent-order delta beat majority but not confidence/entropy; additive-J branching failed at chance; a late semantic anchor wrote names without a valid consequence interface; and early concrete text routed direct operations but failed before proposal selection or any matched-sampling comparison.
• Best next experiment: close the exact-pool visible-selector gap; for training-time policy routing, fit direct cross-fitted advantages and confirm a frozen rule on a third block rather than retrying statewise argmax.
• Strong anchors: qwen35_4b_partial_structure_search, qwen35_4b_generator_verifier_gap, qwen35_4b_code_confidence, qwen35_4b_answer_potential_trace_sft, qwen35_4b_same_prefix_advantage_routing.
• Avoid repeating: pooled-AUROC confidence claims, type-only partial judges, answer-only potential over cap-bound traces, posthoc winner margins, raw ordered-minus-shuffle commit-logit tuning, or gains that hide abstention/commit-rate changes.
• Evidence that advances the program: deployable selection gains under family-held-out candidate pools and adversarial visible examples.

Charter

Backlog

Experiments 107

• 2026-07-27 → 28 Self-Written Verifier Fidelity

  Fill this after the run. Separate deployable evidence from oracle/hidden evaluation.

• 2026-07-19 Qwen35 4B Agentic RLVR Feasibility

  Yes to the machinery: with three fixes (force eager mode so the hybrid architecture doesn't hang, load the model in bf16 not fp32, and split GPU memory 55% to the inference engine with sleep-mode) the loop CLOSES - a mov

• 2026-07-18 Qwen35 4B Self-Repair Install

  The first bet that moved the needle at all — gently. We taught the 4B to debug by training on 504 examples of [buggy code + the real test-failure message] -> [diagnosis + fix], all self-generated by injecting bugs into c

• 2026-07-16 Repair-Verifier Signal Probe

  The gate said no, cleanly. Handed two candidate repairs and the full failure evidence — a task solvable by mentally running each candidate through both trials — the model picked the working one 51.5 percent of the time w

• 2026-07-15 Rank-Capacity Vehicle Cell

  The trial could not answer the capacity question — by design. Its built-in guard required the known nine-point forgetting case to reproduce before trusting any comparison, and on this fresh screen that case measured only

• 2026-07-15 Medium Budget-Probe Measurement

  The probe never got to ask its question. The benchmark's own referee enforces a wall-clock budget per model, and the untouched base — deliberately sent first because this risk was written into the plan — blew past it wit

• 2026-07-15 Interleaved-Replay Dose with Medium Pilot

  The review round did not prevent forgetting: the dosed model lost nine to ten retained answers against both comparisons — almost exactly the cost of dosing directly — so the theory drawn from comparing old receipts is re

• 2026-07-15 Dose-Diversity Mechanism Cell

  Variety does not protect memory: the larger varied practice set also cost nine retained answers, the known ten-point case reproduced exactly, and even pure review cost five on this fresh screen — so the one past case wit

• 2026-07-15 Axis Stack Re-adjudication with Medium Pilot

  Judged fairly on brand-new tasks — with the ceiling quirk handled — the stacked model won the overall skill test for the third straight time (22 vs 15 and 18 of 40) with the cleanest finishing behavior again. But it won

• 2026-07-15 Axis Corpus V2 with Staged Repair

  Even with lessons rebuilt from the autopsy — walking through the search step by step instead of asserting the answer — the repair skills did not stick (a tie and a loss against the comparisons), so the preregistered kill

• 2026-07-14 Policy-Supported Successful-Sibling Universal Curriculum

  This version cannot run its second step. Across 624 first attempts it found 227 failures, but counting and routing had none and selection had only two; the plan required four failures in every skill. It therefore stopped

• 2026-07-14 Natural-Language State-Table Universal Curriculum

  Not known yet. CPU construction produced 80 truth-checked lessons and two 320-row streams with exactly 286,814 tokens each. All 48 smoke tests pass, but no model has been trained or evaluated.

• 2026-07-14 Search-Scaffold Universal Curriculum

  No. On 26 fresh paired cases, the staged-search model tied replay at 16 correct and trailed its parent at 18. All three parsed 23 answers and hit the limit three times; the candidate solved none of four execute/induct ca

• 2026-07-14 Failure-Selected Counterfactual Restart Curriculum

  The data-selection stage passed: from 624 fresh attempts, the frozen rule selected 52 clean restarts, exactly four for each of 13 skills. Forty correct accuracy or cap failures and twelve correct-but-too-long cases are n

• 2026-07-13 Validation-policy counterexample curriculum

  The test could not answer the question. Before the newly trained model was allowed to run, the starting model and the equal-size old-lesson control each fixed all 48 practice cases. The required gains were therefore impo

• 2026-07-13 Low-Density Token-Matched Universal Curriculum

  No. Neither 40 nor 80 abstract lessons passed the fresh local screen. The stronger 80-lesson stream matched the inherited anchor at 54% accuracy and parsed 62% of answers, but still hit the answer cap 10 times; broad eva

• 2026-07-13 Qwen3.5-4B: Installing Universal Features via Designed Synthetic Curricula

  Not with either tested training geometry. Continuing from the strong policy raised fresh synthetic accuracy from 50% to 69% and beat the untrained base overall, but it scored 31% aggregate versus 45% for the strong start

• 2026-07-13 State-Formation Capacity Adjudication

  Not run yet. The reviewed test starts with three compact-update training runs. If any required state check misses, six matched controls become mandatory; the final three runs open only if the full-size version also misse

• 2026-07-13 Semantic-policy headroom tournament

  No rule produced a stable training target after a failed test: every one of 72 cases reached a correct patch. But before seeing test output, the model got zero of 54 ambiguous first patches fully right and never opened t

• 2026-07-13 Qwen3.5-4B Materialized Residual Sibling Search Fresh Replication

  The generation test cannot answer the question because every reasoning trace reached its length cap and the materialized arm parsed only 12 of 52 outputs, with zero successes. The separate one-token ranking test is a cle

• 2026-07-13 Qwen3.5-4B Materialized Residual Sibling Search

  Not yet. The scientific design and every model-free construction check pass, but the model has not run. The frozen test contains 264 fresh functions, and all 38,596 planned prompt renderings fit their assigned context li

• 2026-07-13 Qwen3.5-4B Early Text Hypothesis Forking

  The experiment design and model-free checks now pass, but the model has not run. Review expanded the first draft from twelve operation names to twenty-four fully specified operations, added two fair late-hint comparisons

• 2026-07-13 Qwen3.5-4B Counterfactual Order-Support Selector

  Not reliably. The meaningful-order rule got 43 of 113 puzzles right, clearly better than first attempt or majority vote. But it was only two puzzles ahead of simply choosing the most decisive attempt, and a deliberately

• 2026-07-13 Qwen3.5-4B: The Compute-Optimal Confidence Policy (select + abstain + escalate)

  Partly. Picking the answer with the highest single-token P(True) confidence is the best grader-free selector at every compute level, and abstaining below a confidence threshold trades coverage for accuracy cleanly — both

• 2026-07-12 Transaction-invariant recovery curriculum

  Partly. Focused training lifted practice repairs from 52% to 82% and preserved the test-and-revise loop. But on different transaction APIs it reached 72% versus 70% before training, far below the required ten-point gain.

• 2026-07-12 Verifier-conditioned recovery banking curriculum

  Yes — replaying real failures taught recovery, lifting success from about half of cases to more than nine in ten. But the best version also added a little "explain your fix" text, and that supposedly tiny slice actually

• 2026-07-12 Qwen3.5-4B Same-Prefix Advantage Routing

  Only when it replicates, and here one helper didn't. The test actually ran: the deep specialist's local edge held up on a fresh, separate batch, but the quick specialist looked strong on one batch of stuck points and the

• 2026-07-12 Repository search-compress-bank coding curriculum

  No. It became flawless on the six codebase types it trained on, solving every task in exactly four clean steps. But on four never-seen codebase types its success collapsed from 68% to 35% — worse than just running the un

• 2026-07-12 Public-verifier recovery branch tournament

  No. On four new problem types, both policies repaired 74%, and their combined best reached only 75%—exactly tied with two action-only attempts. The selector was correctly stopped before scoring.

• 2026-07-12 Qwen3.5-4B Pareto Policy Integration

  No. Retested on fresh, uncontaminated tasks, the version built for quick work actually lost at quick work by about two points, while the version built for long work won at both quick AND long tasks. One version quietly d

• 2026-07-12 Qwen3.5-4B Balanced-Core Answer-Potential SFT

  Not run yet — the reasoning bank is fully built (360 tasks, six competing selection rules staged) but no model has been trained or scored. The built test will fine-tune six copies, each fed reasoning chosen a different w

• 2026-07-11 Qwen3.5-4B Specialist Policy Integration

  No — we never found out. Each of four required skills had to gain ten points before merging, but scores top out at 100. One tool-use skill already sat at 99.4%, so its target of 109.4% was impossible. The experiment stop

• 2026-07-11 Interactive policy curriculum: oracle DAgger to execution-reward RL

  No — it backfired badly. Success on the practice tasks fell from about 60% to 35%, and on tasks never touched in training from about 69% to 35%. It kept editing fluently but stopped running its checks: the verify-and-com

• 2026-07-10 Qwen3.5-4B Verified Macro Invention Long-Context Rerun

  No. The model was not short on thinking space, it was stuck repeating itself, and more space only bought longer loops. Handed the blueprint, all 16 tasks finished cleanly. Asked to invent the program from examples alone,

• 2026-07-10 Qwen3.5-4B verified-macro exact CUDA-graph vLLM rerun

  No, it just loops harder. At both budgets, all 48 reasoning attempts ran out of thinking room and had to be force-stopped, none finishing on its own. Widening the budget from 49,152 to 61,440 tokens pushed the share stuc

• 2026-07-10 Qwen3.5-4B verified-macro capacity-fit vLLM rerun

  No. Two hidden limits bit first. Configuring the server for 64 simultaneous long prompts demanded about 2.4 million tokens of fast memory from a pool holding roughly 1 million, forcing constant re-reading. Cutting to 19

• 2026-07-10 Qwen3.5-4B Partial-Structure Recognition-Guided Search

  No. Shown a half-finished program skeleton, the four-billion-parameter model's guess at whether it could still be completed was barely above a coin flip — about 51% correct, where 50% is pure chance — and letting it reas

• 2026-07-10 Qwen3.5-4B Long-Horizon Answer-Potential SFT

  Barely. That confidence signal does sort stronger reasoning above weaker about 60% of the time — better than a coin flip's 50%, and enough to lift a picked chain's success from 16% to 22%. But two dumb rules beat it: jus

• 2026-07-10 Qwen3.5-4B Answer-Potential Trace SFT

  No. The score carried genuine signal — it tracked what the reasoning actually said, not its length or wording — but its top pick raised fresh-answer success only from about 13% to 20%, short of the useful margin needed t

• 2026-07-09 → 10 Gauntlet round 1: breadth-first agentic expert iteration

  Mostly the second. On a blind benchmark the model scored about 14 percent, with six task types near zero, but it had usually reasoned correctly. It simply hit its thinking limit, restarted explaining instead of writing t

• 2026-07-09 Qwen3.5-4B Verified Macro Invention

  No. Even handed the finished plan and asked only to re-express it — no problem to actually solve — the model got it exactly right just one time in four, short of the three-in-four bar. Every output looked flawless: valid

• 2026-07-08 → 09 Does the installable hypothesize-and-verify skill move the structure wall?

  No. Fine-tuning the model on 1,476 worked guess-and-check traces doubled its success at the two-step depth it practiced on (lists jumped from 37% to 70%), yet did nothing one step deeper: three-step success stayed near 5

• 2026-07-08 Qwen3.5-4B: Does Code Confidence Replicate on HumanEval?

  Ask it directly. Averaging the model's confidence across every code token barely helps, nudging correct picks from 77% (blind guessing) to just 79%. But handing the model its own code and reading its confidence in one ye

• 2026-07-07 → 08 Qwen3.5-4B: Can Confidence Replace the Verifier in the Banking Flywheel?

  No. When training on answers checked by actually running the code lifted single-shot accuracy from 8% to 24%, confidence-filtered data — fifteen times purer than a random grab of the model's own outputs — landed right on

• 2026-07-07 Qwen3.5-4B: Does the Confidence Toolkit Survive on Real Code?

  Yes, but not the obvious way. Averaging the model's certainty across every token of a program barely beats a plain majority vote among the tries. The real winner: make the model write the code, then answer one yes/no que

• 2026-07-04 Qwen3.5-4B Tool-Seeded Banking: does tool-search + banking cross the depth-3 wall?

  Partly. A brute-force search found working three-step programs the model never produces itself, and retraining on them lifted three-step success from a hard zero to 5 of 40 fresh, never-seen tasks when it can reason acro

• 2026-07-04 Qwen3.5-4B Depth-3 Dose-Response: data-limited or representational cap?

  Just starved for examples. Feeding a fixed 4-billion-parameter model more search-found solutions lifted its solve rate on fresh three-layer puzzles from 0% to 38% when given sixteen tries — climbing steadily at every dos

• 2026-07-03 Qwen3.5-4B Wall Climbing: does banking shallow composition unlock deeper coverage?

  No. Fine-tuning the model on the two-step solutions it could already produce tripled its two-step success on fresh tasks, from 12% to 36%. But its three-step success stayed at exactly zero, unchanged from before: both mo

• 2026-07-03 Qwen3.5-4B Coverage vs Selection: anatomy of the generation wall

  It never generates it. Whenever a correct program shows up among 32 tries, running each candidate against eight known examples finds it every single time — the model judging its own work, and even a random pick among sur

• 2026-07-03 Qwen3.5-4B Coverage Banking: does banking shift the proposal distribution?

  Yes, but it depends on difficulty. On easy one-step problems it just pulls answers it already knew into its top guess (60% to 80%), gaining no new ground. On harder two-step problems something new happens: on brand-new t

• 2026-06-30 Qwen3.5-4B Verifier vs Visible Selector Showdown

  No. When you can run even a single example test on each candidate, that filter alone lifts the share of shipped programs that fully work from 77% to 85%. Adding a free, instant self-confidence rating reaches 87% — matchi

• 2026-06-30 Qwen3.5-4B Thinking-Budget Scaling

  Yes, but with a sweet spot. Turning on the model's built-in reasoning lifted the accuracy of the answer you would actually ship from 76% to 91% on basic Python tasks — a genuine capability gain, not just a wider pool of

• 2026-06-29 → 30 Qwen3.5-4B Thinking Separability Probe

  Yes, but not for the reason you would expect. From one snapshot of internal activity, whether the model's own code is correct is readable well above a coin-flip, about 64 to 76 percent of the time. Thinking first sharpen

• 2026-06-29 → 30 Qwen3.5-4B Generator-Verifier Gap

  Only after it thinks. Judging on sight, the model rubber-stamps 91% of its tries as correct while just 77% truly pass — barely a check, mostly agreeing with itself. Given room to reason first, it becomes a real critic: l

• 2026-06-28 Qwen Oracle-Distilled Acquisition Policy

  No. The trained picker does read real signal: it beats revealing nothing (50 to 57 percent of tasks fully solved) and crushes a version fed scrambled answers (27 percent). But it lost to simply grabbing a varied spread o

• 2026-06-28 Counterexample-Guided Ephemeral Program

  No. Answering each row directly solved three-quarters of tasks completely, while the best rule-program the method could pick solved only about four in ten — and a perfect picker that peeks at the answers did no better. T

• imported 2026-07-12 Qwen Register Trace Refiner

  Rarely. Even a flawless picker that always grabbed the correct edit reached only 37% on plainly worded problems and 7% on reworded ones, because the correct program usually isn't among the roughly 1,300 nearby edits at a

• imported 2026-07-12 Qwen Progressive Repair Compiler

  Partly. The correct fix sits in the candidate list almost nine times in ten, yet the judge finds it only about half the time, lifting exactly-correct programs from 30% to 49% against the 88% a flawless chooser would reac

• imported 2026-07-12 Qwen Learned Repair Verifier

  Partly. Without ever seeing the true answer, the judge lifted correct execution from 30% to 47%. But a checker allowed to peek at the answer key found a correct fix already sitting in the candidate pile 88% of the time,

• imported 2026-07-12 Qwen Candidate-Trace Verifier

  Yes. A small checker that reads each candidate's worked-out steps — never the true answer — lifted correctly-running programs from 30% to 54%, and to 56% when it cross-checks two wordings of one task. The twist: a picker

• imported 2026-07-12 Qwen 3.5 4B Verified Edit Closure

  Yes, on the hardest unseen tasks. Testing small edits of the model's own near-miss program and keeping whichever passes the example cases raised fully-correct answers from 39% to 52%, while re-sampling the model and re-r

• imported 2026-07-12 Qwen 3.5 4B Static Bridge Ceiling Breaker

  Partly. Folding in just 60 slightly-harder "bridge" examples, a quarter of the training budget, more than doubled success on deeper, never-seen programs, from 20% to 44% fully repaired, with no loss on familiar skills. B

• imported 2026-07-12 Qwen 3.5 4B GraphIR Self Repair

  No. The plain one-line formula fully solved 29% of brand-new tasks; the step-by-step diagram managed only 22%, and the fix-it pass recovered part of that gap to 24% — still behind. That pass genuinely works: on randomly

• 2026-06-27 Real Transform ABI Gate with Counterexamples

  It depends on how messy the data is. For clean, spreadsheet-style pipeline jobs the toolbox covered every one (100%) and held firm even against deliberately tricky examples. For irregular date, ID, and text cleanup, cove

• 2026-06-27 Qwen Verified Skill Memory RAG

  No. Handing the model a matched, verified example made it slightly worse, not better: every value came out correct on 47.5% of jobs, versus 50% when it worked through each value alone with no example. The lookup itself w

• 2026-06-27 Qwen Real Task ABI Coverage Gate

  It depends, and the split is sharp. A frozen kit of reusable office operations, with no training at all, assembled 84% of realistic tasks from stored parts alone, far above the 21% a bare kit managed, and it fully solved

• 2026-06-27 Pairwise Table Judge

  No. Letting the model run a knockout comparison among its own candidate tables did not help: fully-correct tables slipped from 50% (just trusting its first attempt) to 47.5%. On the hard cases where a correct table exist

• 2026-06-27 Counterexample-Guided Consistency Projection

  No. Distilling the model's messy guesses into one reliable formula solved only 20% of tasks completely, versus 50% when it simply transformed each row directly, a 30-point drop. For 29 of the 40 tasks no formula even fit

• 2026-06-27 Qwen3.5-4B Independent Code ABI Coverage Gate

  No, hardly any. The locked toolbox solved only about 14% of brand-new tasks, roughly 1 in 7, versus 37% on the familiar tasks it was shaped around. Reshuffling which tasks are unseen barely moves it, around 18%. And near

• 2026-06-27 Qwen3.5-4B Foofah Selective Program Fallback

  Trust the program the moment it reproduces the visible worked examples. Doing that lifted exact-match accuracy from 55% to 62% across 250 table tasks, rescuing 18 answers the direct route got wrong while losing none it g

• 2026-06-27 Qwen3.5-4B Foofah Program Ensemble Consensus

  No. The simplest rule won: run the first program that passes a single worked example. It solved 52% of tables versus 44% when the model just answered directly, rescuing 23 tables it had otherwise botched while breaking o

• 2026-06-27 Qwen3.5-4B Code ABI Oracle Coverage Ladder

  Yes, mostly. Snapping together at most three verified functions produced a passing solution for 84% of 160 small programming tasks, up from just 13% with a bare-bones library. But the library is a foundation, not a solve

• ~2026-06-27 Qwen 3.5 4B Balanced Discriminative Bridge

  An even mix, clearly. Sixty evenly-spread ordinary examples across ten new program types lifted the model's success on unseen hard problems from 60% (with no examples at all) to 99% fully solved. Hand-picking only the tr

• 2026-06-26 Qwen3.5-4B Verified Algorithm Retrieval Adaptation

  Yes, but modestly. Fetching the most similar already-solved problem and adapting its code cracked 3 of the 8 problems that repeated tries kept missing, lifting overall success from 67% to 79%. Matching by meaning clearly

• 2026-06-26 Qwen3.5-4B Substrate Coverage Ladder

  Only when a building-block was hand-carved for each specific problem. A shared library that reshapes old solutions to fit new ones solved zero of the nine stuck tasks, no better than nothing. Custom-written parts solved

• 2026-06-26 Qwen3.5-4B Retrieval Adapt Verify Scale

  It splits. Pulling a similar solved problem from a 364-entry library and rewriting it recovered 8 of 24 stuck tasks — a third, double the 4 that a random-library control found. But the only safe check, the visible sample

• 2026-06-26 Qwen3.5-4B Pass@k Coverage RL

  No. Retraining the model to spread its tries across different problems actually lowered coverage: it solved about 44% of unseen coding problems within four tries, versus about 69% from simply raising the randomness dial

• 2026-06-26 Qwen3.5-4B Offline Hard Negative Coverage DPO

  It depends — and even the gentle version failed its own bar. A light touch matched the task-solving reach of generating twice as many candidate answers at roughly half the compute, and beat a shuffled-pair placebo. But i

• 2026-06-26 Qwen3.5-4B Independent Retrieval Consensus

  No. Requiring independently reshaped solutions to agree on invented tricky inputs, the system committed to zero of 24 unsolved tasks — the separate versions never converged. Simply taking the first solution that passes t

• 2026-06-26 Qwen3.5-4B Constrained Coverage DPO

  Not quite. The nudged model solved 62% of problems within four tries versus 58% for the untrained model, and kept its first-try accuracy — a real, cheap gain. But simply doubling to eight tries solved 67%, still more. Th

• 2026-06-24 → 26 Qwen3.5-4B Oracle Probe Synthesis MDP

  It's the menu. Just curating which eight test inputs the model chose from raised success from about 43% to 49% — a bigger jump than any training gave. Supervised coaching added a bit more (48% to 51%); preference- and re

• 2026-06-25 Qwen Readable Candidate Verifier

  Barely. Training a picker on the model's reading of each fix's pseudocode plus its claimed output raised the share of tasks fixed correctly from 44% to 51%, closing only about 15% of the distance to a perfect picker's 90

• 2026-06-25 Qwen Candidate-Conditioned Trace Verifier

  No. Doing nothing already solved about 44% of tasks, and always choosing a correct candidate could reach 90%. Yet no trained picker captured any of that headroom; the best merely tied doing nothing. Left unchecked, the m

• 2026-06-25 Qwen3.5-4B Verifier-Guided Self-Improvement Report

  No. After a four-billion-parameter model retrained on its own test-passing code, its success on unseen problems barely moved, going from about 65% to about 65% (a hair lower). Retraining made its several attempts look mo

• 2026-06-25 Qwen3.5-4B Trained vs Frozen Repair MDP Report

  Retry from scratch. Given the same compute budget, a few more fresh attempts rescued 5 of 57 stuck problems, while a fixer trained to patch broken code rescued only 2 — and its repairs secretly failed the hidden tests ne

• 2026-06-25 Qwen3.5-4B Real Sample Verify Commit

  It's the writing. When a 4-billion-parameter model writes several code attempts per problem, choosing is not the bottleneck. On easier problems nearly every attempt is already correct, so the best result sits near 97%. O

• ~2026-06-25 Qwen3.5-4B Oracle-Distilled Semantic Verifier

  Yes, but only on home turf. On the problem set it trained on, the trained judge picked a genuinely-correct program 81% of the time, up from 72% for the same model untrained and just 44% for grabbing the first candidate t

• ~2026-06-25 Qwen3.5-4B HumanEval Adaptive Evidence Budget

  No. Every strategy landed at the same 16.7% correct-pick rate — running no tests, running all eight, and even a strategy allowed to peek at the right answer to time its stop. The trained model did learn to quit sooner, u

• 2026-06-25 Qwen3.5-4B Diversity-Keyed Coverage Gate

  Mostly the second. Of 24 Python problems a 4-billion-parameter model missed on four tries, spending more and more varied sampling recovered 15, lifting the share solved from 70% to nearly 89%. Mixing three creativity set

• 2026-06-24 Qwen3.5-4B Sketch Coverage Shift Probe

  Only when the template explicitly names the new operation. On tasks needing a brand-new operation, hand-written templates that named and typed it kept the correct program every single time; auto-generated templates kept

• 2026-06-24 Qwen3.5-4B Oracle Process GRPO

  Yes, but with sharp limits. Teaching the model to copy a flawless test-picker lifted its success from about a third of puzzles to 43%, versus roughly 31% for picking blindly. Layering on preference-based and reward-based

• 2026-06-24 Qwen3.5-4B Deployable Information Ceiling Sweep

  It's the clues. On the hard hidden rule, the plain strategy got just 4% right when starved of tests and clues, far below the 74% of a rival who secretly knew the answer. But allowing more tests lifted it to 86%, and more

• 2026-06-24 Qwen3.5-4B Adaptive Evidence Budget Policy

  Yes. Trained just for this single decision, the model matched the accuracy of running all ten checks — solving about 92 of every 100 tasks — while using only about five checks on average, near a perfect-hindsight stopper

• 2026-06-24 Qwen3.5-4B Active Counterexample Trace Selection

  Yes, but choose them well. Committing on the visible examples alone left about a quarter of picks secretly wrong, even though every one passed all the examples shown. Requesting six new test cases where the surviving pro

• 2026-06-23 Qwen Typed Bytecode Expert Iteration

  It depends. Training the model only on its own attempts that landed on the correct final answer lifted unaided first-try accuracy from 62% to 73% on fresh problems — a real gain that sticks when it writes programs alone.

• 2026-06-23 Qwen Prefix-State Process Verifier

  Barely. The judge got genuinely good at telling promising partial programs from dead ends, yet it lifted first-try accuracy only a few points, with hard problems going from 41% to 44%. The real gap: on hard problems a co

• 2026-06-23 Qwen On-Policy Repair-to-Compiler

  Yes, but the self-correction isn't the secret ingredient. On brand-new requests the model climbed from about 29% correct programs to 99% after training on its own corrected drafts. The catch: training on plain correct pr

• 2026-06-23 Qwen Mixed-Domain Trace Verifier

  Yes, partly. On fresh tasks the frozen model alone got 46% right; the proofreader lifted that to 57%, and a cross-check that compares reworded versions of the same task reached 61%. But a correct recipe was already among

• 2026-06-23 Qwen Hidden VM Curriculum Repair

  No. Feeding it nearby worksheets that merely land on the correct answer wrecked it. Plain step-by-step training scored 72% on the main mixed test; the same model after answer-chasing repair fell to 35%, and collapsed on

• 2026-06-23 Qwen Context-Conditioned Trace Verifier

  Mostly no. A correct program was almost always sitting in the candidate pile, for 91 to 100 percent of questions, yet the judge barely helped: it nudged easy questions from about 69 to 70 percent and actually made the ha

• 2026-06-23 Qwen Complete-Program Trace Reranker

  Barely, and it backfires on hard cases. A correct program sat in the candidate pile 91 to 100 percent of the time, yet the trained picker nudged easy prompts only from 69 to 70 percent and actively hurt longer ones, drop

• 2026-06-22 Qwen Verifier-Guided Slot Repair

  Yes, mostly, but with a catch. A small model copying 24-step calculations got only about 27% exactly right on its own. A checker that knows the correct running number after every step, allowed to swap one or two bad step

• 2026-06-22 Qwen Teacher-Distilled Slot Compiler

  No. A model trained to copy numbers and operations out of text and run a 24-step calculation got 27% of final answers exactly right; adding the pointing signal landed at 28%, a tie. Worse, agreement between two rewording

• 2026-06-22 Qwen 3.5 4B Model-In-Loop Counterexamples

  No. Building practice cases from the model's actual wrong answers matched, but never beat, simply hand-picking the tricky categories in advance. Both lifted the hardest problems from 64% to a perfect 100% passing every h

• 2026-06-22 Qwen 3.5 4B Counterexample-Directed DSL

  It depends. Hand-picked examples lifted the model's single-best-guess repair rate from 51% to 58% over random examples. But when it generated several candidates and kept the best, the edge vanished (64% slipped to 61%).

• 2026-06-21 Feature-Factorized Rule Diversity

  No. All three practice diets fixed only about 1 in 5 brand-new bug combinations, so mixing bought nothing over drilling either kind alone. What actually mattered was showing worked, step-by-step repair reasoning during t

• 2026-06-20 Counterexample Rule Repair Experiment

  Yes, but only for rule shapes it has practiced. Given the failing examples, a small fine-tuned model fixed 91% of bugs on familiar tasks and got unseen inputs right too, versus 0% untrained and near-zero for every versio

Claims

• ✓ Confirmed C2 · Candidate coverage does not imply deployable accuracy
• ◑ Promising C4 · Active evidence must be coupled to downstream decisions
• ◑ Promising C10 · The C2 selection wall is plumbing not capability: a thinking-verifier closes most of it
• ◑ Promising C29 · Preference training on the model's own failures does NOT close the coverage->deployable gap -- DPO collapses generation; the gap closes with MORE SFT instead
• ◑ Promising C34 · With the interpreter, brute-force structure-search + value-fill + execution-consensus near-solves depth-3 (0.975), dominating the model; banking's structure is a forward-pass-only asset
• ◑ Promising C35 · Brute search dominates tested banked models through depth 4 and remains operationally cheap at depth 5; learned crossover remains open
• ◑ Promising C46 · The confidence toolkit TRANSFERS to real code (MBPP + HumanEval) -- but the hierarchy INVERTS: the single-token P(True) judge readout, not sequence mean-logprob, is the program-level confidence
• ◑ Promising C47 · Confidence cannot replace the execution verifier at the TRAINING seat (conf-filtered banking ~ random banking) -- but the judge itself SURVIVES self-training: rank-filter flywheels stay viable while fixed-threshold filters silently degrade
• ✗ Negative C51 · Answer-only potential after cap-bound thoughts is a real but NON-ACTIONABLE selector: it detects relevant trace content yet fails G0 because the teacher-forced answer seam is rarely deployable
• ◑ Promising C57 · COMPUTE-OPTIMAL CONFIDENCE POLICY = confidence-gated adaptive ALLOCATION (not escalation): on the fixed Qwen3.5-4B, committing the greedy answer when its single-token P(True) is high and sampling+conf-selecting only when it is low reaches full-pool MBPP accuracy (0.762) at ~4.25 avg samples vs 9 for uniform sampling — a ~2x compute saving, strictly beating uniform at 7/9 operating points. COMPUTE-OPTIMAL CONFIDENCE POLICY (corrected, powered-up): on the fixed Qwen3.5-4B for MBPP, single-token P(True) confidence-SELECT is the best verifier-free selector on MODERATE-difficulty MBPP (k=9: 0.762 > majority 0.742 > mean-logprob 0.725; per-cand AUROC 0.77) but only DIFFICULTY-DEPENDENTLY so — it ties majority-vote on easy HumanEval (base pass 0.91, both 0.941), max-P(True) ABSTENTION gives a clean risk-coverage curve (solvability AUROC 0.72), and DEPTH (a higher think budget) modestly beats BREADTH on the overall accuracy-vs-tokens frontier (pure-2048 0.593 > pure-256 0.581). BUT selectively ESCALATING the abstained tail to depth does NOT beat matched-compute breadth: at the powered-up n=400 all four abstain-fraction deltas are +0.004..+0.022 with 95% bootstrap CIs spanning 0 — the initial n=24-60 escalation win (+0.15) was a small-sample artifact, caught by the claim's own pre-registered power-up.
• ◑ Promising C58 · C42 TARGETED REPAIR resolved: a located single-step slip is a NEAR-MISS (the correct digit is the model's RANK-2 choice 85% of the time, median rank 2) and temperature-resampling makes the fix REACHABLE (surfaces d* 80% at n=8), but NO verifier-free CONFIDENCE selector recovers it (greedy re-try and confidence-select both 0.000 -- the erroneous digit IS the argmax). ISOLATED re-computation of just the flagged step, out of the flawed chain context, partially repairs (0.318). So localization enables repair only WITH a verifier or by CONTEXT-REMOVAL, not by confidence selection. END-TO-END: the deployable verifier-free loop (localize min-confidence step -> isolated-recompute -> rerun) lifts whole-chain accuracy +0.048 (oracle-localization ceiling +0.102); repairing a RANDOM step is catastrophic (-0.271), so localization is essential.

Queued proposals 6

• P0 Visible Abstention Selector Benchmark
• P1 Independent Consensus Versus Selector Pool
• P0 Deployable Gap Scorecards
• P1 Operator Bank Shortlister Scaling
• P0 STOP/MORE Controller With Visible Labels
• P1 Multi-Model Role Ensemble Pool