Repair-Verifier Signal Probe
The one idea you need
Every teaching route to the one benchmark family blocking the program's ten-family goal is now closed: the model cannot learn debugging-style repair from demonstrations at any tested dose. One training class remains — letting the model learn from its own attempts with live feedback — but that loop only works if the model can at least RECOGNIZE a correct repair when shown one, even though it cannot produce one. This experiment measures exactly that: two hundred puzzles, each showing a broken procedure's failures on two trials plus two candidate one-step fixes, only one of which works for both. Picking the right one requires mentally running each candidate — the model's one reliably strong skill.
The question
Can the model recognize the correct repair when shown two candidates — the signal an attempt-and-feedback training loop would need?
What we found
The gate said no, cleanly. Handed two candidate repairs and the full failure evidence — a task solvable by mentally running each candidate through both trials — the model picked the working one 51.5 percent of the time with its reasoning enabled and 49 percent without: coin-flip, below even the best word-matching shortcut the design team pre-computed and pinned at 53 percent. It was not running out of thinking room (only 7.5 percent of items hit the cap); it finishes its reasoning and still cannot tell. With recognition absent, a learn-from-your-own-attempts loop would have no signal to climb, and the last training class for the stubborn family closes by the pre-written rule. The program's map is complete: the ten-family goal stands demonstrated on two of four sealed seeds, every route to making it reliable is closed by rule rather than fatigue, and the one blocking skill is now precisely characterized — the model can neither produce, nor buy with thinking time, nor even recognize multi-constraint eliminative repairs.
Why it matters
This one cheap measurement decides whether the program's last mechanism class gets funded on evidence or closed by rule — the difference between a live path to the ten-family goal and an honestly completed map.
On this page
Results at a glance 1
How to read
Two bars for the model's two modes, with the coin-flip floor, the shortcut ceiling, and the funding bar marked.
Takeaway → Both modes sit at the coin-flip floor, below the shortcut ceiling and far from the 65 percent funding bar.
Data table
| arm / frozen reference | accuracy |
|---|---|
| think arm | 0.515 |
| nothink arm | 0.49 |
| shortcut ceiling (pinned) | 0.532 |
| funding bar | 0.65 |
Numbers from experiments/qwen35_4b_repair_verifier_signal_probe/runs/local/probe_readout.json
Technical framing
2AFC repair-verification accuracy vs the frozen bars (200 items/arm) — SIGNAL_ABSENT: with pure failure evidence and two unmarked candidates (marker audit 0 hits x 200 prompts), the composite scored 103/200 thinking and 98/200 without - the coin-flip floor, below the pinned 53.25% collision-shortcut ceiling, cap contacts 7.5% (not budget-limited). The C29 read-only-verifier dissociation is skill-scoped: it does not extend to multi-constraint eliminative bookkeeping. The on-policy class closes for menders; every mechanism class for the goal-gating family is now closed by preregistered rule; the program's goal position stands demonstrated (two 10/10 sweeps across four sealed seeds) and not confirmed.
In the author’s words from the Overview · “Results”
Not yet run. The terminal artifact will be runs/local/probe_readout.json with the ordered SIGNAL_PRESENT / SIGNAL_ABSENT consequence, per-arm accuracy + exact CI, per-formalism accuracy, and the position-bias check.
Overview
Lifecycle 21 — a cheap, eval-only FEASIBILITY GATE for a possible on-policy training charter. The menders-shaped eliminative-repair skill is closed to SFT at every tested dose (three pedagogies, 80–800 rows; the dose-scale candidate scored exactly the untrained controls' guess floor). On-policy training (learn from own attempts + live feedback) is the one remaining mechanism class, and it only has signal if the model can VERIFY repairs it cannot propose (C29 precedent: read-only 2AFC verifier 0.81 while generation collapsed; C47: think-judges rescue substrate-scoped no-think judges). This probe measures that verification signal directly: 400 judgments, no training, no benchmark seed, no promotion.
Research Program
- Program:
agentic_breadth_installation. - Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
- Prior anchors: the dose-scale NULL (
qwen35_4b_menders_dose_scale: 800 rows, candidate at the controls' guess floor, retention taxed) closing the SFT class for the family; C29 (generator-verifier gap: read-only 2AFC 0.81 while generation collapsed); C47 (think-judge rescues substrate-scoped no-think judges); C44 (always give the 4B chain-of-thought).
Question
Does the hygiene_explore composite carry EXECUTION-BASED FIX-VERIFICATION signal for the eliminative-repair skill its generation seat cannot produce — i.e., given a machine spec, a buggy step sequence, and two trials' failure evidence, can it SIMULATE two candidate single-step fixes and pick the unique one that makes both trials come out as wanted, above chance?
Hypothesis
C29-class dissociation predicts verification survives where generation is at floor: the think arm simulates both candidates against the two trials and scores well above the 0.5 chance floor (>= 0.65 with the exact 95% CI excluding 0.5). C47 predicts the no-think arm may sit substantially lower (substrate-scoped judge seat).
Setup
- Model: ONE evaluated checkpoint — the hygiene_explore composite (tree
9eb653d78f05546ca594a831c989fa906d12f3eb7a5a8550d1afcd6bfccc4971), inherited fromqwen35_4b_hygiene_explore_destack_medium, receipt- and tree-authenticated at every boundary. Base: Qwen/Qwen3.5-4B @ 851bf6e8 (the only model this repo ever uses). - Dataset/task source: 200 fresh two-round eliminative-repair episode instances (25 per formalism across all eight: troughline, trinketcord, crankwheel, sigilslate, barrowyoke, balesled, millround, skeinreel), selected deterministically from a 320-instance pool (40 per formalism) generated by the dose-scale cell's reviewed generator machinery (
scripts/gen_feedloop_curriculum.py, copied byte-identically) at construction seed 77160. Frozen selection rule: the first 25 per formalism in stream order whose WRITTEN sequence also fails trial two (pure failure evidence on both trials; 11 coincidental trial-two successes excluded). Each selected instance is rendered fresh as a FULLY SYMMETRIC 2AFC item — machine spec with legality clauses + written steps + both trials' setups/wanted/observed outcomes of the broken run + two candidate single-step changes in identical grammatical form (the unique legal both-trials fix and the legal trial-one-consistent distractor) — with NO repair history, NO attempt narration, a frozen marker-token audit (33 provenance words, zero hits fail-closed), and option self-reference checks. Deciding requires simulating each candidate against both trials. Correct fix is A on exactly 100 items, B on 100 (deterministic per-formalism 13/12 assignment from the construction seed). Every item's fix properties are re-executed against the machine semantics at design time on top of the generator's audits, and a frozen listing-collision audit records the best accuracy any collision-keyed guessing heuristic could reach (0.5325, well under the 0.65 bar). - Train/eval split: no training anywhere. Zero-overlap receipts against the dose-scale corpus, BOTH its 2,280-row streams, its 40-row holdout and three retention screens, every pinned predecessor corpus/stream, every predecessor frozen gate (seeds 88013–88040), and regenerated prior local seeds 88000–88040.
- Baseline: the 0.5 chance floor of a position-balanced 2AFC.
- Controls: within-model decode contrast —
think(natural thinking, 1,024-token cap) vsnothink(runner--thinking off); the nothink arm is descriptive only (C47 substrate-scoping check). - Primary metric: think-arm
2afc_accuracywith the exact (Clopper-Pearson) two-sided 95% binomial CI vs 0.5. Preregistered diagnostic: the think arm's cap-contact rate at the 1,024-token cap (simulating two candidates over two trials each may press the cap). - Oracle-only metrics: none — grading is exact-match of a single letter after the frozen
normalize_answer. - Hidden-label boundary: the runner input carries only
id/messages/meta(no answer, no correct position); answers live in the frozen source file only.
Frozen consequence partition (ordered, total, no third state)
- think accuracy >= 0.65 AND CI excluding 0.5 → SIGNAL_PRESENT: "the on-policy episode charter is fundable — execution-based fix-verification signal exists for the skill generation cannot produce (C29-class dissociation): the model can simulate candidate repairs against two-trial evidence above the frozen bar".
- think accuracy < 0.65 OR CI including 0.5 → SIGNAL_ABSENT: "the eliminative-repair skill lacks even execution-based fix-verification signal at this instrument; the on-policy class closes for menders and the program map is complete at demonstrated-not-confirmed".
- Preregistered scope (never a third state): a SIGNAL_ABSENT reading with think-arm cap contacts on more than 20% of items is annotated as possibly budget-limited at the 1,024-token cap.
Run
Smoke (model-free: design --check, composite pin, lineage --verify-inputs, compile, unit tests):
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B scripts/run.py --smokeLocal event (requires PASS_LOCAL_EVENT in reports/local_design_review.md, committed design receipt, clean pushed green main; two sequential authenticated engine runs; exits 0 on any complete event):
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B scripts/run.py --stage localResults
Not yet run. The terminal artifact will be runs/local/probe_readout.json with the ordered SIGNAL_PRESENT / SIGNAL_ABSENT consequence, per-arm accuracy + exact CI, per-formalism accuracy, and the position-bias check.
Interpretation
Pending the local event. SIGNAL_PRESENT funds an on-policy episode charter (the model can label its own repair attempts using live two-round feedback); SIGNAL_ABSENT closes the last mechanism class for menders and completes the program map at demonstrated-not-confirmed.
Knowledgebase Update
- Program evidence updated: pending the event.
- Program backlog updated: pending the event.
- Claim ledger updated: pending the event (feasibility probes do not manufacture claims by themselves).
Artifacts
src/— the experiment-local vLLM runner (merged-composite--model-overridevariant).scripts/—gen_local_gate.py(design freeze + overlap receipts),check_local.py(frozen readout),eval_local_vllm.py(two-run authenticated event),run.py(checkpointed harness), the copied generator machinery (gen_feedloop_curriculum.py,gen_curriculum.py), and the standalone lineage package scripts (rebuild_lineage.py,merge_adapter.py,lineage_trainers/).configs/— frozen experiment configuration.data/— the frozen probe set + runner input + design receipt;data/lineage/— the evaluated composite's complete six-stage reproduction package.runs/— the local event's raw artifacts and receipts (written by the event).reports/— report, reviews,reports/artifact_manifest.yaml.
Report
Rendered from reports/report.md
Summary
The feasibility gate for the program's last mechanism class returned SIGNAL_ABSENT, cleanly: presented with pure failure evidence and two unmarked candidate repairs — decidable by simulating each against both trials — the hygiene_explore composite scored 103/200 with thinking (51.5%) and 98/200 without (49.0%), both at the coin-flip floor and below the test-pinned 53.25% shortcut ceiling, with cap contacts at 7.5% (no budget limitation). The model cannot verify the repair it cannot produce. The C29 read-only-verifier dissociation is skill-scoped and does not extend to multi-constraint eliminative bookkeeping; an on-policy loop would have no reward signal to climb. Per the frozen consequence the on-policy class closes for menders, completing the program map: every mechanism class for the one goal-gating family — three SFT pedagogies across 80–800 rows and 4–8 formalisms, the deployment thinking budget, and now on-policy selection — is closed by a preregistered rule.
Research Program Fit
agentic_breadth_installation. The dose-scale NULL closed the SFT class for the last blocking family; before funding (or killing) an on-policy episode charter, calibrate-and-diverge doctrine demands a cheap gate on the one precondition every on-policy method shares: a usable execution-based fix-verification signal (C29-class generator-verifier dissociation; C47 think-judge rescue; C44 give the 4B CoT).
Method
- 200 fresh two-round eliminative-repair instances (25 per formalism, eight formalisms) selected deterministically from a 320-instance pool generated by the dose-scale cell's reviewed machinery at construction seed 77160 (grep-fresh); frozen selection rule keeps only instances whose written sequence fails BOTH trials (11 coincidental trial-two successes excluded). Each is rendered fresh as a fully symmetric 2AFC item: spec with legality clauses + written steps + both trials' setups/wanted/observed outcomes of the broken run + two candidates in identical grammatical form (unique legal both-trials fix vs legal trial-one-consistent distractor). No repair history or attempt narration; a frozen 33-word marker-token audit (zero hits, fail-closed) and option self-reference checks keep provenance out of the text; a frozen listing-collision audit records the best collision-keyed guessing accuracy (0.5325, well under the bar). Exact 100/100 position balance assigned deterministically from the construction seed; per-item re-execution of both fixes' properties (and of the re-implemented renderers against the generator's own rendering) on top of the generator's audits; banned-vocabulary scan; zero-overlap receipts vs the dose-scale corpus, both its streams, its holdout, all predecessor corpora and frozen gates (88013-88040), and regenerated prior local seeds.
- ONE evaluated model: the hygiene_explore composite (tree 9eb653d7...), receipt- and tree-authenticated at every boundary; standalone lineage package carried in-cell (six stage datasets + manifest + trainers + merger + vendored root adapter ad2ef4fa.../cd764ae8...).
- Two sequential authenticated engine runs (think natural / nothink off, greedy, 1,024-token cap, identical geometry); grading is exact-match of the letter after the frozen
normalize_answer. - Readings: per-arm 2AFC accuracy with exact Clopper-Pearson 95% CI vs the 0.5 floor; per-formalism accuracy; position-bias check; the preregistered cap-contact diagnostic (a SIGNAL_ABSENT reading with think-arm cap contacts on >20% of items is annotated as possibly budget-limited at the 1,024-token cap — never a third state); the frozen ordered two-state consequence partition on the think arm (SIGNAL_PRESENT iff accuracy >= 0.65 AND CI excludes 0.5; else SIGNAL_ABSENT; measured signal = execution-based fix verification). The nothink arm is descriptive (C47 substrate-scoping check). Exit 0 on any complete event.
Results
Pending the local event.
Controls
- Position balance (100/100) with a preregistered position-bias reading; deterministic assignment neutralizes order-keyed guessing.
- Fully symmetric option presentation: no attempt narration, marker-token audit (zero provenance words), self-reference checks — the item cannot be solved by text matching, only by simulating the candidates.
- Frozen listing-collision artifact ceiling (0.5325) on file next to the 0.65 signal bar.
- Within-model decode contrast (think vs nothink) rather than a cross-model confound.
Oracle Versus Deployable Evidence
No oracle metrics: the letter grading is deployable-style exact match. The probe itself is diagnostic (feasibility), not a deployable capability claim.
Interpretation
Pending. SIGNAL_PRESENT funds the on-policy episode charter; SIGNAL_ABSENT closes the on-policy class for menders and completes the program map at demonstrated-not-confirmed.
Next Experiments
Decided entirely by the frozen consequence partition — no third state exists.
Artifact Manifest
See artifact_manifest.yaml (external composite pin + vendored lineage root recorded).
Experiment log 4
Show the running log (4 entries, 2026-07-16)
Scaffold
Created as a new experiment scaffold.
2026-07-16 — Model-free design freeze (lifecycle 21)
- Opened as the dose-scale NULL's funded successor: menders is closed to SFT at every tested dose (three pedagogies, 80–800 rows; the 800-row candidate scored exactly the untrained controls' guess floor), so on-policy training is the one remaining mechanism class — and it is only fundable if the model can VERIFY repairs it cannot propose (C29 read-only dissociation; C47 think-judge rescue). This cell is the cheap eval-only feasibility gate on that precondition.
Probe set frozen: 200 fresh two-round episode instances (25 per formalism, all eight dose-scale formalisms), selected deterministically from a 320-instance pool (40 per formalism) at construction seed 77160 (grep-fresh; zero seed-context hits; 77150 is the dose-scale construction seed immediately below), generated by the dose-scale cell's reviewed
gen_feedloop_curriculum.pycopied byte-identically. Frozen selection rule: the first 25 per formalism in stream order whose WRITTEN sequence also fails trial two (11 coincidental trial-two successes excluded — pure failure evidence on both trials). Each selected instance is rendered fresh as a FULLY SYMMETRIC 2AFC item: machine spec with legality clauses- written steps + both trials' setups, wanted outcomes, and observed
outcomes of the original (broken) sequence + two candidate single-step changes labeled A/B in identical grammatical form — the unique legal both-trials fix vs the legal trial-one-consistent distractor. NO repair history and NO attempt narration anywhere; a frozen 33-word marker-token audit (tried/attempt/failed/earlier/...) rejects any provenance word fail-closed (zero hits on all 200), and self-reference checks hold each option line and quoted change text to the option block. Deciding requires SIMULATING each candidate against both trials — the measured signal is execution-based fix verification, the self-check an on-policy loop would use as reward. Exact 100/100 position balance assigned deterministically from the construction seed (per-formalism 13/12 splits, formalism-keyed shuffle streams). Every item re-executes both fixes' recorded properties against the machine semantics (and verifies the re-implemented renderers against the generator's own episode rendering) on top of the generator's audits; banned-vocabulary scan on the rewritten prompts. A frozen listing-collision audit records that the best accuracy any collision-keyed guessing heuristic could reach is 0.5325 (both-collide 26 / neither 101 / true-only 30 / wrong-only 43), well under the 0.65 signal bar.
- 2026-07-16 coordinator review, addressed before freeze: the draft item narrated the distractor as the earlier FAILED attempt — a pure text-matching shortcut that could fire SIGNAL_PRESENT on an artifact. The item was redesigned to the fully symmetric shape above, the consequence statements re-worded to name execution-based fix verification, and the cap-contact scope preregistered (below); the probe set, receipts, and prompt-dependent tests were regenerated and re-run.
- Freshness receipts: zero canonical-message overlap vs the dose-scale corpus + replay blend + BOTH 2,280-row streams, the 40-row holdout and three retention screens (88037–88040), 27 pinned predecessor corpora/streams, 25 predecessor frozen gates (seeds 88013–88040), regenerated prior local seeds 88000–88040, and the regenerated dose-scale treatment (800 @ 77150) and holdout (40 @ 88037) rows.
- ONE evaluated model: the hygiene_explore composite (receipt 22a22a68…, tree 9eb653d7…, weights e2112344…), tree-recomputed at every boundary. Standalone-reproducibility gate satisfied by copying the confirmation cell's audited lineage package in-cell (six stage datasets + fixed-seed manifest + three trainer variants + merger) and vendoring the frozen root adapter into this cell's own
large_artifacts/tree (weights ad2ef4fa…, config cd764ae8…, verified);rebuild_lineage.py --verify-inputswired into--smoke. - Two arms frozen (same model, one decode config each):
think(--thinking natural, 1,024-token cap; C44/C47) andnothink(--thinking off, the runner's clean no-think channel used by predecessors' no-think evals — the chat template renders the empty-think suffix, and the raw-arm contract checks every prompt's channel). Greedy, runner seed 77160 (recorded provenance; both runs are greedy), geometry identical to the dose-scale/calibration events (max_model_len 4096, max_num_seqs 16, cudagraph 1/2/4/8/16). - Preregistered readings (no promotion; exit 0 on any complete event): per-arm
2afc_accuracywith the exact Clopper-Pearson two-sided 95% CI vs the 0.5 floor; per-formalism accuracy (25 each); position-bias check (A-correct vs B-correct accuracy); the cap-contact diagnostic (simulating two candidates over two trials each may press the 1,024-token think cap — a SIGNAL_ABSENT reading with think-arm cap contacts on more than 20% of items is annotated as possibly budget-limited; never a third verdict state); and the frozen ORDERED, TOTAL consequence partition on the think arm — >= 0.65 AND CI excluding 0.5 → SIGNAL_PRESENT (the on-policy episode charter is fundable: execution-based fix-verification signal exists); otherwise SIGNAL_ABSENT (the skill lacks even execution-based fix-verification signal; the on-policy class closes for menders and the program map is complete at demonstrated-not-confirmed). No third state; the nothink arm is descriptive (C47 substrate-scoping check). - Verified locally: py_compile green over every script (incl. lineage trainers); 134 unit tests green (symmetric-2AFC construction invariants with fail-closed tampering negatives, oversample/filter selection bookkeeping, marker-token and self-reference audits, exact position balance, listing-collision artifact ceiling, letter grading, CI math with closed-form edge checks, the consequence partition switching exactly at 130/200, cap-contact scoping at its exact boundary, readings schema, receipt-layout negatives, frozen constants coherence, lineage package byte-anchoring + manifest negatives, runner + model-override suites);
gen_local_gate.py --checkbyte-stable across two runs;rebuild_lineage.py --verify-inputsgreen;run.py --smokegreen end-to-end;run.py --stage localrefuses without a clean pushed checkpoint + committed design receipt +PASS_LOCAL_EVENTreview verdict. - No model, GPU, or benchmark event has run; nothing trains in this cell. Next: adversarial local design review → commit/push →
--stage local.
2026-07-16 — Confound caught, redesign frozen
- The first build's 2AFC inherited the episode's narration of the wrong attempt — a text-matching shortcut that would have fired SIGNAL_PRESENT on an artifact. Caught in coordinator review before freeze; redesigned to pure failure evidence with symmetric unmarked candidates; the marker audit (33 tokens × 200 prompts) shows zero hits and the collision-heuristic ceiling is test-pinned at 0.5325.
- Two construction honesty findings handled structurally: trial-two coincidences filtered deterministically; the cap-contact scope annotation preregistered for any SIGNAL_ABSENT reading. 134 tests green; smoke green; PASS_LOCAL_EVENT granted.
2026-07-16 — The event: SIGNAL_ABSENT, cleanly
- Both arms ran authenticated; think 103/200 (51.5%), nothink 98/200 (49.0%) — chance, below even the pinned 53.25% shortcut ceiling, with cap contacts at 7.5% (no budget scoping). Per-formalism max 16/25; position bias noise-shaped.
- The frozen consequence: the eliminative-repair skill lacks even execution-based fix verification — the model finishes its reasoning and still cannot tell which of two handed candidates works. The C29 read-only-verifier dissociation is itself skill-scoped: it does not extend to multi-constraint eliminative bookkeeping. The on-policy class closes for menders; the program map is complete.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_repair_verifier_signal_probe/scripts/run.py --smokeFull run
checkpointed scripts/run.py --stage local onlyRun steps are documented inside the experiment folder (README and scripts).