Public-verifier recovery branch tournament
The one idea you need
Two mechanics only form a useful team when each can fix failures the other misses. If both get stuck on the same repair, choosing between them cannot create the missing solution.
The question
Can visible tests combine two coding policies better than spending the same model budget on more attempts from either one?
What we found
No. On four new problem types, both policies repaired 74%, and their combined best reached only 75%—exactly tied with two action-only attempts. The selector was correctly stopped before scoring.
Why it matters
The remaining gap needs a new kind of solution, not a better chooser. Training should target the shared validate-copy-commit pattern while replaying the reliable recovery behavior already learned.
On this page
Results at a glance 2
How to read
Each horizontal category is a policy or equal-budget control, and taller bars mean more repaired repositories. The final mixed-policy bar should exceed both more-attempt bars to justify selection.
Takeaway → The mixed bar stops at 75%, exactly level with action-only sampling and only one point above either deterministic policy.
Data table
| frozen policy or matched-compute control | prospective development recovery |
|---|---|
| base | 61.3% |
| reason mix | 73.8% |
| action only | 73.8% |
| reason sample more | 73.8% |
| action sample more | 75% |
| mixed-policy union | 75% |
Numbers from experiments/qwen35_4b_recovery_verifier_branch_tournament/reports/result_receipt.json
Technical framing
The mixed-policy ceiling only ties same-policy sampling — The two deterministic source policies have only one exclusive win each, so their 75% union cannot clear the frozen +3-point bars and exactly ties two action-only trajectories. Feasibility stops before selector scoring.
How to read
Problem families run horizontally and recovery success runs vertically, where taller is better. The two colors compare the reason-enhanced and action-only policies family by family.
Takeaway → Both policies fall to zero on atomic reservations and nearly overlap everywhere else, leaving no useful complementary coverage.
Data table
| new procedural family | reason mix | action only |
|---|---|---|
| atomic reservations | 0% | 0% |
| deadline queue | 100% | 100% |
| fallback resolver | 95% | 100% |
| interval coalescer | 100% | 95% |
Numbers from experiments/qwen35_4b_recovery_verifier_branch_tournament/reports/result_receipt.json
Technical framing
Both source policies share the atomic-reservation failure core — Every one of 20 atomic-reservation cases is a shared deterministic failure. The only source-policy differences are one five-point family swap elsewhere, leaving no useful portfolio diversity.
In the author’s words from the Overview · “Results”
Verdict: PROSPECTIVE_DEV_INFEASIBLE; stopped before selector scoring. Confirmation, winner banking, and Menagerie remained sealed. The deterministic source policies each solved 59/80 cases, with only one exclusive win apiece; their union was 60/80. That ceiling could not reach the registered 61.4/80-equivalent bar over either source, much less beat action sample-more by 3pp. All three feasibility checks failed, so the public selector was never scored prospectively. The negative is sharply localized. Fifty-eight cases were solved by both sources and all 20 shared failures were atomic_reservations; both policies retained 100% changed-patch-within-two behavior. … Read the full result →
Overview
Research Program
- Program:
agentic_breadth_installation - Direct predecessor:
qwen35_4b_recovery_payload_budget_harness. - Prior anchors: C46 (execute visible evidence before confidence), C54 (serial-compute capability), and the three conditional-recovery experiments.
Question
Can a frozen public-only rule combine the complementary action-only and λ=.18 recovery policies into a stronger coding agent on wholly new procedural families, while beating two full same-policy trajectories at identical reserved model-token compute?
Hypothesis
The predecessor's two local policies tied at 68.75% on confirmation but had a 78.75% hidden-success union on both independent transfer blocks. Their public visible-test state was informative: choosing action-only only when its final workspace passed visible tests and λ=.18's did not scored 75.0% on both blocks, captured 95.2% of the union, and exceeded the exact randomized-policy expectation by 6.25 points. This rule should prospectively recover useful policy diversity without predicting a latent winner or changing shared weights.
The hypothesis fails if the retrospective rule does not transfer to new algorithm families, if same-policy sample-more matches it, or if its apparent gain requires hidden outcomes.
Frozen Intervention
For each rejected-patch or failed-visible-test state:
- Run one greedy six-call branch with action-only and one with λ=.18.
- Run the public visible suite on each final workspace.
- Choose action-only iff action passes and λ=.18 does not; otherwise choose λ=.18.
No family label, hidden test, model self-report, entropy threshold, or posthoc margin enters the decision. Each branch receives 512 thinking + 512 answer tokens per call. The 12-call tournament reserves 12,288 tokens/case, exactly matching two complete six-call stochastic trajectories from candidate or action-only. The same-policy controls receive an oracle-generous pass-if-either score, making them harder—not easier—to beat.
Setup
- Model: only
Qwen/Qwen3.5-4B, pinned revision851bf6e.... - Frozen checkpoints: C54 apex context control, recovery action-only, and the locality-safe λ=.18 reason mixture; exact hashes are in the config.
- Selector qualification: checksum-frozen predecessor dev/confirm trajectories; this is retrospective design evidence only.
- Prospective substrate: four new procedural repository families—deadline queues, labeled interval coalescing, atomic reservations, and fallback-chain resolution—absent from every prior recovery block.
- Blocks: 80 controlled recovery cases at seed 85000, then an independent 80 at seed 85100. The rule cannot adapt after either.
- Controls: each source policy, C54 apex, candidate pass-if-either sample-more, action pass-if-either sample-more, exact expected random policy choice, and a deterministic random-choice diagnostic.
- Firewall: hidden executable code/output remain host-only. No
benchmarks/source, item, transcript, or result is read or imported.
Gate Order
- Verify all model/source hashes, procedural unresolved-state invariants, selector hidden-label independence, and exact compute reservations.
- Run every branch and same-policy sampling control on prospective dev.
- Before scoring the selector, prove the deterministic action/candidate union can mathematically clear both same-policy sampling controls by 3pp.
- Score the frozen public selector. It must beat the best source, candidate sample-more, action sample-more, and exact random-choice expectation by 3pp; have paired-bootstrap lower bounds ≥0 versus the best source and sample-more; capture ≥85% of the source union; retain both recovery transitions and tool validity; and avoid family collapse.
- Repeat the complete control-first battery unchanged at seed 85100.
- Passing both blocks authorizes a separate transition-balanced winner-bank experiment. It does not authorize Menagerie here because this experiment changes the harness, not a single deployable checkpoint.
Exact thresholds and statistical details are frozen in reports/preregistration.md.
Run
.venv/bin/python experiments/qwen35_4b_recovery_verifier_branch_tournament/scripts/run.py --smoke
.venv/bin/python experiments/qwen35_4b_recovery_verifier_branch_tournament/scripts/run.py --gpu-smoke
.venv/bin/python experiments/qwen35_4b_recovery_verifier_branch_tournament/scripts/run.py --fullResults
Verdict: PROSPECTIVE_DEV_INFEASIBLE; stopped before selector scoring. Confirmation, winner banking, and Menagerie remained sealed.
| Prospective-dev arm | Success | Mean sampled tokens | Atomic reservations |
|---|---|---|---|
| C54 base | 61.25% | 2,248 | 0/20 |
| λ=.18 greedy | 73.75% | 1,439 | 0/20 |
| action-only greedy | 73.75% | 1,537 | 0/20 |
| λ=.18 pass-if-either ×2 | 73.75% | 3,384 | 0/20 |
| action pass-if-either ×2 | 75.0% | 3,395 | 1/20 |
| deterministic mixed-policy union | 75.0% | — | 0/20 |
The deterministic source policies each solved 59/80 cases, with only one exclusive win apiece; their union was 60/80. That ceiling could not reach the registered 61.4/80-equivalent bar over either source, much less beat action sample-more by 3pp. All three feasibility checks failed, so the public selector was never scored prospectively.
The negative is sharply localized. Fifty-eight cases were solved by both sources and all 20 shared failures were atomic_reservations; both policies retained 100% changed-patch-within-two behavior. The traces repeatedly fixed atomic validation or input immutability separately, then regressed the other constraint. Action sample-more found the full conjunction once in 20 cases. Full metrics and hashes are in reports/result_receipt.json.
Interpretation
The predecessor's complementarity was algorithm-specific and did not transfer to four new families. Public branch selection cannot create proposal coverage when both sources share the same failure core; spending the same compute on more action trajectories is at least as good.
This redirects the curriculum from selecting among local policies to installing a missing compositional invariant. The shared failure is not a loop-policy defect: agents inspect, patch, test, and revise, but oscillate between “validate the whole request before any update” and “copy state before updating.” A successor should use executable tool-found solutions across multiple transactional families, preserve the existing conditional transition bank as replay, and require transfer to unseen transactional families plus the broad recovery suite before Menagerie.
Knowledgebase Update
- Program evidence: records the non-transferring union and shared multi-constraint failure core.
- Program backlog: retires public portfolio selection here and queues transaction-invariant scaffold distillation with recovery replay.
- Claim ledger: unchanged; no checkpoint or Menagerie result was produced.
Artifacts
Small selector and gate receipts are committed. Detailed model trajectories live under large_artifacts/qwen35_4b_recovery_verifier_branch_tournament as specified by reports/artifact_manifest.yaml.
Report
Rendered from reports/report.md
Summary
Status: PROSPECTIVE_DEV_INFEASIBLE; stopped before selector scoring. The old policy complementarity did not transfer to four new families, and the mixed union could not beat equal-compute same-policy sampling.
Research program fit
The predecessor repaired the agent interface and found stable complementary coverage, but no globally superior checkpoint. This experiment tests whether the complementarity is deployably capturable before spending on winner-trace curriculum compression.
Method
Action-only and λ=.18 each receive one greedy six-call recovery branch. Action replaces candidate only when action's final workspace passes visible tests and candidate's does not. Candidate is the fixed default. Two complete stochastic trajectories from each single policy receive the same 12,288-token reservation and are scored with the generous pass-if-either hidden coverage upper bound.
Four new procedural families and two seeds provide 80 dev plus 80 confirm cases. Controls and union feasibility precede selector scoring on each block.
Results
Base scored 49/80 (61.25%). Candidate and action-only each scored 59/80 (73.75%). Two complete candidate trajectories still scored 59/80, while two action trajectories scored 60/80 (75.0%). The deterministic candidate/action union was also 60/80. Its preregistered ceiling needed to exceed each comparator by 3pp, so all feasibility checks failed and the selector was not applied.
Paired source outcomes were 58 both-correct, one candidate-only, one action-only, and 20 both-wrong. Every shared failure belonged to atomic_reservations. Both greedy policies nevertheless changed code within two turns in 100% of rejected-patch and failed-test cases, localizing the miss to semantic conjunction rather than process control. Across deterministic and stochastic action runs, only one of 40 atomic trajectories solved.
Controls
C54 apex context, each source policy, two candidate trajectories, two action trajectories, exact random-policy expectation, deterministic random choice, and action-default public selection.
Oracle versus deployable evidence
The hidden union and pass-if-either sample-more are oracle evaluation ceilings. The primary tournament arm selects before hidden tests using only public visible passes. No benchmark source or result is accessed.
Interpretation
Visible-test arbitration is useful only when source policies propose different successful workspaces. On the new tasks, their diversity collapsed: the same transactional conjunction defeated both. The next capability producer must shift proposals with executable supervision for atomic validate-copy-commit patterns, not add branches or tune a selector.
Next experiment
Build diverse transactional repository families, obtain executable tool-found solutions the incumbent almost never proposes, and train an action-seam curriculum mixed with the existing conditional recovery replay. Gate locality, ordinary recovery retention, unseen transactional transfer, and sample-more before Menagerie.
Artifact manifest
See artifact_manifest.yaml.
Experiment log 2
Show the running log (2 entries, 2026-07-12)
2026-07-12 — intake, mechanism audit, and preregistration
- Direct predecessor stopped because λ=.18 tied action-only at 55/80 on confirmation; their hidden union was 63/80 on both transfer blocks.
- Froze the public rule: select action only when action final-visible passes and candidate does not; otherwise candidate.
- Retrospective qualification reproduced 60/80 on both source blocks, 95.2% union capture, and +6.25pp over exact random policy choice. These are design facts, not prospective evidence.
- Added four new procedural families with initial/partial/oracle executable gates and disjoint dev/confirm seeds.
- Matched the two-policy 12,288-token reservation with two full six-call trajectories from each source policy. Pass-if-either hidden coverage is the primary sample-more control.
- Adversarial review fixed exact-random rather than single-draw gating, control-first union feasibility, and the separation between a capability producer and later curriculum compression.
- No prospective Qwen output or Menagerie seed has been exposed.
2026-07-12 — prospective development and feasibility stop
- Base 49/80; λ=.18 greedy 59/80; action greedy 59/80.
- Equal-reservation pass-if-either controls: λ=.18 59/80, action 60/80.
- Deterministic mixed-policy union: 60/80, only one exclusive win per source.
- All three frozen union-feasibility checks failed; recorded
PROSPECTIVE_DEV_INFEASIBLEbefore public selector scoring. - Confirmation, winner-bank production, and Menagerie remained sealed.
- Forensics: all 20 shared failures were
atomic_reservations, despite 100% two-turn changed-patch behavior. Traces oscillated between atomic validation and input-copy invariants; action sample-more solved one of 20 cases. - Strategic pivot: source selection is closed for this line. Queue diverse transactional scaffold-distillation with existing recovery replay to shift proposal coverage while preserving conditional transitions.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
.venv/bin/python experiments/qwen35_4b_recovery_verifier_branch_tournament/scripts/run.py --smokeFull run
.venv/bin/python experiments/qwen35_4b_recovery_verifier_branch_tournament/scripts/run.py --fullRun steps are documented inside the experiment folder (README and scripts).