Payload-capable recovery agent harness
The one idea you need
Two coding policies can resemble two mechanics who fix different failures. A fair test must show that one consistently outperforms the other, not merely that their strengths differ.
The question
Does giving tool commands enough room turn the reason-enhanced recovery model into a consistently stronger coding agent?
What we found
No. It led on the first unseen task block at 71%, but confirmed at 69%, exactly tied with the action-only model and short of the required lead. The external benchmark stayed sealed.
Why it matters
The larger command slot fixed a real interface problem, but changing one global model cannot combine both policies' strengths. The next test should fork both and let visible test evidence choose.
On this page
Results at a glance 3
How to read
Each model or control is grouped along the horizontal axis, with recovery success vertically; taller is better. The two colors compare development with independent confirmation.
Takeaway → The reason-enhanced bar leads during development but exactly matches action-only on confirmation, so it fails the required incumbent win.
Data table
| frozen model or matched control | transfer development | transfer confirmation |
|---|---|---|
| base | 58.8% | 60% |
| happy actions | 61.3% | 56.2% |
| action only | 66.2% | 68.8% |
| matched sampling | 50% | 46.2% |
| runtime scaffold | 66.2% | 63.7% |
| reason mix .18 | 71.2% | 68.8% |
Numbers from experiments/qwen35_4b_recovery_payload_budget_harness/reports/result_receipt.json
Technical framing
Recovery transfers, but confirmation ties the action-only incumbent — The reason mixture passes every development gate at 71.25%, but independently confirms at 68.75%, exactly tied with action-only instead of clearing the frozen +3-point margin. Menagerie remained sealed.
How to read
The horizontal groups are the two unseen blocks, and bar height is recovery success. The third bar counts cases solved by either policy and is only a diagnostic ceiling.
Takeaway → The combined bar repeats at 79% on both blocks, showing stable complementary strengths that a visible test selector might exploit.
Data table
| independent held-out block | reason mix .18 | action only | hidden-success union |
|---|---|---|---|
| transfer development | 71.2% | 66.2% | 78.8% |
| transfer confirmation | 68.8% | 68.8% | 78.8% |
Numbers from experiments/qwen35_4b_recovery_payload_budget_harness/reports/result_receipt.json
Technical framing
The two local policies have the same replicated oracle union — The union uses hidden outcomes only and is therefore an oracle diagnostic, not a deployable ensemble. Its exact replication at 78.75% motivates a new public-verifier branch-selection test.
How to read
Each group is an unseen task block; bar height is the share of generated turns that hit the answer limit, where lower is better. Colors compare base with the reason-enhanced policy.
Takeaway → The reason-enhanced bars stay near 8% while base stays near 30%, confirming that command truncation was a real interface bottleneck.
Data table
| independent held-out block | base answer-cap hits per turn | reason mix answer-cap hits per turn |
|---|---|---|
| transfer development | 31.3% | 7.8% |
| transfer confirmation | 29% | 7.9% |
Numbers from experiments/qwen35_4b_recovery_payload_budget_harness/reports/result_receipt.json
Technical framing
The larger payload removes most candidate truncation — All arms receive 512 answer tokens. The candidate cuts cap hits by roughly 21–23 points versus base, confirming that the old 256-token JSON slot was a real interface bottleneck.
In the author’s words from the Overview · “Results”
Fresh locality passed at 0.114 drift (entropy Δ −0.0059; varentropy Δ −0.0105), candidate answer-cap hits fell to 0.5%/7.8%/7.9% of turns on calibration/dev/confirm, and all candidate rejected- and failed-state cases changed the patch within two turns. The dev block passed every registered gate: candidate was +12.5pp versus base and +21.25pp versus matched sampling, with paired 95% lower bounds +3.75pp and +8.75pp. On confirmation it retained +8.75pp versus base and +22.5pp versus matched sampling, improved all four families, preserved normal success exactly (25.0% versus 25.0%), but tied action-only instead of beating it by 3pp. That single failed check stopped the run. … Read the full result →
Overview
Research Program
- Program:
agentic_breadth_installation - Direct predecessor:
qwen35_4b_recovery_reason_locality_interpolation. - Prior anchors: C50 (answer-emission placement), C54 (serial-compute capability), and the two conditional-recovery experiments immediately above.
Question
Does the frozen locality-safe λ=.18 recovery checkpoint become a deployable, family-transferring coding agent when its tool payload is no longer truncated at 256 answer tokens and rejected-patch recovery is measured across one valid inspection rather than only the next action?
Hypothesis
The predecessor's λ=.18 checkpoint solved 58/60 familiar recovery cases at 0.104 locality drift. Its 24 invalid actions all had closed thinking and ended exactly at the 256-token answer ceiling inside long exact-replacement JSON. Separately, every rejected case changed code within two turns and solved; 20/30 used INSPECT→PATCH. We therefore expect a 512-thinking/512-answer harness to reveal an already-present recovery policy, reduce cap hits and invalid actions, and preserve the candidate's advantage over base, happy, action-only, scaffold, and matched sampling.
The hypothesis fails if the candidate remains payload-bound, loses its advantage when all controls receive the same larger answer slot, or does not transfer to the untouched procedural families.
Frozen Intervention
- Weights: exactly the predecessor's λ=.18 checkpoint, SHA-256 frozen in
configs/default.yaml; no training or further interpolation. - Per call: 512 thinking tokens plus 512 answer tokens for every model arm.
- Deep recovery: six calls = 6,144 reserved sampled tokens/case.
- Matched sampling: two trajectories × three calls × 1,024 = the same 6,144.
- Normal loops: eight deep calls versus two × four sampled calls, both 8,192.
- Rejected transition: a changed patch within two generated turns, with the valid paths restricted to immediate
PATCHorINSPECT→PATCH. Immediate change remains diagnostic.
Setup
- Model: only
Qwen/Qwen3.5-4B, pinned revision851bf6e.... - Calibration: the known six-family/60-case block, rerun for base, happy, action, and candidate under the new identical harness. It is selection/gating evidence only.
- Fresh locality: 48 newly committed non-coding contexts disjoint from both predecessor blocks; the fixed candidate must pass before behavior.
- Transfer: the still-untouched four-family blocks at seeds 84800 and 84900.
- Controls: frozen apex, happy-action, full recovery-action, explicit runtime scaffold, and compute-matched sample-more.
- Metrics: hidden-test recovery, invalids, answer-cap hits, rejected and failed conditional transitions, normal solve/verify/commit, paired bootstrap, and per-family deltas.
- Firewall: hidden executable code/output remain host-only. No
benchmarks/source, item, transcript, or result is read or imported.
Gate Order
- Validate all four model hashes and run the new 48-context locality audit (drift ≤0.15; entropy Δ≥−0.05; varentropy diagnostic).
- Run calibration controls first and prove every threshold reachable before candidate evaluation.
- Candidate must beat base +5pp, happy +3pp, and action +3pp; retain base-level invalid/cap-hit rates; reach ≥80% valid rejected change within two turns and ≥60% failed-test change within two; and retain verification/commit.
- Repeat the complete control-first, feasibility-first battery on
transfer_dev, including scaffold and matched sampling. - Repeat without changes on
transfer_confirm. - Only then assign fresh paired Menagerie quick/medium seeds through the public CLI. One tier must improve ≥2pp and neither may regress >3pp.
Exact thresholds are in configs/default.yaml and the statistical contract is in reports/preregistration.md.
Run
.venv/bin/python experiments/qwen35_4b_recovery_payload_budget_harness/scripts/run.py --smoke
.venv/bin/python experiments/qwen35_4b_recovery_payload_budget_harness/scripts/run.py --gpu-smoke
.venv/bin/python experiments/qwen35_4b_recovery_payload_budget_harness/scripts/run.py --fullResults
Verdict: TRANSFER_CONFIRM_FAIL; Menagerie remained sealed. The matched payload repair worked as an interface intervention. Fresh locality passed at 0.114 drift (entropy Δ −0.0059; varentropy Δ −0.0105), candidate answer-cap hits fell to 0.5%/7.8%/7.9% of turns on calibration/dev/confirm, and all candidate rejected- and failed-state cases changed the patch within two turns.
| Block / arm | base | happy | action | sample-more | scaffold | candidate |
|---|---|---|---|---|---|---|
| calibration | 36.7% | 75.0% | 96.7% | — | — | 100% |
| transfer dev | 58.8% | 61.3% | 66.3% | 50.0% | 66.3% | 71.3% |
| transfer confirm | 60.0% | 56.3% | 68.8% | 46.3% | 63.8% | 68.8% |
The dev block passed every registered gate: candidate was +12.5pp versus base and +21.25pp versus matched sampling, with paired 95% lower bounds +3.75pp and +8.75pp. On confirmation it retained +8.75pp versus base and +22.5pp versus matched sampling, improved all four families, preserved normal success exactly (25.0% versus 25.0%), but tied action-only instead of beating it by 3pp. That single failed check stopped the run.
Exploratory paired forensics expose a stable opportunity rather than a missing global dose. Candidate and action-only each had eight exclusive confirmation wins, and their success union was 78.75% on both independent transfer blocks. The losses were structured: action-only won seven confirmation pattern_router rejected-patch disagreements, while the reason mixture won four rate_buckets rejected-patch disagreements. Full metrics, hashes, and the disagreement cells are committed in reports/result_receipt.json.
Interpretation
The predecessor's two apparent policy failures were indeed measurement and interface failures: a 512-token action slot removes almost all candidate truncation, and valid INSPECT→PATCH recovery transfers perfectly. But the reason mixture is not a uniformly better policy than action-only. It trades wins across held-out algorithms and cannot claim a breakthrough from aggregate dev selection.
The next intervention should exploit the replicated 78.75% union with a public verifier, not another family-specific weight mixture or posthoc router. Fork the two locality-safe policies at recovery states, execute bounded branches, select only from visible test/rejection evidence, and bank the winning trajectories with the same conditional-transition balance. It must beat a compute-matched two-trajectory baseline before any distillation or Menagerie.
Knowledgebase Update
- Program evidence: records the interface repair, independent confirm stop, and replicated candidate/action complementarity.
- Program backlog: replaces global dose tuning with verifier-selected branching and conditional winner banking.
- Claim ledger: unchanged; no Menagerie event was exposed.
Artifacts
Small receipts and final analyses are committed. Detailed trajectories live in large_artifacts/qwen35_4b_recovery_payload_budget_harness as specified by reports/artifact_manifest.yaml.
Report
Rendered from reports/report.md
Summary
Status: TRANSFER_CONFIRM_FAIL; Menagerie sealed. The payload-capable interface validated the candidate's recovery mechanism and passed development transfer, but the independent confirmation block tied rather than beat the action-only incumbent.
Research Program Fit
The predecessor located a high-success, local checkpoint but its 256-token answer slot truncated tool payloads. This follow-up resolves whether that is the remaining deployment bottleneck before spending the untouched transfer and Menagerie instruments.
Method
Candidate, base, happy, action, scaffold, and sample-more arms all use 512 thinking + 512 answer tokens/call on the same vLLM backend. A fresh locality block precedes behavior. Controls and mathematical feasibility precede the candidate on calibration and each transfer block.
Results
Fresh locality passed: centered non-target logit drift was 0.114 against the 0.15 ceiling, mean entropy changed −0.0059 nats, and mean varentropy changed −0.0105. Calibration then passed all gates at 60/60 candidate recovery versus 58/60 action-only, with no invalid actions and one answer-cap hit.
On transfer_dev, candidate recovery was 57/80 (71.25%) versus base 47/80, happy 49/80, action 53/80, matched sample-more 40/80, and scaffold 53/80. Paired bootstrap intervals versus base and sample-more were respectively [+3.75,+22.5]pp and [+8.75,+33.75]pp. Candidate and base both solved 10/40 normal tasks, with perfect verify/commit conditional on success. Every gate passed.
On independent transfer_confirm, candidate scored 55/80 (68.75%) versus base 48/80, happy 45/80, action 55/80, matched sample-more 37/80, and scaffold 51/80. The candidate improved every family versus base, retained normal success at 10/40, reached 100% valid two-turn recovery for both controlled states, and cut answer-cap hits from base's 29.0% to 7.9% of turns. It failed only the registered action-only contrast: 0.0pp versus a required +3pp. The stop label is therefore TRANSFER_CONFIRM_FAIL.
Controls
All frozen controls were rerun before the candidate on each block. Equal-reserved sample-more used two three-call trajectories versus one six-call deep loop; the explicit scaffold used the base checkpoint and identical deep budget. The candidate beat both on both transfer blocks, but did not beat action-only on confirmation.
Oracle Versus Deployable Evidence
Procedural hidden tests are host-side oracles only. Familiar-family calibration cannot establish breadth. Menagerie stays sealed until two family-held-out blocks pass. An exploratory oracle union of candidate and action-only reaches 63/80 (78.75%) on both dev and confirm, versus 57/80 and 55/80 for the candidate alone; it is not deployable evidence until a public verifier selects the branch without hidden outcomes.
Interpretation
The answer cap and immediate-only proxy were real harness bugs, not evidence against recovery learning. Repairing them reveals a local, transferable policy that strongly beats base and sample-more. The remaining failure is policy complementarity: reason mixing helps some algorithmic repairs and hurts others. The paired disagreement pattern replicates in union size across both blocks, which argues for verifier-guided branching rather than another scalar weight dose.
Next Experiments
A new result-bearing experiment should fork action-only and λ=.18 from the same public recovery state, execute bounded branches, select using only visible verifier/rejection signals, and compare against equal-compute independent sampling. Only if that capability producer transfers should its winning traces be compressed into a conditionally balanced curriculum.
Artifact Manifest
See artifact_manifest.yaml.
Experiment log 2
Show the running log (2 entries, 2026-07-12)
2026-07-12 — intake and preregistration
- Parent λ=.18 result: 58/60 recovery, 0.104 drift, but zero eligible candidates.
- Post-stop evidence fixed the intervention: 24/24 invalids hit the 256-token answer cap with closed thinking; 30/30 rejected cases changed within two turns and solved, with 20 INSPECT→PATCH paths.
- Froze candidate weights and a 512-thinking/512-answer interface for every arm.
- Froze valid rejected recovery as PATCH or INSPECT→PATCH within two turns; invalid-first paths do not qualify.
- Added a third, wholly disjoint 48-context locality block.
- No 512-answer behavior, transfer task, or Menagerie seed has been exposed.
2026-07-12 — full run and registered stop
- Fresh locality passed: drift 0.114, entropy Δ −0.0059, varentropy Δ −0.0105.
- Calibration passed at 60/60 candidate versus 58/60 action-only; all candidate conditional transitions passed and payload truncation fell to 1/219 turns.
- Transfer dev passed every gate: candidate 57/80, base 47/80, action 53/80, sample-more 40/80, scaffold 53/80; ordinary tasks were exactly retained.
- Transfer confirm candidate scored 55/80, tying action-only instead of clearing the frozen +3pp contrast. All other checks passed. Recorded
TRANSFER_CONFIRM_FAILand stopped before Menagerie. - Post-stop paired audit: candidate/action union was 63/80 on both dev and confirm. Confirm had eight exclusive wins per policy, concentrated in different family/scenario cells. Queued public-verifier branch selection, not another global interpolation dose.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
.venv/bin/python experiments/qwen35_4b_recovery_payload_budget_harness/scripts/run.py --smokeFull run
.venv/bin/python experiments/qwen35_4b_recovery_payload_budget_harness/scripts/run.py --fullRun steps are documented inside the experiment folder (README and scripts).