Locality-first recovery-reason interpolation
The one idea you need
Two fine-tuned versions share one starting checkpoint: one only learned fix-it actions, the other also learned to explain its plan first. Rather than retrain, you average their learned weight changes on a dial and search for the mix that keeps the skill and drops the mistakes.
The question
If you average two versions of a model, one that just acts and one that also explains, can you dial in a mix that fixes its mistakes without breaking unrelated behavior?
What we found
Yes on skill, no on shipping. One dial setting recovered from broken code 97% of the time, about 12 points above the act-only version and 15 above a matched-training baseline, while barely moving unrelated behavior. But it tripped two automatic release checks, so the test halted before the transfer stage. On inspection, both checks were mis-specified, not the model.
Why it matters
When two fine-tunes each get part of a job right, averaging their weights can capture both without retraining. But write release gates around the real goal, not a proxy like doing it in one step, or genuinely good models get blocked.
On this page
Results at a glance 2
hidden-test recovery success · frozen calibration arm →
Data table
| frozen calibration arm | calibration recovery success |
|---|---|
| base | 48.3% |
| happy actions | 81.7% |
| action endpoint | 85% |
| lambda .10 | 88.3% |
| lambda .18 | 96.7% |
| lambda .24 | 95% |
| lambda .30 | 88.3% |
| reason endpoint | 91.7% |
Numbers from experiments/qwen35_4b_recovery_reason_locality_interpolation/reports/result_receipt.json
share of relevant cases or steps · lambda .18 diagnostic →
Data table
| lambda .18 diagnostic | share |
|---|---|
| rejected: immediate changed patch | 33.3% |
| rejected: changed patch within 2 | 100% |
| invalid steps ending at answer cap | 100% |
Numbers from experiments/qwen35_4b_recovery_reason_locality_interpolation/reports/result_receipt.json
In the author’s words from the Overview · “Results”
Verdict: LOCAL_BUT_NO_BEHAVIOR; stopped at the frozen calibration policy gate. Transfer and Menagerie remained sealed. All four mixtures passed the original locality screen. The path was much safer than endpoint-linear prediction: drift rose only from 0.100 at λ=.10 to 0.121 at λ=.30, while the reason endpoint remained the known 0.303 failure. Entropy and varentropy also stayed inside the registered bounds. “*” Happy locality is the parent's same-method exploratory control, shown only for context. The selector admitted no candidate: every mixture exceeded the base+2pp invalid-turn ceiling and missed the registered 60% immediate rejected-patch transition bar. Full metrics and hashes are in reports/result_receipt.json.
Overview
Research Program
- Program:
agentic_breadth_installation - Program question: can a conditional recovery policy survive locality and family-held-out transfer, then improve the black-box Menagerie instrument?
- Direct predecessor:
qwen35_4b_verifier_conditioned_recovery_bank. - Prior anchors: C50 (emission-seam signal placement), C52 (exact-logit locality), C54 (weight-space interpolation can be non-convex), and C28 (reasoning helps only when its content is useful).
Question
Does the weight-space segment between the parent's locality-safe recovery-action adapter and its behaviorally stronger but non-local plan-supervised sibling contain a point that retains full action learning, repairs invalid recovery behavior, and transfers to unseen procedural coding families?
Hypothesis
The parent arms differ only in a nominal 5% plan loss over byte-identical rows, with the same base, seed, batches, and schedule. Action-only reached 85.0% trained-family recovery at 0.098 locality drift but emitted 19.1% invalid turns; the plan arm reached 91.7%, 5.9% invalid turns, and much shorter trajectories, but drifted 0.303. We hypothesize that low-dose movement from action toward reason corrects tool-call validity and conditional recovery before the broad off-policy lexical pressure crosses the 0.15 locality ceiling.
The hypothesis fails if locality-safe points remain action-like, the chosen point fails a second independent locality block, or the effect does not beat the incumbent and matched-compute controls on untouched families.
Frozen Intervention
For the two parent LoRA deltas from the same frozen C54 apex checkpoint:
W(lambda) = W_apex + (1 - lambda) * delta_action + lambda * delta_reasonThe preregistered scaled candidates are lambda = {0.10, 0.18, 0.24, 0.30}. The full action (0.0) and reason (1.0) endpoints are controls. This preserves the full useful action update while scaling only the learned action→reason contrast; it is not equivalent to retraining with a smaller plan-token weight.
The ladder was chosen before any scaled merge or evaluation. Linear interpolation of the known endpoint drifts predicts the 0.15 frontier near lambda = 0.25, so the ladder brackets rather than searches that boundary.
Setup
- Model: only
Qwen/Qwen3.5-4B, revision851bf6e..., warm-started from the frozen C54apex_replaycheckpoint. - Source data: the parent's own execution-verified, replay-minimized procedural repository trajectories. No new training occurs.
- Calibration: the already-designated 60-case block over six training families; parent base/happy/action results are checksum-frozen and reused.
- Transfer: two untouched 80-case blocks over four algorithmically different procedural families (
transfer_devseed 84800,transfer_confirmseed 84900). No fallback candidate is allowed after selection. - Controls: frozen apex, matched happy-action training, full recovery-action endpoint, explicit runtime recovery scaffold, and two shorter apex trajectories with the same reserved calls/tokens.
- Primary metrics: hidden-test repository success, rejected-patch→changed-patch and failed-test→changed-patch retention, invalid actions, normal-loop solve/verify/commit retention, and paired/family deltas.
- Hidden boundary: hidden executable code and output stay host-only. Only booleans enter receipts.
benchmarks/contents are never read or imported.
Gate Order
- Validate every parent checksum and merge all four frozen scales.
- Run exact next-token locality on the original 48 non-coding contexts for both endpoints and all scales. Record entropy and varentropy. Only passing scales may be behaviorally evaluated.
- Select once on calibration after hard gates for success, tool-call validity, and both recovery transitions. Lexicographic ties prefer fewer invalid actions and then less reason movement.
- Test only that winner on a new disjoint 48-context locality block. Failure stops the experiment; no fallback is selected.
- On each untouched transfer block, run frozen controls first and prove every registered gate mathematically reachable. The winner must beat base, happy, matched sampling, and scaffold; remain non-inferior to action-only while improving its validity or rejected-patch transition; and retain normal agent loops and at least three of four families.
- Menagerie quick/medium remains sealed until both transfer blocks pass.
Exact thresholds and immutable hashes are in configs/default.yaml; the full statistical contract is in reports/preregistration.md.
Run
CPU and invariant smoke:
.venv/bin/python experiments/qwen35_4b_recovery_reason_locality_interpolation/scripts/run.py --smokeOne-scale merge and two-context GPU integration smoke:
.venv/bin/python experiments/qwen35_4b_recovery_reason_locality_interpolation/scripts/run.py --gpu-smokeResumable staged run:
.venv/bin/python experiments/qwen35_4b_recovery_reason_locality_interpolation/scripts/run.py --fullResults
Verdict: LOCAL_BUT_NO_BEHAVIOR; stopped at the frozen calibration policy gate. Transfer and Menagerie remained sealed.
All four mixtures passed the original locality screen. The path was much safer than endpoint-linear prediction: drift rose only from 0.100 at λ=.10 to 0.121 at λ=.30, while the reason endpoint remained the known 0.303 failure. Entropy and varentropy also stayed inside the registered bounds.
| Arm | Recovery | Failed test | Rejected patch | Invalid/turn | Rejected immediate change | Drift |
|---|---|---|---|---|---|---|
| base | 48.3% | 53.3% | 43.3% | 5.7% | 0.0% | — |
| happy action | 81.7% | 80.0% | 83.3% | 2.0% | 33.3% | 0.083* |
| action endpoint | 85.0% | 73.3% | 96.7% | 19.1% | 16.7% | 0.098 |
| λ=.10 | 88.3% | 76.7% | 100% | 11.6% | 16.7% | 0.100 |
| λ=.18 | 96.7% | 93.3% | 100% | 10.4% | 33.3% | 0.104 |
| λ=.24 | 95.0% | 90.0% | 100% | 10.2% | 33.3% | 0.111 |
| λ=.30 | 88.3% | 76.7% | 100% | 10.0% | 33.3% | 0.121 |
| reason endpoint | 91.7% | 100% | 83.3% | 5.9% | 83.3% | 0.303 |
* Happy locality is the parent's same-method exploratory control, shown only for context. The selector admitted no candidate: every mixture exceeded the base+2pp invalid-turn ceiling and missed the registered 60% immediate rejected-patch transition bar. Full metrics and hashes are in reports/result_receipt.json.
Interpretation
The weight direction contains a real, locality-safe behavioral optimum: λ=.18 adds +11.7pp over action-only and +15.0pp over happy on the selection block. It does not yet qualify as deployable capability because the registered policy guards failed.
Exploratory failure forensics change the next intervention. All 24 “invalid” actions ended exactly at the 256-answer-token cap with thinking already closed; they are truncated long patch JSON, not free-form slop. Eleven trajectories had an invalid turn and nine still solved. Likewise, all 30 rejected-patch cases made a changed patch within two turns: 20 sensibly used INSPECT→PATCH, while only ten used immediate PATCH→VERIFY. Thus one failed gate detects a genuine tool-payload budget bottleneck, while the other over-specifies immediacy instead of the requested conditional transition.
The next experiment should freeze λ=.18, give every arm a realistic larger answer payload under matched compute, and gate rejected recovery by changed patch within two turns. That is a new harness experiment, not a reinterpretation of this stopped result.
Knowledgebase Update
- Program evidence: updated with the safe interpolation curve and calibration stop.
- Program backlog: queues a λ=.18 matched-compute answer-budget harness successor with semantically faithful two-turn transition retention.
- Claim ledger/synthesis: deferred; no transfer family or Menagerie seed was exposed.
Artifacts
Small receipts and final analysis are committed. Merged 4B checkpoints, trajectories, and detailed logits live under large_artifacts/qwen35_4b_recovery_reason_locality_interpolation and are tracked by reports/artifact_manifest.yaml.
Report
Rendered from reports/report.md
Summary
LOCAL_BUT_NO_BEHAVIOR: stopped at the frozen calibration policy gate. Every action→reason mixture passed exact-logit/entropy locality, and λ=.18 reached 96.7% trained-family recovery, but no point met the invalid-turn and immediate rejected-patch transition guards. Independent locality confirmation, held-out families, and Menagerie were never opened.
Research Program Fit
The parent found a useful action endpoint (85.0% recovery, 0.098 drift) and a behaviorally stronger but non-local reason endpoint (91.7%, 0.303). This follow-up resolved whether the weight segment contains a locality-safe useful region before spending any fresh family or benchmark evidence.
Method
Both parent LoRA deltas were reconstructed per module in float32. Four frozen points used delta_action + λ(delta_reason − delta_action) for λ=.10/.18/.24/.30, were added once to the common apex checkpoint, and cast once to bfloat16. Full action and reason endpoints anchored the screen.
The full ladder first faced 48 unrelated contexts with median centered non-target logit drift ≤0.15 and mean entropy change ≥−0.05. Only passing points then ran the checksum-frozen 60-case training-family recovery block. Eligibility also required base-level tool validity, immediate rejected-patch change ≥0.60, and failed-test changed patch within two turns ≥0.60.
Results
Interpolation geometry
The endpoint contrast is strongly cancelling, not a scalar dose direction. Summed mixed-delta norms fell from 29.17 at action to 26.57/.24.94/.24.05/.23.47 at the four points, even while behavior changed sharply. All 128 adapted modules were covered and every output hash was recorded.
| Point | Drift | Entropy Δ | Varentropy Δ | Locality |
|---|---|---|---|---|
| action | 0.0982 | +0.0060 | −0.0204 | pass |
| λ=.10 | 0.0999 | −0.0107 | −0.0171 | pass |
| λ=.18 | 0.1039 | −0.0212 | −0.0185 | pass |
| λ=.24 | 0.1107 | −0.0267 | −0.0196 | pass |
| λ=.30 | 0.1207 | −0.0412 | −0.0232 | pass |
| reason | 0.3031 | −0.1058 | −0.0139 | fail |
The safe region extends through λ=.30, far beyond the endpoint-linear prediction. Because selection failed later, the independent confirmation block correctly remained unused.
Calibration behavior
| Arm | Overall | Failed | Rejected | Invalid/turn | Rejected immediate | Failed changed≤2 |
|---|---|---|---|---|---|---|
| base | .483 | .533 | .433 | .057 | .000 | .400 |
| happy | .817 | .800 | .833 | .020 | .333 | .967 |
| action | .850 | .733 | .967 | .191 | .167 | .833 |
| λ=.10 | .883 | .767 | 1.000 | .116 | .167 | .833 |
| λ=.18 | .967 | .933 | 1.000 | .104 | .333 | .967 |
| λ=.24 | .950 | .900 | 1.000 | .102 | .333 | .833 |
| λ=.30 | .883 | .767 | 1.000 | .100 | .333 | .867 |
| reason | .917 | 1.000 | .833 | .059 | .833 | 1.000 |
λ=.18 was a large behavior optimum: +.483 over base, +.150 over happy, and +.117 over action. Yet every candidate failed two frozen checks. Invalid turns had to be ≤.077 (base +.02), and immediate rejected change had to be ≥.60. No candidate was selected, so the registered run stopped.
Failure Forensics (Exploratory)
The stop is valid, but the two failed metrics have different meanings:
- All 24 invalid steps at λ=.18 had closed thinking and consumed exactly all 256 answer tokens. Every parse status was
no_json_tool_call_in_answer; raw tails ended inside long exact-replacement JSON payloads. Nine were also force-closed at the 512-token thinking limit. This is a real deployability failure under the registered harness, caused by payload truncation rather than wandering prose. - Invalids occurred in 11 trajectories, nine of which still ended with a correct workspace. They waste turns and cause two failures, but do not imply absence of the repair capability.
- All 30 rejected-patch cases made a changed patch within two generated turns and all 30 solved. Twenty used the sensible
INSPECT→PATCHsequence; ten usedPATCH→VERIFY. The immediate-only metric therefore rejects successful, context-seeking recovery even though the requested conditional transition is retained within one intervening inspection.
These diagnostics were computed only after the registered stop and cannot rescue this experiment.
Controls
The full action endpoint proves the gain is not merely conditional action training: λ=.18 adds 11.7 points. The happy arm proves it also exceeds generic balanced action SFT by 15.0 points. The reason endpoint verifies the known locality failure exactly. Matched sampling and external scaffold controls were not funded because no candidate cleared the prerequisite policy gate.
Oracle Versus Deployable Evidence
Hidden tests remain host-only and support calibration performance only. No unseen-family task was generated or evaluated in the result-bearing stages, and no Menagerie seed was assigned. This is a mechanism result, not a breadth or black-box capability claim.
Interpretation
The parent trade-off is partially separable: locality-safe interpolation can outperform both endpoints on recovery. The remaining obstacle is now concrete and harness-facing—long exact-replacement actions do not fit a 256-token answer slot—rather than generic collateral or lack of repair knowledge.
The immediate-transition guard also encoded the wrong behavioral preference. After rejection, re-reading the changed file before patching is defensible; the faithful metric is changed patch within two turns, while still measuring the intervening operator and final success.
Next Experiment
Create a separate harness experiment that freezes λ=.18, increases tool-answer payload capacity for every arm under matched total compute, and records rejected-patch changed-within-two as the primary transition. It must retain the existing locality receipt, compare all controls under the same budget/interface, and still pass two untouched family blocks before Menagerie. Do not alter this experiment's thresholds or run its transfer seeds.
Artifact Manifest
Large checkpoints and trajectories remain under large_artifacts/qwen35_4b_recovery_reason_locality_interpolation. The committed result_receipt.json contains compact metrics, failure forensics, weight hashes, and source checksums.
Experiment log 2
Show the running log (2 entries, 2026-07-12)
2026-07-12 — intake and preregistration
- Direct parent stopped at its registered locality gate: action-only drift 0.098, reason drift 0.303, with 85.0% versus 91.7% trained-family recovery.
- Chose action-anchored contrast interpolation over scaling the full reason delta, because weakening the known-local action signal would answer a less useful question.
- Froze lambdas 0.10/0.18/0.24/0.30 from the endpoint-implied locality frontier near 0.25; no scaled checkpoint had been merged or evaluated.
- Added a disjoint locality-confirmation block, hard validity/transition gates, action-endpoint comparison, transfer feasibility receipts, and no-fallback selection.
- Menagerie and both transfer blocks remain unexposed.
2026-07-12 — frozen run
- All four mixtures passed locality; drift was 0.100/0.104/0.111/0.121 and entropy/varentropy remained inside the registered bounds. Full reason reproduced the 0.303 failure.
- λ=.18 peaked at 58/60 recovery (96.7%), versus 29/60 base, 49/60 happy, 51/60 action, and 55/60 full reason.
- The selector admitted no candidate. Every point exceeded base+2pp invalid turns and missed the 60% immediate rejected-patch change gate. Run stopped before confirmation, transfer, scaffold/sampling controls, and Menagerie.
- Post-stop forensics: all 24 invalid steps exhausted exactly 256 answer tokens inside a long JSON patch payload; all had already closed thinking. All 30 rejected cases changed the patch within two turns and solved; 20 used INSPECT→PATCH and 10 PATCH→VERIFY.
- Next strategy: a new λ=.18 harness experiment with realistic tool-payload capacity under matched compute and changed-patch-within-two retention.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
.venv/bin/python experiments/qwen35_4b_recovery_reason_locality_interpolation/scripts/run.py --smokeFull run
.venv/bin/python experiments/qwen35_4b_recovery_reason_locality_interpolation/scripts/run.py --fullRun steps are documented inside the experiment folder (README and scripts).