Research log Small Model Experimentation
GitHub

State-Formation Branch Handoff Recovery

Safe handoff is working

The one idea you need

A retired error slot is like a reserved parking space: once a verified replacement arrives, later checks must recognize the new vehicle without forgetting the old incident.

The question

Can later full-size setup work continue after a successful replacement occupies the file location that previously held a preserved failure?

What we found

Yes, the narrow handoff passed every safety check, then all three full-size setup controls reached 48 of 48 with their learned path and zero of 48 with that path disabled.

Why it matters

Practitioners can continue the promised compact-versus-full-size comparison without rewriting published history or weakening the original safety checks. Later work still waits for each completed setup checkpoint to be published.

Setup seeds complete3 of 3full-size setup pairs passed
Latest learned path intact48 of 48setup cases reached the complete state
Latest learned path disabled0of 48 setup cases reached the complete state
Training presentations4,096exact fixed total per seed
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Status
    2. Method
    3. Results
    4. Interpretation
    5. Next action
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 1

Every handoff lineage check passed

How to read

The six horizontal groups cover the false rejection, retirement, failure archive, missing retired copy, successful setup record, and invocation history. Bar height is the pass indicator; higher is better.

00.250.50.751false rejection reproducedfalse rejection reproduced1retirement exactretirement exact1failure archive exactfailure archive exact1retired mirror absentretired mirror absent1successful G0 exactsuccessful G0 exact1G0 invocation exactG0 invocation exact1

Takeaway → All six bars reach the pass mark before model work, showing the handoff recognizes the verified replacement without losing failure safeguards.

Data table
frozen handoff controlcontrol pass
false rejection reproduced1
retirement exact1
failure archive exact1
retired mirror absent1
successful G0 exact1
G0 invocation exact1

Numbers from experiments/qwen35_4b_state_formation_branch_handoff_recovery/runs/smoke.json

Technical framing

All six handoff lineage controls pass before downstream model work — Operational evidence only. Smoke loads no model, starts no training/evaluation, and opens no benchmark or sealed contrast.

In the author’s words from the Overview · “Results”

Unit controls additionally reject exact failed bytes at the successful slot, any changed successful-G0 bytes, a reappearing mirror, and unregistered invocation shapes. Smoke loaded no model, started no training/evaluation, opened no benchmark or contrast path, and changed no scientific interpretation. No positive-control or downstream scientific result is claimed here. At that checkpoint, use remained blocked until publication and both workflows were green; the later paragraphs record that completed progression. Handoff commit 627254f1…d892 passed both workflows. The exact full-rank seed-7411 positive control then completed once through this wrapper. … Read the full result →

Overview

This operational successor lets immutable producer v11 continue after the first branch recovery successfully replaces the retired failed-G0 receipt at its canonical path. It changes no scientific arm, model, data, metric, threshold, or branch rule.

The handoff question is answered and the setup matrix has passed. Continued Stage-B training through this frozen wrapper belongs to the canonical producer and does not make this recovery experiment active.

Research Program

Question

Can downstream producer stages continue without weakening failure preservation once a successful recovered G0 legitimately occupies the canonical pathname formerly used by the archived failure?

Hypothesis

The defect is limited to invocation orchestration: the first wrapper asks whether the pathname exists, not whether it still contains the exact retired failure. A new wrapper can reuse its frozen, tested path seam while requiring the archived failure and retirement receipts, rejecting a reappearing mirror or exact failed bytes, and accepting only the exact successful G0 plus its STARTED/COMPLETE lineage.

Setup

  • Model-bearing stages remain only Qwen/Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, on producer v11's Transformers backend.
  • Smoke loads no model and runs no training or evaluation.
  • Successful G0: file SHA-256 cdc90cd…c68f, identity e1f1c906…f89dc, status MODEL_SMOKE_PASS, full-rank joint seed 7411.
  • Retired failure: SHA-256 47305826…2c71; it must remain in the first recovery archive and must not reappear at either producer failure path.
  • First recovery: source contract 55d0a455…56f3, smoke SHA-256 8bf5bb36…6849.
  • Allowed stages and argument shapes remain the first recovery's exact registered set. The producer CLI retains canonical path, source snapshot, authorization, setup/control, training, evaluation, lock, and receipt gates.
  • Hidden-label boundary: smoke opens no contrast or benchmark payload. The wrapper cannot analyze.

Recovery boundary

The wrapper imports the exact frozen first-recovery source and reuses its safe reads, immutable writes, producer-context checks, argument-shape policy, stage-specific output validation, and src.analysis._canonical_expected_path context manager. It does not call the first wrapper's pathname-only invoke_producer; it owns new immutable STARTED/COMPLETE receipts and directly calls the same exact producer CLI under the same path seam.

Every invocation first rehashes the retirement, archived failure, successful G0, first-recovery G0 STARTED/COMPLETE receipts, authorization, first-recovery smoke/source, producer source/model, and the handoff's own frozen smoke/source. A producer output cannot be adopted unless its handoff STARTED receipt already exists.

Run

Freeze and smoke without loading the model:

.venv/bin/python -B -m unittest discover \
  -s experiments/qwen35_4b_state_formation_branch_handoff_recovery/tests -v
.venv/bin/python -B \
  experiments/qwen35_4b_state_formation_branch_handoff_recovery/scripts/run.py --smoke

Commit and push the successful G0, its first-recovery invocation receipts, and this frozen handoff; require both repository workflows green. Then retry the seed-7411 positive control through this wrapper. Every later branch-authorized model stage uses the same handoff wrapper; producer analysis continues through the already-frozen analysis recovery.

Results

The focused suite passes 8/8. The frozen smoke emitted BRANCH_HANDOFF_RECOVERY_SMOKE_PASS at file SHA-256 0e0409de…6e5b, receipt identity d6fecf0a…de4c, and source contract 4d2ffde3…8acd. All six controls pass: the first wrapper's pathname-only false rejection is reproduced; retirement, failure archive, successful G0, and its invocation lineage are exact; and the retired mirror remains absent. Unit controls additionally reject exact failed bytes at the successful slot, any changed successful-G0 bytes, a reappearing mirror, and unregistered invocation shapes.

Smoke loaded no model, started no training/evaluation, opened no benchmark or contrast path, and changed no scientific interpretation. No positive-control or downstream scientific result is claimed here. At that checkpoint, use remained blocked until publication and both workflows were green; the later paragraphs record that completed progression.

Handoff commit 627254f1…d892 passed both workflows. The exact full-rank seed-7411 positive control then completed once through this wrapper. Producer file SHA-256 is a0d17e2e…7a16, producer receipt identity 6708a4d4…9649, and handoff COMPLETE identity af0dd15c…2262. The producer control passed oracle and intact joint state at 48/48, adaptation-disabled joint state at 0/48, exactly 256 updates, gradient accumulation 16, and 4,096 singleton presentations. Both the 892,272,640-parameter full- rank path and shared-state parameters changed; early stopping and checkpoint selection were false. The receipt authorizes later producer work only through its unchanged barriers and records zero result payload, benchmark, contrast, or scientific-evidence access.

After seed-7411 setup commit c507488c…3c06 passed both workflows, seed 7412 independently passed the same path. Its G0 file SHA-256 / identity are 10bf22fc…1d18 / 62ecb79e…951dc: zero K=1 and checkpoint-roundtrip error, all 62 full-rank deltas with complete finite optimizer state, all joint trainable groups finite/nonzero, finite K=12, and 22.1 GiB free. Its positive-control file SHA-256 / identity are 1cbbd823…8510 / 6575e1d2…4554: oracle/intact 48/48 and disabled 0/48 after exact 256 updates, accumulation 16, and 4,096 presentations. Handoff G0/control COMPLETE identities are d61d6441…9246 and 6334d72f…c14a. Both receipts retain zero result, benchmark, contrast, and scientific-evidence access.

After seed-7412 setup commit 92473a53…8530 passed both workflows, seed 7413 completed the matrix. Its G0 file SHA-256 / identity are 021a8444…d635 / 4d2316d3…2ff0: zero K=1 and round-trip error, finite/nonzero joint gradients, complete finite optimizer state for all 62 deltas, finite K=12, and 22.1 GiB free. Its control file SHA-256 / identity are 8a4af0d6…fde8 / 976f28ef…14df: oracle/intact 48/48 and disabled 0/48 after exact 256 updates, accumulation 16, and 4,096 presentations, with changed full-rank/shared state. Handoff G0/control COMPLETE identities are fbfb282f…5aef / ccff1520…58c9. All three pairs now pass, remain setup-only, and open no result, benchmark, contrast, or scientific evidence.

Interpretation

The three valid G0/control pairs establish full-rank feasibility and a working state path at every seed, not a capacity result. This recovery safely handed producer authority through setup without changing scientific logic. Capacity interpretation remains unchanged.

Knowledgebase Update

  • Program evidence: no scientific evidence from an invocation repair.
  • Program backlog: producer Stage B now owns the mandatory LoRA-state-only and full-rank-joint work.
  • Claim ledger: unchanged.

Artifacts

  • idea_intake.md: nearest-neighbor and novelty decision.
  • reports/design_review.md: adversarial review and stop conditions.
  • configs/default.yaml: exact producer, recovery, failure, G0, and authorization pins.
  • runs/smoke.json: frozen no-model handoff smoke.
  • runs/invocations/: handoff-owned immutable producer invocation provenance.
  • reports/artifact_manifest.yaml: producer-owned large-artifact boundary.

Report

Rendered from reports/report.md

Status

Finished. The first recovery produced a valid full-rank seed-7411 G0, but its frozen pathname-only retirement guard stopped the following positive control before producer or model work. This handoff then carried all three full-rank G0/control pairs through the unchanged producer barriers. Producer receipts remain scientifically authoritative.

Method

Pin the archived failure, retirement, successful G0, first-recovery G0 invocation lineage, first- recovery source/smoke, producer source/model, and authorization. Reuse the exact frozen path seam while replacing only outer invocation orchestration and emitting handoff-owned STARTED/COMPLETE receipts.

Results

  • Focused tests: 8/8.
  • Smoke status: BRANCH_HANDOFF_RECOVERY_SMOKE_PASS.
  • Smoke file SHA-256 / identity: 0e0409de…6e5b / d6fecf0a…de4c.
  • Frozen source contract: 4d2ffde3…8acd.
  • Six smoke controls pass; unit controls reject failed bytes, changed G0 bytes, a reappearing mirror, and unregistered invocation shapes.
  • Model load, training/evaluation, benchmark/contrast access, and scientific interpretation: zero.
  • Full-rank seed-7411 positive control: producer SHA-256 a0d17e2e…7a16, identity 6708a4d4…9649; oracle/intact 48/48, adaptation-disabled 0/48, 256 updates, accumulation 16, 4,096 presentations, and changed full-rank/shared-state parameters.
  • Handoff COMPLETE identity: af0dd15c…2262; producer result/benchmark/contrast access and scientific evidence remain zero.
  • Full-rank seed-7412 G0: file SHA-256 10bf22fc…1d18, identity 62ecb79e…951dc; zero K=1/ round-trip error, complete finite optimizer state for all 62 deltas, finite K=12, 22.1 GiB free.
  • Full-rank seed-7412 control: file SHA-256 1cbbd823…8510, identity 6575e1d2…4554; oracle/intact 48/48, disabled 0/48, 256 updates, accumulation 16, 4,096 presentations, changed full-rank/shared state, and zero result/benchmark/contrast/scientific access.
  • Full-rank seed-7413 G0: file SHA-256 021a8444…d635, identity 4d2316d3…2ff0; zero K=1/ round-trip error, complete finite optimizer state for all 62 deltas, finite K=12, 22.1 GiB free.
  • Full-rank seed-7413 control: file SHA-256 8a4af0d6…fde8, identity 976f28ef…14df; oracle/intact 48/48, disabled 0/48, 256 updates, accumulation 16, 4,096 presentations, changed full-rank/shared state, and zero result/benchmark/contrast/scientific access.
  • Complete matrix: all three independent full-rank G0/control pairs pass; this remains setup-only.

Interpretation

No capacity conclusion changes. A passing recovery licenses only the preregistered producer stage that its inputs independently authorize.

Next action

No recovery action remains. The complete setup matrix was published and the producer's Stage-B cells are tracked in qwen35_4b_state_formation_capacity_adjudication; their scientific barriers remain unchanged.

Experiment log 7

Show the running log (7 entries, 2026-07-14)

2026-07-14 — handoff defect isolated

  • First recovered full-rank seed-7411 G0 passed and occupied its canonical producer setup path.
  • The following positive-control command stopped before wrapper STARTED publication because the first frozen wrapper treated any occupant of that pathname as the retired failure.
  • No positive-control producer entry, model load, training/evaluation, benchmark access, contrast access, or scientific evidence occurred.
  • Opened this additive operational successor rather than editing the frozen first recovery.

2026-07-14 — design review

  • Frozen boundary requires exact archived-failure and retirement lineage, exact successful G0 and first-recovery invocation lineage, absence of the retired mirror, inherited stage shapes, exact producer validation, and new immutable STARTED/COMPLETE receipts.
  • Next: focused tests, no-model smoke, commit/push, and both workflows green before model work.

2026-07-14 — frozen handoff smoke passes

  • Focused tests pass 8/8, including exact failed-byte, changed-G0, mirror-reappearance, original- false-rejection, and inherited-shape controls.
  • Smoke file SHA-256 is 0e0409de…6e5b, receipt identity d6fecf0a…de4c, under frozen source contract 4d2ffde3…8acd; all six registered lineage/occupancy controls pass.
  • Smoke loaded no model, started no training/evaluation, opened no benchmark or contrast path, and changed no scientific interpretation.
  • Current authorization: publish this complete checkpoint with the successful G0 and its first- recovery invocation receipts. No downstream model stage may begin until both workflows are green.

2026-07-14 — full-rank seed-7411 positive control passes

  • Handoff commit 627254f1…d892 passed Validate Repository and Publish Research Site before use.
  • Producer control file SHA-256 is a0d17e2e…7a16, identity 6708a4d4…9649; handoff STARTED and COMPLETE file hashes are b1b91cb2…ff2b and e0887536…13f1, with COMPLETE identity af0dd15c…2262.
  • Oracle and intact fixed-final joint state are 48/48; adaptation-disabled joint state is 0/48 after exactly 256 optimizer updates, accumulation 16, and 4,096 singleton presentations. Both the full- rank path and shared state changed; no early stop or checkpoint selection occurred.
  • The receipt opens no result payload, benchmark, or contrast and claims no scientific evidence.
  • Current authorization: publish this setup checkpoint and require both workflows green before the full-rank seed-7412 G0/control pair. Result training remains blocked on the complete setup matrix.

2026-07-14 — full-rank seed-7412 G0/control pair passes

  • Seed-7411 setup commit c507488c…3c06 passed both workflows before seed 7412 began.
  • G0 file SHA-256 is 10bf22fc…1d18, identity 62ecb79e…951dc; handoff G0 STARTED/COMPLETE file hashes are 01099e34…aec3 / 0b48b1e5…7d35, COMPLETE identity d61d6441…9246.
  • G0 has zero K=1 and checkpoint-roundtrip error, all 62 deltas have complete finite optimizer state, all joint trainable groups have finite nonzero gradients, K=12 is finite, and 22.1 GiB remains.
  • Control file SHA-256 is 1cbbd823…8510, identity 6575e1d2…4554; handoff STARTED/COMPLETE file hashes are ffc30b05…d55c / 9535f515…f3a2, COMPLETE identity 6334d72f…c14a.
  • Oracle/intact are 48/48 and disabled is 0/48 after exact 256 updates, accumulation 16, and 4,096 presentations. Full-rank/shared-state parameters changed; no early stop or checkpoint selection.
  • Current authorization: publish this setup checkpoint and require both workflows green before seed 7413. Result training remains blocked on the complete setup matrix.

2026-07-14 — full-rank seed-7413 pair passes; setup matrix complete

  • Seed-7412 setup commit 92473a53…8530 passed both workflows before seed 7413 began.
  • G0 file SHA-256 / identity are 021a8444…d635 / 4d2316d3…2ff0; handoff G0 STARTED/COMPLETE file hashes are a38ebcba…38ed / 11229858…cea3, COMPLETE identity fbfb282f…5aef.
  • G0 has zero K=1/round-trip error, finite/nonzero joint gradients, complete finite optimizer state for all 62 deltas, finite K=12, and 22.1 GiB free.
  • Control file SHA-256 / identity are 8a4af0d6…fde8 / 976f28ef…14df; handoff control STARTED/ COMPLETE file hashes are d5dbe18e…afa5 / 8cd6ff71…3c89, COMPLETE identity ccff1520…58c9.
  • Oracle/intact are 48/48 and disabled is 0/48 after exact 256 updates, accumulation 16, and 4,096 presentations. Full-rank/shared-state parameters changed; no early stop or checkpoint selection.
  • All three full-rank setup pairs now pass with zero result, benchmark, contrast, or scientific evidence. Publish/green the complete matrix before any Stage-B training cell.

2026-07-14 — handoff scope closed

  • The branch handoff and all three full-rank G0/control setup pairs completed, the complete matrix was published, both workflows passed, and the producer subsequently launched its first mandatory Stage-B cell. This answers the recovery experiment's operational question.
  • Stage-B execution belongs to the canonical capacity-adjudication producer. Continued use of the frozen wrapper is a producer dependency, not unfinished recovery work.

Reproduce

Smoke test

.venv/bin/python -B experiments/qwen35_4b_state_formation_branch_handoff_recovery/scripts/run.py --smoke

Full run

run only an exact branch-authorized producer command documented in the producer GPU runbook through scripts/run.py

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗