Research log Small Model Experimentation
GitHub

State-Formation Branch Authorization Recovery

Finished2026-07-14Benchmark GeneralizationPosttraining and AdaptationStructured Execution and CompilersGitHub ↗
Retry path passed its safety checks

The one idea you need

A sealed calculator can produce the right decision but still have a plug that downstream equipment cannot use. This recovery repairs that one plug while keeping the calculator and decision unchanged.

The question

Can the frozen training runner verify the existing continue-to-controls decision after its strict file-path check blocked the first attempt?

What we found

Yes for the no-model safety check: all six controls passed, the failed first attempt has a third preserved copy, and the two retry-blocking source paths were retired only after that archive commit passed both checks. The actual full-size model retry still waits for the retirement commit to pass.

Why it matters

This keeps the promised compact-versus-full-size comparison moving without silently changing its model, data, success rule, or training plan, while preserving the failed attempt before retry.

Safety checks6 of 6downstream path and restoration controls passed
Model retries0retry remains blocked on publication
Preserved copies3canonical, mirror, and recovery archive
Sealed cases opened0future comparison rows remain closed
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Status
    2. Method
    3. Current evidence
    4. Interpretation
    5. Next action
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 1

Every downstream recovery safety check passed

How to read

The six groups cover reproducing the defect, recovering the authorization, preserving canonical behavior, rejecting two unsafe path forms, and restoring the original helper. Bar height is the pass indicator.

00.250.50.751defect reproduceddefect reproduced1authorization recoveredauthorization recovered1canonical equivalencecanonical equivalence1unrelated alias rejectedunrelated alias rejected1prefix traversal rejectedprefix traversal rejected1helper restoredhelper restored1

Takeaway → All six bars reach the pass mark before any model retry, so the continuation path is narrow and testable rather than an unreviewed bypass.

Data table
frozen downstream recovery controlcontrol pass
defect reproduced1
authorization recovered1
canonical equivalence1
unrelated alias rejected1
prefix traversal rejected1
helper restored1

Numbers from experiments/qwen35_4b_state_formation_branch_recovery/runs/smoke.json

Technical framing

All six downstream recovery controls pass before model retry — Operational evidence only. Smoke loads no model, starts no training/evaluation, and opens no benchmark or sealed contrast.

In the author’s words from the Overview · “Results”

The recovered producer G0 then passed at file SHA-256 cdc90cd…c68f, identity e1f1c906…f89dc, with exact K=1 parity before/after optimizer, zero checkpoint-roundtrip error, all 62 direct-full-shape delta tensors covered by finite nonzero joint gradients and complete finite optimizer state, finite K=12, and about 22.1 GiB free after G0. It authorizes only the positive control and records zero training/evaluation, benchmark, or contrast access. Recovery STARTED/COMPLETE SHA-256 values are f0c78055…3c87 and 755ad561…f941. … Read the full result →

Overview

This operational successor lets immutable source-v11 branch consumers reopen the authoritative LoRA-miss receipt through the same exact registered-prefix seam already validated for analysis. It does not alter producer science.

Research Program

Question

Can the unchanged producer v11 CLI consume its exact LORA_JOINT_MISS_CONTROLS_REQUIRED receipt and run the already-mandated Stage-B cells after the same registered ../../large_artifacts/... prefix caused downstream branch validation to fail before model load?

Hypothesis

The failure is the same one-function path seam, now reached while src.gate_receipts recomputes branch evidence. Importing the exact producer CLI and temporarily replacing only the exact src.analysis._canonical_expected_path object it uses should preserve every canonical-path, source-snapshot, authorization, model, data, training, evaluation, and branch gate while making the registered external prefix consumable.

Setup

  • Model: only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a; smoke/archive/retirement load no model.
  • Backend: producer's exact frozen transformers backend.
  • Scientific substrate: no new data or arm. The producer retains all rows, splits, fixed seeds, checkpoints, metrics, thresholds, and branch rules.
  • Authorization: producer receipt SHA-256 cb9fee75…818a, identity b973bc01…a862, exact next stage run_lora_state_only_and_fullrank_joint.
  • Observed failure: source-v11 full-rank seed-7411 G0 wrote SETUP_CONTROL_FAILED at branch_authorization, file SHA-256 47305826…2c71, identity 070c23af…aa24, with no completed checks, model load, training/evaluation, benchmark, sealed contrast, or scientific authority.
  • Primary operational gate: BRANCH_RECOVERY_SMOKE_PASS from the real downstream _authorization_for consumer under the narrow seam.
  • Controls: original defect reproduction, recovered authorization, raw/canonical equivalence, unrelated-alias rejection, prefix-traversal rejection, exceptional restoration, exact source and receipt pins, immutable invocation STARTED/COMPLETE lineage.
  • Hidden-label boundary: smoke recomputes only the already-open trigger-side LoRA-miss evidence. No sealed contrast, benchmark path, new metric, or new interpretation is opened.

Recovery boundary

The wrapper imports exact producer scripts/run.py and the top-level src package it actually uses. All producer CLI validation and its reviewed source snapshot stay active. The temporary seam accepts only the exact registered raw external prefix and clean descendants, normalizes once, proves the result remains under the canonical registered prefix, then calls v11's unchanged repository/no- symlink validator. Every other path calls the original helper. The helper is restored in finally.

Only model-smoke, positive-control, train, and evaluate-state may pass through this wrapper, and every invocation must supply an authorization receipt and explicit output. The original CLI still decides whether that exact purpose/capacity/objective/eval-set path is licensed.

Run

Freeze mechanics and validate the already-produced failure without loading the model:

.venv/bin/python -B -m unittest discover \
  -s experiments/qwen35_4b_state_formation_branch_recovery/tests -v
.venv/bin/python -B \
  experiments/qwen35_4b_state_formation_branch_recovery/scripts/run.py --smoke
.venv/bin/python -B \
  experiments/qwen35_4b_state_formation_branch_recovery/scripts/run.py --archive-failure

Commit and push the recovery, smoke, archive, and both original failure receipts; require Validate Repository and Publish Research Site green. Then retire only the two original failed-G0 paths using that full commit SHA, commit/push the retirement, and again require both workflows green:

.venv/bin/python -B \
  experiments/qwen35_4b_state_formation_branch_recovery/scripts/run.py \
  --retire-failure --archive-commit <full-green-archive-commit>

The first recovered producer retry is then the same registered full-rank seed-7411 G0, with the wrapper as the only changed runtime boundary:

.venv/bin/python -B \
  experiments/qwen35_4b_state_formation_branch_recovery/scripts/run.py \
  --stage model-smoke --capacity fullrank --objective joint --seed 7411 \
  --initialization-bundle large_artifacts/qwen35_4b_state_formation_capacity_adjudication/initialization_seed7411.pt \
  --authorization-receipt experiments/qwen35_4b_state_formation_capacity_adjudication/analysis/lora_joint_trigger.json \
  --output experiments/qwen35_4b_state_formation_capacity_adjudication/runs/setup/g0_fullrank_seed7411.json

Every later branch-authorized producer model stage uses the same wrapper; analysis phases continue through the already-frozen analysis recovery.

Results

The focused suite passes 14/14. The exact downstream smoke emitted BRANCH_RECOVERY_SMOKE_PASS at file SHA-256 8bf5bb36…6849, receipt identity d1135ea2…49b5, and frozen recovery source contract 55d0a455…56f3. Its six controls all pass: original downstream defect reproduction, recovered authorization, canonical equivalence, unrelated-alias rejection, prefix-traversal rejection, and helper restoration. It binds the exact producer authorization/failure identities and records no model load, training/evaluation, benchmark path, sealed contrast row, or scientific-interpretation change.

The failure archive then copied the exact 47305826…2c71 bytes to an inode-distinct recovery-owned path and emitted BRANCH_AUTHORIZATION_FAILURE_ARCHIVED at archive-receipt SHA-256 4fcccea3…45ed, identity ff478d40…0ec3. The original canonical/mirror paths remain present, byte-identical, and inode-distinct. The archive correctly authorizes no retry until this whole checkpoint is committed, pushed, and green.

Archive commit bdedabf4ea7d153d21a9b3a0cbe03cad66c8b6b2 passed both workflows. The retirement transaction re-read all four exact Git blobs from that commit, wrote a STARTED receipt, unlinked only the producer canonical/mirror failure paths while holding their verified inodes, and emitted BRANCH_AUTHORIZATION_FAILURE_RETIRED. Retirement-receipt file SHA-256 is 6e4c8ee3…53ad, identity c9abdc59…eae7; the archived failure remains unchanged. It records no model load, training/evaluation, benchmark, or sealed contrast and authorizes recovered producer retry only after this retirement checkpoint is itself published and green.

Retirement commit e69c1960…c79 passed both workflows. The recovered producer G0 then passed at file SHA-256 cdc90cd…c68f, identity e1f1c906…f89dc, with exact K=1 parity before/after optimizer, zero checkpoint-roundtrip error, all 62 direct-full-shape delta tensors covered by finite nonzero joint gradients and complete finite optimizer state, finite K=12, and about 22.1 GiB free after G0. It authorizes only the positive control and records zero training/evaluation, benchmark, or contrast access. Recovery STARTED/COMPLETE SHA-256 values are f0c78055…3c87 and 755ad561…f941.

The next positive-control invocation exposed a frozen orchestration defect before STARTED publication or producer/model work: this wrapper's retirement guard rejects any occupant of the former failure pathname, including the exact successful G0 that producer v11 must place there. The additive qwen35_4b_state_formation_branch_handoff_recovery successor preserves this experiment, pins the exact success/failure/retirement lineage, and passes its no-model smoke. Later stages must use that successor after its publication checkpoint is green.

Interpretation

This is a continuation mechanism, not a LoRA/full-rank capacity result. It successfully established the exact path seam and produced an authoritative full-rank G0, but its pathname-only retirement guard is not a reusable downstream handoff. The producer's G0 remains authoritative; later stages move to the additive handoff successor without editing this frozen result-bearing recovery.

Knowledgebase Update

  • Program evidence: no scientific evidence from recovery alone.
  • Program backlog: Stage B remains mandatory, gated on operational recovery publication.
  • Claim ledger: no claim from an addressing repair.

Artifacts

  • idea_intake.md: nearest-neighbor and novelty decision.
  • reports/design_review.md: adversarial recovery review.
  • configs/default.yaml: exact producer/recovery/authorization/failure pins.
  • runs/smoke.json: frozen no-model downstream-authorization smoke.
  • runs/failures/: preserved failure plus archive/retirement receipts.
  • runs/invocations/: recovery-owned STARTED/COMPLETE provenance for producer calls.
  • reports/artifact_manifest.yaml: ownership of external producer artifacts.

Report

Rendered from reports/report.md

Status

Operational partial. The first full-rank Stage-B G0 failed before model load while immutable v11 recomputed its LoRA-miss authorization through the already-known nonlexical external-prefix defect. This recovery archived and retired the exact failure and then produced a valid full-rank seed-7411 G0. Its frozen pathname-only retirement guard cannot hand off to later stages, so an additive successor is active. This remains mechanics/setup evidence, not a LoRA/full-rank capacity result.

Method

The recovery imports exact producer v11 scripts/run.py, leaves every producer CLI and reviewed source-snapshot gate active, and temporarily replaces only the exact src.analysis path helper used by downstream branch validation. The seam is limited to the one registered raw prefix and clean descendants; all other paths retain v11 behavior. A real no-model authorization smoke and commit-backed failure archival/retirement precede retry.

Current evidence

  • Authoritative producer LoRA-miss receipt: SHA-256 cb9fee75…818a, identity b973bc01…a862.
  • First Stage-B G0 failure pair: SHA-256 47305826…2c71, identity 070c23af…aa24.
  • Failure stage: branch_authorization; completed checks: none.
  • Model load, training/evaluation, benchmark access, sealed contrast access, scientific evidence, and downstream authorization: none.
  • Focused tests: 14/14.
  • Recovery smoke: file SHA-256 8bf5bb36…6849, identity d1135ea2…49b5, source contract 55d0a455…56f3; all six path/consumer/restoration controls pass.
  • Failure archive: file SHA-256 4fcccea3…45ed, identity ff478d40…0ec3; archive bytes match the source pair and occupy a third inode.
  • Retirement: file SHA-256 6e4c8ee3…53ad, identity c9abdc59…eae7; all four archive-commit blobs revalidated, only the producer pair removed, and the recovery archive retained.
  • Recovered G0: MODEL_SMOKE_PASS, file SHA-256 cdc90cd…c68f, identity e1f1c906…f89dc; exact K=1 and round-trip error zero, all 62 full-rank deltas have finite nonzero joint gradients and complete finite optimizer state, with about 22.1 GiB free after G0.
  • Frozen handoff defect: the next stage was rejected before wrapper STARTED publication because the retirement guard tests pathname existence rather than the retired failure's exact bytes/status.

Interpretation

The original analysis recovery was scientifically correct but operationally incomplete. This successor proved the consumer seam and produced a valid G0, then revealed that its one-time retirement guard cannot hand off after the canonical slot is legitimately repopulated. No capacity conclusion changes; an additive byte/status-aware handoff preserves this frozen result.

Next action

Publish and validate qwen35_4b_state_formation_branch_handoff_recovery, then run the already- authorized full-rank seed-7411 positive control through that wrapper.

Experiment log 7

Show the running log (7 entries, 2026-07-14)

2026-07-14 — downstream defect discovered fail-closed

  • The authoritative recovered producer analysis emitted LORA_JOINT_MISS_CONTROLS_REQUIRED and was published at commit 719c8e5a; both repository workflows passed.
  • The exact registered source-v11 full-rank seed-7411 G0 command then stopped at branch_authorization before model load. Downstream authorization re-runs the immutable analyzer's evidence validator, which reaches the same nonlexical registered external prefix.
  • Producer v11 durably wrote byte-identical, inode-distinct canonical/mirror failure receipts at SHA-256 47305826…2c71, identity 070c23af…aa24. They report zero completed checks, no model load, no training/evaluation, zero benchmark/sealed access, and no authority.
  • Source v11 refuses to overwrite the pair and requires archival. Direct retry is prohibited.

2026-07-14 — additive recovery design

  • Created a separate experiment because the analysis recovery is already frozen and result-bearing.
  • Pinned producer v11 source/config/implementation/CLI/GPU-runner/analyzer, the authoritative LoRA receipt, the first recovery sidecar, and the failed G0 pair.
  • Kept exact producer CLI execution and source-snapshot validation. The only temporary runtime change is src.analysis._canonical_expected_path on the module object actually used by downstream branch validation.
  • Required real downstream authorization smoke, alias/traversal/restoration controls, immutable invocation receipts, and commit-backed failure archival before verified retirement.

2026-07-14 — downstream smoke and failure archive pass

  • The focused recovery suite passed 14/14.
  • Real downstream smoke reproduced the original v11 rejection, then recomputed the exact LORA_JOINT_MISS_CONTROLS_REQUIRED receipt through the seam. All six controls pass. File SHA-256 is 8bf5bb36…6849, receipt identity d1135ea2…49b5, and frozen recovery source contract 55d0a455…56f3.
  • Smoke loaded no model, started no training/evaluation, opened no benchmark or sealed contrast, and changed no scientific interpretation.
  • The archive copied exact failure bytes 47305826…2c71 to an inode-distinct recovery path and emitted receipt SHA-256 4fcccea3…45ed, identity ff478d40…0ec3. Both source receipts remain present, byte-identical, and inode-distinct; the archive authorizes no retry by itself.

Current authorization

Only archive-checkpoint documentation, validation, commit, push, and workflow verification are authorized. After both workflows pass, retire exactly the canonical/mirror failure paths using that full commit SHA, then publish and validate the retirement before any model-bearing retry.

2026-07-14 — green archive checkpoint proves and retires exact source pair

  • Archive commit bdedabf4ea7d153d21a9b3a0cbe03cad66c8b6b2 passed Validate Repository and Publish Research Site.
  • Retirement reopened the archive copy, archive receipt, producer canonical, and producer mirror from that exact Git commit and required byte equality before mutation.
  • STARTED receipt SHA-256 is 90cf8dd1…380e. Only the two producer failure paths were unlinked under held-inode verification; the recovery archive remains at exact SHA-256 47305826…2c71.
  • Terminal retirement receipt SHA-256 is 6e4c8ee3…53ad, identity c9abdc59…eae7. It records zero model/training/evaluation/benchmark/sealed access and authorizes recovered producer retry only after retirement publication.

Current authorization

Only retirement documentation, validation, commit, push, and workflow verification are authorized. After both workflows are green, retry the exact full-rank seed-7411 G0 through the frozen wrapper. No positive control or result training is authorized until that G0 passes and is published.

2026-07-14 — recovered G0 passes; frozen wrapper cannot hand off

  • Retirement commit e69c1960…c79 passed both workflows before retry.
  • Full-rank seed-7411 G0 passed at file SHA-256 cdc90cd…c68f, identity e1f1c906…f89dc. Exact K=1 parity and checkpoint restoration have zero error; all 62 direct- full-shape delta tensors have finite nonzero joint gradients and complete finite optimizer state; the finite K=12 path leaves about 22.1 GiB free.
  • The receipt authorizes only the positive control and records no training/evaluation, benchmark, contrast, or scientific evidence. Wrapper STARTED/COMPLETE SHA-256 values are f0c78055…3c87 / 755ad561…f941.
  • The following positive-control attempt stopped before wrapper STARTED publication because the frozen retirement guard rejects any occupant of the canonical G0 pathname, even this exact successful receipt. No producer or model work occurred.
  • Preserve this recovery unchanged. Continue only through the additive byte/status-aware handoff successor after its frozen smoke checkpoint is published and both workflows are green.

Reproduce

Smoke test

.venv/bin/python -B experiments/qwen35_4b_state_formation_branch_recovery/scripts/run.py --smoke

Full run

Use the purpose-specific producer command in README.md through this recovery wrapper.

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗