Research log Small Model Experimentation
GitHub

Interleaved-Replay Dose with Medium Pilot

Finished2026-07-15Agentic Breadth InstallationEvidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptation+2 moreGitHub ↗
Review-round theory refuted; search closed

The one idea you need

The last trial proved the two reliable lessons come back on a fresh start but cost the model ten retained answers when taught directly. Cross-checking every receipt isolated the fix: the one dose that never caused forgetting had a review round between practice doses. That review-round model is already built — this trial teaches the same verified lessons starting from it.

The question

Does a review round at the dose boundary let the lessons land without forgetting — completing a deployable recipe — and does the bigger benchmark tier then credit it?

What we found

The review round did not prevent forgetting: the dosed model lost nine to ten retained answers against both comparisons — almost exactly the cost of dosing directly — so the theory drawn from comparing old receipts is refuted by this direct test. The hygiene lesson won its category for the sixth straight measurement; the route-finding lesson lost this time. Per the preregistered escalation rule, the recipe search is closed: the next step is a mechanics study of WHY doses cost retention, not another schedule shuffle.

Why it matters

A theory built from comparing receipts across trials died the moment it met a controlled test — which is precisely what preregistered direct tests are for. The program keeps the refutation, the six-for-six hygiene result, and a sharply-posed mechanics question, instead of an endless recipe hunt.

Retention cost-9 / -10same as dosing directly — theory refuted
Hygiene lesson6 / 6won its category in every measurement ever
Escalation rulefiredrecipe search closed; mechanics study next
Benchmarknever openedseed 78,149 sealed by the frozen rule
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Next Stage
    8. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 1

The review round did not stop the forgetting

How to read

Bars per model: unseen skill-test total (of 20) and retained-skill total (of 104).

020406080interleaved_parentinterleaved_parent768replay_interleaved2replay_interleaved2669dose_after_replaydose_after_replay1159

Takeaway → The dosed model leads on new skills but pays the same retention cost as before — the schedule was not the cause, and the preregistered rule closes the recipe hunt.

Data table
explicit merged compositeaxis holdout correct (of 20)retention correct (of 104)
interleaved_parent768
replay_interleaved2669
dose_after_replay1159

Numbers from experiments/qwen35_4b_interleaved_replay_dose_medium/reports/report.md

Technical framing

Seed-88019 gate: axis holdout and retention by arm — The direct test refutes the replay-interleaving retention law: the dosed candidate paid the same ~10-point retention cost despite the interleaved parent (hygiene won its sixth consecutive event; explore lost). The frozen escalation rule closes the dose-recipe search; seed 78,149 sealed.

In the author’s words from the Overview · “Results”

Both 1,520-row arms trained for 190 updates and merged into authenticated composites. At the sole fresh gate (seed 88,019), the candidate beat parent and replay on the 20-item axis holdout (11/7/6) and won hygiene (7/2/3), but lost explore (4/5/3). More importantly, retention fell to 59/96/8 versus 68/96/8 for the parent and 69/98/6 for replay, breaking the frozen correct band against both controls by nine and ten points. The candidate was not promoted and conditional benchmark seed 78,149 is permanently sealed.

Overview

The direct test of the replay-interleaving retention law: the same verified two-lesson dose that just recovered its installs (but paid ten retention points when dosed directly), now warm-started from the already-built interleaving replay round — reproducing the only retention-safe dose recipe the line has measured, with the conditional pilot at the medium tier.

Research Program

  • Program: agentic_breadth_installation.
  • Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
  • Prior anchors: recovery confirmed (both flags TRUE, axis 15/20 vs 11/8); the replay-interleaving law isolated across receipts; the interleaving parent already trained, merged, and receipted.

Question

Does replay at the dose boundary protect retention while the proven installs land — completing the recipe for a deployable dosed model — and does the medium tier then convert it?

Hypothesis

Replay re-anchors the retained-skill distribution before the dose perturbs it. Falsified if the dosed candidate still breaks the retention bands or the installs fail to win from the interleaved parent.

Setup

  • Model: only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
  • Parent: the de-stack experiment's replay_clean composite (tree 19759e12...f67, weights 2cef3e5e...0b4); warm start from its adapter (f6f910ed...bb8).
  • Corpus: byte-identical inheritance of the verified 80-row hygiene+explore corpus (construction seed 77,119).
  • Arms: replay_interleaved2 (control) and dose_after_replay (candidate); identical stream geometry; slot seed 55,122; training seed 56.
  • Gate: fresh seed 88,019; two-kind holdout (both must win) + 104-task retention with unchanged bands; normalization; unconditional recovery flags; escalation rule frozen (retention breaking despite interleaving escalates to a mechanism study).
  • Conditional pilot: sealed seed 78,149, MEDIUM tier, think budget 1,024; candidate aggregate strictly above base, control, and parent; every-family-versus-base recorded as the goal gate.
  • Hidden boundary: benchmarks/ unread.

Run

Smoke:

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_interleaved_replay_dose_medium/scripts/run.py --smoke

Checkpointed stages:

.venv/bin/python -B experiments/qwen35_4b_interleaved_replay_dose_medium/scripts/run.py --stage train-control
.venv/bin/python -B experiments/qwen35_4b_interleaved_replay_dose_medium/scripts/run.py --stage train-candidate
.venv/bin/python -B experiments/qwen35_4b_interleaved_replay_dose_medium/scripts/run.py --stage merge-arms
.venv/bin/python -B experiments/qwen35_4b_interleaved_replay_dose_medium/scripts/run.py --stage local
.venv/bin/python -B experiments/qwen35_4b_interleaved_replay_dose_medium/scripts/run.py --stage benchmark

Results

Both 1,520-row arms trained for 190 updates and merged into authenticated composites. At the sole fresh gate (seed 88,019), the candidate beat parent and replay on the 20-item axis holdout (11/7/6) and won hygiene (7/2/3), but lost explore (4/5/3). More importantly, retention fell to 59/96/8 versus 68/96/8 for the parent and 69/98/6 for replay, breaking the frozen correct band against both controls by nine and ten points. The candidate was not promoted and conditional benchmark seed 78,149 is permanently sealed.

Interpretation

Replay at the dose boundary did not protect retention: the candidate paid the same roughly ten-point cost measured for direct dosing. This direct test refutes the replay-interleaving retention law and closes the dose-recipe search. Per the frozen escalation rule, only the separately registered dose-vehicle mechanism study may continue this line.

Knowledgebase Update

  • Program evidence updated: yes.
  • Program backlog updated: yes; the recipe search is closed and the mechanism study owns the successor slot.
  • Claim ledger updated: no.

Artifacts

  • data/sft_hygiene_explore.jsonl, data/corpus_manifest.json: inherited verified corpus.
  • data/stream_manifest.json, data/stream_token_receipt.json: exposure receipts.
  • data/local_tasks_seed88019.jsonl, data/local_input_seed88019.jsonl, data/local_design_receipt.json: frozen gate.
  • reports/preregistration.md, reports/design_review.md: contract and authorization.
  • reports/artifact_manifest.yaml: external parent and conditional model-artifact plan.

Report

Rendered from reports/report.md

Summary

Terminal negative: replay at the dose boundary did not protect retention. The candidate won hygiene but lost explore and fell nine/ten correct answers below the parent/replay retention controls. The replay-interleaving law is refuted, the dose-recipe search is closed, and conditional seed 78,149 remains sealed.

Research Program Fit

The direct test of the replay-interleaving retention law, carrying the session's highest calibrated gate probability; a pass yields a deployable dosed model and the line's first medium-tier paired event.

Method

See the preregistration.

Results

  • Training (1,520 rows, 0 skips, 190 updates each): control 0.3582, candidate 0.4361 train loss.
  • Gate (seed 88,019): axis holdout of 20 — candidate 11, parent 7, replay 6; hygiene 7/2/3 (sixth consecutive win), explore 4/5/3 (loss). Retention: 59/96/8 vs 68/96/8 and 69/98/6 — correct band broke against both controls (−9/−10 vs −5). Not promoted; seed 78,149 permanently sealed.

Controls

interleaved_parent baseline; replay_interleaved2 exact-exposure control; fresh two-kind holdout; retention screen; normalization unchanged.

Oracle Versus Deployable Evidence

Executable truth grades outputs only; benchmarks/ remains unread.

Next Stage

None. The replay-interleaving law is refuted by this direct test; the frozen escalation rule closes the dose-recipe search; the dose-vehicle mechanism study is the only funded successor.

Artifact Manifest

Inherited corpus and model-free artifacts in-repo; parent artifacts external with pinned checksums.

Experiment log 6

Show the running log (6 entries, 2026-07-15)

2026-07-15 — Model-free design freeze

  • Opened after the de-stack test confirmed recovery (both flags TRUE; axis 15/20 vs 11/8) while the direct dose broke the retention bands, isolating the replay-interleaving law. This trial reproduces the retention-safe recipe exactly: the same verified corpus, warm-started from the already-receipted interleaving replay round.
  • Seeds 55122/56/88019/78149 reserved; the escalation rule (mechanism study if retention breaks despite interleaving) is frozen.
  • No model, GPU, training, local, or benchmark event has run.

2026-07-15 — Model-free pipeline run (inherit → measure → materialize → validate → design → gate)

  • Adapted the full pipeline from the de-stack predecessor (build/measure/materialize/validate/train/merge/gate/eval/benchmark/harness) with exactly ONE design change: the parent is the predecessor's own replay_clean arm (the interleaving replay round). Every fail-closed convention kept (hash pins, --check byte-identity, TODO-PIN fail-closed, encoder binding, merge self-pin in the gate receipt, shared finalize_promotion writer, full benchmark CLI, weight recomputation). gen_axis_v2.py, gen_curriculum.py, train_think.py, src/vllm_runner.py copied byte-identical from the predecessor.
  • Inherited the treatment corpus byte-identically (data/sft_hygiene_explore.jsonl sha256 8b3e9791…c22cf1, manifest cbc9ae6d…03903, replay 25a9595f…abf0c2); build_corpus.py authenticates the donor pins AND re-derives the corpus from the copied generator at seed 77,119, requiring exact byte reproduction (verified: byte-identical).
  • Measured exact spans (source_token_lengths.json 1aef3dd5b9020ebfb672efcb8904da0406514984701415a58479e4ac08d3ecf5; treatment vector forward 19,582 / nonzero 5,793 / mass×5 9,665 — identical to the donor, as the bytes demand); the three-axis MILP at slot seed 55,122 solved optimally in 4.6 s: both 240-row variable blocks at forward 147,792 / nonzero 63,001 / mass×5 71,525; arm totals 1,373,106 / 579,624 / 633,716; 1,280 position-aligned shared rows; zero skips. Streams: replay_interleaved2.jsonl 0918144f…3f8d75, dose_after_replay.jsonl 9c91383b…2e77ee; manifest cb617490…790138; independent validation receipt stream_token_receipt.json 0e2197fda5957e9bb260d9e93bd782d343256b9828e9f5a3aa09cce26d2db1b9.
  • Froze the design receipt (data/design_receipt.json 646f5ee2c7a28eafa42625db5a4cc4dcc0ac431b89980a2f07d481496f345ec6) binding the interleaved-parent identity (tracked merge receipt 24367084…b90332, tree 19759e12…8fc67, weights 2cef3e5e…187b04, adapter f6f910ed…6fb8 / 015bb135…4d961), the corpus inheritance, and the lifecycle substring contracts (seeds 56 / 88,019 / 78,149).
  • Froze the 124-task gate at seed 88,019: local_tasks_seed88019.jsonl 6e927f591f9ae9d2edad6e263be3f7c0262b39de4854314a64802e656b98c15b, runner input dcf482cb…c9d568, design receipt d4aeb7fd…a5624; zero canonical-message overlap proven against both inherited corpora, the donor's corpora AND materialized streams, both fresh training streams, regenerated construction rows, prior local seeds 88,000–88,018, and all six predecessor frozen gates (88,013–88,018).
  • Filled the stream pins (materialize/validate/train_trial exposure constants); left fail-closed as TODO-PIN: PUBLISHED_ARM_HASHES (both arms, filled after each training stage publishes) and EXPECTED_TREE_SHA256 (both arms, filled after merge-arms publishes); verified each aborts with the pin message.
  • run.py --smoke green end to end (every --check byte-identical, 50 unit tests, py_compile); training remains sealed behind the pushed-checkpoint gates and the PASS_CONTROL_TRAINING / PASS_CONTROL_MERGE verdicts.
  • No model, GPU, training, local, or benchmark event has run.

2026-07-15 — Authenticated control training

  • train-control ran only after freeze commit 04de32b5 matched origin/main with both workflows green and a clean worktree.
  • replay_interleaved2 trained 1,520/1,520 rows with 0 skipped over 190 updates; receipt/log published and pinned fail-closed.

2026-07-15 — Authenticated candidate training

  • train-candidate ran only after control checkpoint dc201dbd matched origin/main with both workflows green and a clean worktree.
  • dose_after_replay trained 1,520/1,520 rows with 0 skipped over 190 updates; receipt/log published and pinned fail-closed. Merges are next.

2026-07-15 — Authenticated explicit composites

  • merge-arms ran only after candidate checkpoint b5b9b39b matched origin/main with both workflows green; both arms merged (128/128 modules, fingerprint-verified); tree pins filled fail-closed. The one frozen 124-task gate event at seed 88,019 is the only next stage.

2026-07-15 — Gate event: interleaving refuted; escalation rule fired; closed

  • The gate ran from merge checkpoint dd225d33: three authenticated engine runs over the 124-row input at seed 88,019.
  • Axis holdout of 20: candidate 11, parent 7, replay 6; hygiene 7/2/3 (sixth consecutive win), explore 4/5/3 (loss). Retention 59/96/8 vs 68/96/8 and 69/98/6 — the correct band broke against both controls (−9/−10), matching the direct dose's cost despite the interleaved parent.
  • The replay-interleaving retention law is REFUTED by this direct test; the frozen escalation rule fires: the dose-recipe search closes and only a dose-vehicle mechanism study may follow. No promotion; seed 78,149 permanently sealed.

Reproduce

Smoke test

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_interleaved_replay_dose_medium/scripts/run.py --smoke

Full run

checkpointed scripts/run.py stages only

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗