Research log Small Model Experimentation
GitHub

Axis-on-Replay Stack with Medium Pilot

Finished2026-07-14 → 15Agentic Breadth InstallationGitHub ↗
Skills survived the stack; one noisy bar blocked

The one idea you need

Take the two things that worked last time — the targeted practice set that installed new skills, and the plain-review round that scored highest overall — and put them in one model, then judge it on the bigger, finer-grained version of the benchmark where a full sweep has actually happened before.

The question

Do the installed skills and the review boost survive together in one model, and does the bigger benchmark tier finally credit the installed skills at the family level?

What we found

The stacked model kept the installed skills (24 vs 18 and 15 of 40 on unseen tasks, with the cleanest finishing behavior of any model) and lost nothing on the retained skills. But the screen required beating both comparisons on three of four skill types, one type tied at the ceiling for the second straight trial, and another was lost to a one-off spike in a comparison model — so nothing advanced and the bigger-benchmark question was never asked. A second round of plain review made the model noticeably sloppier, undercutting the idea that review keeps compounding.

Why it matters

The installed skills are real and portable across starting models — shown twice now. What blocked promotion was measurement noise, not capability. The queued follow-up re-judges the already-built models with a fairer screen at zero training cost.

Unseen-task result24 vs 18 / 15install replicated on a new starting model
Finishing behavior6 run-onsvs 12 and 18 — cleanest in the event
Blocked by1 of 10 checksa ceiling tie plus a one-off comparison spike
Review round twosloppier86 parsed, 18 run-ons — compounding questioned
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Next Stage
    8. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 1

Unseen-task totals: the stack keeps its skills; review round two slips

How to read

Bars show right answers on 40 unseen skill tasks for the stacked model and both comparisons.

020406080replay_parentreplay_parent186512replay_squaredreplay_squared156418axis_on_replayaxis_on_replay24646

Takeaway → The stacked model leads clearly; the second review round is the weakest arm — but a single noisy sub-check kept the stacked model from advancing.

Data table
explicit merged compositeaxis holdout correct (of 40)retention correct (of 104)cap contacts
replay_parent186512
replay_squared156418
axis_on_replay24646

Numbers from experiments/qwen35_4b_axis_replay_stack_medium_target_match/reports/report.md

Technical framing

Seed-88015 gate: axis holdout and retention by arm — The axis install replicated on the new parent (+6 total; hygiene 9/10 twice across experiments; best termination), but the 3-of-4 kind-breadth bar failed on a ceiling tie plus a control kind-fluke, sealing seed 78145 before the medium pilot. Replay round two drifted locally.

In the author’s words from the Overview · “Results”

Both arms trained cleanly (control 0.3468, candidate 0.4547 train loss; 0 skips) and merged. The frozen 144-task gate event at seed 88,015: axis holdout of 40 — candidate 24, parent 18, replay_squared 15; per-kind candidate/parent/squared: explore 5/4/7, hygiene 9/5/5, protocol 8/8/3, tracefix 2/1/0. Retention of 104: candidate 64/98/6 (correct/parsed/caps), parent 65/92/12, squared 64/86/18. Nine of ten checks passed; the single failure was the 3-of-4 kind-breadth bar (hygiene and tracefix won; protocol tied at the parent's 8 — the second consecutive experiment where protocol ties at that ceiling; explore lost to the control's 7/10). No promotion; seed 78,145 permanently sealed; no benchmark event ran.

Overview

Stack the proven axis-atom install on the strongest replay-compounded parent (the 0.5081 composite), against a replay-squared control that simultaneously measures whether replay compounding continues at round two — with the conditional pilot at the MEDIUM tier, where the all-families goal is empirically reachable.

Research Program

  • Program: agentic_breadth_installation.
  • Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
  • Prior anchors: the goal-gap axis experiment (first local promotion; axis flipped warren, its replay control flipped rites and posted the line's best aggregate); three consecutive replay-compounding observations; 8-of-92 medium-tier all-families passes versus 1-of-65 at quick.

Question

Do the axis install and replay compounding stack in one model without interference — and does the medium tier convert installed axis skills into family scores that quick-tier atoms miss?

Hypothesis

The two effects have disjoint family footprints at seed 78,144 and different mechanisms (content install versus distribution refresh), so training the inherited axis corpus from the replay-compounded parent should preserve both; the replay-squared control separates compounding from content; medium-tier episodes reward the multi-turn-flavored axes the atoms install.

Setup

  • Model: only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
  • Parent: the replay_repeat composite of the goal-gap axis experiment (weights 3df45004...0072, tree 4c4f3561...acd7); warm start from its adapter (20be87b5...cd36). Runtime LoRA forbidden.
  • Treatment corpus: byte-identical inheritance of the frozen 160-row axis corpus (e7a95d73...686e; construction seed 77,117).
  • Arms: replay_squared (exact-exposure replay continuation — the control AND the round-two compounding measurement) and axis_on_replay (the stack candidate); 1,280-core + 240-block streams, exact three-axis MILP, slot seed 55,119; training seed 53, 190 updates, zero skips.
  • Local gate: fresh seed 88,015, the same two-instrument achievable design (40-task axis holdout + 104-task retention screen; relative wins, non-inferiority bands, no absolute per-kind floors), three composites (parent + both arms).
  • Conditional pilot: sealed seed 78,145, MEDIUM tier, think budget 1,024, four weight-authenticated composites (base, parent, replay_squared, candidate). Gates: candidate aggregate strictly above base, replay_squared, and parent; the every-family-versus-base record is the goal gate at the tier where it has been passed 8 times in 92 events.
  • Hidden boundary: benchmarks/ unread; independent seeds and matched-compute sample-more remain mandatory before any universal claim.

Run

Smoke:

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_axis_replay_stack_medium_target_match/scripts/run.py --smoke

Checkpointed stages (each requires its prerequisite committed at a clean, pushed, green main):

.venv/bin/python -B experiments/qwen35_4b_axis_replay_stack_medium_target_match/scripts/run.py --stage train-control
.venv/bin/python -B experiments/qwen35_4b_axis_replay_stack_medium_target_match/scripts/run.py --stage train-candidate
.venv/bin/python -B experiments/qwen35_4b_axis_replay_stack_medium_target_match/scripts/run.py --stage merge-arms
.venv/bin/python -B experiments/qwen35_4b_axis_replay_stack_medium_target_match/scripts/run.py --stage local
.venv/bin/python -B experiments/qwen35_4b_axis_replay_stack_medium_target_match/scripts/run.py --stage benchmark

Results

Both arms trained cleanly (control 0.3468, candidate 0.4547 train loss; 0 skips) and merged. The frozen 144-task gate event at seed 88,015: axis holdout of 40 — candidate 24, parent 18, replay_squared 15; per-kind candidate/parent/squared: explore 5/4/7, hygiene 9/5/5, protocol 8/8/3, tracefix 2/1/0. Retention of 104: candidate 64/98/6 (correct/parsed/caps), parent 65/92/12, squared 64/86/18. Nine of ten checks passed; the single failure was the 3-of-4 kind-breadth bar (hygiene and tracefix won; protocol tied at the parent's 8 — the second consecutive experiment where protocol ties at that ceiling; explore lost to the control's 7/10). No promotion; seed 78,145 permanently sealed; no benchmark event ran.

Interpretation

Three frozen readings land clearly. (1) Stack survival: the axis install transfers across parents — total +6 over parent for the second time, hygiene at 9/10 twice, with the best termination in the event (6 caps versus 12 and 18) and retention inside every band. (2) Replay compounding: round two DRIFTED at the local instrument (parse 86, caps 18, axis 15/40 with wild kind variance including the 7/10 explore fluke that broke the breadth bar) — the aggregate compounding observed at seed 78,144 does not show up as local quality, so compounding is either aggregate-specific or seed-fortunate. (3) Instrument design: the protocol holdout has tied at the parent ceiling in two independent experiments (8/8/8 twice), making it undetectable and effectively converting the 3-of-4 bar into 3-of-3 on the remaining kinds — a measured design flaw for successors to fix prospectively, not a capability fact.

Terminal Disposition

No later event is authorized here. Seed 78,145 was never opened and is spent-by-sealing. The published axis_on_replay composite carries the strongest local numbers of any artifact in the line (24/40 axis, 64/98/6 retention) and is the presumptive subject of a fresh-instrument re-adjudication with a detectability-corrected breadth bar; any such test is a new experiment with its own gate seed and lifecycle.

Knowledgebase Update

  • Program evidence updated: stack survival, replay round-two drift, and the ceiling-tie instrument flaw recorded.
  • Program backlog updated: fresh-instrument re-adjudication of the published composites queued with calibration notes.
  • Claim ledger updated: no.

Artifacts

  • data/sft_axis160.jsonl, data/corpus_manifest.json: inherited frozen corpus.
  • data/stream_manifest.json, data/stream_token_receipt.json: exposure receipts.
  • data/local_tasks_seed88015.jsonl, data/local_input_seed88015.jsonl, data/local_design_receipt.json: frozen gate.
  • reports/preregistration.md, reports/design_review.md: prospective contract and authorization.
  • reports/artifact_manifest.yaml: external parent and conditional model-artifact plan.

Report

Rendered from reports/report.md

Summary

Model-free construction is under way: the frozen axis corpus inherited byte-identically onto the strongest replay-compounded parent (0.5081 at seed 78,144), a replay-squared exact-exposure control that doubles as the round-two compounding measurement, the same achievable two-instrument gate at fresh seed 88,015, and a conditional pilot preregistered at the MEDIUM tier. No model event has run; seed 78,145 is sealed.

Research Program Fit

The predecessor proved the axis atoms install (first local promotion) and that its replay control compounds aggregate; their family flips were disjoint (warren versus rites). This trial tests whether the two effects stack in one model and whether medium-tier granularity converts installed skills that quick-tier atoms miss — the goal's own confirm tier, where all-families has passed 8 of 92 events.

Method

Identical machinery to the predecessor with three deltas: parent (the 0.5081 composite), control (replay squared), pilot tier (medium, think budget 1,024, seed 78,145). Frozen secondary readings: replay round-two compounding (replay_squared minus parent), stack interference (candidate axis-holdout counts versus the predecessor's 28/40), and the family-union reading (both flips held in one model).

Results

  • Training (1,520 rows, 0 skips, 190 updates each): control 0.3468, candidate 0.4547 train loss.
  • Gate (seed 88,015), axis holdout of 40: candidate 24, parent 18, replay_squared 15; per-kind candidate/parent/squared: explore 5/4/7, hygiene 9/5/5, protocol 8/8/3, tracefix 2/1/0. Retention of 104: 64/98/6 vs 65/92/12 vs 64/86/18 (correct/parsed/caps).
  • Nine of ten checks passed; the 3-of-4 kind-breadth bar alone failed (protocol ceiling tie — second consecutive experiment — plus the control's explore 7/10). No promotion; seed 78,145 permanently sealed; the medium pilot never ran.

Controls

The parent composite is the baseline; replay_squared is the mechanism-falsifying control and the compounding probe; the axis holdout at unseen seed 88,015 separates installation from memorization; the retention screen guards the thirteen retained skills.

Oracle Versus Deployable Evidence

Executable truth grades outputs only; benchmarks/ remains unread.

Next Stage

None. Closed per the frozen contract. The queued successor is a training-free fresh-instrument re-adjudication of the published composites with a prospectively detectability-corrected breadth bar and a conditional medium pilot.

Artifact Manifest

Inherited corpus and model-free artifacts tracked in-repo; parent artifacts external with pinned checksums; future weights under large_artifacts/ with receipts.

Experiment log 5

Show the running log (5 entries, 2026-07-14 → 15)

2026-07-14 — Model-free design freeze

  • Opened after the goal-gap axis experiment closed green (first local promotion; pilot negative with disjoint family flips between the axis candidate and its replay control). This trial claims both queued directions: stack the axis install on the 0.5081 replay-compounded parent, and measure round-two replay compounding inside the same event.
  • Inherited the frozen 160-row axis corpus byte-identically (e7a95d73...686e); reserved fresh slot/training/gate/aggregate seeds 55119/53/88015/78145; pilot preregistered at the MEDIUM tier, where the all-families goal has passed 8 of 92 historical events.
  • No model, GPU, training, local, or benchmark event has run.

2026-07-14 — Authenticated control training

  • train-control ran only after freeze commit bcce7472 matched origin/main with both workflows green and a clean worktree.
  • replay_squared trained 1,520/1,520 rows with 0 skipped over 190 updates; receipt/log published and pinned fail-closed. The candidate arm remains untrained until this checkpoint publishes green.

2026-07-15 — Authenticated candidate training

  • train-candidate ran only after control checkpoint f4fa0701 matched origin/main with both workflows green and a clean worktree.
  • axis_on_replay trained 1,520/1,520 rows with 0 skipped over 190 updates; receipt/log published and pinned fail-closed. Merges are the only next stage.

2026-07-15 — Authenticated explicit composites

  • merge-arms ran only after candidate checkpoint 02155e08 matched origin/main with both workflows green; PASS_CONTROL_MERGE and the merge self-pin were required and verified.
  • Both arms merged (scale 2.0, 128/128 nonzero modules, fingerprint-verified); merged-tree pins filled fail-closed in the evaluator. The one frozen 144-task gate event at seed 88,015 is the only next stage.

2026-07-15 — Gate event: no promotion; experiment closed

  • The frozen gate event ran from merge checkpoint 7183fa9e: three authenticated engine runs over the 144-row input at seed 88,015.
  • Axis holdout: candidate 24/40, parent 18, replay_squared 15. Per-kind candidate/parent/squared: explore 5/4/7, hygiene 9/5/5, protocol 8/8/3, tracefix 2/1/0. Retention: 64/98/6 vs 65/92/12 vs 64/86/18.
  • Nine of ten checks passed; the 3-of-4 kind-breadth bar failed (protocol tied at the parent ceiling for the second consecutive experiment; explore lost to the control's 7/10). No promotion; seed 78,145 permanently sealed per the frozen contract; no benchmark event ran.

Reproduce

Smoke test

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_axis_replay_stack_medium_target_match/scripts/run.py --smoke

Full run

checkpointed scripts/run.py stages only

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗