Retention-Screen Calibration Study
The one idea you need
Every training trial in this program ends with a forgetting check: the new model retakes a 104-task quiz beside its parent, and a drop of more than five tasks fails it. But the quiz itself is regenerated fresh each time, and the guard on the last trial caught the same model's score moving four points between two fresh quizzes. This study re-measured five already-built models on four brand-new quizzes each — twenty measurement runs, zero training — to size that wobble directly.
The question
How much does a model's forgetting gap versus its parent wobble between fresh screens, what pass/fail margin does that imply, and which past readings survive?
What we found
The gap wobbles with a standard deviation of 4.3 tasks — the five-task pass/fail margin was barely one wobble wide, so single-quiz forgetting verdicts were close to coin flips. Every historical 'this model forgot 5-10 tasks' reading sits inside the measured noise; averaged over four fresh quizzes the real forgetting is only 1-4 tasks. The frozen fix: future trials must average three fresh quizzes, where the same five-task margin becomes correctly sized.
Why it matters
Every forgetting conclusion this program made — including the priced 5-10-point dose tax and the paused capacity question — was read off a single quiz. This calibration shrinks the tax to 1-4 points, vindicates the guard that paused the last trial, and hands every future trial a sound adjudication rule.
On this page
Results at a glance 1
How to read
Each group is one already-built model; the four bars are its gap to the parent on four freshly generated quizzes.
Takeaway → Single readings swing from -10 to +4 for the same unchanged model — the historical one-quiz forgetting verdicts were reading this noise.
Data table
| published composite (unchanged between screens) | screen 88022 | screen 88023 | screen 88024 | screen 88025 |
|---|---|---|---|---|
| axis160_direct | -7 | -10 | 0 | 2 |
| axis160_r64 | -4 | 4 | 0 | -3 |
| hygiene_explore_direct | 3 | -1 | -4 | -7 |
| replay_clean | 1 | 2 | -1 | -5 |
Numbers from experiments/qwen35_4b_retention_screen_calibration/runs/local/calibration_readout.json
Technical framing
Retention delta vs clean_parent across four fresh screens — Twenty authenticated eval runs, zero training: the same frozen composites swing from -10 to +4 against the same parent purely by screen draw. Governing estimand delta_sd_pooled = 4.27 (level SD 4.81 descriptive) -> recommended_band 9, adjudication_protocol pooled_k3; all five historical single-screen tax readings (-9, -10, -10, -7, -5) fall inside pooled +/- 2 SD intervals; pooled deltas are only -3.75 / -0.75 / -2.25 / -0.75. The +/-5 band is correctly sized only on the mean of three pooled screens (2 x 4.27/sqrt(3) = 4.9).
In the author’s words from the Overview · “Results”
All 20 runs completed and weight-authenticated (receipt runs/local/calibration.json, readout runs/local/calibration_readout.json) (table on the experiment page). Governing estimand delta_sd_pooled = 4.27 → recommended_band = 9, adjudication_protocol = pooled_k3 (every future retention adjudication must pool three fresh screens). Level SD 4.81 (descriptive). Stability flags: all five historical single-screen readings (−9 axis160_direct@88020, −7 axis160_r64@88021, −10/−10 hygiene_explore_direct@88018/88020, −5 replay_clean@88020) fall inside their arms' pooled-delta ± 2·SD intervals. Vehicle (descriptive, not gated): rank-64 pooled delta −0.75 vs rank-32's −3.75 (difference +3.0 favoring rank 64, within noise).
Overview
Measure the measuring stick: five published composites re-run across four fresh 104-task retention screens (20 authenticated eval events, zero training) to size the screen's seed-to-seed variance directly, derive the band that separates real retention effects from draws, and freeze the adjudication protocol every future dose and vehicle cell will use.
Research Program
- Program:
agentic_breadth_installation. - Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
- Prior anchors: the SCREEN_INSTABILITY verdict (the known −9 re-measured at −5); pooled same-composite scatter of ±3–4 points across four gates; the paused rank-64 vehicle question awaiting calibrated bands.
Question
What is the retention screen's true per-arm seed variance, what band does it imply, and which historical single-screen readings survive pooled re-measurement?
Hypothesis
Screen SD is on the order of 2–4 points, implying either wider bands or pooled multi-screen adjudication; the dose-tax direction survives pooling while individual band-edge calls do not.
Setup
- Model: only
Qwen/Qwen3.5-4B, revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Arms (published, weight-authenticated):
clean_parent,replay_clean,hygiene_explore_direct,axis160_direct,axis160_r64. - Screens: four frozen retention-only instruments (seeds 88,022–88,025; 104 rows each from the original 13-skill generator), oracle-free inputs, overlap-receipted against all prior gates and each other.
- Event: 20 sequential authenticated engine runs in frozen screen-major order; consolidated receipt with per-arm/per-screen table, the governing pooled delta-vs-parent SD (level SD reported descriptively), recommended band (⌈2·delta SD⌉, min 5), adjudication protocol tier, and historical stability flags.
- No training, merging, promotion, or benchmark stage.
Run
Smoke:
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_retention_screen_calibration/scripts/run.py --smokeCheckpointed stage:
.venv/bin/python -B experiments/qwen35_4b_retention_screen_calibration/scripts/run.py --stage localResults
All 20 runs completed and weight-authenticated (receipt runs/local/calibration.json, readout runs/local/calibration_readout.json):
| arm | correct by screen (88022/23/24/25) | delta vs parent by screen | pooled delta | delta SD |
|---|---|---|---|---|
| clean_parent | 69 / 67 / 69 / 64 | — | — | — |
| axis160_direct | 62 / 57 / 69 / 66 | −7 / −10 / 0 / +2 | −3.75 | 5.68 |
| axis160_r64 | 65 / 71 / 69 / 61 | −4 / +4 / 0 / −3 | −0.75 | 3.59 |
| hygiene_explore_direct | 72 / 66 / 65 / 57 | +3 / −1 / −4 / −7 | −2.25 | 4.27 |
| replay_clean | 70 / 69 / 68 / 59 | +1 / +2 / −1 / −5 | −0.75 | 3.10 |
- Governing estimand
delta_sd_pooled= 4.27 →recommended_band= 9,adjudication_protocol=pooled_k3(every future retention adjudication must pool three fresh screens). Level SD 4.81 (descriptive). - Stability flags: all five historical single-screen readings (−9 axis160_direct@88020, −7 axis160_r64@88021, −10/−10 hygiene_explore_direct@88018/88020, −5 replay_clean@88020) fall inside their arms' pooled-delta ± 2·SD intervals.
- Vehicle (descriptive, not gated): rank-64 pooled delta −0.75 vs rank-32's −3.75 (difference +3.0 favoring rank 64, within noise).
Interpretation
- The ±5 single-screen retention band every prior gate used was ~1.2 SD wide — a true-zero-cost arm fails it roughly one screen in eight, and the program adjudicated four arms per event. The rank cell's SCREEN_INSTABILITY guard is vindicated and quantified.
- The "intrinsic retention tax ~5–10 points per dose" reading must be revised: pooled over four screens the taxes are 0.75–3.75 points; the historical 5–10-point readings were single-screen draws from a ±4.3-SD process around those small means. A tax likely exists (all four pooled deltas are negative) but is several times smaller than the single-screen readings suggested.
- Coherence check: ±5 is almost exactly 2 × (4.27/√3) = 4.9 — the historical band size is right if and only if it is applied to the MEAN of three pooled fresh screens, which is precisely the frozen
pooled_k3protocol. - Screen 88025 ran hard for every arm (parent 64 vs 67–69 elsewhere) — a real common-difficulty component that same-screen deltas cancel; the estimand correction from the adversarial review (delta SD, not level SD) is what kept this event's outputs from being inflated by it.
Knowledgebase Update
- Program evidence updated: calibration outputs + the revised tax reading recorded.
- Program backlog updated: instability successor closed; vehicle resumption and any future dose gate now require the
pooled_k3protocol. - Claim ledger updated: no new claim; the tax law's evidence note revised in synthesis.
Artifacts
data/local_tasks_seed8802{2,3,4,5}.jsonl+ inputs +data/local_design_receipt.json: four frozen screens.reports/preregistration.md,reports/design_review.md: contract and authorization.reports/artifact_manifest.yaml: the five composite pins.
Report
Rendered from reports/report.md
Summary
The line's first instrument-calibration study is closed: five published composites across four fresh retention screens (20 authenticated eval events, zero training) measured the retention screen's delta-vs-parent noise at SD 4.27 — the ±5 single-screen band every prior gate used was ~1.2 SD wide. Outputs: recommended_band 9, adjudication_protocol pooled_k3 (three fresh screens pooled before any retention adjudication; ±5 on the pooled mean is correctly sized at 2 × 4.27/√3 = 4.9), and all five historical single-screen "forgetting tax" readings (−9, −10, −10, −7, −5) inside measured noise around much smaller pooled deltas (−3.75, −2.25, −0.75, −0.75). The per-dose retention tax revises from 5–10 points to 1–4 points pooled; the rank-64 vehicle reads −0.75 vs rank-32's −3.75 descriptively (+3.0, within noise).
Research Program Fit
The SCREEN_INSTABILITY verdict's funded successor; every future retention adjudication depends on it, and the paused rank-64 vehicle reading resumes descriptively under it.
Method
See the preregistration.
Results
runs/local/calibration_readout.json: delta_sd_pooled 4.27 (per-arm delta SDs 5.68 / 4.27 / 3.59 / 3.10), level SD 4.81 descriptive, band 9, protocol pooled_k3, five/five stability flags inside, vehicle descriptive +3.0 for rank 64. Full per-arm/per-screen table in the README.
Controls
All arms published and weight-authenticated; screens overlap-receipted against all prior gates and each other; frozen run order; normalization unchanged.
Oracle Versus Deployable Evidence
Executable truth grades outputs only; benchmarks/ remains unread.
Next Stage
Closed. Successors inherit the pooled_k3 protocol: the vehicle question and any future dose gate adjudicate retention on the mean of three fresh screens with the ±5 band.
Artifact Manifest
Four frozen screens in-repo; five composite pins external with committed receipts.
Experiment log 3
Show the running log (3 entries, 2026-07-15)
2026-07-15 — Model-free design freeze
- Opened as the SCREEN_INSTABILITY verdict's funded successor: four gates' pooled scatter showed same-composite retention deltas moving ±3–4 points between fresh screens, rivaling the ±5 band.
- Four retention-only screens frozen (seeds 88,022–88,025) with overlap receipts; five published composites pinned; the 20-run screen-major event order, the pooled-SD outputs, the band formula, the protocol tiers, and the historical stability flags are preregistered.
- No model, GPU, or benchmark event has run; nothing trains in this study.
2026-07-15 — Adversarial design review: estimand corrected pre-freeze
- Three-lens review (contract, statistics, fail-closed) with adversarial verification confirmed one MAJOR finding and refuted nothing: the draft derived
recommended_bandandadjudication_protocolfrom the pooled SD of retention-correct LEVELS, but every band this program adjudicates is a same-screen DELTA versus a parent — common screen-difficulty variance inflates the level SD yet cancels exactly in deltas, while independent per-arm noise makes the delta SD ~√2 × the level SD, so the draft's outputs were calibrated against the wrong noise process in an unknowable direction. - Corrected before any commit or model event: the governing estimand is now
delta_sd_pooled(pooled across-screen sample SD of the per-screen delta-vs-clean_parent series over the four non-parent arms, ddof=1); the level SD stays in the receipt descriptively. Preregistration, code, config, README, and tests amended together; a regression test pins the cancellation case (levels wobbling ±4 with a constant −5 delta must read band 5 / single_screen, not band 7 / pooled_k2). - 59/59 unit tests green after the amendment.
2026-07-15 — Calibration event (the only model event) and closure
- CI green on the freeze commit;
run.py --stage localexecuted the 20 authenticated engine runs in the frozen screen-major order; every composite tree recomputed and matched its receipt at the boundary. - Readings:
delta_sd_pooled4.27 →recommended_band9 andadjudication_protocolpooled_k3; level SD 4.81 (descriptive); all five historical single-screen tax readings fall inside their arms' pooled ± 2·SD intervals; pooled deltas −3.75 (axis160_direct), −2.25 (hygiene_explore_direct), −0.75 (axis160_r64), −0.75 (replay_clean). - Vehicle, descriptive: rank-64 −0.75 vs rank-32 −3.75 (+3.0, within noise).
- Closure: the ±5 single-screen band was ~1.2 SD wide; ±5 on a pooled three-screen mean is correctly sized (2 × 4.27/√3 = 4.9), which is the frozen protocol going forward. The 5–10-point per-dose tax reading revises to 1–4 points pooled.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_retention_screen_calibration/scripts/run.py --smokeFull run
checkpointed scripts/run.py --stage local onlyRun steps are documented inside the experiment folder (README and scripts).