Axis Stack Re-adjudication with Medium Pilot
The one idea you need
The skill-stacked model already exists and has twice shown its skills are real; both times a measurement quirk — one comparison category stuck at a ceiling where wins are impossible — blocked its advance. This trial re-judges the same three built models on brand-new tasks with that quirk handled explicitly, at zero training cost.
The question
Judged fairly on new tasks, does the skill-stacked model clear the program's bars — and does the bigger benchmark tier then credit its skills?
What we found
Judged fairly on brand-new tasks — with the ceiling quirk handled — the stacked model won the overall skill test for the third straight time (22 vs 15 and 18 of 40) with the cleanest finishing behavior again. But it won only two of the four skill categories: document-hygiene and route-finding practice stick every time, while program-repair practice never took (down to 1 of 10) and procedure-following practice teaches nothing the starting model didn't already know. Two wins fell short of the required three, so the bigger benchmark stayed sealed.
Why it matters
Three independent screens now agree: the question is no longer whether designed practice installs skills (it does, without forgetting), but which lessons carry weight. Half this set needs rewriting from the recorded failures — a content problem, not a measurement problem.
On this page
Results at a glance 1
How to read
Per-category right answers (of 10) for the stacked model and both comparisons.
Takeaway → Hygiene and route-finding practice beat both comparisons; program-repair and procedure practice do not — the practice set needs a content revision, not another measurement.
Data table
| axis lesson kind | axis_on_replay | replay_parent | replay_squared |
|---|---|---|---|
| explore | 7 | 3 | 6 |
| hygiene | 7 | 5 | 5 |
| protocol | 7 | 7 | 5 |
| tracefix | 1 | 0 | 2 |
Numbers from experiments/qwen35_4b_axis_stack_readjudication_medium_pilot/reports/report.md
Technical framing
Seed-88016 corrected gate: per-kind axis correct (of 10) — Third fresh-instrument replication: hygiene and explore install (candidate wins), protocol ties the parent for the third time (redundant dose), tracefix trends to chance. Axis total 22 vs 15/18; retention 65/98/5 — best termination again. Two kind wins < 3 required: not promoted; seed 78,146 sealed.
In the author’s words from the Overview · “Results”
The corrected gate event ran at fresh seed 88,016 with all three inherited composites weight-authenticated. All four kinds were DETECTABLE (no control ceiling this time), so the corrected bar required 3 kind wins. Axis holdout of 40: candidate 22, parent 15, replay_squared 18 — the axis total win replicated for the THIRD time on a third fresh instrument. Per-kind candidate/parent/squared: explore 7/3/6 (win), hygiene 7/5/5 (win), protocol 7/7/5 (tie with parent — third consecutive event), tracefix 1/0/2 (loss). Retention: candidate 65/98/5 (correct/parsed/caps) vs 61/92/12 and 66/91/13 — best termination in every one of the three events. Two kind wins < 3 required: NOT_PROMOTED; seed 78,146 is permanently sealed; the medium pilot never ran.
Overview
Re-judge the published axis-stack composites on a fresh instrument with the measured ceiling-tie flaw corrected prospectively — control-ceiling kinds excluded and reported as not-detectable, wins required on two-thirds of detectable kinds — then fund the medium-tier pilot conditionally. Training-free; both prior failures remain recorded and their seeds sealed.
Research Program
- Program:
agentic_breadth_installation. - Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
- Prior anchors: the axis install replicated across two parents (24/40 vs 18/15; hygiene 9/10 twice) and was blocked once by the aggregate pilot's replay comparison and once by a single breadth check whose protocol kind tied at the parent ceiling in both experiments.
Question
Measured fairly on fresh tasks — with undetectable kinds excluded rather than silently tightening the quota — do the already-installed axis skills clear the program's relative bars, and does medium-tier granularity then convert them at the family level?
Hypothesis
The install is real (replicated twice); the prior block was instrument noise (a systematic ceiling tie plus a control kind-fluke). The corrected bar removes exactly that noise without weakening any other condition, adds a fail-closed GATE_UNDETECTABLE outcome, and leaves retention bands unchanged.
Setup
- Model: only
Qwen/Qwen3.5-4B, revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Arms (all inherited published composites, weight- and tree-pinned):
replay_parent(3df45004...0072),replay_squared(e43b885c...069e), candidateaxis_on_replay(7ebcad39...d0e4). No training, merging, or exposure matching. - Gate: fresh seed 88,016, the standard two instruments (40-task axis holdout, 104-task retention screen). Corrected promotion: axis total strictly above both controls; strict wins on at least two-thirds (rounded up) of DETECTABLE kinds (a kind is undetectable if either control scores ≥ 9/10; undetectable kinds are reported, not counted); retention non-inferiority bands unchanged; route abstentions ≤ 4; zero detectable kinds fails closed as
GATE_UNDETECTABLE. - Conditional pilot: sealed seed 78,146, MEDIUM tier, think budget 1,024, four weight-authenticated composites (base, both controls, candidate); candidate aggregate strictly above base, replay_squared, and replay_parent; the every-family-versus-base record is the goal gate at the tier where it has passed 8 of 92 events.
- Hidden boundary:
benchmarks/unread.
Run
Smoke:
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_axis_stack_readjudication_medium_pilot/scripts/run.py --smokeCheckpointed stages:
.venv/bin/python -B experiments/qwen35_4b_axis_stack_readjudication_medium_pilot/scripts/run.py --stage local
.venv/bin/python -B experiments/qwen35_4b_axis_stack_readjudication_medium_pilot/scripts/run.py --stage benchmarkResults
The corrected gate event ran at fresh seed 88,016 with all three inherited composites weight-authenticated. All four kinds were DETECTABLE (no control ceiling this time), so the corrected bar required 3 kind wins. Axis holdout of 40: candidate 22, parent 15, replay_squared 18 — the axis total win replicated for the THIRD time on a third fresh instrument. Per-kind candidate/parent/squared: explore 7/3/6 (win), hygiene 7/5/5 (win), protocol 7/7/5 (tie with parent — third consecutive event), tracefix 1/0/2 (loss). Retention: candidate 65/98/5 (correct/parsed/caps) vs 61/92/12 and 66/91/13 — best termination in every one of the three events. Two kind wins < 3 required: NOT_PROMOTED; seed 78,146 is permanently sealed; the medium pilot never ran.
Interpretation
Across three preregistered fresh-instrument events the mechanism map is no longer noise: the hygiene and explore lessons install reliably (hygiene beat both controls in all three events; explore in two), the termination benefit is unconditional (caps roughly halved every time), and the axis TOTAL always wins — but the tracefix lesson never installed (4/10, 2/10, 1/10 — trending to chance) and the protocol lesson adds nothing the parent lacks (tied in all three events). The corrected bar did exactly its job: with no ceiling excuse available, it exposed that the corpus installed half its content. The correct successor is a content revision — replace the two dead blocks using this line's own raw failure outputs — not another measurement.
Terminal Disposition
No later event is authorized here. Seed 78,146 is spent-by-sealing. Three published composites and three gate events' raw outputs (432 graded completions per arm family) are preserved for the successor's failure forensics.
Knowledgebase Update
- Program evidence updated: the three-replication installable-lesson map recorded.
- Program backlog updated: axis corpus v2 (tracefix/protocol blocks redesigned from failure forensics) queued with calibration notes.
- Claim ledger updated: no.
Artifacts
data/local_tasks_seed88016.jsonl,data/local_input_seed88016.jsonl,data/local_design_receipt.json: frozen gate.reports/preregistration.md,reports/design_review.md: contract and authorization.reports/artifact_manifest.yaml: inherited composite pins.
Report
Rendered from reports/report.md
Summary
Training-free re-adjudication of the published axis-stack composites on a fresh instrument (gate seed 88,016) with the measured ceiling-tie flaw corrected prospectively, and a conditional MEDIUM-tier pilot at sealed seed 78,146. Both predecessor failures remain recorded with their seeds sealed. No model event has run.
Research Program Fit
The axis install replicated across two parents and was blocked once by the quick-tier aggregate comparison and once by a single breadth check whose protocol kind tied at the parent ceiling in both experiments — a measured instrument flaw, queued for exactly this correction in the program backlog before this experiment opened.
Method
Three inherited composites (weight- and tree-pinned), one fresh two-instrument gate with the detectability-corrected breadth bar (undetectable kinds excluded and reported; two-thirds of detectable kinds required; GATE_UNDETECTABLE fails closed; retention bands unchanged), then the conditional medium pilot (candidate aggregate strictly above base and both controls; every-family-versus-base recorded as the goal gate).
Results
- Gate (seed 88,016; all arms weight-authenticated): all four kinds detectable; required wins 3.
- Axis holdout of 40: candidate 22, parent 15, replay_squared 18. Per-kind candidate/parent/squared: explore 7/3/6 (win), hygiene 7/5/5 (win), protocol 7/7/5 (tie — third consecutive event), tracefix 1/0/2 (loss).
- Retention of 104: 65/98/5 vs 61/92/12 vs 66/91/13 (correct/parsed/caps) — bands all passed; best termination of the event.
- Two kind wins < 3: NOT_PROMOTED; seed 78,146 permanently sealed; the medium pilot never ran.
Controls
replay_parent is the baseline; replay_squared the exposure-matched control from the stack trial; the fresh task seed removes any reuse of graded items.
Oracle Versus Deployable Evidence
Executable truth grades outputs only; benchmarks/ remains unread.
Next Stage
None. Closed per contract. The queued successor is axis corpus v2: keep hygiene/explore, replace the redundant protocol block, redesign trace-repair from this line's own raw failure outputs.
Artifact Manifest
All arms are inherited published composites with pinned checksums; the frozen gate and receipts are tracked in-repo.
Experiment log 2
Show the running log (2 entries, 2026-07-15)
2026-07-15 — Model-free design freeze
- Opened after the stack trial closed green, blocked by one of ten checks: a kind-breadth bar where the protocol kind tied at the parent ceiling for the second consecutive experiment and one control produced a 7/10 explore fluke. The detectability correction and this re-adjudication were queued in the program backlog before this experiment opened.
- Training-free: the three published composites are inherited with recomputed weight and tree pins; the fresh gate at seed 88,016 uses the corrected bar (undetectable kinds excluded and reported; two-thirds of detectable kinds required; GATE_UNDETECTABLE fails closed); retention bands unchanged.
- The conditional pilot is preregistered at the MEDIUM tier, sealed seed 78,146. Both prior failures remain recorded; their seeds remain sealed.
- No model, GPU, or benchmark event has run.
2026-07-15 — Corrected gate event: not promoted; experiment closed
- The gate event ran from freeze commit
3fc5fb6c: three authenticated engine runs over the fresh 144-row input at seed 88,016. - All four kinds detectable (no control ceiling); required wins 3. Candidate axis 22/40 vs parent 15 and squared 18 (third consecutive axis-total win); kind wins: explore and hygiene (2 of 3 required); protocol tied the parent for the third consecutive event; tracefix lost (1/10). Retention bands all passed with the best termination of the event (caps 5 vs 12/13).
- NOT_PROMOTED under the corrected bar; seed 78,146 permanently sealed; the medium pilot never ran. The three-replication mechanism map is recorded: hygiene/explore/termination install; tracefix/protocol do not.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_axis_stack_readjudication_medium_pilot/scripts/run.py --smokeFull run
checkpointed scripts/run.py stages onlyRun steps are documented inside the experiment folder (README and scripts).