Menders/Sirens + Tier Forensics
The one idea you need
The program's goal needs one trained model to beat the untouched base on every one of ten benchmark families at once. The last attempt failed on exactly two families that looked frozen: one always scored zero, the other always exactly one half, for every model at every attempt on the quick benchmark tier. Before designing another training treatment against that wall, this study dug through every benchmark receipt the repository has ever produced — including hundreds from an older experiment line that also ran the bigger, finer-grained medium tier — to check whether the wall is real or an artifact of the quick tier's coarse scoring.
The question
Are the two frozen family scores a real model wall, or an artifact of the quick benchmark tier — and on which tier is the all-families goal actually winnable?
What we found
An artifact. The constants already have committed counterexamples on the line's own instrument, and the all-ten-families-at-once win happened 9 times in 94 historical medium-tier comparisons versus once in 84 quick-tier ones. On medium, the base model never maxes out any family (so strict wins are always available) and the always-one-half family spreads out to a normal range. The catch: those nine winners trained on benchmark-family data, which this program forbids — so the next step is measuring this line's own clean models on medium for the first time.
Why it matters
Every future attempt at the program goal was about to be aimed at two families that cannot be beaten at quick granularity. This redirect — costing zero GPU hours — moves the goal gate to a tier where history says it is winnable.
On this page
Results at a glance 1
How to read
Each bar counts historical paired comparisons; the rightmost group (10 of 10) is the program goal.
Takeaway → Medium-tier comparisons cluster at eight and nine family wins with nine full passes; quick-tier ones cluster at six and seven with one pass.
Data table
| families strictly above base (of 10) | quick tier (84 events) | medium tier (94 events) |
|---|---|---|
| <=5 | 3 | 0 |
| 6 | 20 | 5 |
| 7 | 35 | 14 |
| 8 | 17 | 46 |
| 9 | 8 | 20 |
| 10 (goal gate) | 1 | 9 |
Numbers from experiments/qwen35_4b_menders_sirens_tier_forensics/runs/constants_analysis.json
Technical framing
Strict family wins vs base per paired event, by tier — Every committed gateway receipt paired within its event: the all-ten-families goal gate passed 9/94 medium arm-events vs 1/84 quick. Base never sits at a family ceiling at medium (0/95 events) and sirens leaves its exactly-0.500 sticking point (14/95 vs 49/82 at quick). The menders=0 / sirens=0.500 constants have three committed counterexamples at quick/tb1024. Caveat: medium passers were gym-trained arms from the older line; the contamination-free universal arms have never been measured at medium.
In the author’s words from the Overview · “Results”
The constants were never universal. At the line's own quick/tb1024 instrument: base sirens 0.375 (seed 78,131), candidate menders 0.021 (78,131), replay_refresh menders 0.125 (78,133) — all in committed receipts. The goal-gap event (78,144) drew base at exactly menders 0 / sirens 0.500. The goal gate is ~8x more passable at medium. Strict-win-all-ten passed 9/94 medium arm-events vs 1/84 quick. Strict-win distribution (medium): mode 8/10 (46 events), 20 at 9/10, 9 at 10/10. Base never ceilings at medium (0/95 events have any family at 1.0; quick had 2/82), and sirens un-sticks: base exactly-0.5 in 14/95 medium events vs 49/82 quick; base sirens spans 0.2–0.6 at medium. Menders at medium: base zero in 54/95, max 0.3; treated arms reached 0.4. … Read the full result →
Overview
Adjudicate the two "frozen constants" blocking the all-families goal gate (menders = 0, sirens = 0.500 at quick/tb1024) from committed gateway receipts only: zero GPU, zero seeds, benchmarks/ never read.
Research Program
- Program:
agentic_breadth_installation. - Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
- Prior anchors: the goal-gap pilot (7 families up, 0 down, gate failed on menders/sirens ties); the backlog's queued menders/sirens forensics prerequisite; the earlier gym line's medium-tier receipt corpus.
Question
Are menders = 0 and sirens = 0.500 structural walls of the 4B, or artifacts of the quick instrument's granularity and item draws — and at which tier is the all-ten-families goal gate actually passable?
Method
Sweep every committed gateway receipt (experiments/*/runs/**/*.json, 2,278 files), extract every ten-family score block with tier/seed/budget provenance (380 raw rows, 356 cleaned), then: constant check with exact counterexamples; per-tier base family profiles (floor/ceiling frequencies); paired within-event strict-win adjudication of the goal gate. Cleaning rules and their one honest post-first-look refinement are recorded in the preregistration; the harness re-derives the analysis byte-identically from the committed table.
Results
- The constants were never universal. At the line's own quick/tb1024 instrument: base sirens 0.375 (seed 78,131), candidate menders 0.021 (78,131), replay_refresh menders 0.125 (78,133) — all in committed receipts. The goal-gap event (78,144) drew base at exactly menders 0 / sirens 0.500.
- The goal gate is ~8x more passable at medium. Strict-win-all-ten passed 9/94 medium arm-events vs 1/84 quick. Strict-win distribution (medium): mode 8/10 (46 events), 20 at 9/10, 9 at 10/10.
- Base never ceilings at medium (0/95 events have any family at 1.0; quick had 2/82), and sirens un-sticks: base exactly-0.5 in 14/95 medium events vs 49/82 quick; base sirens spans 0.2–0.6 at medium. Menders at medium: base zero in 54/95, max 0.3; treated arms reached 0.4.
- Near-miss blockers: menders/sirens/warren at quick; menders/rites/warren at medium.
Interpretation
The two constants are quick-instrument artifacts — coarse 1/8-step granularity plus item draws — not model walls. The all-families goal gate's realistic venue is the medium tier, where every family has strict-win headroom against base in every historical event. Honest limit: all nine medium passers came from the old gym line, whose arms trained ON menagerie-family data; the contamination-free universal arms (best aggregate 0.5081 at quick) have never been measured at medium. The funded successor is that measurement: base + the line's best published composites, one fresh sealed medium seed, tb1024, paired same-backend, goal gate recorded.
Run
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_menders_sirens_tier_forensics/scripts/run.py --smoke # analysis reproduces byte-identically
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_menders_sirens_tier_forensics/scripts/run.py --full # sweep + analysis both reproduceArtifacts
runs/receipt_table.json: the raw 380-row sweep with provenance sha256 per receipt.runs/constants_analysis.json: cleaned analysis — constant check, base profiles, paired goal-gate adjudication.reports/preregistration.md: frozen questions, cleaning rules, and the honest ordering note.
Report
Rendered from reports/report.md
Summary
Analysis-only forensics over every committed gateway receipt (2,278 files, 356 cleaned family-score rows, zero GPU): the two "frozen constants" blocking the all-families goal gate — menders = 0 and sirens = 0.500 for every arm at quick/tb1024 — are instrument artifacts, not model walls. Three committed counterexamples exist at the line's own instrument, and the paired within-event strict-win analysis shows the goal gate (all ten families strictly above base) passed 9 of 94 historical medium-tier arm-events versus 1 of 84 at quick. At medium the base never sits at a family ceiling and sirens leaves its 0.500 sticking point (exactly 0.5 in 14/95 base events vs 49/82 at quick).
Research Program Fit
The backlog's queued prerequisite before any new menders/sirens treatment; it converts "the wall is two families wide" into "the venue was one tier too coarse," and de-risks the goal-gate path's next benchmark event.
Method
See the preregistration: frozen sweep + cleaning rules (one honest post-first-look refinement recorded), constant check with exact counterexamples, per-tier base family profiles, paired strict-win adjudication.
Results
Quick: 84 paired arm-events, 1 goal-gate pass, 8 near-misses at 9/10 (blockers menders/sirens/warren); base at a ceiling in 2/82 events. Medium: 94 paired arm-events, 9 passes, 20 near-misses at 9/10 (blockers menders/rites/warren); base ceilings 0/95; base menders zero in 54/95 (max 0.3), sirens 0.2–0.6. Counterexamples at quick/tb1024: base sirens 0.375 (78,131), candidate menders 0.021 (78,131), replay_refresh menders 0.125 (78,133).
Controls
Raw and cleaned tables both committed; every row carries its source receipt sha256; the harness re-derives the analysis (smoke) and the sweep (full) byte-identically.
Oracle Versus Deployable Evidence
Receipt analysis only; benchmarks/ never read; no new model evidence of either kind.
Next Stage
Closed. Funded successor: the universal line's first medium-tier paired measurement — base + best published composites, one fresh sealed medium seed, tb1024, goal gate recorded. Caveat carried: the nine historical medium passers were gym-trained arms; instrument feasibility is established, line transfer is not.
Artifact Manifest
Everything in-repo (two JSON artifacts under runs/); no external artifacts.
Experiment log 1
Show the running log (1 entry, 2026-07-15)
2026-07-15 — Sweep, analysis, and closure (no model event)
- Swept 2,278 committed JSON files under
experiments/*/runs/; 380 raw family-score rows; 356 after the frozen cleaning (one post-first-look refinement — summary-file exclusion — recorded honestly in the preregistration; headline readings unchanged by it). - Constant check: the goal-gap pilot's "menders = 0 and sirens = 0.500 for every arm at every seed at quick/tb1024" has three genuine committed counterexamples (base sirens 0.375 at seed 78,131; candidate menders 0.021 at 78,131; replay_refresh menders 0.125 at 78,133) — the constants are item-draw artifacts of the quick instrument, not structural walls.
- Tier adjudication: paired within-event strict-win analysis gives the goal gate (10/10 families strictly above base) 9 passes in 94 medium arm-events versus 1 in 84 quick arm-events; at medium the base never sits at a family ceiling (0/95 events) and sirens leaves its 0.500 sticking point (exactly-0.5 in 14/95 base events vs 49/82 at quick).
- Blocking families in 9/10 near-misses: menders/sirens/warren at quick; menders/rites/warren at medium.
- Caveat preserved: all nine medium passers are gym-trained arms from the old line; the contamination-free universal arms have never been measured at medium — that measurement is the funded successor.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_menders_sirens_tier_forensics/scripts/run.py --smokeFull run
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_menders_sirens_tier_forensics/scripts/run.py --fullRun steps are documented inside the experiment folder (README and scripts).