Sweep-Rate Consolidation (Erratum: 2/6, not ~50%)
The one idea you need
As the program closed its campaign, its documents cited a roughly-fifty-percent rate for the flagship result — the trained model beating the base on all ten benchmark families at once. That figure came from a favorable four-seed window and quietly omitted an earlier miss. This cell collected every recorded reading, recomputed each verdict from raw family scores under tamper-proof pins, and corrected the figure everywhere it appears with visible errata.
The question
Over every recorded all-families reading, how often does the reference model actually sweep — and what blocks it?
What we found
Two sweeps in six readings — one in three, not one in two — with a wide honest confidence band (roughly 4 to 78 percent at 95%). The texture matters more than the point estimate: the model never lost a single family to the base in sixty comparisons and won the overall score all six times; every failed sweep was a zero-zero tie at the same one family. The corrected figure now stands in every document that carried the old one, as visible errata; no underlying fact changed.
Why it matters
Terminal claims deserve all the evidence. The program's flagship number is now computed over the complete record with proper uncertainty, and the correction trail is public — closed by arithmetic, not by narrative.
On this page
Results at a glance 1
How to read
One bar per seed: how many of the ten families the reference model strictly won.
Takeaway → Eight to ten wins every time, never a loss — the distance to a perfect sweep is always the same single family.
Data table
| sealed medium seed (tb 1024) | families strictly won | families lost |
|---|---|---|
| 78150 | 8 | 0 |
| 78154 | 10 | 0 |
| 78155 | 9 | 0 |
| 78156 | 8 | 0 |
| 78157 | 10 | 0 |
| 78159 | 9 | 0 |
Numbers from experiments/qwen35_4b_sweep_rate_consolidation/runs/sweep_rate_analysis.json
Technical framing
Strict family wins vs base per sealed seed (of 10) — all six recorded readings — The consolidated record: sweep rate 2/6 = 0.333 (exact 95% CI [0.043, 0.777]; Beta posterior mean 0.375), correcting the informal ~50% window figure (2/4 over 78154-78157). Menders blocked all four misses at 0-margin; rites and warren once each (warren WON +0.267 at 78155); zero strict losses in sixty family comparisons; aggregate strict win 6/6. Visible errata landed in every carrier document; no per-seed fact changed.
In the author’s words from the Overview · “Results”
Sweep rate 2/6 = 0.333 (exact Clopper–Pearson 95% CI [0.043, 0.777]; Beta(1,1) posterior mean 0.375, 95% CrI [0.099, 0.710]). Menders blocks 4/4 misses, all 0-margin draws; rites and warren once each. Zero strict losses across all sixty family comparisons; aggregate strict win 6/6. Base draw distribution computed per family (base rites 0.0 on all six seeds — the intake's example corrected; chronicle drew >0 for base on exactly the two passing seeds).
Overview
Terminal bookkeeping for the agentic-breadth program's headline sweep claim: the informal "~50% sweep rate" was computed over a favorable four-seed window (78154–78157); the full committed record holds SIX goal-gate readings of the hygiene_explore composite vs base at sealed medium/tb1024 seeds, and the honest all-events rate is 2/6 (exact Clopper-Pearson 95% CI [0.043, 0.777]; Beta(1,1) posterior mean 0.375, 95% CrI [0.099, 0.710]). Analysis-only: zero GPU, zero seeds, no model, benchmarks/ never read.
Research Program
- Program:
agentic_breadth_installation - Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
- Prior anchors: the medium-tier measurement (78150, 8/10); the statechain dose's sealed sweep (78154, 10/10); the three-seed confirmation (78155–78157: 9/10, 8/10, 10/10 — AGGREGATE_ONLY); the zero-root rebuild's original reading (78159, 9/10).
Question
What is the program's honest all-events goal-gate sweep rate, with calibrated uncertainty — and which documents carried the favorable-window figure that needs the erratum?
Method
Byte-copy the six committed benchmark summaries into data/source_summaries/ and hard-pin their sha256s; recompute every goal gate from per_family scores (strict wins/ties/losses vs base; FAMILIES byte-identical to the tier-forensics analyzer, enforced by test); cross-check against any goal-gate block the summary already records (78154's recorded block agrees; the others record none); then compute the rate with exact interval math (integer-shape beta quantiles via the binomial-tail identity, deterministic bisection, stdlib only — verified externally against scipy to 6 decimals). The harness re-derives both artifacts byte-identically.
Results
| seed | source cell | strict wins | ties | losses |
|---|---|---|---|---|
| 78,150 | universal_medium_tier_measurement | 8/10 | menders, rites | none |
| 78,154 | statechain_only_dose | 10/10 | — | none |
| 78,155 | goal_gate_confirmation | 9/10 | menders (warren WON +0.267) | none |
| 78,156 | goal_gate_confirmation | 8/10 | menders, warren | none |
| 78,157 | goal_gate_confirmation | 10/10 | — | none |
| 78,159 | zero_root_lineage_rebuild | 9/10 | menders | none |
Sweep rate 2/6 = 0.333 (exact Clopper–Pearson 95% CI [0.043, 0.777]; Beta(1,1) posterior mean 0.375, 95% CrI [0.099, 0.710]). Menders blocks 4/4 misses, all 0-margin draws; rites and warren once each. Zero strict losses across all sixty family comparisons; aggregate strict win 6/6. Base draw distribution computed per family (base rites 0.0 on all six seeds — the intake's example corrected; chronicle drew >0 for base on exactly the two passing seeds).
Interpretation
The terminal claim, on all the evidence: the reference model beats base on the aggregate always and on every family on one seed in three, with the entire distance between "one in three" and "always" being menders draws. The earlier "~50%" framing was a favorable window (2/4), later held as "a fifth data point" at 2/5 — corrected here with visible errata in every carrier document; no per-seed fact changed anywhere. The wide CI is the honest cost of six readings and is reported as such.
Knowledgebase Update
- Program evidence updated: the corrected 2/6 rate with exact CI, the blocker table, and the erratum target list recorded in
runs/sweep_rate_analysis.json. - Program backlog updated: the documents on the erratum list need their figures amended to cite 2/6 (or to scope their window explicitly).
- Claim ledger updated: no new claim — this corrects the informal figure attached to existing entries.
Artifacts
data/source_summaries/: the six byte-identical summary copies (sha-pinned inputs).runs/readings_table.json: one verified row per seed — aggregates, per-family scores/deltas, wins/ties/losses, blockers, provenance sha, cross-check status.runs/sweep_rate_analysis.json: rate + exact CI + posterior, blocker table, base-draw distribution, and the ERRATUM block.scripts/: collect (fail-closed pins) + analyze (--verifybyte-identity) + harness.tests/: 30 unittest cases — pinned expectations for all six readings, CI math at boundaries, provenance-drift negatives, FAMILIES byte-identity, erratum presence.
Report
Rendered from reports/report.md
Summary
The program's informal "~50% sweep rate" was computed over a favorable four-seed window. The full committed record holds SIX goal-gate readings of the headline hygiene_explore composite against base at sealed medium/tb1024 seeds, and the honest all-events rate is 2/6 = 0.333 — exact Clopper-Pearson 95% CI [0.043272, 0.777222]; under a Beta(1,1) prior the posterior is Beta(3,5) with mean 0.375 and 95% credible interval [0.098988, 0.709579]. Every reading was re-derived from its sha256-pinned committed summary; the one recorded goal-gate block (78154) agrees with the recomputation. No verdict changes — AGGREGATE_ONLY already recorded the frozen confirmation bar as failed — but the quotable rate is corrected, and the carrier documents are listed for amendment.
Research Program Fit
agentic_breadth_installation, lifecycle 24: terminal bookkeeping. The program's map closes with calibrated instruments and closed doors; this cell makes the headline replication number honest before the record is quoted onward.
Method
Analysis-only (no model, zero GPU, zero seeds, benchmarks/ never read). The six summaries are byte-copied into data/source_summaries/ and hard-pinned by sha256; collect_readings.py fails closed on any drift (local copy or a present-but-different original), recomputes strict wins/ties/losses per family (FAMILIES byte-identical to the tier-forensics analyzer, enforced by test), and cross-checks recorded goal-gate blocks. analyze_sweep_rate.py computes the rate, exact CI (integer-shape beta quantiles via the binomial-tail identity and deterministic bisection — stdlib only, externally cross-checked against scipy to six decimals), the posterior, the blocker table, the base draw distribution, and the erratum; --verify and the harness require byte-identical re-derivation of both artifacts.
Results
Six verified readings (all: aggregate win vs base; zero strict family losses):
- 78150 (universal_medium_tier_measurement, hygiene_explore): 8/10 — menders + rites ties. MISS.
- 78154 (statechain_only_dose, hygiene_explore_parent): 10/10. PASS (recorded goal-gate block agrees).
- 78155 (goal_gate_confirmation, hygiene_explore): 9/10 — menders tie; warren WON +0.267. MISS.
- 78156 (goal_gate_confirmation, hygiene_explore): 8/10 — menders + warren ties. MISS.
- 78157 (goal_gate_confirmation, hygiene_explore): 10/10. PASS.
- 78159 (zero_root_lineage_rebuild, hygiene_explore_original): 9/10 — menders tie. MISS.
Blockers: menders in ALL FOUR misses (0-margin draws — both arms at zero); rites once; warren once. Base draws (computed): zero on all six seeds for lockpick/menders/mirage/rites/siftstack; chronicle above zero on exactly the two passing seeds (78154, 78157); stockade and warren on four; sirens and toolsmith on all six. The intake's guess "base rites>0 on 2 seeds" was wrong — chronicle is that family.
Erratum: the "~50%" / "two of four sealed seeds" figure is a window artifact — 2/4 over 78154–78157, a window that starts at the first pass and omits the earlier 78150 miss; the zero-root docs extended it to a "fifth data point" (2/5 = 0.4) while keeping the "~50%" label. Corrected: 2/6. Carriers: experiments/qwen35_4b_goal_gate_confirmation/README.md + reports/report.md, knowledge/synthesis.md (confirmation, dose-scale, and zero-root entries), knowledge/experiment_brief.json (menders_dose_scale and repair_verifier_signal_probe briefs), knowledge/experiment_viz.json (zero-root note), experiments/qwen35_4b_zero_root_lineage_rebuild/README.md + experiment_log.md + reports/preregistration.md.
Controls
Fail-closed provenance (pinned sha256s on both the local copies and the originals when present); recomputation from raw per-family scores rather than trusting derived blocks, with mandatory agreement where a block exists; FAMILIES byte-identity to the forensics analyzer; byte-identical re-derivation of both artifacts; negative tests for every drift path.
Oracle Versus Deployable Evidence
Not applicable — no model event. All inputs are committed receipts from prior sealed events; this cell adds no new measurement, only honest arithmetic over the existing record.
Interpretation
The correction is small in mechanism and large in honesty: the composite's aggregate transfer remains unconditional (6/6, margins 3–6×, zero strict losses in sixty comparisons), and the gate remains draw-gated at exactly menders — but "roughly half of sealed seeds sweep" overstates the record. The right sentence going forward: two full sweeps in six all-time readings (2/6, CI [0.04, 0.78]), every miss a menders 0-margin draw.
Next Experiments
None funded from here — this is terminal bookkeeping. The carrier documents listed in the erratum should be amended (or their windows scoped explicitly) whenever they are next touched.
Artifact Manifest
No external artifacts; everything is committed in-repo. reports/artifact_manifest.yaml records the smoke/full commands.
Experiment log 1
Show the running log (1 entry, 2026-07-16)
2026-07-16 — Collection, analysis, errata, closure
- Six readings collected fail-closed from sha-pinned copied summaries and recomputed from per-family scores: 8/10, 10/10, 9/10, 8/10, 10/10, 9/10 — zero strict losses in all sixty family comparisons; aggregate win 6/6.
- Sweep rate 2/6 = 0.333 (exact 95% CI [0.043, 0.777]; Beta posterior mean 0.375, 95% CrI [0.099, 0.710]). Menders blocks 4/4 misses (all 0-margin draws); rites once; warren once (and warren WON +0.267 at 78,155). Base draw note: base rites was 0.0 on all six seeds — the intake's example was wrong and is corrected here; chronicle is the family where base drew >0 on exactly the two passing seeds.
Visible errata appended to the carrier documents (confirmation README
- report, zero-root README, three synthesis paragraphs, and the brief/
viz figures updated); per-seed facts unchanged everywhere.
- 30 tests green; --smoke and --full reproduce both artifacts byte-identically.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_sweep_rate_consolidation/scripts/run.py --smokeFull run
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_sweep_rate_consolidation/scripts/run.py --fullRunnable scripts exist in the experiment folder, but the exact invocation was not written down.