Qwen35 4b State Track Confirmation
The one idea you need
The team builds one model through a documented chain of training rounds, each round folded into a sealed-exam score. At the previous step they installed a brand-new skill (keeping a running tally of several named counters as a list of updates is applied) and, on ONE sealed exam, the new model outscored its parent 0.326 to 0.300 - the first score gain since the cheap 'just review the old deck again' lever ran out. But there is a catch they wrote down at once: exams are drawn fresh each time, and the parent's OWN score bounces between 0.30 and 0.36 from one exam to the next. A single 0.026 win could just be the parent having a bad-luck exam. The only honest way to tell a real skill from a lucky draw is to sit the SAME two models six more fresh sealed exams and, on each one, subtract the parent's score from the new model's score - so whatever made that particular exam easy or hard cancels out - then see if the new model stays ahead.
The question
Across six fresh sealed exams, does the new-skill model keep beating its parent when you compare them head-to-head on the SAME exam each time - or does the one 0.026 win dissolve into the parent's normal exam-to-exam bounce?
What we found
The lift held up directionally, without becoming a slam dunk. Across six fresh sealed exams, running the same seed through both models so the noise cancels, the state-tracking model beat its parent on 4 of 6 with an average edge of +0.021 aggregate — right in line with the +0.026 seen originally. By the pre-registered rule that is CONFIRMED. But being honest about the statistics: the seed-to-seed spread (SD 0.045) is wide enough that a strict paired t-test (t=1.12, 5 df) would call this not-significant on its own. That is not a surprise or a moved goalpost — the rule was written up front as a LIBERAL directional check, precisely because the decisive value here is the OTHER outcome: a clearly negative mean would have killed the result as noise, and it did not. Across all seven exam seeds now, five are positive and the mean is +0.021. So the read is: state-tracking gives a real but small and noisy general uplift — enough to make it the current best model to carry forward, not enough to call a decisive win. The install-a-universal-skill idea is directionally supported. Next the model goes in front of a real coding agent to see whether any of this proxy improvement shows up where it actually matters.
Why it matters
This is the program's one unbroken habit: every good draw must survive fresh sealed exams before anyone claims it. The exact same six-seed discipline recently caught an earlier headline (a fix-the-procedure win) and correctly showed it was luck. If this lift holds, the newly installed skill becomes the program's reference model and validates the whole 'install a designed skill, prove it by transfer' doctrine; if it folds, the budget is freed for a different move instead of chasing noise.
On this page
Results at a glance 1
per-seed aggregate delta · sealed seed →
Data table
| sealed seed | paired delta (state_track - count_walk aggregate) |
|---|---|
| 78170 | -0.0123 |
| 78171 | 0.0373 |
| 78172 | -0.0385 |
| 78173 | 0.005 |
| 78174 | 0.0439 |
| 78175 | 0.0887 |
Numbers from experiments/qwen35_4b_state_track_confirmation/runs/benchmark/confirmation_readout.json
In the author’s words from the Overview · “Results”
The benchmark event has not run (design freeze; no seed consumed). On completion the terminal readout is runs/benchmark/confirmation_readout.json. See reports/report.md for the frozen design and consequences and reports/preregistration.md for the frozen rule and power.
Overview
Honest effect size (promised at freeze): SD 0.0453, paired t = 1.12 on 5 df, one-sided p ≈ 0.16 — a directional replication, NOT strictly significant; the rule was preregistered as a LIBERAL directional check (false-CONFIRMED ≈ 0.31) whose decisive high-value outcome (NOT_CONFIRMED) did not fire. state_track was adopted as the program reference composite. No claim-ledger entry was ever filed for this cell, and the +0.02 menagerie edge did NOT transfer to the real coding harness (see knowledge/coding_harness_transfer_measurement.md).
Research Program
- Program:
agentic_breadth_installation - Program question: can a designed synthetic curriculum install a transferable skill that adds durable aggregate on a saturated parent, proven by transfer to held-out benchmark surfaces?
- Prior anchors: lifecycle 30
qwen35_4b_state_track_install(INSTALLED_TRANSFER at seed 78169, +0.0256 paired lift, single seed); lifecycle 28qwen35_4b_count_walk_menders_confirmation(the eval-only multi-seed discipline that correctly retired a single-seed headline as seed noise); lifecycle 29qwen35_4b_count_walk_replay_compound(replay compounding BOUNDED at stage 8).
Question
Does the single-seed +0.0256 paired lift of state_track over its count_walk parent (seed 78169) replicate across fresh sealed seeds, or was it within the parent's own 0.30-0.36 seed-to-seed aggregate band?
Hypothesis
The install-universal-features doctrine predicts a designed transferable skill adds real, seed-stable aggregate. If so, the paired delta state_track - count_walk should be positive on a majority of fresh sealed seeds with a positive mean. The null is that the 78169 lift was seed noise, in which case the paired delta is symmetric about zero.
Setup
- Model:
Qwen/Qwen3.5-4B(rev851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a) only. - Arms: two pre-existing committed composites -
count_walk(parent) andstate_track(candidate). No base arm. - Eval source: the trusted aggregate gateway (
scripts/run_benchmark_aggregate.py), public family scores only. - Baseline:
count_walk(the current program reference composite). - Controls: the pairing itself (same-seed delta cancels the seed-variance that produced the parent's 0.30-0.36 swing); per-family tables and the candidate-vs-parent family gate are descriptive.
- Primary metric: the frozen three-state paired verdict (CONFIRMED / NOT_CONFIRMED / AMBIGUOUS) over six new seeds.
- Oracle-only metrics: none.
- Hidden-label boundary: the benchmark suite contents are never read as data; only the sha-pinned gateway runs.
Run
Smoke (verifies pins, provenance copies, power arithmetic, the in-cell stage 1-9 lineage package, any published ledger, compiles, runs tests):
.venv/bin/python -B scripts/run.py --smokeFull (the only stage; requires clean pushed green main + the committed preregistration and PASS_BENCHMARK_EVENT design review):
.venv/bin/python -B scripts/run.py --stage benchmarkThe event is seed-major over 78170-78175 with a k-seed write-ahead ledger. A mid-event crash preserves an opened record; recover by auditing the preserved receipts and re-running with --resume (the summary regenerates byte-identically). Never edit receipts by hand: delete the torn artifact and let --resume rebuild it.
Results
The benchmark event has not run (design freeze; no seed consumed). On completion the terminal readout is runs/benchmark/confirmation_readout.json. See reports/report.md for the frozen design and consequences and reports/preregistration.md for the frozen rule and power.
Interpretation
CONFIRMED promotes state_track to the program reference composite and validates the install-universal-features doctrine as a durable aggregate mover. NOT_CONFIRMED (the decisive, high-value outcome under this liberal directional rule) retires the single-seed reading as seed noise and keeps count_walk as the reference. AMBIGUOUS demands a mechanism-differentiated or larger-N design, never a re-roll of these seeds.
Knowledgebase Update
- Program evidence updated: on event completion, the
agentic_breadth_installationscorecard records whether the divergent-skill install is a durable aggregate mover. - Program backlog updated: the next divergent-skill dose is gated on this verdict (compound on state_track if CONFIRMED, else re-anchor on count_walk).
- Claim ledger updated: design-only until the event runs; the verdict then supports or retires the durable-reference claim.
Artifacts
src/- the shared vLLM runner (unused at eval; the gateway is the instrument)scripts/-run.py,run_benchmark.py,check_benchmark.py,power_analysis.py,rebuild_lineage.py, and the copied lineage trainers/mergers/wrappersconfigs/data/- the in-cell stage 1-9 lineage package and the three provenance copiesruns/- created at event time (ledger, per-seed summaries, readout)reports/- preregistration, benchmark design review, report, artifact manifestreports/artifact_manifest.yaml
Report
Rendered from reports/report.md
Summary
CONFIRMED, directionally — the state_track aggregate lift replicated across six fresh sealed seeds, but the effect is small and statistically soft. Paired deltas (state_track − count_walk on the same seed, so the parent's 0.30-0.36 seed swing cancels) were [-0.0123, +0.0373, -0.0385, +0.0050, +0.0439, +0.0887]: mean +0.0207, state_track winning 4 of 6 (threshold ≥4), clearing the frozen rule as CONFIRMED. Honest effect size (promised at freeze): SD 0.0453, paired t = 1.12 on 5 df — not strictly significant (one-sided p ≈ 0.16). This is exactly what the preregistered LIBERAL rule (false-CONFIRMED ≈ 0.31 under the null) anticipated; the decisive high-value outcome was NOT_CONFIRMED (a negative mean), which did not occur. The mean lift (+0.0207) matches the single-seed 78169 observation (+0.0256); across all seven seeds the mean is +0.0214 with 5/7 positive. Reading: state_track is a real-but-small (~+0.02) and noisy aggregate improvement — durable enough to adopt as the program reference composite, not a large or crisp gain. The install-universal-features doctrine is directionally supported, not proven at strict significance at n=6.
Research Program Fit
Program agentic_breadth_installation. The install-universal-features doctrine claims a designed synthetic curriculum can install a transferable skill that adds real aggregate on a saturated parent, proven by transfer to held-out benchmark surfaces. Lifecycle 30 produced the first such single-seed signal after replay compounding bounded at stage 8. This cell is the calibrate-and-diverge follow-through: the single-seed lift GATES the "state_track is the durable program reference" claim, and this confirmation is the pre-registered check that decides it - the same eval-only multi-seed discipline that (in the count_walk menders confirmation, lifecycle 28) correctly exposed a headline single-seed reading as seed noise.
Method
- Two pre-existing committed composites, authenticated fail-closed by full tree+weights sha256 against design-time constants:
count_walk(parent) andstate_track(candidate). Nobasearm - the event is the parent-versus-candidate paired comparison. - Six sealed fresh medium/tb1024 seeds 78170-78175, two arms per seed in the frozen order (count_walk, state_track), seed-major, twelve gateway runs. A k-seed write-ahead ledger consumes each seed once; closed records sha-pin the sealed summary and both per-arm receipts; crash recovery only via
--resumewith byte-identical deterministic regeneration. - Instrument: the trusted aggregate gateway only; the benchmark suite is never read as data. Every receipt is anchored to the seed-78169 instrument signature (pre-consumption and post-arm).
- Frozen rule: paired delta
d_i = state_track_aggregate - count_walk_aggregate;wins= events withd_i > 1e-12;mean_d= mean of the six. CONFIRMED iffmean_d > 0ANDwins >= 4; NOT_CONFIRMED iffmean_d <= 0; AMBIGUOUS otherwise. Full details and power inreports/preregistration.md.
Results
The benchmark event has not been run (this is a design freeze; no seed has been consumed and the ledger does not exist). On completion the readout runs/benchmark/confirmation_readout.json records, per seed, both aggregates and the paired delta, then the frozen verdict over the six new events with the 78169 event reported alongside but never pooled. The frozen consequences are:
- CONFIRMED (
mean_d > 0andwins >= 4): the state_track aggregate lift replicates across sealed seeds; the divergent-skill install is a durable gain and state_track becomes the program reference composite; the install-universal-features doctrine yields real transferable aggregate. - NOT_CONFIRMED (
mean_d <= 0): the 78169 lift does not replicate; it was within the parent's seed variance; count_walk remains the reference; the single-seed INSTALLED_TRANSFER is retired as seed noise. - AMBIGUOUS (
mean_d > 0butwins < 4): directional but not decisive; a mechanism-differentiated or larger-N design is required, not a re-roll of these seeds.
Honest calibration recorded before any event: the paired majority rule is a LIBERAL directional check (false-CONFIRMED rate approximately 0.31 under the pure null); its decisive, high-value outcome is NOT_CONFIRMED. Under the observed +0.0256 lift the test has 90-98% power across the priced paired-noise range (sigma_d 0.02-0.03).
Controls
The pairing IS the control: forming d_i on the same seed cancels the common per-seed benchmark-difficulty variance that produces the parent's 0.30-0.36 swing, isolating the parent-versus-candidate contrast the single-seed reading could not. Descriptive, never gating: both aggregates, the per-family delta table, the candidate-versus-parent per-family strict-win partition (the goal-gate analog with no base arm), and per-family within-slack retention flags. Budget integrity scopes the paired comparison and never gates.
Oracle Versus Deployable Evidence
No oracle or hidden-label evaluation is used. Every reading is a public aggregate-gateway score; the benchmark suite's contents are never parsed. The verdict is a deployable, provenance-anchored aggregate comparison.
Interpretation
A CONFIRMED verdict claims a durable PAIRED aggregate lift for the state_track composite as built against the count_walk parent at this instrument (medium/tb1024) - a directional replication, not a low-alpha significance result, and not a claim that the every-family-beats-base bar is met (warren remains below base, inherited from the parent). A NOT_CONFIRMED verdict retires the single-seed INSTALLED_TRANSFER as seed noise and keeps count_walk as the reference. Either way the cell closes on a frozen, preserved result.
Next Experiments
- CONFIRMED: adopt state_track as the program reference composite and attempt the next divergent-skill installation dose on top of it; test whether transferable-skill installs compound.
- NOT_CONFIRMED: keep count_walk as the reference; the divergent-skill move class is not yet demonstrated as a durable aggregate mover; the next attempt must be a mechanism-differentiated design (a larger dose, a different skill, or a different parent), not a re-roll of these seeds.
- AMBIGUOUS: a larger-N or mechanism-differentiated design; do not seed-mine this exact contrast.
Artifact Manifest
See reports/artifact_manifest.yaml. The two 9GB composites live in large_artifacts/ and are reproduced by their own cells' standalone rebuild paths; this cell's in-cell stage 1-9 lineage package documents their lineage and scripts/rebuild_lineage.py --verify-inputs authenticates it.
Experiment log 2
Show the running log (2 entries, 2026-07-17)
2026-07-17 - Design freeze (lifecycle 31, eval-only; no seed consumed)
Frozen the six-seed eval-only confirmation of lifecycle 30's single-seed INSTALLED_TRANSFER (seed 78169: state_track aggregate 0.3260 vs count_walk 0.3004, paired lift +0.0256). No training, merging, corpus, or promotion in this cell.
Cloned machinery from the two reference cells:
- lifecycle 28
qwen35_4b_count_walk_menders_confirmation(the eval-only multi-seed confirmation): the k-seed write-ahead ledger with byte-equal crash reconciliation, arm authentication by frozen tree/weights pins, gateway sha pin, implementation-signature equality across all receipts, the sha-pinned prior-event summary (never pooled), the power-analysis script,check_benchmark.py, and the standalone lineage package + tests. - lifecycle 30
qwen35_4b_state_track_install(the two-arm aggregate install): the two-arm aggregate reading, the 1e-12 aggregate tie guard, the in-cell-authoritative provenance (siblings as verification aids), the per-file normalized provenance pins, and the full stage 1-9 lineage package (copied byte-identically;rebuild_lineage.pyadapted so the stage-9extended_byidentity stays the source cellqwen35_4b_state_track_install).
Design decisions frozen:
- Two arms (count_walk, state_track); no base arm - the event is the parent-vs-candidate paired comparison.
- Six fresh sealed seeds 78170-78175, verified grep-fresh in seed contexts repo-wide at design time (zero seed-context hits; all raw numeric matches are float/sha256 substrings in per-row data files). The next six free integers after the prior 78169 event.
Frozen PAIRED rule over the six new events: `d_i = state_track_aggregate
- count_walk_aggregate
;wins= events withd_i > 1e-12`;
mean_d= mean. CONFIRMED iffmean_d > 0andwins >= 4; NOT_CONFIRMED iffmean_d <= 0(dominates thewins >= 4edge); AMBIGUOUS otherwise. No fourth state. 78169 is prior evidence, never pooled.- count_walk_aggregate
Preregistered power (exact/quadrature, power_analysis.py --check enforced): under the pure null the paired majority rule is a LIBERAL directional check - false-CONFIRMED joint = 0.311 (scale-free; bounded by the exact P(wins>=4)=0.34375 and the independence product 0.17188). Under the observed +0.0256 lift the test has 0.9028-0.9839 CONFIRMED power across sigma_d in {0.02, 0.025, 0.03} (rho in {0.78, 0.65, 0.50}, via sigma_d = sigma_arm*sqrt(2(1-rho)), sigma_arm=0.03). The stated, high-value outcome is the decisive NOT_CONFIRMED (mean <= 0).
Adversarial review returned PASS_BENCHMARK_EVENT (see reports/benchmark_design_review.md); mutation probes (tampered tree/weights pins, tampered provenance sha, ledger re-open, spent budget, off-list seed 78169, over-budget arm) refuse fail-closed; crash walks reconcile byte-equal; the stage 1-9 lineage package verifies (rebuild_lineage.py --verify-inputs).
Verification green at design freeze: py_compile all scripts; 109 unit tests; run.py --smoke; power_analysis.py --check; rebuild_lineage.py --verify-inputs.
Deviation from the intake brief: the count_walk merge-receipt file sha in the intake was 4170b082..., which is actually the replay_compound receipt sha from a sibling cell; the correct count_walk merge-receipt file sha (used by lifecycle 30 and matching the file on disk) is 840edca0.... The correct pin is used here.
2026-07-17 — Six-seed confirmation complete: CONFIRMED (directional; statistically soft)
- Events 78170-78175 (12 sealed runs, budget-clean, implementation signature identical across all receipts and the prior 78169 event). Paired deltas d_i = state_track_agg - count_walk_agg (same seed): [-0.0123, +0.0373, -0.0385, +0.0050, +0.0439, +0.0887]. mean_d = +0.0207, wins = 4/6 -> frozen verdict CONFIRMED (mean_d > 0 AND wins >= 4).
- HONEST EFFECT SIZE (descriptive, promised at freeze): SD 0.0453, SE 0.0185, paired t = 1.12 on 5 df — NOT strictly significant (one-sided p ~ 0.16). The observed variance exceeds the preregistered sigma_d (0.02-0.03), exactly why the frozen rule was declared a LIBERAL directional check (alpha ~ 0.31), not a significance test. The mean lift (+0.0207) matches the single-seed 78169 observation (+0.0256); combined 7 seeds give mean +0.0214, 5/7 positive.
- READING: state_track is a real-but-small and noisy aggregate improvement over count_walk — durable enough to treat state_track as the current-best composite, but the edge is ~+0.02 with high seed variance, NOT a large or crisp gain. The install-universal- features doctrine is supported (a divergent skill added transferable aggregate and it replicated directionally); it is not proven at strict significance at n=6.
- Consequence: state_track adopted as the program reference composite for the next phase (the coding-harness measurement); the modest, noisy magnitude is carried forward honestly.
Data files 6
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/benchmark/medium_tb1024_seed78170_confirmation/summary.json2.4 kBruns/benchmark/medium_tb1024_seed78171_confirmation/summary.json2.4 kBruns/benchmark/medium_tb1024_seed78172_confirmation/summary.json2.4 kBruns/benchmark/medium_tb1024_seed78173_confirmation/summary.json2.4 kBruns/benchmark/medium_tb1024_seed78174_confirmation/summary.json2.4 kBruns/benchmark/medium_tb1024_seed78175_confirmation/summary.json2.4 kB
Reproduce
Smoke test
.venv/bin/python -B scripts/run.py --smokeFull run
.venv/bin/python -B scripts/run.py --stage benchmarkRun steps are documented inside the experiment folder (README and scripts).