Research log Small Model Experimentation
GitHub

Low-Density Token-Matched Universal Curriculum

Finished2026-07-13Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned SelectionPosttraining and Adaptation+3 moreGitHub ↗
Low-density doses failed the local gate

The one idea you need

The previous designed mix may have been too dense, and its replay comparison also saw more tokens. This follow-up changes only a tiny number of rows while holding total training exposure exactly equal, so the later difference can be tied much more cleanly to curriculum content.

The question

Can a tiny dose of abstract skill lessons improve the replay-refreshed model without erasing any of its broad gains?

What we found

No. Neither 40 nor 80 abstract lessons passed the fresh local screen. The stronger 80-lesson stream matched the inherited anchor at 54% accuracy and parsed 62% of answers, but still hit the answer cap 10 times; broad evaluation stayed sealed.

Why it matters

Exact exposure removes the prior compute ambiguity: these gentle lesson doses were too small to install reliable concise execution. The next attempt must bridge the dose gap or target answer commitment directly, while keeping a replay-only comparison and fresh evaluation seeds.

best local accuracy54%80-lesson stream tied the inherited anchor
best answer parse rate62%below the registered 90% gate
fewest answer-cap contacts10 of 26the registered ceiling was two
broad events consumedzeroall arms stopped at the local screen
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Next Experiments
    9. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 2

Only the final 80 training slots change

How to read

Every bar totals 1,520 rows. Moving from replay only to the 40- and 80-lesson doses replaces one or both token-matched replay blocks while the 1,440-row core stays fixed.

050010001500shared replayshared replay144014401440designed curriculumdesigned curriculum04080matched replay replacementsmatched replay replacements80400

Takeaway → The ladder isolates a very small designed-content increment at exactly equal forward-token exposure.

Data table
frozen dose componentreplay_repeatdesigned40designed80
shared replay144014401440
designed curriculum04080
matched replay replacements80400

Numbers from experiments/qwen35_4b_universal_low_density_token_match/data/dose_manifest.json

Technical framing

Nested doses replace exactly 0, 40, or 80 replay rows — All arms shared slot order, contained 1,520 rows, ran 190 optimizer steps, and totaled exactly 1,429,053 forward tokens. All trained cleanly before the fresh local screen.

Every exact-exposure stream missed the local gate

How to read

Two bars per model show exact-answer accuracy and answer parse rate on the same 26 fresh cases. The 80-lesson stream is best among the new streams, but neither bar reaches its registered threshold.

00.20.40.60.80.5380.50.50.538accuracy0.5770.5380.5380.615parse rate

Takeaway → Adding 80 lessons modestly improves readable answers over replay alone, yet accuracy only ties the inherited anchor and answer-cap failures remain frequent.

Data table
fresh local metricinherited anchorreplay_repeatdesigned40designed80
accuracy0.5380.50.50.538
parse rate0.5770.5380.5380.615

Numbers from experiments/qwen35_4b_universal_low_density_token_match/runs/local/seed88004_promotion.json

Technical framing

All exact-token arms fail the fresh local screen — Frozen gates required at least 65% accuracy, at least 90% parse rate, and at most two answer-cap contacts. Cap contacts were 11, 13, 12, and 10 respectively; no candidate was eligible.

In the author’s words from the Overview · “Results”

All three arms completed the exact 190-update, 1,429,053-forward-token contract with Every candidate missed the frozen accuracy ≥0.65, parse ≥0.90, and cap-contact ≤2 gates; all passed the feasible-route abstention check. The 80-row dose was the least degenerate candidate, but it only tied the inherited anchor's accuracy and remained far from deployable parsing. No arm became eligible, so the merge stage and aggregate-only seed 78,134 event did not run.

Overview

Outcome: completed negative on 2026-07-13; all arms failed the fresh local gate and the benchmark remained sealed.

Research program

Question

Can an order-of-magnitude lower density of truth-audited designed procedures add transfer to the replay-refreshed policy without giving back its broad gains, once the designed and replay-only arms are matched exactly on rows, optimizer steps, slot order, and forward-token exposure?

Hypothesis

The preceding candidate replaced 400 replay rows (26.3% of its dose) and learned the local procedures but lost to replay alone. At 40 or 80 designed rows (2.6% or 5.3%), the broad replay policy should remain dominant while the abstract procedures act as a small regularizing increment. If designed content is useful at all, one of these doses should beat an exact-token replay continuation; if neither does, the prior gap was not merely a compute mismatch or an excessive designed fraction.

Setup

  • Model: only Qwen/Qwen3.5-4B, revision 851bf6e....
  • Start: authenticated replay_refresh adapter from the parent.
  • Arms: replay_repeat (0 designed), designed40, and designed80.
  • Shared dose: 1,440 identical replay rows in identical shuffled slots.
  • Replacements: two 40-row, all-skill designed halves and two replay blocks whose token sums match those halves exactly.
  • Compute: every arm has 1,520 rows, 190 effective-batch-8 steps, and exactly 1,429,053 forward tokens.
  • Training: one epoch, learning rate 1e-5, rank 32 / alpha 64, w_think=0.2, seed 43, max length 4,096, zero skips.
  • Local screen: fresh synthetic seed 88,004; every arm is gated independently.
  • Paired pilot: aggregate-only quick@1,024 seed 78,134 on one merged vLLM backend.
  • Controls: base, blend, inherited replay-refresh anchor, and exact-token replay continuation. Both designed doses are prospectively registered, not adaptively chosen from benchmark results.
  • Hidden-label boundary: invoke only the trusted aggregate gateway. Never read or import benchmark items, sources, transcripts, or private output.

Run

Smoke:

.venv/bin/python experiments/qwen35_4b_universal_low_density_token_match/scripts/run.py --smoke

Full:

.venv/bin/python experiments/qwen35_4b_universal_low_density_token_match/scripts/run.py --stage train-control
.venv/bin/python experiments/qwen35_4b_universal_low_density_token_match/scripts/run.py --stage train-d40
.venv/bin/python experiments/qwen35_4b_universal_low_density_token_match/scripts/run.py --stage train-d80
.venv/bin/python experiments/qwen35_4b_universal_low_density_token_match/scripts/run.py --stage local
.venv/bin/python experiments/qwen35_4b_universal_low_density_token_match/scripts/run.py --stage merge
.venv/bin/python experiments/qwen35_4b_universal_low_density_token_match/scripts/run.py --stage benchmark

Results

All three arms completed the exact 190-update, 1,429,053-forward-token contract with zero skips. Fresh local seed 88,004 then produced:

ModelAccuracyParse rateCap contactsGate
inherited replay_refresh anchor0.5380.57711diagnostic only
replay_repeat0.5000.53813fail
designed400.5000.53812fail
designed800.5380.61510fail

Every candidate missed the frozen accuracy ≥0.65, parse ≥0.90, and cap-contact ≤2 gates; all passed the feasible-route abstention check. The 80-row dose was the least degenerate candidate, but it only tied the inherited anchor's accuracy and remained far from deployable parsing. No arm became eligible, so the merge stage and aggregate-only seed 78,134 event did not run.

Interpretation

Exact token matching closes the prior compute-exposure ambiguity for these doses: replacing 40 or 80 of 1,520 replay rows was insufficient to install robust concise execution into the replay-refreshed policy. The 80-row trend in parse rate and cap contacts is directional, not a pass and not evidence of broad transfer. The result does not say whether an intermediate dose, a termination-focused loss, or a lower-collateral integration mechanism can cross the local threshold while preserving the replay anchor; each requires a new result-separated experiment.

Knowledgebase Update

  • Program evidence: records the exact-token low-density local negative.
  • Program backlog: closes this ladder and requires a new result-separated mechanism.
  • Shared synthesis: low density alone does not resolve the install/retention tradeoff.
  • Claim ledger: unchanged; no benchmark event ran and no universal-feature claim exists.

Artifacts

  • data/source_token_lengths.json: tokenizer-frozen per-source row lengths.
  • data/dose_manifest.json: exact nested selection and hashes.
  • data/dose_token_receipt.json: zero-skip proof and exact arm exposures.
  • scripts/materialize_doses.py: deterministic nested token-matched constructor.
  • runs/local/seed88004.json: complete fresh local receipt.
  • runs/local/seed88004_promotion.json: authenticated fail-closed promotion decision.
  • reports/design_review.md and reports/preregistration.md: frozen review and gates.
  • reports/artifact_manifest.yaml: external adapter and merged-model locations.

Report

Rendered from reports/report.md

Summary

Completed negative. All three exact-token arms trained cleanly, but none passed the fresh local gate. The benchmark remained sealed.

Research Program Fit

This is the direct result-separated follow-up to the replay-anchor negative. It asks whether the designed procedures failed because their 26.3% density was too high or because they provide no broad increment beyond replay continuation.

Method

Every arm shares 1,440 replay rows in the same training slots. replay_repeat adds two 40-row replay blocks; designed40 swaps one for a stratified all-skill designed half; designed80 swaps both. The replay blocks match the two designed halves at 16,732 and 16,543 forward tokens exactly. Thus all arms contain 1,520 rows and 1,429,053 forward tokens and receive the same 190 updates from the same parent.

Results

All arms completed 190/190 updates over 1,520 rows with zero skips. Final training losses were 0.4069 (replay_repeat), 0.5128 (designed40), and 0.5864 (designed80); wall times were 1,380.519, 1,362.717, and 1,334.550 seconds.

On fresh local seed 88,004, replay repeat and the 40-row arm each scored 0.500 accuracy and 0.538 parse with 13 and 12 cap contacts. The 80-row arm scored 0.538 accuracy, 0.615 parse, and 10 cap contacts. The inherited replay-refresh anchor was 0.538 / 0.577 / 11. Every candidate missed the registered accuracy, parse, and cap bars, while all passed the feasible-route abstention check. No arm was eligible; explicit merge and paired benchmark stages did not run.

Controls

  • Exact-token replay continuation from the same anchor.
  • Inherited replay-refresh anchor, C53 blend, and pinned base.
  • Fresh local seed 88,004 and paired aggregate-only seed 78,134.
  • Explicit merges and a single qwen_vllm benchmark backend.
  • Prospectively registered 40- and 80-row doses with independent local eligibility.

Oracle Versus Deployable Evidence

Only deployable greedy local outputs and trusted aggregate benchmark fields are admissible. There is no oracle selector, private-item inspection, or benchmark-shaped training signal.

Interpretation

The exact-token comparison shows that 40 and 80 designed rows are below the local installation threshold from this anchor. The 80-row arm directionally improved parseability and cap behavior over replay repeat, but it did not improve accuracy over the inherited anchor and remained far below every absolute gate. This is a local mechanism negative, not a broad-retention measurement.

Next Experiments

Use a new result-separated experiment. The next design should explicitly bridge the large gap between the locally passing 400-row parent and the locally failing 80-row arm, or target concise answer commitment directly while preserving an exact-token replay control. It must use fresh local and benchmark seeds and keep the benchmark sealed until a prospectively frozen local mechanism gate passes.

Artifact Manifest

See artifact_manifest.yaml. All adapters and local receipts are authenticated; no merged checkpoint was produced.

Experiment log 5

Show the running log (5 entries, 2026-07-13)

2026-07-13 — intake, design freeze, and smoke

  • Created as the result-separated successor to qwen35_4b_universal_replay_anchor; the closest near-duplicate is named in idea_intake.md.
  • Copied the parent's checksum-pinned 800-row designed source and 2,240-row replay source. No benchmark content was read or copied.
  • Froze a common 1,440-row replay core and two 40-row designed halves covering all 13 skills. Deterministic local-swap selection found two disjoint 40-row replay blocks with exactly the same token sums: 16,732 and 16,543.
  • Materialized three position-aligned 1,520-row arms. Each has exactly 1,429,053 forward tokens, zero skipped rows, and 190 optimizer steps.
  • Froze training seed 43, local seed 88,004, and aggregate-only quick@1,024 seed 78,134 before training.
  • Adversarial design review passed with the explicit limitation that target-token composition differs by mechanism even though forward compute is exact.
  • Non-GPU smoke and six dose/local-gate tests passed. No training or new benchmark event has run.

2026-07-13 — replay-only exact-token control training

  • Ran scripts/run.py --stage train-control from design commit 740f30a3.
  • The wrapper reauthenticated the parent replay-refresh adapter, the checked-in replay_repeat bytes, and the zero-skip token receipt before launching training.
  • Completed 190/190 optimizer steps over 1,520 rows with zero skips in 1,380.519 wall seconds. Final training loss was 0.4069.
  • Adapter weights: 169,903,320 bytes, SHA-256 bb4f0f8d35ce51e59fb06e8fc835ef043ac8960a5c178e6a511ec75c0a622a07. Adapter config SHA-256: 8a89c0cc0ec7d7d9db475479a916c51c8b442c46b354991aa2eb27ab91017f17.
  • Durable receipt: runs/training/replay_repeat.json, SHA-256 e6f041e8b77dee0c80a625a7191a8609fed914238f3017e68ecb8f9517d6be5c. Full log: runs/training/replay_repeat.log, SHA-256 1083db586b0278745930eadb6088ca9c4b64f72d77d188714d33d5c340e76f1a.
  • No local or benchmark evaluation was performed at this checkpoint.

2026-07-13 — 40-row designed arm training

  • Ran scripts/run.py --stage train-d40 from replay-control checkpoint 668366b5 after both of that checkpoint's GitHub workflows passed.
  • Reauthenticated the same parent adapter and token receipt. The position-aligned stream differed from replay_repeat in exactly 40 rows and retained the exact 1,429,053-forward-token exposure.
  • Completed 190/190 optimizer steps over 1,520 rows with zero skips in 1,362.717 wall seconds. Final training loss was 0.5128.
  • Adapter weights: 169,903,320 bytes, SHA-256 b4ca4c0187797f57ae3259f7de1817be34aad927583c0a8728786c56b40ac4a9. Adapter config SHA-256: d70536028419d199d5ca4a273ad8af18a4819d3a1e40898cec2f74625eb1a964.
  • Durable receipt: runs/training/designed40.json, SHA-256 820e6df4aec64639e3fb1639a799ba94645c2297810950bc3e5c95291586773f. Normalized full log: runs/training/designed40.log, SHA-256 9dbb302ee6072b24abd39803ea85b80cab37099e400a51eb3c442ff5876470e8.
  • No local or benchmark evaluation was performed at this checkpoint.

2026-07-13 — 80-row designed arm training

  • Ran scripts/run.py --stage train-d80 from 40-row checkpoint 83d9f078 after that checkpoint was rebased, fully checked, and pushed to main.
  • Reauthenticated the same parent adapter and token receipt. The position-aligned stream differed from replay_repeat in exactly 80 rows and retained the exact 1,429,053-forward-token exposure.
  • Completed 190/190 optimizer steps over 1,520 rows with zero skips in 1,334.550 wall seconds. Final training loss was 0.5864.
  • Adapter weights: 169,903,320 bytes, SHA-256 ba82457d127c63662b5b86b4a2e1d94ed18014651b59aefd6512690eef1dabc4. Adapter config SHA-256: 6f3710860c78a559d3592d4a43389f0347b171d1f9bfc083737d448e2fe04520.
  • Durable receipt: runs/training/designed80.json, SHA-256 8f1ab673b7570452474639d351ac8e5d756fc574dce607c221d84edacd3838b5. Normalized full log: runs/training/designed80.log, SHA-256 23685837e14702657bcb941d34166e9997ed19d5c249f81fe30a3a6c553a9af1.
  • All three arms are now trained. No local or benchmark evaluation was performed at this checkpoint.

2026-07-13 — fresh local gate negative

  • Ran scripts/run.py --stage local from pushed checkpoint f30098ce after all three training arms and their receipts were independently published.
  • Evaluated the inherited replay-refresh anchor and all three arms on the same 26 fresh procedural cases at seed 88,004, greedy decoding, and a 1,024-token cap.
  • replay_repeat scored 0.500 accuracy, 0.538 parse, and 13 cap contacts; designed40 scored 0.500, 0.538, and 12; designed80 scored 0.538, 0.615, and 10. The inherited anchor scored 0.538, 0.577, and 11.
  • Every candidate passed the feasible-route abstention check but failed the frozen accuracy ≥0.65, parse ≥0.90, and cap-contact ≤2 checks. The promotion receipt therefore contains eligible: [] and exited nonzero as designed.
  • Full local receipt: runs/local/seed88004.json, SHA-256 5a012f15225d93cca00e7ea974472e3319a66e39b071b54418d323750631fcbb. Promotion receipt: runs/local/seed88004_promotion.json, SHA-256 fc90df3c56c861819c2bc6f8776fd659d6d37484c4357a3320bc6598288a1da7.
  • The merge stage did not run and benchmark seed 78,134 remains unconsumed.

Reproduce

Smoke test

.venv/bin/python experiments/qwen35_4b_universal_low_density_token_match/scripts/run.py --smoke

Full run

.venv/bin/python experiments/qwen35_4b_universal_low_density_token_match/scripts/run.py --stage all

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗