Research log Small Model Experimentation
GitHub

Mid-Density Token-Matched Universal Curriculum

Finished2026-07-13Agentic Breadth InstallationBenchmark GeneralizationPosttraining and AdaptationProcess Control and Tool Use+1 moreGitHub ↗
160 lessons nearly pass; benchmark stays sealed

The one idea you need

Very small doses of abstract lessons did not install reliable execution, while a much larger earlier dose learned locally but displaced broad behavior. This experiment tests two doses between those endpoints while keeping training exposure exactly equal to replay-only practice.

The question

Can a middle-sized dose of abstract skill lessons cross the local installation threshold without sacrificing the replay practice that protects broad capability?

What we found

The 160-lesson stream improved fresh local accuracy from 17 to 19 of 26 cases, raised readable answers from 18 to 23, and cut answer-limit contacts from nine to three. It still missed the readability and length gates by one case each. Adding 80 more lessons erased the accuracy gain, so no model advanced to broad evaluation.

Why it matters

The middle dose moved the right behaviors and came close, while the larger dose moved backward. That narrows the next intervention to concise answer commitment at the 160-lesson mix instead of adding more generic lessons or weakening the gate.

best local accuracy19 of 26160 lessons versus 17 for replay only
best readable answers23 of 26one case short of the registered gate
fewest answer-limit contactsthreeone above the registered ceiling
broad events consumedzeropromotion receipt was empty
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Next Experiments
    9. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 2

Two middle doses replace matched replay blocks

How to read

Every bar totals 1,520 rows. The designed streams replace two or three token-matched 80-row replay blocks while the 1,280-row replay core stays fixed.

050010001500shared replayshared replay128012801280designed curriculumdesigned curriculum0160240matched replay replacementsmatched replay replacements240800

Takeaway → The frozen ladder isolates lesson density at equal token exposure; all three streams trained cleanly.

Data table
frozen dose componentreplay_repeatdesigned160designed240
shared replay128012801280
designed curriculum0160240
matched replay replacements240800

Numbers from experiments/qwen35_4b_universal_mid_density_token_match/data/dose_manifest.json

Technical framing

Mid-density doses replace exactly 0, 160, or 240 replay rows — All arms share slot order, contain 1,520 rows, ran 190 optimizer steps, and totaled exactly 1,405,510 forward tokens. All three trained with zero skips; the fresh local comparison is pending.

The 160-lesson stream misses two local bars narrowly

How to read

Each model has bars for exact-answer accuracy and readable-answer rate on the same 26 fresh cases. The 160-lesson stream leads both, but readable answers remain one case short; its three answer-limit contacts are also one above the ceiling.

00.250.50.7510.6540.6540.7310.654accuracy0.6920.6920.8850.846parse rate

Takeaway → A middle dose improves correctness and concise completion, while 240 lessons reverse part of the gain; no arm is eligible for broad evaluation.

Data table
fresh local metricinherited anchorreplay_repeatdesigned160designed240
accuracy0.6540.6540.7310.654
parse rate0.6920.6920.8850.846

Numbers from experiments/qwen35_4b_universal_mid_density_token_match/runs/local/seed88005_promotion.json

Technical framing

The 160-row dose nearly clears the fresh local gate — Frozen gates required at least 65% accuracy, at least 90% parse rate, and at most two cap contacts. Cap contacts were 9, 9, 3, and 5 respectively; no arm was eligible.

In the author’s words from the Overview · “Results”

Every arm cleared the 0.65 accuracy bar and feasible-route abstention check. No arm simultaneously cleared parse ≥0.90 and cap contacts ≤2. designed160 was one parsed case and one cap contact short; designed240 was worse on all three primary metrics. The promotion receipt is empty, no model was merged, and aggregate seed 78,135 remained sealed.

Overview

Research program

Question

Can a representative 160- or 240-row truth-audited designed dose cross the fresh local installation gate from the strong replay-refresh anchor without returning to the parent 400-row density that failed broad retention?

Hypothesis

The 80-row arm improved parseability and cap behavior directionally but was below the local threshold, while the earlier 400-row arm passed locally from a weaker start and then displaced broad capability. A mid-density dose should cross the installation threshold while leaving more replay mass than the 400-row mixture. If neither 160 nor 240 rows passes, representative dose interpolation alone is not the missing mechanism.

Setup

  • Model: only Qwen/Qwen3.5-4B, revision 851bf6e....
  • Start: authenticated replay_refresh adapter from the replay-anchor experiment.
  • Arms: replay_repeat (0 designed), designed160, and designed240.
  • Shared dose: 1,280 identical replay rows in identical shuffled slots.
  • Replacements: three disjoint 80-row designed blocks, each covering all 13 skills, and three disjoint replay blocks with exactly matching forward-token sums.
  • Feasibility boundary: a representative 320-row arm was rejected before freeze because 320 proportional designed rows were shorter than the 320 shortest replay rows; exact matching would have required an unregistered length-biased curriculum.
  • Compute: every arm has 1,520 rows, 190 effective-batch-8 steps, and exactly 1,405,510 forward tokens with zero tokenizer skips.
  • Training: one epoch, learning rate 1e-5, rank 32 / alpha 64, w_think=0.2, seed 43, and max length 4,096.
  • Local screen: fresh procedural seed 88,005; every arm is gated independently at accuracy ≥0.65, parse ≥0.90, cap contacts ≤2, and no repeated feasible-route abstention.
  • Conditional paired pilot: aggregate-only quick@1,024 seed 78,135 on one explicitly merged vLLM backend.
  • Controls: base, blend, inherited replay-refresh anchor, and a new exact-token replay continuation. Both designed doses are prospectively registered.
  • Hidden-label boundary: invoke only the trusted aggregate gateway. Never read or import benchmark items, sources, transcripts, or private output.

Run

Smoke and staged run:

.venv/bin/python experiments/qwen35_4b_universal_mid_density_token_match/scripts/run.py --smoke
.venv/bin/python experiments/qwen35_4b_universal_mid_density_token_match/scripts/run.py --stage train-control
.venv/bin/python experiments/qwen35_4b_universal_mid_density_token_match/scripts/run.py --stage train-d160
.venv/bin/python experiments/qwen35_4b_universal_mid_density_token_match/scripts/run.py --stage train-d240
.venv/bin/python experiments/qwen35_4b_universal_mid_density_token_match/scripts/run.py --stage local
.venv/bin/python experiments/qwen35_4b_universal_mid_density_token_match/scripts/run.py --stage merge
.venv/bin/python experiments/qwen35_4b_universal_mid_density_token_match/scripts/run.py --stage benchmark

Results

All three arms completed 190/190 updates over their authenticated 1,520-row, 1,405,510-forward-token streams with zero skips. Fresh local seed 88,005 produced:

  • inherited anchor: 17/26 accuracy, 18/26 parsed, 9 cap contacts;
  • replay_repeat: 17/26 accuracy, 18/26 parsed, 9 cap contacts;
  • designed160: 19/26 accuracy, 23/26 parsed, 3 cap contacts;
  • designed240: 17/26 accuracy, 22/26 parsed, 5 cap contacts.

Every arm cleared the 0.65 accuracy bar and feasible-route abstention check. No arm simultaneously cleared parse ≥0.90 and cap contacts ≤2. designed160 was one parsed case and one cap contact short; designed240 was worse on all three primary metrics. The promotion receipt is empty, no model was merged, and aggregate seed 78,135 remained sealed.

Interpretation

Representative dose interpolation has a sharp, nonmonotonic local effect. The 160-row dose adds two correct cases over exact-token replay and cuts missing answers from eight to three, cap contacts from nine to three, and mean generation from 729 to 511 tokens. Adding another 80 designed rows reverses the accuracy gain and worsens parse and cap behavior. The next result-separated experiment should keep the 160-row capability mix fixed and target the remaining answer-commit/termination seam with exact-token controls; it must not lower these observed gates or spend the sealed benchmark seed.

Knowledgebase Update

  • Program evidence: exact-token 160-row near-miss and 240-row reversal recorded.
  • Program backlog: termination-focused successor queued with fresh seeds.
  • Claim ledger: unchanged; a single pilot cannot establish a universal-feature claim.

Artifacts

  • data/source_token_lengths.json: tokenizer-frozen source-row lengths.
  • data/dose_manifest.json: exact representative selection, slot, and hash receipt.
  • data/dose_token_receipt.json: zero-skip proof and exact arm exposure.
  • scripts/materialize_doses.py: deterministic nested exact-token constructor.
  • reports/design_review.md and reports/preregistration.md: frozen threats and gates.
  • reports/artifact_manifest.yaml: external adapter and merged-model locations.
  • runs/local/seed88005.json: complete experiment-owned local receipt.
  • runs/local/seed88005_promotion.json: empty promotion decision; benchmark sealed.

Report

Rendered from reports/report.md

Summary

Completed local negative. A representative nested 0/160/240-row designed-dose ladder was trained from replay refresh with exact forward-token matching; no arm passed the fresh local gate, so merge and benchmark remained sealed.

Research Program Fit

This is the direct result-separated follow-up to the 0/40/80 local negative. It bridges the known gap between an 80-row dose that did not install locally and the earlier 400-row mixture that installed locally but lost broad capability.

Method

Every arm shares 1,280 replay rows in the same slots. Three disjoint 80-row designed blocks each cover all 13 skills and have disjoint replay counterparts with exactly the same token sums. Replay repeat uses all replay blocks, designed160 replaces A and B, and designed240 replaces A, B, and C. Every arm contains 1,520 rows, 1,405,510 forward tokens, and 190 updates from the same parent.

Results

All three exact-token arms completed 190/190 updates over 1,520 rows with zero skips. On fresh local seed 88,005, anchor and replay repeat each scored 17/26 accuracy, 18/26 parse, and 9 cap contacts. designed160 scored 19/26, 23/26, and 3; designed240 scored 17/26, 22/26, and 5. Every arm passed accuracy ≥0.65 and the route check, but none passed parse ≥0.90 and cap contacts ≤2. Promotion was empty, so no merge or paired benchmark occurred.

Controls

  • Exact-token replay continuation from the same anchor.
  • Inherited replay-refresh anchor, blend, and pinned base.
  • Fresh local seed 88,005 and conditional aggregate-only seed 78,135.
  • Explicit merges and one qwen_vllm benchmark backend.
  • Both doses registered before any model training.

Oracle Versus Deployable Evidence

Only deployable greedy local outputs and trusted aggregate benchmark fields are admissible. There is no oracle selector, private-item inspection, or benchmark-shaped training signal.

Interpretation

The 160-row dose is a real near-threshold local improvement: versus exact-token replay it adds two correct cases, five parsed answers, removes six cap contacts, and shortens mean output by about 218 tokens. The effect is not monotonic: 240 rows lose the accuracy gain and regress parse/cap behavior relative to 160. This rejects representative dose interpolation as sufficient, not the curriculum signal. The remaining local bottleneck is concise answer commitment at the 160-row capability mix. There is no broad-transfer evidence because the benchmark correctly stayed sealed.

Next Experiments

Use a new experiment and fresh local seed to hold the 160-row capability mix fixed while adding a small truth-audited answer-commit/termination intervention with an exact-token active control. Do not add more representative dose, lower the observed gate, reuse seed 88,005, or spend aggregate seed 78,135. Any later pilot pass still requires independent quick replication, medium@2,048, paired uncertainty, and matched-compute sampling.

Artifact Manifest

See artifact_manifest.yaml; planned entries must be replaced with authenticated checksums immediately after each successful stage.

Experiment log 5

Show the running log (5 entries, 2026-07-13)

2026-07-13 — intake, feasibility revision, and design freeze

  • Created as the result-separated successor to the exact-token 0/40/80 local negative; the closest near-duplicate is named in idea_intake.md.
  • Copied the authenticated 800-row designed and 2,240-row replay sources. No benchmark content was read or copied.
  • Rejected the initially proposed 320-row arm before freeze: proportional designed rows were collectively shorter than the shortest row-matched replay selection.
  • Froze a representative 0/160/240 ladder with a 1,280-row common replay core and three all-skill 80-row blocks. Each designed/replay block matches forward tokens exactly at 33,613, 34,091, and 33,015.
  • Materialized three position-aligned 1,520-row arms. Each has exactly 1,405,510 forward tokens, zero skipped rows, and 190 optimizer steps.
  • Froze training seed 43, local seed 88,005, and conditional aggregate-only quick@1,024 seed 78,135 before training.
  • Adversarial design review passed after the 320-row feasibility revision. No model training or new evaluation event has run.

2026-07-13 — replay-only exact-token control training

  • Ran scripts/run.py --stage train-control from design commit 49e42f0b.
  • The wrapper reauthenticated the replay-refresh parent, checked-in replay_repeat bytes, and zero-skip token receipt before launching training.
  • Completed 190/190 optimizer steps over 1,520 rows with zero skips in 1,396.686 wall seconds. Final training loss was 0.4199.
  • Adapter weights: 169,903,320 bytes, SHA-256 7db84c6313fbf479ec7d08334cfa41a1b4883c95a7a0215eb40b2059e55d2ac5. Adapter config SHA-256: 12943a035cdbe3cbded978903911fad6e135dddeb1872a1298371269ade3cd4f.
  • Durable receipt: runs/training/replay_repeat.json, SHA-256 950d30cd3a69ebe1957729e6da099913096f7eb2a2443585c4e25bed6c053eed. Normalized full log: runs/training/replay_repeat.log, SHA-256 e8a72ef7fde6cdb7d3f642ffeb5e93755a8783f2c1bfe9feaf6f2b7c7a6c83f8.
  • No local or benchmark evaluation was performed at this checkpoint.

2026-07-13 — 160-row exact-token arm training

  • Ran scripts/run.py --stage train-d160 from published replay-control commit 0ad7ca07.
  • The wrapper reauthenticated the replay-refresh parent, checked-in designed160 bytes, and zero-skip token receipt before launching training.
  • Completed 190/190 optimizer steps over 1,520 rows with zero skips in 1,390.190 wall seconds. Final training loss was 0.6606.
  • Adapter weights: 169,903,320 bytes, SHA-256 f05c13ae66e19bbd29abbd2b62ae3c1a577642efefbdba435879012bf4494654. Adapter config SHA-256: 0cd3ca7c710e48c264fd1d4c019c304ec0b9e5b13098b89a5b4aa0b171191e58.
  • Durable receipt: runs/training/designed160.json, SHA-256 485e3a76a8ef45d92df0a60dbcc338d5c1d4ddfc53fbb4e83acb10af7e75d258. Normalized full log: runs/training/designed160.log, SHA-256 34ce943073908585c933991161f84a076a734b98cc8a23c0fc953dde1db995f3.
  • No local or benchmark evaluation was performed at this checkpoint.

2026-07-13 — 240-row exact-token arm training

  • Ran scripts/run.py --stage train-d240 from published 160-row checkpoint 71d6e641.
  • The wrapper reauthenticated the replay-refresh parent, checked-in designed240 bytes, and zero-skip token receipt before launching training.
  • Completed 190/190 optimizer steps over 1,520 rows with zero skips in 1,373.185 wall seconds. Final training loss was 0.7284.
  • Adapter weights: 169,903,320 bytes, SHA-256 9b159156a7fd59f259454427371e7eb6f72dc2bb1f4e51d6cb6c9dc169af0116. Adapter config SHA-256: 66fb435d789ce43a741a04cdbceca1f3c133f157310e2751cbead79d7e51531c.
  • Durable receipt: runs/training/designed240.json, SHA-256 8bc2f0528f75766c0954ec55455cf9873e4396324c55707f8b1a617af707d82d. Normalized full log: runs/training/designed240.log, SHA-256 3c2e0692742ac13f3712a0825d86fb728ef561eca22c3a551b1d8923aca77838.
  • All three exact-token arms are now trained. No local or benchmark evaluation was performed at this checkpoint.

2026-07-13 — fresh local gate negative

  • Ran scripts/run.py --stage local from published all-arms checkpoint 68667bea.
  • Consumed the single registered experiment-owned seed 88,005 event across the inherited anchor and all three arms at greedy decode and 1,024 generated tokens.
  • Anchor and replay_repeat each scored 17/26 accuracy, 18/26 parse, and 9 cap contacts. designed160 scored 19/26, 23/26, and 3; designed240 scored 17/26, 22/26, and 5. All candidates had zero feasible-route abstentions.
  • Every arm passed accuracy ≥0.65 and the route check. Every arm failed parse ≥0.90 and cap contacts ≤2. The 160-row arm missed each remaining bar by one case.
  • Full receipt: runs/local/seed88005.json, SHA-256 ca1a33612ba7dcd430c05a90ee3953358b1137b345101ec78e829e64137bffb3. Promotion receipt: runs/local/seed88005_promotion.json, SHA-256 3bdad3e9e7536972e7d6178484d02098503d5c77a7a14cb530f46af0173ea41a.
  • Gate receipt hashes: replay 5e2861186d11a454695a47b9e806391d58950fd12d1dfacc3b8fc7c6a8f39279, 160 rows ee70d0fd88400a4909b4e0b7b45537f2bfe4697ed47521126a56f971bcba03b0, and 240 rows 7fca078a2050586f91d0eaa233932f9991704315175a98fa5a48b3830082dbd8.
  • Eligible list was empty. No merge or benchmark ran; seed 78,135 remains sealed.

Reproduce

Smoke test

.venv/bin/python experiments/qwen35_4b_universal_mid_density_token_match/scripts/run.py --smoke

Full run

.venv/bin/python experiments/qwen35_4b_universal_mid_density_token_match/scripts/run.py --stage all

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗