Research log Small Model Experimentation
GitHub

Fresh-Surface Budget-Commit Universal Curriculum

Finished2026-07-14 → 15Agentic Breadth InstallationBenchmark GeneralizationGitHub ↗
New words worked; rule-guessing wall stayed

The one idea you need

Take the practice set that nearly worked before, rewrite every exercise with brand-new vocabulary, and add a new lesson that teaches working to a strict step budget and committing an answer the moment the budget says stop.

The question

Does the near-miss practice set still work when its vocabulary is completely replaced, and does budget-and-commit practice fix the run-on answers that kept failing the screen?

What we found

The rewritten practice set beat both comparison models on the big screen — more right answers (69 vs 63 and 62 of 104), many fewer run-on answers (7 vs 18 and 13), and 31 percent shorter output — on vocabulary it had never trained on. But the screen also required four of eight rule-guessing tasks right, and every model scored zero of eight there, so nothing advanced to the benchmark. The budget lesson made things worse, not better.

Why it matters

The practice set's benefit binds to structure, not to specific words — the first positive mechanism reading in this line. And the screen's rule-guessing requirement is now known to be impossible for this model family, so future tests must either attack rule-guessing differently or measure what can actually move.

Right answers69 / 104vs 63 (parent) and 62 (pure review) — all strict wins
Run-on answers7vs 18 (parent) and 13 (review); output 31% shorter
Rule-guessing0 / 8every model, including both comparisons
Benchmarknever openedseed 78,143 sealed by the frozen rule
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Next Stage
    8. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 1

The rewritten practice set wins the screen; the rule-guessing floor blocks everyone

How to read

Grouped bars per model: right answers and cleanly finished answers (of 104), and run-on answers (fewer is better).

0255075100parentparent638718replay_repeatreplay_repeat629113designed_freshdesigned_fresh69977budget_commitbudget_commit628816

Takeaway → The rewritten set beats both comparisons on every headline number, but the screen's rule-guessing requirement was zero-for-eight for every model, so nothing advanced.

Data table
explicit merged compositecorrect (of 104)parsed (of 104)cap contacts
parent638718
replay_repeat629113
designed_fresh69977
budget_commit628816

Numbers from experiments/qwen35_4b_universal_fresh_surface_budget_commit_target_match/reports/report.md

Technical framing

104-task local gate: totals by arm (seed 88013) — designed_fresh won all four preregistered strict comparisons on never-trained surfaces and cut generation 31%, but the induct floor (>=4/8) was 0/8 for every arm, so no candidate promoted and aggregate seed 78,143 stays sealed. budget_commit was at-or-below replay everywhere.

In the author’s words from the Overview · “Results”

All three arms trained from green checkpoints with zero skipped rows and exactly matched exposure (losses: replay 0.4063, designed 0.4634, budget 0.5106), merged as authenticated explicit composites, and evaluated in the one frozen 104-task local gate event at seed 88,013. Totals (correct / parsed / cap contacts of 104): parent 63/87/18; replay_repeat 62/91/13; designed_fresh 69/97/7; budget_commit 62/88/16. Mean generated tokens: parent 515.8, replay 534.0, designed_fresh 357.2, budget_commit 396.2. designed_fresh passed the correct (≥68), parse (≥96), caps (≤8), and route-abstention bars and won ALL FOUR preregistered strict comparisons (total and the 24-row execute+induct+probe subtotal, versus both parent and replay). … Read the full result →

Overview

Re-dose the proven 160-row designed distribution on six disjoint fresh surfaces from the authenticated parent, and ablate 40 of those rows into a bounded-check budget-commit lesson, under a three-axis exact-exposure replay control and a quadrupled 104-task local gate.

Research Program

  • Program: agentic_breadth_installation.
  • Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
  • Prior anchors: mid-density near-miss (19/26 with parse and caps each one item short); close-weight redistribution negative; terminal successful-sibling availability stops closing harvest-based supervision.

Question

Is the 160-row designed dose surface-general — does it install when rendered on six entirely fresh surfaces from the current parent — and does substituting 40 rows with a designed bounded-computation budget-commit lesson repair the termination seam (cap contacts, unparsed answers) at exactly matched exposure?

Hypothesis

The designed dose binds to structure, not surface tokens, so a fresh-surface re-dose from the replay-refreshed parent should reproduce the mid-density accuracy gain on a gate that renders only original surfaces (a built-in transfer test). The remaining deployment seam is termination; a lesson whose content is monitored bounded computation with a mandatory canonical commit — never a loss-weight knob and never an idealized long trace — should reduce cap contacts and unparsed answers without losing semantics. Exact three-axis exposure matching (forward tokens, loss-bearing targets, absolute loss mass) against an active replay continuation makes any win attributable to content.

Setup

  • Model: only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
  • Parent: authenticated explicit composite replay_after_close (weights 7ab4c419...e2e6); runtime LoRA is forbidden.
  • Treatment corpora (construction seed 77,116): designed_fresh = 160 rows at the frozen designed160 per-skill quotas on six fresh surfaces (greek, elements, animals, ordinals, gems, digraphs) with fresh separators, attributes, and routing capabilities; budget_commit = a deterministic 120-row subset of the same corpus plus 40 budget lessons (hard check allowance, stop-on-first-hit, mandatory BUDGET commit on exhaustion, decoy planted immediately past the cutoff). A banned-vocabulary audit proves zero leakage of predecessor surfaces, gym family names, and public benchmark family names.
  • Streams: 1,280 shared position-aligned replay rows plus a 240-row variable block per arm; the two treatment blocks (160 treatment + 80 replay filler) and the 240-row replay control block are matched EXACTLY on forward tokens, nonzero loss-bearing targets, and absolute loss mass (MILP, namespace seed 55,117).
  • Training: one event per arm, 1,520 rows, 190 optimizer steps, batch 1, accumulation 8, LR 1e-5, rank 32, alpha 64, think/close weights 0.2/0.2, seed 51, warm start from the parent adapter (bb59d3bd...154d), zero skipped rows required.
  • Local gate: fresh seed 88,013, 104 tasks (8 per each of 13 skills) from the ORIGINAL-surface generator, explicit merged composites on the pinned vLLM geometry, greedy, natural thinking, 1,024-token cap. Bars per candidate: parsed ≥ 96, correct ≥ 68, cap contacts ≤ 8, route abstentions ≤ 4, execute/induct/probe ≥ 4/8 each, plus strict wins over parent AND replay on total correct and the 24-row execute+induct+probe subtotal. Single-winner promotion: higher total, then target subtotal, then fewer caps, then budget_commit.
  • Conditional aggregate: sealed fresh seed 78,143, quick tier, think budget 1,024, same-backend paired event over base, parent, replay control, and the promoted candidate. Promotion requires the candidate to strictly lift the aggregate AND every one of the ten public families versus base, and to strictly beat both the replay control and the parent on aggregate.
  • Hidden boundary: benchmarks/ remains unread; independent higher-tier confirmation and matched-compute sample-more remain mandatory before any universal claim.

Run

Smoke:

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_universal_fresh_surface_budget_commit_target_match/scripts/run.py --smoke

Checkpointed stages (each requires its prerequisite committed at a clean, pushed, green main):

.venv/bin/python -B experiments/qwen35_4b_universal_fresh_surface_budget_commit_target_match/scripts/run.py --stage train-control
.venv/bin/python -B experiments/qwen35_4b_universal_fresh_surface_budget_commit_target_match/scripts/run.py --stage train-designed
.venv/bin/python -B experiments/qwen35_4b_universal_fresh_surface_budget_commit_target_match/scripts/run.py --stage train-budget
.venv/bin/python -B experiments/qwen35_4b_universal_fresh_surface_budget_commit_target_match/scripts/run.py --stage merge-arms
.venv/bin/python -B experiments/qwen35_4b_universal_fresh_surface_budget_commit_target_match/scripts/run.py --stage local
.venv/bin/python -B experiments/qwen35_4b_universal_fresh_surface_budget_commit_target_match/scripts/run.py --stage benchmark

Results

All three arms trained from green checkpoints with zero skipped rows and exactly matched exposure (losses: replay 0.4063, designed 0.4634, budget 0.5106), merged as authenticated explicit composites, and evaluated in the one frozen 104-task local gate event at seed 88,013.

Totals (correct / parsed / cap contacts of 104): parent 63/87/18; replay_repeat 62/91/13; designed_fresh 69/97/7; budget_commit 62/88/16. Mean generated tokens: parent 515.8, replay 534.0, designed_fresh 357.2, budget_commit 396.2.

designed_fresh passed the correct (≥68), parse (≥96), caps (≤8), and route-abstention bars and won ALL FOUR preregistered strict comparisons (total and the 24-row execute+induct+probe subtotal, versus both parent and replay). It failed only the per-kind floors: execute 3/8 and induct 0/8 (probe 6/8 passed). budget_commit failed broadly and was at-or-below replay on every headline number. Decisively: induct was 0/8 for every arm including the parent, so the induct ≥ 4/8 floor was structurally unpassable in this event. No candidate promoted; per the frozen contract, aggregate seed 78,143 is permanently sealed and no benchmark event ran.

Interpretation

Three separable findings.

  1. The preregistered surface-generality reading is POSITIVE: the designed dose, re-rendered on six entirely fresh surfaces, strictly beat both the parent and the exactly-matched replay control on total correct and on the target subtotal, on gate tasks whose surfaces it never saw — while also cutting cap contacts by 11, raising parses by 10, and shortening generation by 31% versus the parent. The designed-160 effect binds to structure, not to its original surface vocabulary. This is a single-seed local mechanism screen, not a capability claim.
  2. The budget lesson is falsified as designed: substituting 40 designed rows for bounded-scan lessons made termination WORSE (16 caps vs 7), not better — the stop-on-contract behavior did not leave its narrow format, and the lost designed rows cost semantics.
  3. The gate's per-kind induct floor (≥4/8) is unpassable by this lineage: at eight induction tasks the wall (C38/C39) shows as exactly zero for every arm. Under the 26-task predecessor gate the same floor (≥1/2) was occasionally passable by luck; quadrupling the screen exposed it as structural. Any successor must either treat induction with a fundamentally different mechanism or preregister a gate whose floors are achievable given the known wall.

Terminal Disposition

No later event is authorized here. Aggregate seed 78,143 was never opened and is spent-by-sealing; do not lower the induct floor, re-run the local event, or benchmark the unpromoted composites from this directory. The designed_fresh adapter, composite, and receipts are published and may serve as a parent for successors with their own lifecycle.

Knowledgebase Update

  • Program evidence updated: terminal local negative with the positive surface-generality mechanism reading and the budget-lesson negative recorded.
  • Program backlog updated: the generic-dose line closes; the goal-gap axis-curriculum successor is queued.
  • Claim ledger updated: no; local mechanism screens do not mint claims.

Artifacts

  • data/sft_fresh_designed160.jsonl, data/sft_fresh_budget160.jsonl, data/corpus_manifest.json: frozen fresh-surface corpora.
  • data/sft_blend.jsonl: frozen replay pool (byte-identical to predecessors).
  • data/stream_manifest.json, data/stream_token_receipt.json: three-axis exposure receipts.
  • data/local_tasks_seed88013.jsonl, data/local_input_seed88013.jsonl, data/local_design_receipt.json: frozen local gate.
  • reports/preregistration.md, reports/design_review.md: prospective contract and adversarial authorization.
  • reports/artifact_manifest.yaml: external parent and conditional model-artifact plan.

Report

Rendered from reports/report.md

Summary

The experiment is closed as a terminal local negative with a positive preregistered mechanism reading. All three exactly-matched arms trained and merged cleanly; in the one frozen 104-task surface-transfer gate event, designed_fresh won all four strict comparisons over parent and replay (69 vs 63/62 correct; 9 vs 6/7 on the 24 target rows) with 7 cap contacts and 97 parses, but the induct floor (≥4/8) was 0/8 for every arm — structurally unpassable — so no candidate promoted and aggregate seed 78,143 is permanently sealed. The budget-commit substitution was worse than replay across the board.

Research Program Fit

This trial claims the program's queued bounded-computation plus canonical-answer commitment successor slot with a designed, non-harvested mechanism, after the two successful-sibling stops closed harvest-based supervision. It simultaneously tests the doctrine's open surface-diversity question: whether the near-miss designed dose binds to structure rather than to its original surface vocabulary.

Method

  • Arm D (designed_fresh): the frozen designed160 per-skill distribution rendered over six fresh surface pools (greek, elements, animals, ordinals, gems, digraphs) with fresh separators, record attributes, and routing capabilities; construction seed 77,116; zero banned-vocabulary occurrences.
  • Arm B (budget_commit): a deterministic 120-row largest-remainder subset of arm D plus 40 budget lessons — ordered scan under a hard check allowance, stop at the first hit, mandatory BUDGET commit on exhaustion, and a decoy satisfier planted immediately past the cutoff so violating the contract is parseable-but-wrong. 27 hit / 13 exhaust outcomes.
  • Control (replay_repeat): 240 replay rows solved jointly with the two 80-row filler sets by one MILP so that all three 240-row variable blocks are EXACTLY equal on forward tokens (142,211), nonzero loss-bearing targets (63,739), and absolute loss mass in fifths (72,755). Solve time 7.1 s; deterministic re-solve verified.
  • Streams: 1,280 shared position-aligned replay rows + the arm block = 1,520 rows and 1,356,964 forward tokens per arm; zero encoder-skipped rows; slot permutation seed 55,117.
  • Training plan: one event per arm from the authenticated replay_after_close parent adapter, 190 optimizer steps, LR 1e-5, think/close weights 0.2/0.2, seed 51.
  • Local gate: 104 fresh tasks (8 per skill) from the ORIGINAL-surface generator at seed 88,013 — training never sees these surfaces, so the gate is itself a surface-transfer test. Frozen bars: parsed ≥ 96, correct ≥ 68, caps ≤ 8, route abstentions ≤ 4, execute/induct/probe ≥ 4/8, strict wins over parent AND replay on totals and the 24-row target subtotal; single-winner promotion with a frozen tiebreak.
  • Conditional aggregate: sealed seed 78,143, quick tier, think budget 1,024, four merged composites (base, parent, replay control, promoted candidate); promotion needs a strict lift of the aggregate and every public family versus base plus aggregate wins over replay and parent.

Results

  • Training (each 1,520 rows, 0 skipped, 190 updates, seed 51): replay 0.4063, designed 0.4634, budget 0.5106 train loss.
  • Local gate totals (correct/parsed/caps of 104): parent 63/87/18; replay 62/91/13; designed_fresh 69/97/7; budget_commit 62/88/16. Mean generated tokens 515.8/534.0/357.2/396.2.
  • designed_fresh strict wins: total 69>63 and 69>62; target24 9>6 and 9>7 — all four preregistered comparisons, on surfaces it never trained on.
  • Per-kind floors: designed_fresh execute 3/8, induct 0/8, probe 6/8. Induct was 0/8 in every arm including the parent.
  • No promotion; seed 78,143 sealed; no benchmark event. Frozen artifact hashes: corpora e599f156...84d5 / ecece8e2...9800; exposure receipt c90873af...bb54; local receipt and promotion record under runs/local/.

Controls

The explicit parent composite is the baseline; the exact-exposure replay continuation is the mechanism-falsifying control; arm B versus arm D is a 40-row content-substitution ablation at identical exposure. Retention is measured on all 13 skills, not inferred.

Oracle Versus Deployable Evidence

Executable truth grades outputs only and is stripped from every model-facing byte. benchmarks/ remains unread; only the trusted gateway's aggregate and public per-family scores will be consumed.

Next Stage

None. The experiment is closed terminally under the frozen no-promotion rule. The published designed_fresh adapter and composite may parent a successor with its own lifecycle; the program pivot (goal-gap axis curriculum) is recorded in the backlog.

Artifact Manifest

All model-free artifacts are tracked in-repo. The parent composite, parent adapter, and reserialized base are external with pinned checksums. Future adapters and merged composites will live under large_artifacts/ with tracked receipts.

Experiment log 7

Show the running log (7 entries, 2026-07-14 → 15)

2026-07-14 — Model-free design freeze

  • Opened after the residual successful-sibling terminal stop published green; this trial claims the program's queued bounded-computation plus canonical-answer commitment successor slot with a designed, non-harvested mechanism.
  • Wrote scripts/gen_fresh_curriculum.py: the thirteen predecessor lesson constructors over six fresh surface pools (greek, elements, animals, ordinals, gems, digraphs), fresh separators/attributes/capabilities, plus the new u_budget bounded-check lesson with a planted decoy immediately past the allowance cutoff.
  • Froze the corpora at construction seed 77,116: arm D sft_fresh_designed160.jsonl (160 rows, designed160 quotas, sha e599f156...84d5) and arm B sft_fresh_budget160.jsonl (120-row deterministic subset of arm D plus 40 budget lessons, 27 hits / 13 exhausts, sha ecece8e2...9800). Banned-vocabulary audit passes; --check regenerates byte-identically.
  • Reserved fresh construction/slot-match/training/local/aggregate seeds 77116/55117/51/88013/78143; aggregate seed sealed.
  • Preregistered the full contract in reports/preregistration.md: three-axis exact exposure match, one training event per arm from the authenticated replay_after_close parent, a 104-task original-surface local gate with strict wins over parent and replay, single-winner promotion, and a four-model conditional aggregate pilot (strictly lift aggregate and every public family versus base; strictly beat replay and parent on aggregate).
  • No model, GPU, training, local, or benchmark event has run.

2026-07-14 — Authenticated control training

  • train-control ran only after design-freeze commit 1d82b6c7 matched origin/main with both workflows green and a clean worktree.
  • replay_repeat trained 1,520/1,520 rows with 0 skipped over 190 updates (train loss 0.4063, 1,341.9 wrapper seconds); receipt and log published under runs/training/ and their hashes pinned fail-closed in train_trial.py.
  • No evaluation ran; the treatment arms remain untrained until this checkpoint publishes green.

2026-07-14 — Authenticated designed-fresh training

  • train-designed ran only after control checkpoint cd1cba9e matched origin/main with both workflows green and a clean worktree.
  • designed_fresh trained 1,520/1,520 rows with 0 skipped over 190 updates (train loss 0.4634, 1,334.7 wrapper seconds); receipt/log published and pinned.
  • The budget arm remains untrained until this checkpoint publishes green.

2026-07-15 — Authenticated budget-commit training

  • train-budget ran only after designed checkpoint c54a5378 matched origin/main with both workflows green and a clean worktree.
  • budget_commit trained 1,520/1,520 rows with 0 skipped over 190 updates (train loss 0.5106, 1,393.4 wrapper seconds); receipt/log published and pinned.
  • All three arms are trained; merges are the only next stage.

2026-07-15 — Merge-gate pin amendment (model-free)

  • The merge stage refused to open: merge_trained_arm.py demands its own hash under the receipt's code_sha256.merge, but the receipt generator had listed it as pin-deferred — an implementation inconsistency between two harness files, caught by the gate itself failing closed. No composite was produced.
  • merge_trained_arm.py carries no orchestrator-filled constants, so it is now pinned at receipt level; the receipt regenerated with the frozen local tasks and oracle-free input byte-identical (be817bd0..., 7cba75dc...). Bars, seeds, and every other frozen field are unchanged.
  • No model, GPU, or evaluation event ran during the amendment.

2026-07-15 — Authenticated explicit composites

  • merge-arms ran only after the amendment checkpoint 6a6f7ee7 matched origin/main with both workflows green; the PASS_CONTROL_MERGE verdict and the merge self-pin were both required and verified.
  • All three arms merged through the pinned external merger (scale 2.0, 128/128 nonzero modules, fingerprint-verified): tree hashes f2aa4a76...2523 (replay_repeat), 93433aa2...0255 (designed_fresh), 8faf6f68...ff18 (budget_commit); receipts and logs under runs/merges/.
  • The three merged-tree pins are now filled fail-closed in the local evaluator. The one frozen local gate event is the only next stage.

2026-07-15 — Terminal local gate event and closure

  • The one frozen local event ran from merge checkpoint e443682a (clean synced green main): four sequential authenticated vLLM engine runs over the frozen 104-task oracle-free input, greedy, natural thinking, 1,024-token cap.
  • Totals (correct/parsed/caps of 104): parent 63/87/18; replay 62/91/13; designed_fresh 69/97/7; budget_commit 62/88/16. Mean generated tokens 515.8 / 534.0 / 357.2 / 396.2.
  • designed_fresh passed correct/parse/caps/abstention bars and all four strict wins, failing only the execute (3/8) and induct (0/8) floors; budget_commit failed broadly. Induct was 0/8 for every arm including the parent — the floor was structurally unpassable in this event.
  • No candidate promoted. Aggregate seed 78,143 is permanently sealed; no benchmark event ran; runs/local/ holds the complete raw outputs, metadata, logs, receipt, and promotion record.
  • Harness note: the eval and stage exit nonzero on no-promotion by design after publishing complete receipts; the preserved traceback in the stage log is the fail-closed signal, not an artifact loss.

Reproduce

Smoke test

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_universal_fresh_surface_budget_commit_target_match/scripts/run.py --smoke

Full run

checkpointed scripts/run.py stages only

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗