Research log Small Model Experimentation
GitHub

Search-Scaffold Universal Curriculum

Finished2026-07-14Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned SelectionPosttraining and Adaptation+3 moreGitHub ↗
Staged search fails its local mechanism gate

The one idea you need

Earlier lessons showed a complete worked search trace, but the model still wandered among candidates and ran out of room. This design turns the missing intermediate moves—apply, fit, reject, and execute—into separately graded practice before recombining them.

The question

Can teaching each step of a bounded two-operation search help the model solve and finish fresh abstract tasks better than another equal replay update?

What we found

No. On 26 fresh paired cases, the staged-search model tied replay at 16 correct and trailed its parent at 18. All three parsed 23 answers and hit the limit three times; the candidate solved none of four execute/induct cases and neither probe-selection case.

Why it matters

Separately grading search substates was not enough to make them reusable. The candidate often reached a correct execution state but failed to commit, while its hypothesis simulation regressed. The next mechanism must match the natural-language variable-depth deployment interface rather than add more canonical two-operation lessons.

Candidate local accuracy16 of 26ties replay and trails the parent at 18
Readable answers23 of 26identical across all arms; one short of the gate
Execute plus induct0 of 4the parent solves two of four
Probe selection0 of 2both controls solve both cases
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Next Experiments
    9. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 1

Staged search ties replay and trails the parent locally

How to read

The parent solves 18 of 26 fresh cases; replay and scaffold solve 16. All three parse 23 answers and finish within the cap on 23 cases, so the scaffold does not improve the registered emission seam.

0102030exact answersexact answers181616parsed answersparsed answers232323within token capwithin token cap232323

Takeaway → The candidate fails five gates and loses all execute, induction, and probe-selection cases; broad evaluation remains sealed.

Data table
fresh local outcomeclose_xi parentreplay after closesearch scaffold after close
exact answers181616
parsed answers232323
within token cap232323

Numbers from experiments/qwen35_4b_universal_search_scaffold_token_match/runs/local/seed88007.json

Technical framing

Staged search ties replay and trails its parent locally — Greedy seed 88007 at a 1,024-token cap. Scaffold is also 0/2 execute, 0/2 induct, and 0/2 probe; it fails five gates and aggregate seed 78137 remains sealed.

In the author’s words from the Overview · “Results”

Its final loss is 0.4215; adapter weights/config SHA-256 are 10155232...fc538 / 373c1426...ac9b. The scaffold candidate then trained independently on all 320 rows with zero skips and 40/40 updates. Its final loss is 1.492; adapter weights/config SHA-256 are e7957d90...84618 / 22859c76...2c4ce. Losses are not compared across the different target distributions and carry no capability interpretation. Fresh paired local seed 88,007 produced: close_xi_parent: 18/26 correct, 23/26 parsed, 3 cap contacts; replay_after_close: 16/26 correct, 23/26 parsed, 3 cap contacts; scaffold_after_close: 16/26 correct, 23/26 parsed, 3 cap contacts. The candidate scored 0/2 on execute, 0/2 on induct, and 0/2 on probe, versus parent 1/2, 1/2, and 2/2. … Read the full result →

Overview

This experiment tests whether independently supervised, executable search substates compose into a bounded general reasoning procedure better than an exact-token replay continuation from the same parent.

Research Program

  • Program: agentic_breadth_installation
  • Program question: can engineered synthetic curricula install substrate-general procedures that improve every held-out benchmark family without displacing the incumbent broad policy?
  • Parent result: qwen35_4b_universal_close_weight_token_match.
  • Prior anchors: C44/C59 (serial reasoning content is load-bearing), C56 (oracle trace narration is not a reusable induction circuit), and the exact-token mid-density/close-weight local negatives.

Question

Can a staged executable curriculum that teaches the component operations of two-step search—apply a proposed first operation, fit or reject a second operation, execute a verified pair, then solve—cross the unchanged fresh local gate where full narrated induction traces and heavier close loss did not?

Hypothesis

The current curriculum jumps from primitive lessons to a full decomposition trace. At deployment the model repeatedly explores candidates without a bounded decision procedure. Training the intervening states as independently scored, truth-audited subproblems should make candidate evaluation and rejection addressable features. A fixed compact ledger in the final lessons should then compose those features and commit within 1,024 tokens. This is a curriculum-structure intervention, not another close-weight or generic-dose sweep.

Setup

  • Only model: Qwen/Qwen3.5-4B, pinned revision 851bf6e....
  • Parent: authenticated close_xi adapter from the completed predecessor (weights 16e9dc75...c179, config de953bd5...7ff).
  • Synthetic source: new executable abstract-index tasks over disjoint randomized surfaces; no benchmark content, outputs, or family implementation is read.
  • Candidate block: 80 rows, 16 each of apply-first, fit-second, reject-first, execute-pair, and bounded full-search lessons.
  • Frozen exposure: 200 common replay rows plus the 80 staged rows and 40 matched replay fillers. The replay arm uses the same 200 rows plus 120 replay rows. Both arms have 320 rows, exactly 286,814 forward tokens, zero skips, and 40 updates; exactly 200 shuffled positions are byte-identical.
  • Training seed: 45. Fresh local seed: 88,007. Conditional aggregate seed: 78,137.
  • Local gate: accuracy ≥0.65, parse ≥0.90, cap contacts ≤2, no repeated feasible-route abstention, plus accuracy ≥0.50 (at least one of two) in both u_execute and u_induct.
  • Hidden-label boundary: local cases are fresh procedural experiment data. Benchmark access is conditional and aggregate-only through the trusted gateway; benchmark sources, items, transcripts, and private outputs remain unread.

Run

Smoke:

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B \
  experiments/qwen35_4b_universal_search_scaffold_token_match/scripts/run.py --smoke

Full:

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B \
  experiments/qwen35_4b_universal_search_scaffold_token_match/scripts/run.py \
  --stage train-control
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B \
  experiments/qwen35_4b_universal_search_scaffold_token_match/scripts/run.py \
  --stage train-candidate
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B \
  experiments/qwen35_4b_universal_search_scaffold_token_match/scripts/run.py \
  --stage local

The local stage writes a promotion receipt even on failure. Merge and benchmark remain sealed unless the sole candidate passes; then run --stage merge and --stage benchmark. Each natural stage is committed, rebased, fully checked, pushed to main, and verified in both GitHub workflows before the next starts.

Results

CPU feasibility passed. The deterministic source has 80 truth-audited rows over six surface families. The two frozen streams each contain 320 trainable rows and exactly 286,814 forward tokens at max length 4,096, with zero skips. All 43 experiment tests and the staged smoke harness pass. The replay control has now trained from the authenticated parent on all 320 rows with zero skips and 40/40 updates. Its final loss is 0.4215; adapter weights/config SHA-256 are 10155232...fc538 / 373c1426...ac9b. The scaffold candidate then trained independently on all 320 rows with zero skips and 40/40 updates. Its final loss is 1.492; adapter weights/config SHA-256 are e7957d90...84618 / 22859c76...2c4ce. Losses are not compared across the different target distributions and carry no capability interpretation.

Fresh paired local seed 88,007 produced:

  • close_xi_parent: 18/26 correct, 23/26 parsed, 3 cap contacts;
  • replay_after_close: 16/26 correct, 23/26 parsed, 3 cap contacts;
  • scaffold_after_close: 16/26 correct, 23/26 parsed, 3 cap contacts.

The candidate scored 0/2 on execute, 0/2 on induct, and 0/2 on probe, versus parent 1/2, 1/2, and 2/2. It failed accuracy ≥0.65, parse ≥0.90, caps ≤2, execute ≥0.50, and induct ≥0.50; only the route-abstention check passed. Promotion is empty. No merge occurred and conditional aggregate seed 78,137 remains sealed.

Interpretation

The registered scaffold does not install its target behavior. Relative to the parent it gains one trace and one repair case but loses one execute, one induct, and both probe cases; relative to replay it is a 3-win/3-loss redistribution. Mean output grows to 520.5 tokens (parent 434.2; replay 471.6), and both execute failures visibly reach the correct final state before over-explaining to the cap. Canonical two-operation scaffolds did not transfer to variable-depth natural-language execution or hypothesis discrimination. Do not repeat this interface or consume the sealed benchmark seed.

Knowledgebase Update

  • Program evidence: staged search recorded as a local mechanism negative.
  • Program backlog: require a fresh natural-language state-table/compiler successor.
  • Claim ledger: unchanged.

Artifacts

  • src/
  • scripts/
  • configs/
  • data/
  • runs/
  • analysis/
  • reports/
  • reports/artifact_manifest.yaml
  • analysis/local_forensics.md
  • runs/local/seed88007.json
  • runs/local/seed88007_promotion.json
  • idea_intake.md

Report

Rendered from reports/report.md

Summary

The design is frozen after successful CPU feasibility. The same-parent exact-token replay control and staged-search candidate both trained successfully. The scaffold then failed the single fresh local mechanism gate; no merge or benchmark event ran.

Research Program Fit

This is the result-separated successor to the close-weight negative. It preserves the strong emission near-miss but changes mechanism: independently supervised search substates rather than more close loss or another representative dose.

Method

The five stages are apply-first, fit-second, reject-first, execute-pair, and bounded two-branch search. Every target is recomputed by an experiment-local executable specification over abstract surfaces. The candidate contains 200 inherited replay rows, 80 scaffold rows (16/stage), and 40 replay fillers. The control contains the same 200 rows plus 120 replay rows. Both streams have 320 rows, 286,814 forward tokens, zero skips, and 40 frozen updates from the same authenticated parent.

Results

CPU construction passed: source SHA-256 5854c218...a093; candidate stream 79a8d7c9...0b90; replay stream c157fb13...355d; exact token receipt eeb12b95...e4a0f. Forty-three experiment tests and the full smoke harness pass. The replay control completed 40/40 updates over 320/320 rows with zero skips in 281.2 seconds. Final train loss was 0.4215. Its adapter weights/config SHA-256 are 10155232...fc538 / 373c1426...ac9b; receipt/log SHA-256 are 5b293eb6...5a66 / 7d3bc262...d5f7.

The scaffold candidate then completed 40/40 updates over 320/320 rows with zero skips in 291.4 seconds. Final train loss was 1.492. Its adapter weights/config SHA-256 are e7957d90...84618 / 22859c76...2c4ce; receipt/log SHA-256 are 13ba8897...6dd0 / ccaffa7b...99c1. The losses are not a capability comparison because the targets differ.

Fresh paired local seed 88,007 gave parent 18/26 correct, 23/26 parsed, and three cap contacts; replay 16/26, 23/26, and three; scaffold 16/26, 23/26, and three. The candidate was 0/2 on execute, 0/2 on induction, and 0/2 on probe. It failed five of six registered checks—accuracy, parse, cap, execute, and induction—and passed only route abstention. Promotion is empty, so no checkpoint was merged and aggregate seed 78,137 remains sealed.

Controls

  • Authenticated close_xi parent.
  • Authenticated newly trained replay-only continuation from that parent.
  • Exact row, forward-token, update, seed, optimizer, close-weight, and parent matching.
  • Two hundred byte-identical replay positions; batch size one avoids padding-compute differences.
  • Fresh local procedural seed 88,007 before conditional aggregate seed 78,137.

Oracle Versus Deployable Evidence

Executable generators may use hidden construction state only to truth-audit rows. Promotion uses autonomous greedy model behavior. Benchmark access remains behind the aggregate-only firewall.

Interpretation

The intervention is executable but fails its intended mechanism. Against parent, candidate has two paired wins and four losses; against replay, three wins and three losses. Mean generation length is 520.5 tokens versus 434.2 parent and 471.6 replay. Both candidate execute failures compute the correct final state in visible thought but continue to the cap without an answer. Both probe cases regress from correct in both controls to wrong in candidate, and both induction cases are wrong. The package therefore neither commits after verified execution nor improves branch simulation.

The post-result diagnosis is an interface mismatch: training uses exactly two canonical-coded operations and a two-branch demonstration, while the local executor uses natural-language procedures of variable depth and probe selection requires independent simulation/scoring. This does not justify tuning the observed arm. It justifies a new result-separated natural-language state-table/compiler mechanism.

Next Experiments

Preserve and publish this negative. Start a new experiment with fresh seeds if the next natural-language state-table/compiler mechanism survives idea intake and design review. Do not lower the gate, reuse seed 88,007, merge this adapter, or consume aggregate seed 78,137.

Artifact Manifest

The parent, frozen data identities, commands, and future external-artifact locations are recorded in artifact_manifest.yaml. Both adapters exist externally; the full local receipt and empty promotion receipt are committed. No merge or benchmark artifact exists.

Experiment log 5

Show the running log (5 entries, 2026-07-14)

2026-07-14 — Intake and scaffold

  • Ran repository related-work discovery before creation.
  • Named the closest near-duplicates: the completed universal curriculum, mid-density exact-token ladder, and close-weight successor.
  • Chose a different mechanism: independently scored decomposition-search substates followed by a bounded compact ledger.
  • Reserved construction seed 77,111, training seed 45, fresh local seed 88,007, and conditional aggregate seed 78,137.
  • No data materialization, model access, training, evaluation, merge, or benchmark event has run.

Next: publish the intake checkpoint, then implement and truth-audit the smallest runnable staged-search stream before adversarial review.

2026-07-14 — Feasibility and design freeze

  • Implemented an experiment-local executable operation universe and 80 deterministic lessons: 16 each of apply, fit, reject, execute, and search across colors, digits, letters, nonce strings, Romans, and syllables.
  • Independently recomputed every operation, fitting second, unique fitting pair, dead branch, intermediate state, and final answer in tests. The reject stage is balanced 8 FIT / 8 NO_FIT.
  • Reused the predecessor's authenticated 200-row replay core and 120-row control partition. Selected a disjoint 40-row candidate filler with an exact token sum.
  • Froze replay SHA-256 c157fb13...355d and candidate SHA-256 79a8d7c9...0b90: 320 rows and 286,814 forward tokens each, zero skips, max sequence 2,991, 40 updates, and exactly 200 byte-identical shuffled positions.
  • Preserved the non-equivalent target allocation: replay has 116,036 prompt, 167,411 thought, 640 close, and 2,727 answer tokens; candidate has 124,245 prompt, 158,311 thought, 640 close, and 3,618 answer tokens.
  • Removed the predecessor's target-specific close-weight interface. Both arms use ordinary thought/close weight 0.2; unit tests lock span behavior.
  • Froze wrappers for authenticated same-parent training, local seed 88,007, a sole candidate promotion, explicit merges, and one aggregate-only quick@1,024 paired event at seed 78,137.
  • Completed adversarial review with a narrowed claim boundary: the full target shows one dead and one true branch, not exhaustive search. All 43 experiment tests and the staged smoke harness pass.
  • No GPU model load, training, local generation, merge, or benchmark event ran.

Next: commit, rebase, run the full repository check, push this design freeze to main, verify both workflows, and only then train the replay control.

2026-07-14 — Replay control training

  • Began only after design-freeze commit 603b8107 was pushed to main and both Validate Repository and Publish Research Site completed successfully.
  • Authenticated the close_xi warm start and exact replay stream/token receipt.
  • Trained replay_after_close for the frozen 320 rows, one epoch, 40 optimizer steps, seed 45, and ordinary thought/close weights 0.2. All 320 rows encoded and zero skipped.
  • Completed in 281.2 wall seconds with final train loss 0.4215.
  • Preserved receipt/log hashes 5b293eb6...5a66 / 7d3bc262...d5f7. External adapter weights/config hashes are 10155232...fc538 / 373c1426...ac9b; weight size is 169,903,320 bytes.
  • No candidate training, local generation, merge, or benchmark event ran.

Next: publish and CI-verify this control checkpoint, then train the frozen candidate.

2026-07-14 — Scaffold candidate training

  • Began only after control commit b8f11db6 was pushed to main and both GitHub workflows completed successfully.
  • Independently restarted from the authenticated close_xi parent; did not continue from or inspect capability behavior of the replay control.
  • Trained scaffold_after_close for the frozen 320 rows, one epoch, 40 optimizer steps, seed 45, and ordinary thought/close weights 0.2. All 320 rows encoded and zero skipped.
  • Completed in 291.4 wall seconds with final train loss 1.492. This loss is not compared causally with replay loss because target composition differs.
  • Preserved receipt/log hashes 13ba8897...6dd0 / ccaffa7b...99c1. External adapter weights/config hashes are e7957d90...84618 / 22859c76...2c4ce; weight size is 169,903,320 bytes.
  • No local generation, merge, or benchmark event ran.

Next: publish and CI-verify this candidate checkpoint, then consume the single frozen local seed 88,007 over parent, replay control, and candidate together.

2026-07-14 — Fresh local negative

  • Began only after candidate commit 9e34c675 was pushed to main and both GitHub workflows completed successfully.
  • Consumed the single registered experiment-owned seed 88,007 over parent, active replay, and scaffold in one greedy Transformers process at a 1,024-token cap.
  • Parent scored 18/26 correct, 23/26 parsed, and three caps; replay scored 16/26, 23/26, and three; scaffold scored 16/26, 23/26, and three.
  • Scaffold was 0/2 execute, 0/2 induct, and 0/2 probe. It failed accuracy, parse, cap, execute, and induction checks; route abstention alone passed. Promotion is empty and the harness stopped nonzero as registered.
  • Preserved local/gate/promotion receipts. Full local receipt SHA-256 is 156acd37...acdb; promotion SHA-256 is 7e1fd417...f1c1.
  • Post-decision paired forensics show 2 wins/4 losses versus parent and 3/3 versus replay. Candidate mean output grew to 520.5 tokens, and both execute failures computed the correct state before running to cap without a parsed answer.
  • No merge ran. Aggregate seed 78,137 remains sealed and no benchmark data was read.

Next: publish this completed negative, then create a fresh result-separated successor for natural-language variable-depth state execution and hypothesis scoring.

Reproduce

Smoke test

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_universal_search_scaffold_token_match/scripts/run.py --smoke

Full run

run.py --stage train-control; commit/push/verify; --stage train-candidate; commit/push/verify; --stage local; conditionally --stage merge; conditionally --stage benchmark

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗