Research log Small Model Experimentation
GitHub

Natural-Language State-Table Universal Curriculum

Finished2026-07-14Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnostics+2 moreGitHub ↗
Design frozen; no model result

The one idea you need

A state table is a compact scratchpad: after each instruction, it records exactly what changed. This trial pairs that scratchpad with a separate scorecard for competing explanations, then asks for one short verified answer.

The question

Can explicit natural-language state tracking and hypothesis scoring teach a small model to execute, compare, and finish more reliably than another equal replay update?

What we found

Not known yet. CPU construction produced 80 truth-checked lessons and two 320-row streams with exactly 286,814 tokens each. All 48 smoke tests pass, but no model has been trained or evaluated.

Why it matters

The preceding staged-search curriculum often computed the right state but failed to answer, and it damaged branch selection. This successor changes the training interface to match those observed deployment failures while keeping benchmark data sealed.

Model resultszeroCPU design only; training waits for a green checkpoint
Truth-audited lessons8020 each execute, score, repair, and commit
Tokens per arm286,814exact match, zero skips, 40 planned updates
Aligned replay rows200byte-identical and in the same positions
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Next Experiments
    9. Artifact Manifest
  4. Experiment log
  5. Data files
  6. Reproduce
  7. Related

Results at a glance 1

Intake and CPU design are complete

How to read

Bars at one mark completed admission stages. Training, local evaluation, and broad evaluation remain at zero.

00.250.50.7511intake1CPU design1training0local gate0broad gate

Takeaway → The generator, exact-compute control, gates, and adversarial review are frozen without creating a model result or consuming an evaluation event.

Data table
preregistered experiment stagecompleted checkpoint
intake1
CPU design1
training1
local gate0
broad gate0

Numbers from experiments/qwen35_4b_universal_state_table_compiler_token_match/data/design_receipt.json

Technical framing

Intake and exact-compute CPU design are complete — CPU design froze 80 truth-audited lessons and two 320-row, 286,814-token arms with zero skips and 200 aligned replay rows. No model result, merge, local event, or benchmark event exists.

In the author’s words from the Overview · “Results”

The active replay control then trained from the authenticated parent for all 40 steps over all 320 rows with zero skips and final loss 0.4226. Its adapter weights/config hashes are 83a741e4...409a / 13838f2e...843; receipt/log hashes are b05dc72e...e99a / 5f4d1fe3...60ba. The candidate independently restarted from the same parent and completed the same 320 rows, zero skips, and 40 steps with final loss 1.059. Its adapter weights/config hashes are 36e54804...5d0f / 7101cc87...4b34; receipt/log hashes are 6aab42b3...2be2 / 26907944...c059. Training losses are operational evidence only. Fresh paired local seed 88,008 rejected the candidate. … Read the full result →

Overview

Fresh local negative on 2026-07-14; aggregate sealed.

This result-separated successor tests whether truth-audited, variable-depth natural-language state tables plus independent hypothesis scoring and a short verified commit install a reusable reasoning procedure better than an exact-token replay continuation from the same parent.

Research Program

Question

Does matching the training interface to variable-depth natural-language execution teach the model to maintain explicit state, compare independently simulated hypotheses, and stop with a concise answer—without sacrificing broad replay behavior?

Hypothesis

The failed predecessor often reached the correct final state but did not commit, and it regressed hypothesis selection. A truth-audited table that records each natural- language transition should make execution inspectable; separate rows that score each hypothesis on every probe should preserve discrimination; an answer-only commit after verification should train the missing emission seam. The mechanism is false if the candidate cannot beat both its parent and an exact-token replay control on a fresh unchanged local gate.

Setup

  • Model: only Qwen/Qwen3.5-4B, pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
  • Parent: authenticated close_xi adapter from qwen35_4b_universal_close_weight_token_match; the failed scaffold adapter is not inherited.
  • Dataset/task source: fresh deterministic procedural synthesis owned by this experiment. No benchmark source, item, transcript, or result detail may be read.
  • Candidate: variable-depth natural-language state execution, independent hypothesis scoring, verification/repair, and concise commit lessons.
  • Mechanism-falsifying control: same-parent replay continuation with identical forward tokens, optimizer steps, backend, seed, and position-aligned shared replay.
  • Frozen arms: 320 rows and exactly 286,814 forward tokens each, zero skips, 40 optimizer steps, and 200 byte-identical replay rows at the same positions. Candidate contains 80 curriculum rows plus 40 replay filler; control contains 120 replay rows.
  • Primary admission: the inherited absolute local capability gate, a new explicit probe ≥0.50 check, and strict paired wins over parent and active replay both overall and on execute/induct/probe combined.
  • Conditional broad admission: aggregate-only same-backend evaluation only after the sole candidate passes every local check; all reported families must improve before higher-tier confirmation or matched-compute sample-more.
  • Reserved seeds: construction 77112, training 46, fresh local 88008, and conditional aggregate 78138.

Run

Frozen smoke:

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_universal_state_table_compiler_token_match/scripts/run.py --smoke

The adversarial review passed and the harness now exposes exactly one expensive stage per invocation: train-control, train-candidate, local, merge, or benchmark. Each stage requires a clean worktree; every predecessor receipt must already be committed at HEAD. Follow reports/preregistration.md and publish/CI-verify every stage before starting the next.

Results

CPU construction produced 80 truth-audited rows: 20 each execute, score, repair, and commit. All 80 answers recompute from executable state; all score rows evaluate three hypotheses on five probes; correct hypothesis position is balanced 7/7/6. Exact-token materialization succeeded at 320 rows, 286,814 tokens, zero skips, and 200 aligned replay positions per arm. The frozen smoke passes 48 tests. The active replay control then trained from the authenticated parent for all 40 steps over all 320 rows with zero skips and final loss 0.4226. Its adapter weights/config hashes are 83a741e4...409a / 13838f2e...843; receipt/log hashes are b05dc72e...e99a / 5f4d1fe3...60ba. The candidate independently restarted from the same parent and completed the same 320 rows, zero skips, and 40 steps with final loss 1.059. Its adapter weights/config hashes are 36e54804...5d0f / 7101cc87...4b34; receipt/log hashes are 6aab42b3...2be2 / 26907944...c059. Training losses are operational evidence only.

Fresh paired local seed 88,008 rejected the candidate. Parent, replay, and candidate scored 19/26, 16/26, and 16/26 correct; parsed 23/26, 21/26, and 22/26; and contacted the 1,024-token cap 3, 5, and 5 times. Candidate execute/induct/probe was 0/2, 0/2, and 1/2, for 1/6 target cases versus replay 2/6 and parent 4/6. It failed accuracy, parse, cap, execute, induction, and every strict relative check. Promotion is empty, so no merge or benchmark event ran and conditional aggregate seed 78,138 remains sealed.

Interpretation

Truth-audited natural-language tables did not install a reusable deployed procedure. The candidate sometimes improved isolated computation: it solved one parent/replay trace miss, fixed one optimization case, and computed a state exactly before losing only on spaces. But it also treated a reference cycle declaration as an operation, repeated both induction cases to the cap, miscounted a probe score, and reached the correct execute result without committing before the cap. The idealized training interface therefore remained off-policy relative to the model's actual failure prefixes. Retire another hand-authored trace surface; the next result-separated test should use fresh on-policy failure-prefix correction with executable oracle continuations and exact serialization, under the same controls and gates.

Knowledgebase Update

  • Program evidence: records the fresh exact-token local negative and failure anatomy.
  • Program backlog: retires idealized state-table surfaces and queues on-policy failure-prefix correction.
  • Shared synthesis: adds the off-policy-interface boundary.
  • Claim ledger: unchanged.

Artifacts

  • idea_intake.md
  • configs/default.yaml
  • data/design_receipt.json
  • data/stream_token_receipt.json
  • runs/training/replay_after_close.json
  • runs/training/replay_after_close.log
  • runs/training/state_table_after_close.json
  • runs/training/state_table_after_close.log
  • runs/local/seed88008.json
  • runs/local/seed88008_promotion.json
  • analysis/local_failure_forensics.md
  • scripts/run.py
  • reports/design_review.md
  • reports/preregistration.md
  • reports/report.md
  • reports/artifact_manifest.yaml

Report

Rendered from reports/report.md

Summary

The truth-audited state-table curriculum failed its fresh local gate. The candidate tied exact-token replay at 16/26 correct, lost to its 19/26 parent, and was worse than both controls on the targeted execute/induct/probe subtotal. No arm was promoted and the aggregate benchmark remained sealed.

Research Program Fit

The experiment belongs to agentic_breadth_installation and must beat same-parent exact-token replay before it can contribute broad evidence.

Method

The 80-row intervention contains 20 each of variable-depth natural-language execution tables, independent three-hypothesis/five-probe score tables, first-error repair, and concise commit from a verified table. Both candidate and replay start independently from authenticated close_xi. Each arm has 320 rows, 286,814 exact forward tokens, zero skips, 40 updates, and 200 position-aligned identical replay rows.

Results

Construction source SHA-256 is a7b453af...e88bb; replay/candidate streams are 2727e29a...a2b5 / 8e1b8fdc...1355; token and design receipts are 163e40a6...f0b8 / 0bac3340...ef837. All truth audits recompute, all absolute gates are reachable, and 48 tests plus the frozen smoke pass. The replay control completed 40/40 updates over 320 rows with zero skips, final loss 0.4226, and 294.1 wrapper wall seconds. Its 169,903,320-byte external adapter has weights/config hashes 83a741e4...409a / 13838f2e...843; the preserved receipt/log hashes are b05dc72e...e99a / 5f4d1fe3...60ba. The candidate independently restarted from the same authenticated parent and completed 40/40 updates over 320 rows with zero skips, final loss 1.059, and 290.9 wrapper wall seconds. Its 169,903,320-byte adapter weights/config hashes are 36e54804...5d0f / 7101cc87...4b34; receipt/log hashes are 6aab42b3...2be2 / 26907944...c059.

At fresh seed 88,008, parent/replay/candidate scored 19/16/16 correct, 23/21/22 parsed, 3/5/5 cap contacts, and 438.1/508.1/522.5 mean generated tokens. The target subtotal over execute, induct, and probe was 4/6, 2/6, and 1/6. Candidate execute was 0/2, induction 0/2, and probe 1/2. It failed the absolute accuracy, parse, cap, execute, and induction checks; every strict win over parent and replay was false. Promotion is empty. Local/promotion receipt hashes are 027c0f63...f2869 / 429770fd...70f5; no merge or benchmark ran and seed 78,138 remains sealed.

Controls

The required active control is an independently trained same-parent replay continuation matched on exact forward tokens, optimizer steps, and 200 aligned replay positions. Candidate minus control target-token deltas are +1,196 prompt, -1,955 thought, 0 close, and +759 answer. The failed predecessor adapter is not a parent or control.

Oracle Versus Deployable Evidence

Generator execution and table recomputation are construction oracles. Model outputs must be scored without supplying hidden state. Benchmark data stays behind the aggregate-only firewall and is unavailable unless the sole candidate passes every frozen local check. It did not, so no benchmark source, item, transcript, result detail, or aggregate event was accessed.

Interpretation

The package hypothesis is rejected at this dose and interface. Paired forensics show two narrow improvements—one trace and one optimization flip versus both controls—but five parent wins regressed, including both execute cases and one probe. One state answer was semantically exact but failed serialization only; one execute thought held the exact target but never committed before cap. Those seam failures coexist with semantic failures: a cycle declaration became a spurious operation, both induction cases repeated to cap, and a probe counted two distinct outputs as three. Idealized truth-audited traces therefore did not align training with the model's deployed failure prefixes. See analysis/local_failure_forensics.md.

Next Experiments

Preserve and publish this negative. A new experiment may test fresh on-policy failure-prefix correction with executable oracle continuations, explicit bounded commit targets, and exact answer serialization. It must use new seeds, exclude this held-out local event from training, retain a same-parent exact-token replay control, and pass the unchanged local gate before any aggregate event.

Artifact Manifest

The parent identity, data/receipt hashes, reserved seeds, and one-stage checkpoint order are recorded in artifact_manifest.yaml and preregistration.md.

Experiment log 5

Show the running log (5 entries, 2026-07-14)

2026-07-14 — Intake

  • Opened only after the staged-search negative was preserved in commit d68f0042, pushed to main, and both GitHub workflows passed.
  • Ran related-work discovery and named qwen35_4b_universal_search_scaffold_token_match as the closest near-duplicate.
  • Selected the existing agentic_breadth_installation program and the authenticated close_xi parent; the failed scaffold adapter will not be inherited.
  • Reserved fresh construction/training/local/conditional aggregate seeds 77112/46/88008/78138.
  • Authorized CPU feasibility and adversarial design review only. No GPU, merge, local capability, or benchmark event ran.

Next: publish and CI-verify this intake, then implement and adversarially review the smallest truth-audited generator and exact-token control before any training.

2026-07-14 — CPU feasibility and design freeze

  • Began only after intake commit a9689c52 was pushed to main and both GitHub workflows passed.
  • Deterministically generated 80 rows at construction seed 77112: 20 each execute, score, repair, and commit; source SHA-256 is a7b453af...e88bb.
  • Recomputed every transition, answer, hypothesis prediction/score, and first repair error. Correct hypothesis position is balanced 7/7/6 and every false hypothesis matches 1–4 of five probes.
  • Materialized 320-row replay and candidate arms at exactly 286,814 forward tokens, zero skips, 40 planned updates, and 200 byte-identical aligned replay positions.
  • Froze replay/candidate hashes 2727e29a...a2b5 / 8e1b8fdc...1355; token/design receipt hashes are 163e40a6...f0b8 / 0bac3340...ef837.
  • Proved all absolute local gates reachable and added fail-closed strict wins over both controls overall and on execute/induct/probe. Aggregate seed 78138 stays conditional.
  • Adversarial review returned PASS_EXPENSIVE_RUN. The harness permits one expensive stage per clean, incrementally committed checkpoint. Frozen smoke passes 48 tests.
  • No model/GPU, local capability, merge, or benchmark event ran.

Next: publish and CI-verify this design checkpoint, then train only the active replay control.

2026-07-14 — Replay control training

  • Began only after design-freeze commit 956a307a was pushed to main and GitHub Validate Repository run 29336429602 and Publish Research Site run 29336429945 both completed successfully.
  • Fetched and fast-forwarded the clean worktree to current origin/main commit e21aa1bb before starting; the incoming change was unrelated to this experiment.
  • Re-ran the frozen generator, exact-token materialization, design receipt, and all 48 experiment tests before loading the authenticated close_xi warm start.
  • Trained replay_after_close for the frozen 320 rows, one epoch, 40 optimizer steps, seed 46, and ordinary thought/close weights 0.2. All 320 rows encoded and zero skipped.
  • Completed in 294.1 wrapper wall seconds with final train loss 0.4226. This loss is operational evidence only and is not a capability comparison.
  • Preserved receipt/log hashes b05dc72e...e99a / 5f4d1fe3...60ba. The external 169,903,320-byte adapter weights/config hashes are 83a741e4...409a / 13838f2e...843 and independently match the receipt.
  • No candidate training, local generation, merge, or benchmark event ran.

Next: publish and CI-verify this replay-control checkpoint, then train only the frozen candidate.

2026-07-14 — State-table candidate training

  • Began only after replay-control commit 5b906a4e was pushed to main and GitHub Validate Repository run 29337458833 and Publish Research Site run 29337458951 both completed successfully.
  • Verified the clean worktree was exactly aligned with origin/main, revalidated the committed replay-control receipt, and re-ran the frozen design plus all 48 tests.
  • Independently restarted from authenticated close_xi; the candidate did not continue from the replay control or inspect any capability behavior.
  • Trained state_table_after_close for the frozen 320 rows, one epoch, 40 optimizer steps, seed 46, and ordinary thought/close weights 0.2. All 320 rows encoded and zero skipped.
  • Completed in 290.9 wrapper wall seconds with final train loss 1.059. This loss is not compared causally with replay loss because the target composition differs.
  • Preserved receipt/log hashes 6aab42b3...2be2 / 26907944...c059. The external 169,903,320-byte adapter weights/config hashes are 36e54804...5d0f / 7101cc87...4b34 and independently match the receipt.
  • No local generation, merge, or benchmark event ran; fresh local seed 88,008 remains unconsumed.

Next: publish and CI-verify this candidate checkpoint, then run the single frozen paired local evaluation.

2026-07-14 — Fresh local gate negative

  • Began only after candidate commit ed68aa81 was pushed to main and GitHub Validate Repository run 29338242755 and Publish Research Site run 29338241500 both completed successfully.
  • Fetched current origin/main, confirmed a clean aligned worktree, authenticated both committed training receipts, and re-ran the frozen design plus all 48 tests.
  • Consumed local seed 88,008 exactly once across parent, replay control, and candidate together in one Transformers process at batch size 4 and 1,024 generated tokens.
  • Parent/replay/candidate scored 19/16/16 correct, parsed 23/21/22, contacted the cap 3/5/5 times, and used 438.1/508.1/522.5 mean generated tokens. All had zero repeated feasible-route abstentions.
  • Candidate execute/induct/probe was 0/2, 0/2, and 1/2: 1/6 targeted cases versus replay 2/6 and parent 4/6. It failed accuracy, parse, cap, execute, induction, and all four strict parent/replay win checks. Promotion is empty.
  • Preserved complete local receipt 027c0f63...f2869; parent/replay/candidate gate receipts 8af2f171...c964 / d767cf2b...cdc0c / 76dcd96a...b0957; and promotion receipt 429770fd...70f5.
  • Paired forensics found a mixed but non-general pattern: candidate gained one trace, one optimization, and one abstention case, but lost both execute cases, one probe, one order case, and replay's repaired case. One state result was semantically exact but failed only on spaces; one execute result was present in thought but never committed before cap. Both induction cases repeated to cap.
  • No merge or benchmark event ran. Conditional aggregate seed 78,138 is sealed.

Next: publish and CI-verify this negative, then open a result-separated on-policy failure-prefix correction experiment with fresh seeds and unchanged controls/gates.

Data files 2

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

Smoke test

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_universal_state_table_compiler_token_match/scripts/run.py --smoke

Full run

run.py --stage train-control; commit/push/verify; --stage train-candidate; commit/push/verify; --stage local; commit/push/verify; conditionally --stage merge; commit/push/verify; conditionally --stage benchmark

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗