Research log Small Model Experimentation
GitHub

Policy-Supported Successful-Sibling Universal Curriculum

Finished2026-07-14Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnostics+3 moreGitHub ↗
Balanced sibling mining cannot start

The one idea you need

When the model misses a fresh problem on its first try but solves it on another sampled try, that successful attempt may be a better lesson than a hand-written solution because it already uses reasoning the model can produce.

The question

Can the model learn broadly useful behavior from its own shortest correct sampled attempts on fresh problems it first gets wrong?

What we found

This version cannot run its second step. Across 624 first attempts it found 227 failures, but counting and routing had none and selection had only two; the plan required four failures in every skill. It therefore stopped before sampled retries or training.

Why it matters

Already-strong skills should not be forced into a repair-only data quota. The next test can focus sampled-success lessons on the ten skills that still fail while ordinary replay protects the skills the model already handles.

Fresh design pool624 tasks48 for each of 13 skills
Sampled tries per failed task16fixed before the first model run
Required supported lessons52never constructed because the first availability gate failed
Hard first-attempt failures227spread unevenly across the 13 skills
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Terminal Disposition
    9. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 2

Only intake, fresh data, and review are complete

How to read

The groups follow the planned checkpoints from intake through two collection runs, training, local testing, and broad testing. A bar at one means that checkpoint is complete.

00.250.50.751intakeintake1fresh sourcefresh source1design reviewdesign review1greedy collectiongreedy collection1failure freezefailure freeze1sibling collectionsibling collection0sibling selectionsibling selection0exposure freezeexposure freeze0paired trainingpaired training0local gatelocal gate0broad gatebroad gate0

Takeaway → The first five checkpoints completed, but the availability gate failed. Every sampled-retry, training, and evaluation stage correctly stayed closed.

Data table
preregistered experiment stagecompleted checkpoint
intake1
fresh source1
design review1
greedy collection1
failure freeze1
sibling collection0
sibling selection0
exposure freeze0
paired training0
local gate0
broad gate0

Numbers from experiments/qwen35_4b_universal_successful_sibling_target_match/reports/report.md

Technical framing

The all-skill sibling design stops before sampling — Intake, source, review, greedy collection, and failure freezing completed. The four-per-skill prerequisite failed, so sibling sampling and every downstream stage stayed sealed.

Counting, routing, and selection fall short of the quota

How to read

For each skill, the taller or shorter first bar counts hard failures in 48 fresh first attempts, while the second bar marks the fixed requirement of four. A failure bar below four blocks the whole balanced design.

0204060inductinduct464executeexecute394selectselect24tracetrace124verifyverify194countcount04repairrepair304optimizeoptimize244abstainabstain64statestate174orderorder114probeprobe214routeroute04

Takeaway → Ten skills have enough failures, but counting and routing have zero and selection has two, so an all-skill failure-only curriculum is impossible without changing the rules.

Data table
universal procedural skillhard greedy failuresrequired quota
induct464
execute394
select24
trace124
verify194
count04
repair304
optimize244
abstain64
state174
order114
probe214
route04

Numbers from experiments/qwen35_4b_universal_successful_sibling_target_match/data/greedy_failure_inventory_seed66115.json

Technical framing

Three skills cannot supply the frozen failure quota — The parent produced 227 hard failures overall, but count and route produced zero and select produced two. Because every skill needed four, the experiment stopped before sibling sampling.

In the author’s words from the Overview · “Results”

Model-free construction and the sole greedy event completed. The authenticated parent produced 624/624 rows and 296,259 sampled tokens at 859.6 tok/s with no recovery or rerun. The frozen failure gate found 227 hard failures overall, but per-skill availability was count=0, route=0, and select=2, below the mandatory four for each skill. The outcome is STOP_INSUFFICIENT_GREEDY_FAILURES; inventory/selection-receipt hashes are 8e21caf8...d783 / 3397b773...2a6e. No sibling input was emitted, sibling seed 66,116 was not consumed, and training, local evaluation, and benchmark access never opened.

Overview

Test whether the deployed parent’s own short verifier-correct sampled trajectories can turn fresh greedy failures into transferable supervision under exact-exposure replay.

Research Program

  • Program: agentic_breadth_installation.
  • Program question: can synthetic training data install a general feature that improves the held-out aggregate without regressing any reported family?
  • Prior anchors: C54's shortest-success compression signal; the terminal on-policy-prefix negative; the terminal clean-restart negative.

Question

Does the prior local failure come from teaching hand-authored trajectories outside the deployed parent’s policy support? This trial keeps the fresh greedy-failure selector but replaces every oracle restart with a short successful trajectory sampled from the same authenticated parent.

Hypothesis

On greedy-failure tasks, a verifier-correct sibling proves that the complete reasoning and answer path already lies inside the parent’s sampling support. Distilling the shortest such path from the original prompt should bank a reachable decision policy rather than merely teach answer emission, and should therefore beat an independently trained exact-exposure replay continuation.

Setup

  • Model: only Qwen/Qwen3.5-4B, pinned at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
  • Parent: authenticated explicit composite replay_after_close from qwen35_4b_universal_on_policy_prefix_repair_token_match; runtime LoRA is forbidden.
  • Task source: 624 fresh executable procedural tasks, 48 per each of 13 universal skills, construction seed 77,115.
  • Collection: one natural-thinking greedy event at seed 66,115; then, only for its published hard failures, one natural-thinking n=16 event at seed 66,116 with temperature/top-p/top-k 0.6/0.95/20.
  • Candidate source: four tasks per skill whose sampled sibling naturally stops, closes thinking, exactly matches executable truth, uses a canonical answer tail, and stays within 768 thinking tokens. The shortest qualified sibling wins. There is no oracle-trace fallback.
  • Baseline: unchanged authenticated parent.
  • Active control: independent same-parent replay continuation, later matched exactly on forward tokens, loss-bearing target tokens, absolute loss mass, updates, and aligned replay rows.
  • Fresh local gate: unchanged 26-task, two-per-skill gate at seed 88,011; strict total and execute/induct/probe wins over both parent and replay are mandatory.
  • Hidden-label boundary: benchmarks/ remains unread. Aggregate seed 78,141 stays sealed until local promotion.
  • Final claim boundary: even a strict all-family aggregate pilot still owes independent higher-tier confirmation and a matched-compute sample-more baseline.

Run

Smoke:

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_universal_successful_sibling_target_match/scripts/run.py --smoke

Checkpointed stages:

.venv/bin/python -B experiments/qwen35_4b_universal_successful_sibling_target_match/scripts/run.py --stage collect-greedy
.venv/bin/python -B experiments/qwen35_4b_universal_successful_sibling_target_match/scripts/run.py --stage prepare-siblings
.venv/bin/python -B experiments/qwen35_4b_universal_successful_sibling_target_match/scripts/run.py --stage collect-siblings
.venv/bin/python -B experiments/qwen35_4b_universal_successful_sibling_target_match/scripts/run.py --stage select-siblings

Every stage requires its prerequisite receipt committed on clean, synchronized main. Each result is checked, rebased, pushed, and held until both repository workflows are green before the next event.

Results

Model-free construction and the sole greedy event completed. The authenticated parent produced 624/624 rows and 296,259 sampled tokens at 859.6 tok/s with no recovery or rerun. The frozen failure gate found 227 hard failures overall, but per-skill availability was count=0, route=0, and select=2, below the mandatory four for each skill. The outcome is STOP_INSUFFICIENT_GREEDY_FAILURES; inventory/selection-receipt hashes are 8e21caf8...d783 / 3397b773...2a6e. No sibling input was emitted, sibling seed 66,116 was not consumed, and training, local evaluation, and benchmark access never opened.

Interpretation

This is a prerequisite negative, not a test of successful-sibling distillation. A balanced failure-only curriculum is ill-posed when the deployed parent already clears some procedural skills greedily: count and route supply no failures, while select supplies only two. The useful result is to separate residual repair from retention. A successor may reuse this published failure pool in a new directory, sample only the ten skills with at least four failures, and use exact-exposure replay plus the unchanged all-skill local gate to protect mastered skills.

Knowledgebase Update

  • Program evidence updated: terminal availability result recorded.
  • Program backlog updated: residual-only successor is queued; this record is closed.
  • Claim ledger updated: no; no capability result exists.

Artifacts

  • data/collection_tasks_seed77115.jsonl: executable truth, never passed whole to the model.
  • data/greedy_input_seed66115.jsonl: oracle-free model input.
  • data/collection_task_manifest.json and data/design_receipt.json: frozen provenance.
  • reports/preregistration.md and reports/design_review.md: prospective contract and adversarial authorization.
  • reports/artifact_manifest.yaml: external parent and future adapter/composite plan.

Terminal Disposition

No later event is authorized here. Do not lower the quota, borrow across skills, sample the two select failures, or add oracle rows. Any residual-skill design is a new experiment with its own intake, receipts, and lifecycle.

Report

Rendered from reports/report.md

Summary

The experiment stopped at its first prospective availability gate. The authenticated greedy collection completed, but three skills could not supply the required four hard failures: count and route supplied zero, and select supplied two. No sibling sampling, training, local capability evaluation, or benchmark access occurred.

Research Program Fit

This trial is the result-separated successor registered by agentic_breadth_installation after clean hand-authored restarts improved termination but lost semantic competence. It changes the teacher’s policy support while preserving fresh failure selection, exact-exposure replay, and the strict local promotion rule.

Method

Construction seed 77,115 creates 48 truth-audited tasks for each of 13 universal skills. One greedy event will identify hard failures. A separately published oracle-free input will then request 16 natural-thinking samples from the same parent on those failures only. Four short, naturally closed, exactly correct task winners per skill are required. No successful sibling means no training example; an oracle trace cannot fill the hole.

Results

  • 624/624 source rows regenerate byte-for-byte.
  • Exactly 48 rows exist for every registered skill.
  • The greedy runner input exposes only IDs, user messages, and public metadata.
  • Canonical-message overlap is zero against the two closest predecessor sources and reserved local seeds 88,000–88,011.
  • Twenty-seven experiment tests pass in the repository environment.
  • The greedy event from pushed-green commit 0038fba1 completed 624/624 requests, 296,259 sampled tokens, 859.6 tok/s, and 392.0 wrapper seconds.
  • Raw output, metadata, log, and receipt SHA-256 are e91313c04fb851bc760f7b080e3411b120fdf78b8a6e89fafbd9a9011a10f556, 0e82ae73995033e80350e9cf340ea852b342fd819e35b2cfbf75222e125b15ce, f16571516efeae1feef5eec2a7dc065f940e231dee34fc0a271a917c39dafca4, and cee1f19d1319e01a9aa69cedb1e9261de29a3c24d9e07477b77a1f5d9cc94962.
  • Collection used the frozen explicit composite and runner geometry with no recovery or generation rerun.
  • Model-free grading found 227 hard failures. Counts by skill were induct 46, execute 39, select 2, trace 12, verify 19, count 0, repair 30, optimize 24, abstain 6, state 17, order 11, probe 21, and route 0.
  • The frozen four-per-skill prerequisite failed on select, count, and route. Outcome: STOP_INSUFFICIENT_GREEDY_FAILURES.
  • Failure inventory and selection receipt SHA-256 are 8e21caf82b29ee800a4846401a7561e6624461457324e19f8a50a42175eed783 and 3397b7738eefcc12a0c6a8c6687e627e9a6125d7186d48f4a5c73e6e2a9d2a6e.
  • No sibling model input was emitted; sibling seed 66,116 and aggregate seed 78,141 remain sealed.
  • No capability result exists yet.

Controls

The failure is upstream of the mechanism-falsifying replay control. No training stream or arm exists, so this result says nothing about replay versus successful-sibling supervision.

Oracle Versus Deployable Evidence

Procedural truth constructs tasks and grades outputs. It is excluded from all model-facing collection input. Selected training trajectories, if available, must be raw same-parent samples; the oracle contributes only the correctness bit. benchmarks/ remains unread and aggregate seed 78,141 is sealed.

Interpretation

The all-skill failure quota assumed every skill still had a repairable greedy residual. That assumption is false for this parent and substrate: count and route are already saturated under the hard-failure definition. Forcing balanced failure supervision would require manufacturing failures or adding correct rows, both prohibited. This is a clean feasibility stop, not evidence against the proposed distillation mechanism.

Terminal Disposition

No further stage is authorized. A result-separated successor may reuse the published immutable collection, define residual skills prospectively as the ten with at least four hard failures, sample only those failures, and rely on exact-exposure replay plus the unchanged all-skill local gate for retention. It may not reinterpret or rescue this directory.

Artifact Manifest

The parent composite is external and immutable. All source, greedy output, metadata, log, receipts, and the terminal failure inventory are tracked. No experiment-local adapter, merged composite, sibling output, local result, or benchmark result exists.

Experiment log 3

Show the running log (3 entries, 2026-07-14)

2026-07-14 — Intake and model-free freeze

  • Claimed the queued successor from the terminal clean-restart result in a new experiment directory.
  • Identified qwen35_4b_universal_failure_selected_restart_target_match as the closest near-duplicate and active replay as the mechanism-falsifying control.
  • Reserved fresh construction/greedy/sibling/selection/training/local/aggregate seeds 77115/66115/66116/55115/49/88011/78141.
  • Materialized 624 fresh truth-audited tasks, 48 per skill, plus an oracle-free greedy input.
  • Froze two separate parent events: greedy failure identification first; n=16 sibling sampling only after those failures are committed.
  • Froze a four-per-skill successful-sibling gate, 768-thinking-token ceiling, shortest-qualified selection, and an absolute prohibition on oracle-trace fallback.
  • Added adversarial review verdict PASS_GREEDY_COLLECTION; no later event is authorized.
  • Experiment test suite: 27/27 passes under .venv with bytecode disabled.
  • No model, GPU, training, local, or benchmark event ran during this stage.

2026-07-14 — Authenticated greedy collection

  • Launched only after design commit 0038fba1 passed Validate Repository 29371704674 and Publish Research Site 29371704828.
  • The frozen parent completed 624/624 rows and 296,259 sampled tokens at 859.6 tok/s in 392.0 wrapper seconds.
  • Raw/metadata/log/receipt hashes are e91313c0...f556 / 0e82ae73...15ce / f1657151...ca4 / cee1f19d...4962.
  • Recovery was unused and generation was not rerun. Benchmark data was not read; aggregate remains sealed.
  • Failure grading and the oracle-free sibling input remain unopened until this collection checkpoint is published green.

2026-07-14 — Terminal greedy-failure availability stop

  • Opened model-free grading only after greedy commit 5b784ac5 passed Validate Repository 29372380559 and Publish Research Site 29372380538.
  • Found 227 hard failures overall, but the mandatory per-skill quota failed: count=0, route=0, and select=2 versus four required.
  • Preserved outcome STOP_INSUFFICIENT_GREEDY_FAILURES; inventory/receipt hashes are 8e21caf8...d783 / 3397b773...2a6e.
  • Emitted no sibling input and consumed no sibling, training, local, or aggregate seed.
  • Closed the experiment immediately. The validation-only harness now treats an authenticated terminal gate's expected exit code 2 as a passing smoke condition; selection logic and the historical design receipt are unchanged.

Reproduce

Smoke test

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_universal_successful_sibling_target_match/scripts/run.py --smoke

Full run

terminal at prepare-siblings; no later command is authorized

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗