Research log Small Model Experimentation
GitHub

Residual-Skill Successful-Sibling Universal Curriculum

Finished2026-07-14Agentic Breadth InstallationBenchmark GeneralizationPosttraining and AdaptationGitHub ↗
Stopped: too few correct retries

The one idea you need

Use the model’s own correct sampled retries as lessons only where its first answer still fails. Keep ordinary broad practice for skills it already handles, then test every skill again.

The question

Can short correct retries improve the ten skills that still fail without damaging counting, routing, or selection?

What we found

Grading the 3,600 retries found 855 short fully-correct ones, and nine of ten weak skills had plenty. But rule-guessing (induction) yielded a usable correct retry on only 2 of its 46 failed tasks, below the required four, so the frozen rule stopped the experiment before any training. No model was trained and no benchmark was touched.

Why it matters

The skill the model most needs to repair is the one where its own retries are never correct: rule-guessing. Together with the earlier stop this closes learning-from-own-retries as the path, and points the program back to purpose-designed practice data that does not depend on the model already being able to succeed.

Qualified retries855 / 3,600short, fully correct, cleanly finished
Rule-guessing coverage2 / 46failed tasks with any usable correct retry
Required per skill4nine of ten skills met it; induction did not
Training and benchmarksnever openedseeds 50 / 88,012 / 78,142 unconsumed
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Next Stage
    9. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 1

Usable correct retries per skill: rule-guessing falls below the bar

How to read

Each bar counts failed tasks where at least one same-model retry was short and fully correct. Four per skill were required.

0102030executeexecute29optimizeoptimize21probeprobe21repairrepair21statestate12tracetrace12verifyverify12orderorder11abstainabstain6inductinduct2

Takeaway → Nine skills clear the bar easily; rule-guessing manages two of forty-six, so the preregistered rule stopped the experiment before training.

Data table
residual treatment skillfailure tasks with a qualified sibling
execute29
optimize21
probe21
repair21
state12
trace12
verify12
order11
abstain6
induct2

Numbers from experiments/qwen35_4b_universal_residual_successful_sibling_target_match/reports/report.md

Technical framing

Qualified successful-sibling availability by residual skill (quota 4) — The frozen selector required four tasks per skill. Induct supplied two of its 46 failure tasks from 736 samples, so the experiment stopped terminally (STOP_INSUFFICIENT_SUCCESSFUL_SIBLINGS) before any training, local, or benchmark event.

In the author’s words from the Overview · “Results”

It produced 3,600/3,600 completions over 225 prompts, sampled 2,337,087 tokens at 739.2 tok/s, and took 3,210.1 wrapper seconds. Raw/metadata/log/receipt hashes are 688c4f7e...c332 / 56951f00...9cdf / d0b31be8...f29 / c3a3a297...f614. Recovery was unused and generation was not rerun. The frozen model-free selection ran from the published-green collection checkpoint 915a7c62. Grading found 855/3,600 qualified siblings overall, but per-task availability was induct=2, below the mandatory four tasks per residual skill; every other skill met quota (execute 29, optimize/probe/repair 21, state/trace/verify 12, order 11, abstain 6). … Read the full result →

Overview

Test same-parent shortest-success distillation only on procedural skills with a live greedy residual, while exact-exposure replay and an unchanged all-skill gate protect saturated skills.

Research Program

  • Program: agentic_breadth_installation.
  • Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
  • Prior anchors: C54 shortest-success compression; terminal clean-oracle restarts; terminal balanced-sibling availability stop.

Question

Can shortest verifier-correct trajectories from the same deployed parent improve the ten procedural skills with real greedy residuals, while active replay preserves count, route, and select without manufacturing failures for them?

Hypothesis

A correct sampled sibling supplies a complete reasoning path already inside the parent’s support. Restricting this signal to actual residual skills avoids off-policy oracle language and avoids wasting treatment capacity on saturated skills. Exact-exposure replay and the unchanged all-skill local gate make retention part of the success criterion.

Setup

  • Model: only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
  • Parent: authenticated explicit composite replay_after_close; runtime LoRA is forbidden.
  • Inherited collection: immutable 624-task source and 227-failure inventory from qwen35_4b_universal_successful_sibling_target_match.
  • Residual treatment skills: induct, execute, trace, verify, repair, optimize, abstain, state, order, and probe. These prospectively have at least four hard failures each.
  • Retention skills: select, count, and route. They receive no manufactured failure rows; active replay and a fresh all-13-skill gate protect them.
  • Collection: all 225 residual hard failures, same parent and pinned vLLM runner, natural thinking, n=16, seed 66,117, temperature/top-p/top-k 0.6/0.95/20, 1,024-token cap.
  • Selection: four tasks per residual skill; shortest naturally stopped, closed, canonical, exact-answer sibling within 768 thinking tokens. No oracle fallback or second sampling event.
  • Planned active control: independent same-parent replay continuation matched exactly on forward tokens, loss-bearing targets, absolute loss mass, updates, and 200 aligned replay rows.
  • Fresh local gate: seed 88,012, two tasks for every one of all 13 skills, with the unchanged strict candidate wins over both parent and replay.
  • Hidden boundary: benchmarks/ remains unread; conditional aggregate seed 78,142 stays sealed until local promotion.
  • Claim boundary: independent higher-tier confirmation and matched-compute sample-more remain mandatory.

Run

Smoke:

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_universal_residual_successful_sibling_target_match/scripts/run.py --smoke

Checkpointed stages:

.venv/bin/python -B experiments/qwen35_4b_universal_residual_successful_sibling_target_match/scripts/run.py --stage collect-siblings
.venv/bin/python -B experiments/qwen35_4b_universal_residual_successful_sibling_target_match/scripts/run.py --stage select-siblings

Every model event requires clean synchronized main, its prerequisite committed at HEAD, and both required workflows green. Every result gets its own checked, rebased, pushed checkpoint before the next stage.

Results

Model-free inheritance and residual input construction are complete. The source, failure inventory, greedy receipt, and terminal stop receipt retain their published hashes. The derived input contains exactly 225 hard failures across the ten registered residual skills and no answer, reference thought, audit, or truth field. Input/manifest/design hashes are dafeb012...1119 / cee88012...c7e7 / e1066596...93d7.

The sole sibling event ran from synchronized published commit fc5a333b after Validate Repository 29373498273 and Publish Research Site 29373498296 passed. It produced 3,600/3,600 completions over 225 prompts, sampled 2,337,087 tokens at 739.2 tok/s, and took 3,210.1 wrapper seconds. Raw/metadata/log/receipt hashes are 688c4f7e...c332 / 56951f00...9cdf / d0b31be8...f29 / c3a3a297...f614. Recovery was unused and generation was not rerun.

The frozen model-free selection ran from the published-green collection checkpoint 915a7c62. Grading found 855/3,600 qualified siblings overall, but per-task availability was induct=2, below the mandatory four tasks per residual skill; every other skill met quota (execute 29, optimize/probe/repair 21, state/trace/verify 12, order 11, abstain 6). The outcome is STOP_INSUFFICIENT_SUCCESSFUL_SIBLINGS; inventory/selection-receipt hashes are 60c95b7a...083e / d3926daf...ad01. No training corpus was emitted, training seed 50 and local seed 88,012 were not consumed, and benchmark aggregate seed 78,142 was never opened.

Interpretation

This is an availability negative for policy-supported successful-sibling distillation, concentrated exactly at the program's known wall. Across 46 induct hard failures and 736 samples, the parent produced a qualified short correct sibling on only two tasks, while nine of ten residual skills supplied quota easily. The signal a same-parent curriculum needs is missing precisely where repair matters most: induction failures are not near-misses the parent can re-sample its way out of (C38/C39). Lowering the quota, dropping induct, raising n, or relaxing the 768-token ceiling would each abandon the preregistered design rather than test it. The residual-vs-retention separation itself worked: the stop is now attributable to one skill's policy support, not to saturated skills polluting the quota.

Knowledgebase Update

  • Program evidence updated: terminal availability result recorded.
  • Program backlog updated: the successful-sibling line is closed; a designed-curriculum successor is queued.
  • Claim ledger updated: no; no capability result exists.

Artifacts

  • data/inherited_*: self-contained published lineage artifacts.
  • data/residual_sibling_input_seed66117.jsonl: oracle-free model input.
  • data/residual_collection_manifest.json and data/design_receipt.json: frozen provenance.
  • runs/sibling_collection/seed66117.*: complete raw output, metadata, log, and authenticated receipt.
  • data/successful_sibling_inventory_seed66117.json and data/successful_sibling_selection_receipt.json: terminal model-free grading and stop receipt.
  • reports/preregistration.md and reports/design_review.md: prospective contract and collection authorization.
  • reports/artifact_manifest.yaml: external parent and conditional model-artifact plan.

Terminal Disposition

No later event is authorized here. Do not lower the four-task quota, drop induct from the treatment set, resample deficient skills at larger n, relax the 768-thinking-token ceiling, or add oracle rows. The published 3,600-output bank is immutable and may be reused only by a new experiment with its own intake, prospective policy, receipts, and lifecycle. Together with the balanced predecessor, this closes same-parent successful-sibling mining as a curriculum source: the skills that keep failing greedily are the skills whose successes the parent cannot sample.

Report

Rendered from reports/report.md

Summary

The residual design, its one authenticated sibling collection, and the frozen model-free selection are complete, and the experiment is closed terminally. Grading found 855/3,600 qualified siblings, but only two of 46 induct failure tasks supplied any qualified sibling, below the mandatory four per skill. The outcome is STOP_INSUFFICIENT_SUCCESSFUL_SIBLINGS: no training corpus exists, and training, local evaluation, and benchmark access never opened.

Research Program Fit

This is the result-separated correction to the balanced sibling prerequisite stop. It tests policy-supported successful paths without manufacturing failures for saturated skills.

Method

The inherited parent event supplied 227 hard failures. A prospective rule retains the ten skills with at least four failures, totaling 225 prompts, and excludes select/count/route from treatment. One n=16 same-parent event created the fixed bank from which the frozen selector will test whether each residual skill supplies four short, naturally closed, exactly correct siblings. Active replay and an unchanged all-skill local gate protect excluded skills.

Results

  • Inherited source, inventory, greedy receipt, and stop receipt reproduce their published hashes.
  • The residual input has 225 unique hard-failure prompts across exactly ten skills.
  • Residual counts are induct 46, execute 39, trace 12, verify 19, repair 30, optimize 24, abstain 6, state 17, order 11, and probe 21.
  • Model input excludes oracle answers, thoughts, audits, truth flags, and expected answers.
  • Input/manifest/design hashes are dafeb012...1119, cee88012...c7e7, and e1066596...93d7.
  • The sibling event ran from clean synchronized published commit fc5a333b after both required workflows passed.
  • It completed 225/225 prompts and 3,600/3,600 outputs, sampling 2,337,087 tokens at 739.2 tok/s in 3,210.1 wrapper seconds.
  • Raw output, metadata, log, and receipt SHA-256 are 688c4f7ed461228ac7fad7f4f484c178fe07b205b1305fd39439e18c8e70c332, 56951f0044ffd53208f78916eb1620bfc1d64b20d9c650a5d7c992d7e5bf9cdf, d0b31be82a11bea55af205b6e4b73f4c0ac9cefbe92262158e901add4683df29, and c3a3a297ba11ce9242f5b842d353cdb60171e6376e4ea9d3efd68b0e75a5f614.
  • Collection used the frozen explicit parent and sampling geometry with no recovery or generation rerun. Aggregate seed 78,142 remains sealed.
  • Twenty-nine experiment tests pass.
  • The frozen selection, run from published-green checkpoint 915a7c62, qualified 855/3,600 siblings (natural stop, closed canonical thinking, exact answer, ≤768 thinking tokens).
  • Per-task availability was execute 29, optimize/probe/repair 21, state/trace/verify 12, order 11, abstain 6, and induct 2 — induct alone fell below the four-task quota.
  • The outcome is STOP_INSUFFICIENT_SUCCESSFUL_SIBLINGS with zero selected rows; inventory/receipt hashes are 60c95b7a...083e / d3926daf...ad01. Training, local, and aggregate seeds 50/88,012/78,142 were never consumed.

Controls

The explicit parent is the baseline. The future mechanism-falsifying control is an independent exact-exposure replay continuation from the same parent. Retention is not inferred from excluded training rows; it is measured on fresh all-skill tasks.

Oracle Versus Deployable Evidence

Executable truth grades outputs only. It is absent from model-facing bytes. The benchmark directory is unread and aggregate seed 78,142 remains sealed.

Interpretation

The availability negative lands exactly on the program's wall skill. Nine of ten residual skills supplied their quota easily, so the residual/retention separation did its job; the design failed only where the parent's policy support is genuinely empty. Induction failures are not near-misses recoverable by re-sampling at n=16 within a short-thinking budget — consistent with C38/C39's finding that induction, not execution, is the modality-general deficit. Same-parent successful-sibling mining is therefore structurally unable to cover the full residual set: the skill that most needs repair is the skill whose successes the parent cannot produce.

Next Stage

None. The experiment is closed terminally under the preregistered no-fallback rule. Any successor — a nine-skill sibling curriculum, a larger sampling budget, or a designed synthetic curriculum that does not depend on parent policy support — is a new experiment with its own intake, design review, and receipts.

Artifact Manifest

Inherited lineage, all model-free artifacts, and the complete raw sibling collection/metadata/log/receipt are tracked. The parent composite is external and immutable. Future adapters/composites, if authorized, belong under this experiment’s large_artifacts tree with tracked receipts.

Experiment log 3

Show the running log (3 entries, 2026-07-14)

2026-07-14 — Result-separated residual freeze

  • Opened only after the balanced predecessor closed and published green.
  • Copied immutable source, failure inventory, greedy receipt, and terminal stop receipt into this experiment.
  • Prospectively fixed the ten skills with at least four published hard failures as treatment and select/count/route as replay-protected retention skills.
  • Derived 225 oracle-free sampling rows. Input/manifest hashes: dafeb012...1119 / cee88012...c7e7.
  • Reserved fresh sibling/selection/training/local/aggregate seeds 66117/55116/50/88012/78142.
  • Froze one n=16 same-parent event, four qualified tasks per residual skill, a 768-thinking-token ceiling, shortest-success selection, and no oracle fallback or resampling.
  • Design receipt hash: e1066596...93d7; adversarial verdict PASS_SIBLING_COLLECTION.
  • Experiment tests pass 29/29. No model, GPU, training, local, or benchmark event ran.

2026-07-14 — Authenticated residual sibling collection

  • Launched only after design commit fc5a333b passed Validate Repository 29373498273 and Publish Research Site 29373498296; local HEAD and origin/main matched and the preflight worktree was clean.
  • The frozen explicit replay composite completed all 225 prompts with exactly 16 outputs each: 3,600 completions and 2,337,087 sampled tokens at 739.2 tok/s in 3,210.1 wrapper seconds.
  • Raw/metadata/log/receipt hashes are 688c4f7e...c332 / 56951f00...9cdf / d0b31be8...f29 / c3a3a297...f614.
  • Recovery was unused and generation was not rerun. Benchmark data was not read and aggregate seed 78,142 remains sealed.
  • Correctness grading and the frozen model-free selection remain unopened until this collection checkpoint is published green. No training or evaluation stage ran.

2026-07-14 — Terminal successful-sibling availability stop

  • Ran the frozen model-free select-siblings stage only after collection checkpoint 915a7c62 matched origin/main with Validate Repository 29376623518 and Publish Research Site 29376623499 green.
  • Grading qualified 855/3,600 siblings overall (natural stop, closed canonical thinking, exact answer, ≤768 thinking tokens). Dominant rejections: over the short budget 1,527, wrong answer 1,359, non-canonical answer tail 1,352.
  • Per-task availability: execute 29, optimize 21, probe 21, repair 21, state 12, trace 12, verify 12, order 11, abstain 6, induct 2. The mandatory four-tasks-per-skill gate failed on induct alone: 2 of 46 induct failure tasks yielded any qualified sibling from 736 samples.
  • Outcome STOP_INSUFFICIENT_SUCCESSFUL_SIBLINGS; zero rows selected. Inventory/receipt hashes: 60c95b7a96e0c325d25aa28e5583e42935081d9c372e0ce334f81ec7693a083e / d3926dafb70148ee40ba63e1d4140aed7842d80820884452164834f608f3ad01.
  • Selection seed 55,116 governed no sampling; training/local/aggregate seeds 50/88,012/78,142 were never consumed. Benchmark data was not read.
  • Experiment closed terminally per the preregistered no-fallback rule; the sample bank remains immutable for any successor with its own lifecycle.

Reproduce

Smoke test

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_universal_residual_successful_sibling_target_match/scripts/run.py --smoke

Full run

checkpointed scripts/run.py stages only

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗