Research log Small Model Experimentation
GitHub

Transaction-invariant recovery curriculum

Finished2026-07-12Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnostics+4 moreGitHub ↗
The structure transferred but one exception did not

The one idea you need

A shop should reject a negative order as malformed, but simply decline an ordinary order when stock is too low. Both outcomes leave stock unchanged, yet they are different rules. The model learned the safe update pattern but confused these two kinds of failure.

The question

Can focused worked repairs teach one small coding model to validate a whole request, copy state, and commit changes atomically without erasing its existing test-and-revise loop?

What we found

Partly. Focused training lifted practice repairs from 52% to 82% and preserved the test-and-revise loop. But on different transaction APIs it reached 72% versus 70% before training, far below the required ten-point gain. Every target first patch copied state, checked the whole request, and updated atomically, yet all missed the separate negative-amount exception.

Why it matters

A worked repair can install the broad shape of a safe program without installing the exact policy for each failure case. The next training set should begin from almost-correct failed tests and teach the smallest distinction-preserving correction, rather than add more generic success examples.

Practice repairs before and after52% → 82%strong local installation with the focused examples
Different-API repairs before and after70% → 72%only one additional success out of 64 cases
First target patches with the safe structure16 of 16all copied, checked every resource, and updated atomically
External benchmark callszerostopped after the different-API gain missed its bar
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Verdict
    2. Results
    3. Mechanism Forensics
    4. Operational Correction
    5. Interpretation
    6. Artifacts
  4. Experiment log
  5. Data files
  6. Reproduce
  7. Related

Results at a glance 2

A large practice gain shrinks to one new success

How to read

The two groups compare practice families with different transaction APIs. Within each, bars show the starting model, equal extra recovery training, and focused transaction training. Taller is better.

0%25%50%75%100%trained familiestrained families51.7%38.3%81.7%unseen transaction devunseen transaction dev70.3%64.1%71.9%

Takeaway → Focused training dominates on practice tasks but barely moves the different APIs, so the learned rule shape did not become reliable task success.

Data table
evaluation blockrecovery parentreplay-only controltransaction curriculum
trained families51.7%38.3%81.7%
unseen transaction dev70.3%64.1%71.9%

Numbers from experiments/qwen35_4b_transaction_invariant_recovery_curriculum/reports/result_receipt.json

Technical framing

The trained-family install barely transfers to new transaction APIs — The curriculum clears every calibration gate (+30 points over parent), then adds only one success out of 64 on unseen transaction dev (+1.56 points). Parent sample-more also scores 0.7031, so transfer gates stop confirmation and Menagerie.

Every first patch learned three parts and missed one

How to read

Each bar is the share of 16 target first patches containing one required behavior. Full-height bars mean every patch had it; the missing bar is the exception policy.

0%25%50%75%100%copy input statecopy input state100%check all resourcescheck all resources100%atomic false decisionatomic false decision100%negative amount errornegative amount error0%

Takeaway → All patches copy state, check the whole request, and update atomically, but none reject negative amounts correctly—the precise residual for the next curriculum.

Data table
required transaction behaviorshare of 16 first changed patches
copy input state100%
check all resources100%
atomic false decision100%
negative amount error0%

Numbers from experiments/qwen35_4b_transaction_invariant_recovery_curriculum/reports/result_receipt.json

Technical framing

All target first patches learn the transaction structure but miss negative handling — After visible failure, all 16 trajectories overcorrect by raising on every unavailable or insufficient request. The next intervention must teach validation-policy distinctions while preserving the installed structure.

In the author’s words from the Overview · “Results”

Verdict: TRANSACTION_DEV_FAIL. The primary passed locality (0.119 drift), then installed the training families strongly: 81.7% versus parent 51.7% and replay-only 38.3%, with perfect two-turn recovery and improved interface validity. On unseen transaction dev it reached 71.9%, versus parent and equal-compute sample-more 70.3% and replay-only 64.1%. It therefore missed the registered +10/+5 bars and paired lower-bound gate. Confirmation, broad retention, and Menagerie remained sealed. … Read the full result →

Overview

Teach a single Qwen3.5-4B coding policy to validate whole requests, copy state, and commit atomically while preserving its verifier-conditioned recovery loop.

Research Program

  • Program: agentic_breadth_installation.
  • Direct predecessor: qwen35_4b_recovery_verifier_branch_tournament.
  • Prior anchors: C49 (merged-checkpoint deployment), C52 (token-local uncertainty steering is not context-local), C54 (current apex incumbent), and the conditional recovery line.

Question

Can a low-dose, action-seam curriculum install the missing transactional coding invariant—validate the complete request, copy state, then commit atomically—into the locality-safe recovery-action checkpoint without deleting its general tool-loop behavior, and does that transfer to unseen transaction families and the Menagerie?

Hypothesis

The predecessor localized all 20 shared deterministic failures to atomic reservations. The agents continued inspecting, patching, testing, and revising, but alternated between whole-request validation and input nonmutation instead of producing their conjunction. This is proposal failure, so a selector, extra sampling, or another loop-recovery update cannot create the missing program.

Twenty-four fresh procedural repositories provide executable full-conjunction repairs. Mixing their seven transition-complete action targets with 24 frozen recovery task blocks should install the semantic invariant while retaining the conditional loop. A matched control trains on 48 recovery-only blocks with the same rows, epochs, optimizer, transition counts, operator mass, and parent checkpoint. The hypothesis fails if the control matches the transaction arm, if the gain does not transfer to unseen transaction families, or if locality or broad recovery regresses.

Frozen Intervention

  • Parent: the merged recovery-action checkpoint, exact weight hash 991d2d...aea; this already contains the C54 apex and the locality-safe seven-transition recovery update.
  • Primary bank: 24 programmatic transaction tasks (six families × four) plus 24 deterministic task blocks from the frozen recovery bank.
  • Matched control: 48 deterministic frozen recovery task blocks.
  • Each task contributes exactly one row for each of seven state→action transitions. Think-block loss is zero; only the JSON tool-action seam is supervised.
  • Both arms are calibrated to 38,248 weighted action tokens per operator per epoch and receive six epochs from the identical parent.
  • Primary selection is fixed to transaction_replay; the replay-only arm is a mechanism control, not a model-selection candidate.

The train families cover inventory orders, ledger transfers, seat groups, multidimensional claims, flag batches, and rename batches. Transfer uses atomic reservations as the predecessor sentinel plus unseen debit, membership-move, and document-patch families. Every initial and partial implementation fails both executable suites; only the full repair passes. Hidden executables and repair objects remain host-side.

Evaluation and Gate Order

  1. Verify hashes, exact seven-transition task blocks, equal weighted action mass, procedural replay, and benchmark-firewall cleanliness.
  2. Train and merge both arms; compare the fixed primary against the C54 apex on 48 fresh non-coding contexts. Require centered non-target drift ≤0.15 and mean entropy delta ≥−0.05; record varentropy without using it as token pressure or a selection label.
  3. On trained transaction families, run the parent and replay-only control first, prove the frozen bars attainable, then require primary success ≥80%, +15 points over the parent, and +10 over replay-only while retaining both recovery transitions and interface validity.
  4. On four transfer families, compare primary against parent, replay-only, and equal-reservation parent sample-more. Require +10/+5/+5 points respectively, a nonnegative paired-bootstrap lower bound versus parent, transition retention, and no family collapse. Repeat unchanged on an independent seed.
  5. On the four older broad-recovery families, require recovery and normal-loop success within three points of the parent plus verification, commit, transition, invalid-action, and payload-cap retention.
  6. Only an all-pass white-box battery authorizes fresh paired Menagerie quick and medium events through the public CLI. Compare the single primary checkpoint to the frozen C54 apex: at least one tier must gain two points and neither may lose more than three.

Exact thresholds and stop labels are frozen in reports/preregistration.md.

Run

python experiments/qwen35_4b_transaction_invariant_recovery_curriculum/scripts/run.py --smoke
.venv/bin/python experiments/qwen35_4b_transaction_invariant_recovery_curriculum/scripts/run.py --gpu-smoke
.venv/bin/python experiments/qwen35_4b_transaction_invariant_recovery_curriculum/scripts/run.py --full

Results

Verdict: TRANSACTION_DEV_FAIL. The primary passed locality (0.119 drift), then installed the training families strongly: 81.7% versus parent 51.7% and replay-only 38.3%, with perfect two-turn recovery and improved interface validity. On unseen transaction dev it reached 71.9%, versus parent and equal-compute sample-more 70.3% and replay-only 64.1%. It therefore missed the registered +10/+5 bars and paired lower-bound gate. Confirmation, broad retention, and Menagerie remained sealed.

The mechanistic result is more specific than “no transfer.” On all 16 atomic- reservation cases, the first candidate patch newly contained whole-request validation, copied state, and atomic commit—the intended proposal shift—but omitted the distinct negative-amount ValueError rule. After visible failure, all 16 overcorrected by raising on every insufficient request. The next iteration should teach verifier-faithful validation-policy distinctions from near-correct failed-test states, not add generic transaction dose. Full results are in reports/report.md and reports/result_receipt.json.

Knowledgebase Update

  • Program evidence and backlog record proposal-structure installation without task-success transfer and queue the counterexample-policy successor.
  • Shared synthesis and program scorecard now distinguish transaction structure from validation-policy fidelity.
  • Claim ledger remains unchanged: the transfer gate failed and no Menagerie event ran.

Artifacts

Committed code, frozen design, compact gates, and result receipts live here. Banks, adapters, merged checkpoints, logits, and detailed trajectories live at large_artifacts/qwen35_4b_transaction_invariant_recovery_curriculum under reports/artifact_manifest.yaml.

Report

Rendered from reports/report.md

Verdict

TRANSACTION_DEV_FAIL. The fixed primary passed exact locality and produced a large trained-family install, but gained only 1/64 over the parent and equal-compute sample-more on unseen transaction families. Confirmation, broad retention, and Menagerie remained sealed.

Results

Stage / armParentReplay-onlySample-moreTransaction curriculum
Apex-relative locality drift———0.119 (pass ≤0.15)
Trained-family calibration51.7%38.3%—81.7%
Unseen transaction dev70.3%64.1%70.3%71.9%

Calibration passed every frozen gate: +30.0 points over the parent, +43.3 over matched replay-only, 100% for both changed-patch-within-two transitions, and lower invalid-action and answer-cap rates. Locality was also clean: entropy rose 0.011 nats and varentropy changed by only −0.0002.

Transfer did not meet the claim. Candidate minus parent was +1.56 points with a paired 95% bootstrap interval [−3.13,+7.81]; candidate minus sample-more was also +1.56 [−4.69,+7.81]. It beat replay-only by +7.81 [1.56,15.63] and did not regress any family, but the registered bars were +10/+5/+5.

Mechanism Forensics

The aggregate null hides a sharp partial success. On all 16 atomic-reservation cases, the candidate's first changed patch:

  • copied the capacity mapping;
  • validated every resource in a request before subtracting;
  • committed a request atomically and returned False when it did not fit.

That is the conjunction the predecessor never proposed. Every patch omitted the separate negative-amount rule (ValueError). After the visible failure, all 16 trajectories overcorrected by raising on every invalid or insufficient request, destroying the required per-request False behavior. Final success there remained 0/16. The curriculum shifted proposals, but it taught a generic transaction template rather than verifier-faithful validation-policy discrimination.

The three newly skinned transfer families were already near saturation: candidate/parent was 100/100% on debits, 100/100% on membership moves, and 87.5/81.25% on document patches. The original sentinel therefore remained the only headroom and the only shared failure.

Operational Correction

The control-first analyzer caught mismatched procedural manifests before candidate exposure. Python's randomized set rendering changed seat-test bytes across processes and, on Ada, altered deterministic batch outputs. The invalid controls were quarantined; every official child now freezes PYTHONHASHSEED=0; corrected manifests match. This is codified in tests and the design review.

Interpretation

The promising next unit is not more transaction families or more dose. It is a counterexample-conditioned validation-policy curriculum: start from the local candidate, create near-correct public-failure states where exactly one policy distinction is wrong (negative→raise, unavailable→False, malformed→None, duplicate→reject), and teach the smallest changed patch that preserves the other transaction invariants. Mix complete recovery replay and compare against matched extra transaction/recovery dose.

This result supports proposal-structure installation but not transferable task success or a general capability unlock. No benchmark seed was consumed.

Artifacts

Compact metrics and hashes are in result_receipt.json. Detailed banks, checkpoints, locality rows, valid trajectories, and quarantined invalid controls remain under large_artifacts/qwen35_4b_transaction_invariant_recovery_curriculum.

Experiment log 2

Show the running log (2 entries, 2026-07-12 → 13)

2026-07-12 — design and CPU validation

  • Created the successor after the public branch tournament found 20/20 shared deterministic failures on atomic reservations.
  • Added six transaction training families and three new transfer families; retained atomic reservations as the predecessor sentinel.
  • Fixed five hidden tests whose partial implementation initially passed the hidden half even though it failed the visible suite. Revalidated 30 generated tasks across all ten families: initial and partial fail both checks; full repairs pass both.
  • Built the matched bank path: 24 transaction + 24 recovery blocks versus 48 recovery blocks, seven transitions per task, equal action mass, zero plan loss, exact replay-source hash.
  • Added exact parent-weight validation, explicit merge lineage, 512+512 looping evaluation, locality entropy/varentropy audit, staged feasibility and candidate gates, unit tests, and resumable CPU/GPU/full runners.
  • CPU smoke: PASS. No candidate training or result-bearing generation yet.
  • Full bank/tokenization preflight: PASS. Each arm has 48 tasks/336 rows, exactly 152,992 weighted action-token mass per epoch, zero whole-task padding, a 1,179-token maximum row, and 72 registered optimizer steps. The actual 8.5 GB parent weight hash matched before tokenization.

2026-07-13 — GPU smoke, training, and control-manifest correction

  • GPU smoke passed two optimizer steps, an explicit 128-module merge, and a real six-turn vLLM recovery block.
  • Restored the two documented Qwen training fast paths in .venv; both import checks and an actual forward pass succeeded.
  • Trained both 72-step result arms. Primary aggregate loss 0.0612, merge delta norm sum 16.735; replay-only loss 0.0419, delta norm sum 5.683. No padding, truncation, fallback, OOM, or retry occurred.
  • Primary passed apex-relative locality: drift 0.1194, entropy delta +0.0112, varentropy delta −0.0002.
  • Parent and replay-only calibration controls each scored 25/60, but their procedural manifest hashes differed. Forensics localized the byte difference to randomized Python set-literal rendering in seat-group visible tests. The feasibility analyzer stopped before candidate exposure. Quarantined both invalid payloads, froze PYTHONHASHSEED=0 for every official child process, and reran controls before continuing. No threshold or model changed.
  • Corrected calibration: parent 31/60, replay-only 23/60, primary 49/60. Primary passed every calibration and locality gate.
  • Transaction dev: parent 45/64, replay-only 41/64, equal-compute parent sample-more 45/64, primary 46/64. Primary passed transition, interface, and family-retention checks but failed the +10 parent, +5 sample-more, and paired lower-bound gates. Stopped confirmation, broad retention, and Menagerie.
  • Atomic-reservation forensics: all 16 first candidate patches installed copy + all-resource validation + atomic per-request commit, but all omitted the negative-amount exception policy. After feedback, all 16 overcorrected by raising on every invalid/insufficient request. This selects a near-correct counterexample-policy curriculum as the next iteration.

Data files 1

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

Smoke test

python experiments/qwen35_4b_transaction_invariant_recovery_curriculum/scripts/run.py --smoke

Full run

.venv/bin/python experiments/qwen35_4b_transaction_invariant_recovery_curriculum/scripts/run.py --full

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗