Research log Small Model Experimentation
GitHub

Validation-policy counterexample curriculum

Finished2026-07-13Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnostics+4 moreGitHub ↗
The lesson was already mastered

The one idea you need

A training experiment needs room to improve. If the starting model already fixes every practice case, even a perfect trained model cannot prove that the new lessons helped.

The question

Can a tiny set of almost-correct failed-test revisions teach that exact distinction without erasing the coding loop already learned?

What we found

The test could not answer the question. Before the newly trained model was allowed to run, the starting model and the equal-size old-lesson control each fixed all 48 practice cases. The required gains were therefore impossible, so the run stopped before transfer or the external benchmark.

Why it matters

Turning the old failure into a clear instruction plus an almost-correct program also made it easy. Future curricula must first prove that the starting model still fails the exact new practice distribution; failure forensics alone do not guarantee useful training headroom.

Starting model practice score48 of 48no room for the required fifteen-point gain
Equal extra-training control48 of 48no room for the required ten-point gain
Unrelated-behavior drift0.109the trained candidate stayed below the 0.15 safety ceiling
External benchmark callszerostopped before candidate behavior and transfer
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Verdict
    2. What ran
    3. Locality
    4. Controls-first calibration
    5. Mechanism forensics
    6. Interpretation
    7. Exposure ledger
  4. Experiment log
  5. Data files
  6. Reproduce
  7. Related

Results at a glance 2

Both controls already sit at the maximum score

How to read

The first two bars show practice success for the starting model and the equal extra-training control; the third is the best score any candidate could possibly achieve. All bars reach the same ceiling.

0%25%50%75%100%transaction parenttransaction parent100%extra-training controlextra-training control100%candidate ceilingcandidate ceiling100%

Takeaway → With both controls at 100 percent, the promised improvement cannot be measured and the candidate must remain unopened.

Data table
controls-first calibration armtrained-family recovery success
transaction parent100%
extra-training control100%
candidate ceiling100%

Numbers from experiments/qwen35_4b_validation_policy_counterexample_curriculum/reports/result_receipt.json

Technical framing

Both calibration controls saturate the available score — Parent and matched control each solve all 48 cases. No candidate can clear the frozen +15/+10 point mechanism bars, so candidate behavior and every downstream gate remain sealed.

The small update stayed inside its unrelated-behavior limit

How to read

The first bar is measured drift from the older reference model; the second is the largest allowed drift. Shorter is better, and the measured bar remains below the limit.

00.050.10.15measured driftmeasured drift0.109drift ceilingdrift ceiling0.15

Takeaway → The narrow training update was local enough, but locality cannot rescue an evaluation set the parent already solves perfectly.

Data table
locality quantitymedian centered non-target logit drift
measured drift0.109
drift ceiling0.15

Numbers from experiments/qwen35_4b_validation_policy_counterexample_curriculum/reports/result_receipt.json

Technical framing

The candidate passes direct locality before the infeasibility stop — Candidate drift is 0.1094 versus the preregistered 0.15 ceiling; unrelated entropy rises by 0.0214. This validates locality at the chosen dose, not capability benefit.

In the author’s words from the Overview · “Result”

Verdict: CALIBRATION_INFEASIBLE. Both fixed arms trained and merged, and the candidate passed direct C54 locality at 0.109 centered-logit drift with unrelated entropy retained. The controls-first calibration then found that the learned transaction parent and matched extra-transaction control each solved 48/48 fresh train-skin recovery cases. The required +15/+10 point bars were therefore mathematically impossible. Candidate scientific behavior, transfer, retention, and Menagerie remained sealed. The useful correction is mechanistic. On all 48 parent cases, the first changed patch already included the negative check, copied state, and the ordinary False rejection policy. … Read the full result →

Overview

Teach Qwen3.5-4B the semantic distinction left by the transaction curriculum: negative quantities are malformed and raise ValueError, while unknown or insufficient resources are ordinary False decisions.

Research program

Question

Can a residual action-seam intervention teach verifier-faithful exception versus rejection policy from near-correct failed-test states, transfer across new repository schemas, and retain the transaction structure and conditional tool loop already installed in the parent?

Why this is the next test

The predecessor installed the intended proposal structure: on all 16 atomic- reservation cases, the first changed patch copied state, checked the whole request, committed atomically, and returned False for ordinary rejection. Every patch nevertheless omitted the separate negative-amount exception. After visible failure, every trajectory overcorrected by raising on all unavailable or insufficient requests. Generic transaction dose is therefore the wrong next lever; the remaining error is policy discrimination after verifier feedback.

Frozen intervention

  • Parent: the local transaction candidate, exact weight SHA-256 1cf5fbca317808d6d00225f5cd533c82c7e1602b2b2e5e2da8f4307b01941ba3.
  • Control source: the predecessor's exact 48-task, 336-row transaction-plus-recovery bank, SHA-256 9c196d1e7e49881bbf151e1575c98811bcdca66e6ef38858c34f60f1256b9315.
  • Candidate bank: the same 48 complete task blocks, except the diagnosis_to_changed_patch row in each of 24 recovery blocks is replaced by one fresh near-correct validation-policy revision. The other 312 rows are retained from the prior bank.
  • Matched control: all 336 prior rows receive the same extra update from the same parent.
  • Both arms have zero think loss, identical transition/operator counts, 38,248 weighted action tokens per operator per epoch, and three epochs at LR 2e-5, rank 32/alpha 64, batch 4 × seven transition-stratified accumulation steps.

Training uses six API skins spanning bundle mappings, record objects, and tuple requests. Transfer uses three new skins across the same three representations. atomic_reservations is a separately gated predecessor sentinel and is never counted as unseen transfer.

Evidence hygiene

Every generated initial and partial repository fails visible and hidden executables; every oracle passes. Public-content hashes prove all 24 bank and 24 calibration repositories unique and disjoint, all 32 development and 32 confirmation repositories unique and mutually disjoint, and all official manifests stable across Python hash seeds. Hidden tests and repair objects stay host-side. Nothing under benchmarks/ is read or imported; Menagerie is run only through its public CLI and only aggregate/per-family scores are retained.

Gate order

  1. Lock the frozen design commit and verify exact hashes, bank composition, transition completeness, action-mass parity, executable replay, and firewall cleanliness.
  2. Train and merge both fixed arms. Before behavior generation, compare the candidate directly with C54 apex on 48 fresh non-coding contexts; require centered non-target drift ≤0.15 and entropy delta ≥−0.05, while recording varentropy diagnostically.
  3. Generate parent and matched-control calibration receipts first. If the bars are feasible, expose the candidate and require ≥80% success, +15 points over parent, +10 over control, transition retention, and interface validity.
  4. On one known sentinel plus three fresh transfer families, compare candidate with parent, matched extra-transaction training, and equal-reservation parent sample-more. Require +10/+5/+5 points, nonnegative paired-bootstrap lower bound versus parent, ≥50% sentinel success, nonnegative transfer on all three fresh families, and verify/commit/transition/interface retention. Repeat unchanged on a content-disjoint seed.
  5. Require broad recovery and normal-loop retention on four older families.
  6. Only an all-pass white-box battery runs frozen paired Menagerie quick seed 71301 and medium seed 71302 against C54 apex. At least one tier must gain two points and neither may lose more than three.

Exact thresholds and stop labels are frozen in reports/preregistration.md.

Run

python experiments/qwen35_4b_validation_policy_counterexample_curriculum/scripts/run.py --smoke
.venv/bin/python experiments/qwen35_4b_validation_policy_counterexample_curriculum/scripts/run.py --lock-design <design-commit>
.venv/bin/python experiments/qwen35_4b_validation_policy_counterexample_curriculum/scripts/run.py --gpu-smoke
.venv/bin/python experiments/qwen35_4b_validation_policy_counterexample_curriculum/scripts/run.py --full

Result

Verdict: CALIBRATION_INFEASIBLE. Both fixed arms trained and merged, and the candidate passed direct C54 locality at 0.109 centered-logit drift with unrelated entropy retained. The controls-first calibration then found that the learned transaction parent and matched extra-transaction control each solved 48/48 fresh train-skin recovery cases. The required +15/+10 point bars were therefore mathematically impossible. Candidate scientific behavior, transfer, retention, and Menagerie remained sealed.

The useful correction is mechanistic. On all 48 parent cases, the first changed patch already included the negative check, copied state, and the ordinary False rejection policy. Making the contract explicit and the partial state otherwise correct removed the predecessor's failure core. The predecessor did not demonstrate a general inability to express the semantic distinction; it demonstrated a miss under its more implicit atomic-reservation contract and initial proposal dynamics.

Before another training run, a separate qualification stage must establish parent headroom on multiple semantic-conflict families and representations. Only axes with replicated nontrivial failure should enter a counterexample curriculum. Full evidence is in reports/report.md and reports/result_receipt.json.

Knowledgebase update

Program evidence, backlog, scorecard, and shared synthesis record the qualification failure and the new headroom-first rule. No claim ID is added: the treatment was never behaviorally exposed and no transfer or Menagerie event ran.

Artifacts

Committed design, code, tests, and compact receipts live here. Banks, adapters, merged checkpoints, logits, and detailed trajectories live under large_artifacts/qwen35_4b_validation_policy_counterexample_curriculum, as documented in reports/artifact_manifest.yaml.

Report

Rendered from reports/report.md

Verdict

CALIBRATION_INFEASIBLE. The candidate was local, but the parent and matched control were already perfect on the fresh trained-family instrument. The registered effect bars could not be earned, so candidate behavior, transfer, retention, and Menagerie remained sealed.

What ran

The immutable design at e0b19f5d used the learned transaction checkpoint as parent. Candidate and control each trained for 36 transition-stratified optimizer steps over 336 rows with identical 152,992 total weighted action mass per epoch and zero think loss. Candidate replaced only 24 post-failure revision rows; 312 rows retained the prior bank. Control retained all 336 prior rows.

Candidate training loss was 0.013996 versus control 0.018505; merged delta-norm sums were 2.908 and 2.957. These are diagnostics, not capability results. GPU smoke passed 12/12 and exercised training, explicit composite merge, and the real vLLM looping harness.

Locality

Candidate versus C54 apex passed on all 48 fresh non-coding contexts:

  • median centered non-target logit drift: 0.10944 (ceiling 0.15);
  • mean entropy delta: +0.02142 (floor −0.05);
  • mean varentropy delta: −0.01082 (diagnostic only).

This supports the narrow engineering point that one-transition, three-epoch action-only replay can stay local. It does not show behavioral benefit.

Controls-first calibration

The parent and matched control were generated before candidate behavior on 24 fresh train-skin repositories × two controlled recovery states:

ArmSuccessFailed-test changed within twoRejected valid changed within twoInvalid actions/turn
learned transaction parent48/4824/2424/240.000
extra-transaction control48/4824/2424/240.000

The theoretical candidate ceiling is 48/48. It cannot exceed parent by 15 points or control by 10 points, so the feasibility analyzer failed exactly success_vs_start and success_vs_control and stopped.

Mechanism forensics

This is not merely “the tasks were easy.” The task construction changed the meaning of the predecessor's residual:

  • the issue explicitly stated negative quantities must raise ValueError;
  • the failed-test partial already copied state, validated existence/capacity, returned False for ordinary rejection, committed atomically, and preserved inputs;
  • visible output directly identified the missing negative exception.

On every one of 48 parent trajectories, the first changed patch contained the negative check, copied state, and the ordinary false-decision path. In rejected- patch states the parent frequently wrote the complete correct program directly from the initial source; in failed-test states it made the preserving revision immediately. The original atomic-reservation failure had a more implicit contract (nonnegative input domain without an explicit exception sentence) and different proposal dynamics. Converting that failure into explicit instruction converted it into an already-solved editing task.

Interpretation

The counterexample mechanism remains untested, not disproven. The experiment establishes a reusable ordering rule: failure forensics are not sufficient to design a training substrate. Before capability-production spend, the exact procedural task distribution must show replicated parent headroom under the intended prompt and verifier feedback. This is the same portfolio principle seen in specialist qualification, now at the curriculum-substrate level.

The next step is a no-training headroom tournament over multiple semantic conflicts and public representations. It should distinguish explicit contract following from inference under verifier evidence, then admit only axes with a non-saturated parent band into a separately preregistered curriculum.

Exposure ledger

  • Candidate scientific calibration: not generated.
  • Policy transfer development/confirmation: not generated.
  • Broad recovery/normal retention: not generated.
  • Menagerie: not invoked; seeds 71301/71302 unconsumed.
  • Claim ledger: unchanged.

Experiment log 8

Show the running log (8 entries, 2026-07-13)

2026-07-13 — intake and scaffold

  • Attached the experiment to agentic_breadth_installation; direct predecessor is qwen35_4b_transaction_invariant_recovery_curriculum.
  • Copied the predecessor's local looping harness, merged-checkpoint trainer, vLLM runner, locality audit, and conditional transition bank into this self-contained directory.
  • Pinned the learned transaction parent (1cf5fb...41ba3), C54 apex anchor (c93316...608d5), prior primary bank (9c196d...9315), and prior bank receipt (8c2c33...e63e).

2026-07-13 — adversarial preflight repairs

  • The first task generator had only five content variants and nominally fresh splits overlapped. Replaced it before preregistration with high-entropy procedural values/resources and three distinct public data representations.
  • Made the reused atomic-reservation sentinel high-entropy and changed its partial state to the exact near-correct residual: copy, whole-request check, atomic update, and False rejection are already correct; only negative handling is absent.
  • The copied locality builder reproduced the predecessor's content. Replaced every stem and prefix; all 48 new hashes are disjoint from prior locality blocks.
  • Narrowed the treatment from full seven-row policy-task blocks to a single replaced diagnosis_to_changed_patch row in each of 24 otherwise frozen predecessor blocks. This removes direct whole-solution reteaching while retaining every conditional transition through 312 unchanged rows.
  • Added public-content digests excluding task ID/split, cross-process manifest stability, within/cross-block uniqueness, sentinel separation, verification and commit gates, frozen Menagerie seeds, aggregate-only benchmark storage, and an immutable design-file receipt.

2026-07-13 — CPU preflight

  • Unit suite: 21/21 passed.
  • Harness smoke: passed across ten policy families; initial/partial fail, oracle pass, firewall clean, locality fresh.
  • Full deterministic banks built: candidate/control 336 rows each; 24 injected candidate revisions; all transition and operator counts matched; 38,248 weighted action tokens per operator per epoch; no overlength row.
  • Bank SHA-256: candidate 940da9...305a; control 524240...c10e; receipt 45ce6e...d6e8.
  • No model output existed during any of these corrections or checks.

Next recorded event

2026-07-13 — immutable design boundary

  • Rebased the design onto current upstream main, resolved generated catalog conflicts by regeneration, and pushed directly to main at e0b19f5d.
  • Wrote runs/preregistration_receipt.json over 17 design-critical files. The receipt records model_output_precedes_lock: false; every GPU/model mode now fails closed on file-digest or ancestry drift.

2026-07-13 — receipt handoff

  • Committed and pushed the immutable receipt at 39413cea; only then started GPU smoke and the full staged pipeline.

2026-07-13 — staged run

  • GPU smoke passed 12/12 with perfect two-turn recovery and zero invalid/cap contacts.
  • Candidate/control trained for 36 steps each; merged hashes are 4ca7d5...4aa2 and 9aef44...cdec.
  • Candidate passed direct C54 locality: drift 0.10944, entropy +0.02142, varentropy −0.01082.
  • Parent controls ran before candidate behavior. Parent and matched control each scored 48/48 on trained-family recovery, making the frozen +15/+10 bars impossible. Analyzer stopped at CALIBRATION_INFEASIBLE.
  • Candidate behavior, transfer, retention, and Menagerie remained sealed; no benchmark seed was consumed.

2026-07-13 — learned lesson

All 48 parent first changed patches already contained negative handling, copy, and ordinary false rejection. The explicit contract and one-line partial made the intended residual trivial. A successor must qualify non-saturated semantic axes on the exact prompt/verifier distribution before it trains any arm.

Data files 1

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

Smoke test

python experiments/qwen35_4b_validation_policy_counterexample_curriculum/scripts/run.py --smoke

Full run

.venv/bin/python experiments/qwen35_4b_validation_policy_counterexample_curriculum/scripts/run.py --full

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗