Semantic-policy headroom tournament

Results at a glance 3

Overview

This no-training tournament found no replicated post-failure semantic axis and stopped on its answer-cap instrument gate. Its trajectory contrast moves the capability target earlier, to evidence acquisition before the first patch.

Research program

• Program: agentic_breadth_installation.
• Direct predecessor: qwen35_4b_validation_policy_counterexample_curriculum.
• Parent: its unchanged learned transaction checkpoint, exact weight SHA-256 1cf5fb...41ba3.

Question

Which validation-policy conflicts still produce replicated failed-test headroom in the transaction-trained model, under the exact looping coding harness, and are therefore legitimate substrates for a later counterexample curriculum?

Design

This experiment trains nothing and cannot invoke Menagerie. It crosses three conflicts with three public representations:

• negative quantity: malformed ValueError versus ordinary insufficiency False;
• non-integer quantity: malformed TypeError versus ordinary insufficiency False;
• blank resource: malformed ValueError versus ordinary unknown resource False;
• bundle mappings, record dictionaries, and tuple sequences.

Nine inferred-contract families state the valid input domain and ordinary rejection policy but require the agent to infer malformed behavior from visible tests/failure output. Three explicit-contract controls state the exception verbatim. Every partial implementation is otherwise correct and fails visible and hidden tests only at the semantic conflict; every oracle passes.

Two content-disjoint blocks each contain 36 unique repositories (12 families × three tasks) and 72 controlled recovery cases. An inferred axis qualifies only if failed-test success is 15–80% in both blocks, at least two of three shapes are individually inside that band in each block, explicit-control success is ≥85%, and invalid/cap contacts remain ≤5%. At least one replicated axis must qualify.

Why this is different

The predecessor trained before proving that its rewritten substrate retained the historical failure. Parent and control were already 48/48, so the treatment effect was unidentifiable. Here, exact-substrate parent headroom is the only outcome. Eligible axes and families are emitted mechanically by frozen rules for use in a separate future experiment; no update, threshold change, or benchmark escalation occurs here.

Firewall and compute

All repositories are fresh procedural fixtures. Hidden tests and repair objects stay host-side; only public issue/source/test/tool output reaches the model. Both blocks use the same merged checkpoint, copied vLLM 0.24 runner, 512 think

• 512 answer tokens, one greedy trajectory, and six turns. Nothing under

benchmarks/ is read/imported, and Menagerie authorization is hard-coded false.

Run

python experiments/qwen35_4b_semantic_policy_headroom_tournament/scripts/run.py --smoke
.venv/bin/python experiments/qwen35_4b_semantic_policy_headroom_tournament/scripts/run.py --lock-design <commit>
.venv/bin/python experiments/qwen35_4b_semantic_policy_headroom_tournament/scripts/run.py --gpu-smoke
.venv/bin/python experiments/qwen35_4b_semantic_policy_headroom_tournament/scripts/run.py --full

Result

Verdict: INSTRUMENT_FAIL. The preregistered answer-cap gate failed in both parent blocks, so no semantic axis is licensed for training. The runner stopped with its registered gate code; no checkpoint was trained and Menagerie remained sealed.

Negative and non-integer handling were saturated after direct failed-test evidence in both blocks. Blank-resource repair was uneven: record was the only shape inside the 15–80% band in A, while tuple was the only one in B. Thus no axis met the frozen requirement for two supported shapes in both blocks even apart from the interface failure. Explicit controls passed at 9/9 and 8/9, and invalid actions stayed at 1.69% and 1.38% of turns.

The cap gate failed at 43/356 turns (12.08%) in A and 46/363 (12.67%) in B, against a 5% ceiling. Forensics localize the problem: 78/89 capped answers contained a valid first tool call, and 77 of those continued with post-call run-on. All capped cases still retained the targeted recovery transition, but all 12 end-to-end failures contacted the cap. The formal stop therefore stands; the association cannot be dismissed or interpreted as the semantic mechanism.

Interpretation

This qualification does not support another post-failure policy curriculum. The parent usually converts explicit verifier evidence into the correct semantic revision already, and the remaining blank-resource misses do not replicate across representations. The more promising frontier is earlier in the loop: acquiring ambiguous public evidence and binding it to the initial proposal before a failed test supplies the answer.

The trajectory contrast makes that pivot concrete. Every one of 72 failed-test cases reached a fully correct patch; the four terminal misses were destructive regressions after correctness. In the rejected-patch condition, by contrast, none of 54 inferred-contract cases produced a fully correct first patch, and zero of 72 rejected trajectories inspected visible tests before first patching. The model can use decisive evidence once handed to it, but does not acquire that evidence before committing to an ambiguous proposal.

A successor should use counterfactual pairs whose issue and source are held constant while visible evidence flips the required policy, then balance the inspect→patch, rejected_patch→changed_patch, and failed_test→diagnose/revise transitions. It should repair the measurement with a payload-safe, parse-aware answer allowance, while keeping response closure diagnostic rather than turning slop suppression into the capability objective. Exact metrics and hashes are in reports/result_receipt.json.

Knowledgebase update

Program evidence, backlog, scorecard, and shared synthesis record the formal instrument failure and the earlier-loop pivot. The claim ledger is unchanged: this no-training qualification produced no checkpoint or benchmark result.

Artifacts

Committed design and compact receipts live here. Detailed parent trajectories will live under large_artifacts/qwen35_4b_semantic_policy_headroom_tournament per reports/artifact_manifest.yaml.

Report

Rendered from reports/report.md

Verdict

INSTRUMENT_FAIL. The exact transaction-trained parent was evaluated on both frozen blocks, but answer-cap contacts reached 12.08% and 12.67% of turns, above the preregistered 5% ceiling. No axis independently met the replicated headroom rule. Training, checkpoint creation, and Menagerie were never authorized.

What ran

The immutable design at 391dadc1 and lock receipt at 9cc90db4 evaluated the unchanged Qwen3.5-4B transaction checkpoint on two content-disjoint blocks. Each block contained 36 fresh procedural repositories and two controlled states per repository, for 72 trajectories. Nine inferred-contract cells crossed negative-quantity, non-integer-quantity, and blank-resource policies with bundle, record, and tuple representations. Three explicit-contract cells served as instrument controls.

The model received one greedy six-turn trajectory per case with 512 thinking and 512 answer tokens per turn. The host alone executed hidden tests. No benchmark source, item, transcript, or result entered the run.

Primary qualification

An axis required 15–80% failed-test success in both blocks and at least two of three shape-family rates inside that band in each block.

The shape-level failed-test rates were:

Negative and non-integer recovery were descriptively saturated. Blank-resource recovery had nontrivial misses, but its supported representation changed from record in A to tuple in B. This is not the replicated two-shape substrate the frozen rule required.

Global controls and interface

Because the preregistration gives instrument failures precedence, the terminal label is INSTRUMENT_FAIL, not the cleaner NO_QUALIFIED_AXIS label.

Answer-cap forensics

Across both blocks, 89/719 turns reached the 512-token answer allowance. This was usually not an unusable truncation: 78/89 capped turns contained a complete parseable tool object, and 77/78 then continued with post-call run-on. Eleven capped turns were invalid. Capped answers comprised 41 inspections, 35 patches, 11 invalid calls, one verification, and one commit.

At case level, 54/66 cap-contact trajectories succeeded versus 78/78 without a cap contact. Every one of the 12 failures contacted the cap. Yet all targeted local transitions still occurred among cap-contact cases: failed-test changed- patch-within-two was 38/38 and rejected-patch valid-changed-within-two was 28/28. Failures instead exhausted six turns without a successful workspace or submission.

This supports two bounded conclusions. First, the cap gate is partly measuring valid-call run-on rather than parser failure. Second, cap contact remains associated with every end-to-end miss, so it cannot be waived after seeing the data. A fresh successor should use a parse-aware payload allowance and preserve both raw cap and validity diagnostics.

Earlier-proposal forensics

The raw trajectories expose a cleaner state contrast than the terminal score. All 72 failed-test cases reached a fully correct executable patch at some point; the four failed-test endpoint misses were destructive regressions after an earlier correct workspace. First-patch correctness was already 29/36 in A and 30/36 in B.

The rejected-patch condition provided no test output. Across its 54 inferred- contract cases, the first changed patch was fully correct 0 times. Across all rejected cases, only 6/36 first patches per block were fully correct, all from the explicit negative/non-integer controls. None of the 72 rejected trajectories inspected visible tests before first patching; the usual action was another source read. Nevertheless, 64/72 eventually reached a correct workspace after later verifier evidence.

This localizes the capability gap to specification acquisition and evidence- conditioned proposal formation. It is not a missing ability to write the correct patch once the public failure states the distinction, and it is not a deleted recovery transition.

Interpretation

Direct failed-test evidence is no longer the best place to seek this capability. The transaction-trained parent repaired negative and non-integer semantic conflicts in every inferred-contract cell, while blank-resource errors did not replicate across shapes. Training on these states would either target already-mastered behavior or chase an unstable representation-specific residual.

The unresolved seam lies earlier: deciding when public evidence must be inspected and binding that evidence to an initial proposal before the verifier states the discrepancy. The strongest next strategy is counterfactual evidence binding. Hold issue and source nearly constant, flip visible tests between opposed but equally plausible policies, and require the model's patch to flip with the evidence. Train complete, transition-balanced trajectories only after the exact parent shows replicated initial-proposal headroom on fresh families. Compare against matched extra replay and equal-compute sampling, then require locality, conditional transition retention, family-held-out transfer, and only then paired Menagerie.

Exposure ledger

• Training: not run.
• New checkpoint: none.
• Menagerie: not invoked; no benchmark seed was consumed.
• Claim ledger: unchanged.
• Compact qualification and result receipts: committed.
• Detailed trajectories: external and checksummed in the artifact manifest.

Experiment log 7

Data files 1

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

• previewreports/result_receipt.json6.4 kB

Reproduce

Smoke test

python experiments/qwen35_4b_semantic_policy_headroom_tournament/scripts/run.py --smoke

Full run

.venv/bin/python experiments/qwen35_4b_semantic_policy_headroom_tournament/scripts/run.py --full

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Related experiments

Nearest by shared name, tags, and programs.

• 2026-07-11 Interactive policy curriculum: oracle DAgger to execution-reward RL
• 2026-07-13 Validation-policy counterexample curriculum
• 2026-06-28 Qwen Oracle-Distilled Acquisition Policy
• 2026-06-24 Qwen3.5-4B Adaptive Evidence Budget Policy
• 2026-07-13 Qwen3.5-4B: The Compute-Optimal Confidence Policy (select + abstain + escalate)
• 2026-07-12 Qwen3.5-4B Pareto Policy Integration