Counterfactual evidence-acquisition curriculum

Results at a glance 1

Overview

This experiment tests whether transition-balanced action supervision can teach Qwen3.5-4B to search decisive public repository evidence before its first patch, bind that evidence to the correct policy, and retain the complete coding loop.

Research programs

• Primary: agentic_breadth_installation.
• Supporting: active_evidence_acquisition, process_control_and_tool_use, posttraining_and_adaptation, and benchmark_generalization.
• Direct predecessor: qwen35_4b_semantic_policy_headroom_tournament.
• Training parent: the exact transaction-replay checkpoint from qwen35_4b_transaction_invariant_recovery_curriculum.
• Closest newly landed conceptual near-neighbor: qwen35_4b_early_text_hypothesis_forking.

The near-neighbor supplies exhaustive first-operation hypotheses before thinking on a depth-two list DSL and performs visible-only selection. It does not train weights or acquire evidence with tools. This experiment therefore does not claim novelty for generic early hypothesis shaping; it asks whether an autonomous repository evidence-search policy can be installed and transferred.

Question

Can one Qwen3.5-4B checkpoint learn the conditional sequence

ambiguous source -> search discriminator -> evidence-faithful first patch

across unseen coding families, evidence paths, and query wording while preserving rejected_patch -> changed_patch, failed_test -> diagnose/revise, patch_ok -> verify, and passed_test -> commit?

Result

The experiment stopped at its first model-bearing gate, before interface sampling or training. On the frozen 48-context block, the transaction-replay start checkpoint had median centered non-target logit drift 0.110735 from the apex anchor, above the preregistered 0.100000 ceiling. Entropy retention passed: mean entropy changed by +0.013636 against a -0.050000 floor, and mean varentropy changed by +0.000297.

The terminal verdict is LINEAGE_LOCALITY_INFEASIBLE. The answer-band ladder, qualification, all three training arms, transfer, retention, uncertainty diagnostics, and Menagerie remained sealed. This does not test whether the evidence-acquisition curriculum works; it shows that the chosen start/anchor pair was not local enough for the frozen causal comparison. Any repair requires a new intake and experiment rather than a wider post-result threshold here.

Why this question is next

The predecessor found no usable post-failure semantic axis and formally stopped on its answer-cap gate. Its opened trajectories nevertheless localized the remaining proposal problem: every failed-test case eventually reached a correct patch, but inferred rejected states produced 0/54 correct first patches and none of 72 trajectories inspected visible tests before first patching. The model appeared able to use decisive evidence once supplied, but did not seek it before committing.

That contrast is only a hypothesis generator. This experiment first requires two clean blocks to reproduce low unassisted acquisition, high evidence- injected reachability, and a large advantage over matched-operator nondiscriminating search before any training is authorized.

Counterfactual repositories

Each inferred dyad has byte-identical issue text, source, file tree, path names, and all non-discriminating public bytes. Exactly one public evidence file differs and requires the opposite semantic patch. A successful dyad requires both branches to:

• acquire the designated evidence before their first changed patch;
• make a first patch that passes that branch's visible and hidden checks; and
• produce a patch that fails on the counterfactual counterpart.

The model never sees hidden executables, hidden results, branch labels, or oracle patches. Evidence spans public tests, documentation, and callsites. Bank, qualification, and transfer use disjoint path-name regimes; transfer also uses a held-out signature-query skin.

Training design

The fixed primary arm, evidence_binding, mixes 24 inferred counterfactual tasks with 24 complete blocks from the existing transaction/recovery replay. Two matched controls use explicit-contract redundant acquisition or exchange the evidence-to-patch labels only within each dyad.

Every arm has 432 rows: 48 rows for each of nine conditional transitions. Each transition receives exactly 16,000 weighted answer-action tokens per epoch. Think loss is zero. Training uses rank 32, alpha 64, LR 2e-5, three epochs, and complete nine-transition optimizer supercycles. Logical batch-four rows are executed as serial unpadded physical batch-of-one forwards to avoid the known hybrid-architecture padding divergence.

The controls distinguish aligned evidence binding from generic extra training, unconditional search, or nonspecific shuffled-label damage. The primary is fixed before controls run and cannot be replaced after outcomes.

Staged gates

1. Model-free task, firewall, bank, batching, compute-accounting, and context- geometry smoke.
2. Exact start-to-apex locality feasibility on the new 48-context block, before any behavioral exposure or training.
3. Outcome-free interface selection over answer rungs 1,024/2,048/4,096.
4. Two independent start-model qualification blocks: unassisted, correctly injected evidence, matched-operator nondiscriminating search, and explicit-contract control.
5. Equal-mass three-arm training, merge, and direct candidate-to-apex locality.
6. Trained-family calibration against start and both controls.
7. Held-out-family development and untouched confirmation against start, apex, both controls, nondiscriminating search, injected reachability, and the stronger of start/apex actual-compute-matched sample-more pools.
8. Explicit conditionality plus broad-recovery and transaction-loop retention.
9. Entropy/varentropy diagnostics only after outcomes are already open.
10. Fresh paired Menagerie quick/medium CLI events only after every white-box gate passes.

The actual sample-more comparator is selected by a fixed trajectory-index prefix that overmatches each primary case on both sampled and logical model tokens. Outcomes never choose prefix membership; full six-trajectory pools are oracle-only.

Exact thresholds, stop labels, and prohibited rescues are frozen in reports/preregistration.md. The adversarial attacks that shaped the design are in reports/design_review.md.

Firewall and interpretation boundary

Only Qwen/Qwen3.5-4B is permitted. Every behavior arm uses the same pinned vLLM runner. Fresh procedural repositories are the only training and white-box evaluation substrate. Nothing under benchmarks/ may be read or imported, and Menagerie may be invoked only through its public CLI after authorization.

A white-box pass would support a narrow learned policy for ambiguity-triggered search and evidence-faithful proposal on these procedural coding families. It would not establish open-world active search, verifier-free correctness, internal uncertainty measurement, or a universal capability unlock. A Menagerie pass would add cross-instrument evidence without erasing those limits.

Frozen run protocol

The commands below record the preregistered protocol. The terminal disposition now blocks every scientific command in this directory; they are retained for auditability, not as pending work.

Model-free smoke:

.venv/bin/python experiments/qwen35_4b_counterfactual_evidence_acquisition_curriculum/scripts/run.py --smoke

After committing and pushing the immutable design, create the lock receipt:

.venv/bin/python experiments/qwen35_4b_counterfactual_evidence_acquisition_curriculum/scripts/run.py --lock-design <commit>

Commit and push that receipt to main before any Qwen output. The stages can then be run separately or resumed end to end:

.venv/bin/python experiments/qwen35_4b_counterfactual_evidence_acquisition_curriculum/scripts/run.py --interface
.venv/bin/python experiments/qwen35_4b_counterfactual_evidence_acquisition_curriculum/scripts/run.py --qualify
.venv/bin/python experiments/qwen35_4b_counterfactual_evidence_acquisition_curriculum/scripts/run.py --full

The full orchestrator must stop at every failed ancestor gate. Its Menagerie stage remains sealed unless it has just assembled the final white-box authorization receipt.

Every scientific stop writes runs/terminal_disposition.json with lifecycle_closed: false. After the result is documented, checked, committed, and pushed directly to main, seal the documentation commit with:

.venv/bin/python experiments/qwen35_4b_counterfactual_evidence_acquisition_curriculum/scripts/run.py --closeout

Regenerate the catalog to account for the changed receipt, run make check, commit and push the closed receipt and generated fixpoint, then prove that exact clean commit is on origin/main:

make catalog && make catalog && make check
.venv/bin/python experiments/qwen35_4b_counterfactual_evidence_acquisition_curriculum/scripts/run.py --verify-closeout

The semantic closeout rejects a lingering in-progress registry entry, design-only brief/headline chart, missing program evidence, or stale generated indexes. The experiment is not finished until the post-push verification succeeds and CI is green.

Final state

The deterministic model-free smoke passed, the immutable 35-file design was locked to commit 7311bbeeef2bffe72024eae5b4136c07bbaa7704, and the exact lineage-locality gate then failed on drift while passing entropy retention. Only symmetric next-token logit measurements were produced. No behavioral trajectory, adapter, trained checkpoint, transfer score, or benchmark event exists for this experiment, and zero Menagerie seeds were consumed.

Artifacts

• idea_intake.md: program routing, prior work, and non-duplication decision.
• reports/preregistration.md: immutable arms, thresholds, stage order, and stop taxonomy.
• reports/design_review.md: adversarial objections and mandatory controls.
• configs/default.yaml: exact lineage, seeds, budgets, and gates.
• reports/context_geometry_receipt.json: model-free context-fit check.
• reports/smoke_receipt.json: deterministic bank, tokenizer, batching, firewall, and counterfactual-invariant smoke.
• analysis/locality_start_vs_anchor.json: terminal symmetric logit-locality result.
• runs/terminal_disposition.json: open/closed lifecycle record for LINEAGE_LOCALITY_INFEASIBLE.
• reports/artifact_manifest.yaml: exact external-bank checksums, tracked result artifacts, absent downstream artifacts, and lifecycle commands.

Report

Rendered from reports/report.md

Status

Finished — LINEAGE_LOCALITY_INFEASIBLE (2026-07-13). The first model-bearing gate stopped the experiment before interface sampling, behavioral qualification, or training. The chosen transaction-replay start checkpoint was not sufficiently local to the frozen apex anchor for the registered causal comparison. No Menagerie event occurred.

Research-program fit

The experiment is a capability-production successor in agentic_breadth_installation, with supporting roles for Active Evidence Acquisition, Process Control and Tool Use, Posttraining and Adaptation, and Benchmark Generalization.

The direct predecessor found that post-failure semantic recovery was generally already strong once visible verifier evidence was in context. Its opened rejected-state trajectories instead failed before proposal: 0/54 inferred cases made a correct first patch and 0/72 inspected visible tests before patching. Because that predecessor formally failed its answer-cap instrument gate, this successor does not inherit the observation as evidence. It independently qualifies both acquisition headroom and evidence-utilization reachability before training.

The newly landed qwen35_4b_early_text_hypothesis_forking is an explicit near-neighbor but not the same intervention. It supplies a complete bank of first-step hypotheses before thinking on a small DSL and performs no training or tool acquisition. The present design tests autonomous repository search and weight-installed evidence-to-patch binding. It makes no broad novelty claim about early textual proposal shaping.

Frozen method

Counterfactual causal unit

Each inferred pair holds the issue, source, tree, paths, and all non-discriminator files byte-identical. One public evidence file flips between opposed policies. The primary event requires the agent to acquire that evidence before its first changed patch, pass both public and host-private checks on its own branch, and cross-fail the paired branch. Both branches must succeed, so the dyad is the statistical unit.

Evidence crosses test, documentation, and callsite channels. The bank, qualification, and transfer splits have disjoint path skins. Transfer uses two unseen families and a signature-based search query absent from the reference- and symbol-query training/qualification regimes.

Qualification before production

An outcome-free interface ladder chooses the smallest answer allowance among 1,024, 2,048, and 4,096 that passes invalid-action and all-length-contact gates. Then two independent start-model blocks compare unassisted acquisition with a host-injected correct search, a matched-operator nondiscriminating search, and an explicit-contract control. The control's output must exclude the designated evidence path and marker. Training is authorized only if both blocks show low unassisted success, high injected reachability, a 30-point injected advantage over both unassisted and nondiscriminating search, broad family/channel/query support, and healthy interface behavior.

Transition-balanced training

The fixed primary evidence_binding arm mixes 24 inferred counterfactual tasks with 24 prior complete-loop task blocks. explicit_redundant provides equal dose when the issue already states the answer. shuffled_binding preserves the primary prompts and target multiset but exchanges evidence-to-patch targets within each pair.

All three arms contain 432 rows: 48 at each of nine conditional transitions, including ambiguous-source acquisition, evidence-to-policy patching, rejected-patch revision, failed-test diagnosis and changed revision, verification, and commit. Every transition receives exactly 16,000 weighted answer tokens per epoch. Think loss is zero, physical model forwards are unpadded batch one, and optimizer steps consume full nine-transition supercycles.

Transfer and sampling burden

The primary must pass direct apex-relative locality, trained-family calibration, held-out development, untouched confirmation, explicit conditionality, and two old-family loop-retention blocks. It must beat the exact start checkpoint, C54 apex incumbent, both trained controls, and the stronger matched sample-more baseline.

Sample-more pools contain six shallow trajectories for start and apex. A fixed trajectory-order prefix is selected using only actual sampled and logical model token costs, at the first point meeting both costs of the primary deep case. Outcomes do not determine prefix membership. The full pools are reported only as favorable oracle coverage.

Key registered gates

• Start and primary each must clear direct apex locality on the exact new 48-context set: centered non-target drift at most 0.10 and entropy delta at least −0.05.
• Trained-family primary paired success must be at least 0.65, +0.15 over start, and +0.10 over each trained control.
• Both held-out blocks require +0.08 over start and apex, +0.05 over each control, matched-operator nondiscriminating search, and the stronger dual-overmatched sample-more prefix; they also require injected reachability, nonnegative transfer on both families, per-channel and held-out-query support, and a nonnegative paired bootstrap lower bound versus start.
• Rejected and failed transition retention must each reach 0.95; verification and commit must each reach 0.90.
• On both legacy suites, normal and recovery success may regress at most 0.03 versus start, and no family may regress more than 0.10.
• Invalid and unusable-cap deltas may not exceed +0.02 after training.
• Menagerie remains sealed until every white-box gate passes. Its final strategy gate requires at least one tier at +0.02 and neither below −0.03 versus apex.

All thresholds and stop labels are specified in preregistration.md. Controls cannot replace the fixed primary, development and confirmation cannot be pooled, and a stopped stage cannot be repaired in this directory.

Controls and causal interpretation

The injected-evidence condition proves only that the current interface can use the public discriminator; it is not deployable. Matched-operator nondiscriminating search separates specific evidence from a generic search action and extra tool-result context. Explicit redundant acquisition tests generic extra task/search dose. Within-dyad label shuffling tests whether the observed evidence-to-patch direction matters while preserving frequency and prompt structure. Start, apex, matched sampling, locality, and legacy retention protect against nonspecific training damage.

The strongest permitted white-box conclusion is that this action-only recipe installed an ambiguity-triggered search-and-binding policy that transferred across the registered procedural family/path/query shifts. It would not prove open-world search, internal confidence, verifier-free correctness, or general coding superiority. A benchmark pass would add cross-instrument evidence, not authorize those stronger claims.

Firewall and artifact policy

Only Qwen/Qwen3.5-4B is allowed. Hidden tests and patches remain host-side. No benchmark source, item, transcript, or result detail may be read. White-box behavior uses one pinned vLLM backend; Transformers is limited to symmetric logit audits and training internals.

The three deterministic banks live under large_artifacts/qwen35_4b_counterfactual_evidence_acquisition_curriculum and are bound by compact checksums. The compact locality summary is tracked in analysis/locality_start_vs_anchor.json; row-level logits were not retained. No adapter, merged candidate, behavior trajectory, sample pool, uncertainty table, or benchmark artifact was created. The exact inventory is in artifact_manifest.yaml.

Result

The deterministic model-free smoke and exact prompt-token equivalence checks passed. The immutable 35-file design was locked to commit 7311bbeeef2bffe72024eae5b4136c07bbaa7704, and the locality gate evaluated the registered 48 contexts symmetrically on the exact frozen checkpoints.

The drift failure triggered the preregistered terminal verdict LINEAGE_LOCALITY_INFEASIBLE. The answer-band ladder, qualification blocks, all three training arms, candidate locality, calibration, transfer, retention, uncertainty analysis, and Menagerie were sealed immediately. Zero benchmark seeds were consumed.

Interpretation

This result does not test evidence-acquisition behavior or the proposed curriculum. It establishes only that the registered start/anchor pair cannot support the intended attribution under the frozen 0.10 locality contract. The entropy guard rules out the specific failure mode of a broad entropy collapse; it does not rescue the failed drift gate, and varentropy remains diagnostic.

No new shared claim is warranted because the comparison is specific to this checkpoint pair and context distribution. The durable operational lesson is to measure exact parent-to-anchor locality before investing in a downstream causal curriculum. A successor would need a new intake and preregistration using either a genuinely local reference pair or a design that treats their pre-existing difference as an explicit baseline. The threshold cannot be widened or the run rescued in this result-bearing directory.

Experiment log 8

Reproduce

Smoke test

.venv/bin/python experiments/qwen35_4b_counterfactual_evidence_acquisition_curriculum/scripts/run.py --smoke

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Related experiments

Nearest by shared name, tags, and programs.

• 2026-07-11 Interactive policy curriculum: oracle DAgger to execution-reward RL
• 2026-07-13 Validation-policy counterexample curriculum
• 2026-07-12 Transaction-invariant recovery curriculum
• 2026-06-28 Qwen Oracle-Distilled Acquisition Policy
• 2026-06-24 Qwen3.5-4B Adaptive Evidence Budget Policy
• 2026-07-13 Qwen3.5-4B: Installing Universal Features via Designed Synthetic Curricula