Interactive policy curriculum: oracle DAgger to execution-reward RL

Finished2026-07-11Active Evidence Acquisition Agentic Breadth Installation Benchmark Generalization Evidence-Conditioned Selection+5 moreGitHub ↗

The one idea you need

An interactive repair agent runs a loop: look, edit, run the test, commit the fix. Its correction clips were nearly all edits and almost none of the run-the-test moments — like an apprentice who watches endless bolt-tightening but rarely sees the engine started and checked.

The question

If you correct a small agent at the exact moments it stumbles mid-task, one step at a time, does it become a better multi-step problem-solver?

What we found

No — it backfired badly. Success on the practice tasks fell from about 60% to 35%, and on tasks never touched in training from about 69% to 35%. It kept editing fluently but stopped running its checks: the verify-and-commit step was only about 1 in 40 correction examples, so it washed out entirely. On one repair family it never verified once, scoring zero.

Why it matters

Before fine-tuning a looping agent, count how often each decision type appears in your training data. Rare but decisive steps — verify, commit, stop — get drowned out by common edits while parsing, formatting, and surface scores keep looking healthy.

Practice-task success60% → 35%score on task families used for the correction training

Untouched-task success69% → 35%families never used in training, tested for transfer

How rare the verify step wasabout 1 in 4055 of 2,270 correction examples were the verify-and-commit action

Checks run on one repair family0 of 600 turnsthe retrained model never ran its test there, scoring zero

On this page

Results at a glance
Overview
Report
Experiment log
Data files
Reproduce
Related

Results at a glance 1

How much the retrained model gained or lost on each task group

How to read

Three bars show the retrained model's score minus the original model's, for trained task families, untouched task families, and basic single-step skills. The zero line means no change; bars above zero are better, below zero are worse.

Takeaway → Both multi-turn task bars plunge far below zero, roughly a quarter to a third of the scale, while the basic-skill bar barely dips — the damage is to in-task decision-making, not underlying knowledge.

Data table

frozen proxy slice	DAgger minus C53 incumbent
trained task families	-0.253
untouched task families	-0.333
atom retention	-0.0215

Numbers from experiments/qwen35_4b_interactive_policy_curriculum/analysis/dagger_gate.json

Technical framing

Live-state DAgger damaged the interactive policy — Parsing remained perfect and natural closure improved, but trained and untouched multi-turn policies collapsed. The preregistered gate stopped reward training and Menagerie.

In the author’s words from the Overview · “Results”

Verdict: negative training-recipe result; DAgger gate failed. The frozen Paired bootstrap 95% intervals exclude zero by a wide margin: train macro [-0.2954, -0.2103], unseen macro [-0.3804, -0.2869]. Mean action validity fell 5.7pp on trained families and 25.2pp on unseen families, even though natural thinking closure improved 10.3pp and 12.9pp. This is not a global parser or atom-capability collapse; it is a multi-turn policy collapse. The collection itself was valid: 400 live trajectories yielded 1,386 unique visited-state corrections, 203 expert demonstrations (12.8%, below the 20% cap), and 681 C53 replay rows. All 2,270 training rows fit the 4,096-token window. … Read the full result →

Overview

This completed preregistered study tested whether correction at the model's own live multi-turn states could improve Qwen3.5-4B beyond the incumbent C53 blend and unlock guarded whole-episode reward training. It did not: the DAgger warm start failed its mechanism gate decisively, so RL, controls, and Menagerie were correctly cancelled.

Research Program

Primary program: agentic_breadth_installation
Supporting programs: process_control_and_tool_use, posttraining_and_adaptation
Program question: after C53's one-time emission-policy install, can training on live state transitions and their consequences buy a second increment of substrate-general agentic capability?
Prior anchors: C50/C53 (blackbox install and second wall), C5 (adaptation must beat frozen alternatives), C11 (test-time feedback alone does not beat matched sampling), and qwen35_4b_oracle_process_grpo (compact executable process state is learnable).

Question

Can Qwen3.5-4B learn one shared observe -> state -> probe/tool/revise -> verify -> commit policy across firewall-clean interactive environments when supervision follows the states the policy actually visits and the final update is driven by multi-turn execution reward? Does that policy transfer to unseen proxy families and beat the incumbent C53 blend on fresh paired Menagerie events?

Hypothesis

C53 saturated because it trains successful actions as isolated completions; it does not optimize the state distribution induced by the model's own prior actions. Programmatic-oracle DAgger should repair recoverable visited states, and guarded group-relative policy optimization on complete episodes should then favor sequences of decisions that cause success rather than merely look like successful traces.

The mechanism is falsified if a matched additional-DAgger/SFT control performs as well as execution-reward training, shuffled trajectory rewards perform as well as real rewards, or gains stay confined to trained environment families.

Setup

Model: only Qwen/Qwen3.5-4B, pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
Starting policy: a regenerated merged checkpoint of the committed C53 sft_blend.jsonl recipe. It is the incumbent for every comparison.
Environment source: a self-contained copy of the C53 procedural gym. No benchmark family implementation, item, transcript, or result detail is read or imported.
DAgger/RL train families: kilnrite, glyphgate, loomfix, ferrier, and burrowmaze.
Proxy-transfer families never used for DAgger or RL: gatepost, patchwheel, and spindle.
Train/eval seed namespaces and exact level mixtures are frozen in configs/curriculum.yaml.
Inner-loop primary metric: greedy terminal episode score on frozen unseen seeds, macro-averaged by family; exact-success, action validity, natural close, turn use, and train-vs-transfer slices are co-primary diagnostics.
Blackbox primary metric: paired Menagerie medium aggregate against the C53 blend on two fresh seeds. Quick is the regression guard; slow is confirmation only.
Oracle-only quantities: expert action, expert state summary, pass@K coverage, and hidden environment reward. They label training or establish ceilings but are never model inputs at deployment.

Registered Curriculum

Reproduce and merge the C53 blend from its committed training rows.
Roll out that policy on fresh train-family episodes. At every model-visited nonterminal state, a state-aware programmatic expert emits a compact common process trace and the next action. Mix these DAgger rows with a frozen C53 replay anchor and train an emission-weighted QLoRA update.
Evaluate the DAgger checkpoint on frozen train-family and unseen-family proxy episodes. If its mechanism gate fails, diagnose the interface and make at most one preregistered repair before any RL spend.
From the gated DAgger checkpoint, sample grouped on-policy trajectories. Use exact terminal environment score, group-relative advantages, PPO clipping, a reference-KL tether, and supervised replay guards. Constant- reward groups contribute diagnostics but no fabricated dense reward.
Compare the RL checkpoint against the DAgger checkpoint, a compute- overmatched additional-DAgger/SFT update, shuffled trajectory rewards, and frozen matched-compute sampling.
Open the Menagerie firewall only if all whitebox mechanism and retention gates pass. Benchmark invocations are CLI-only and stored aggregate-only.

Full frozen decisions, reward definition, entropy routing, and stop rules are in reports/preregistration.md; adversarial review is in reports/design_review.md.

Gates

DAgger gate: train-family macro score improves at least +0.08 over the incumbent, proxy-transfer macro is nonnegative with at least one family improving >=+0.03, and atom/closure/parse retention regressions are each no worse than -0.03 absolute.
RL mechanism gate: real-reward RL beats both DAgger and matched additional-SFT by >=+0.05 train-family macro, is nonnegative on the unseen-family macro, improves at least three train families, and exceeds shuffled-reward training by >=+0.03; retention guards remain green.
Menagerie strategy win: candidate minus incumbent is >=+0.05 on the mean of two fresh paired medium seeds, quick is no worse than -0.02, and a slow confirmation is positive. Anything weaker is recorded as partial or negative evidence, not a breakthrough.

Run

CPU-only smoke:

.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --smoke

The result-bearing pipeline is staged so every expensive phase can stop at its registered gate:

.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --stage incumbent
.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --stage dagger-collect
.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --stage dagger-train
.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --stage proxy-eval
.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --stage dagger-gate
.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --stage rl-collect
.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --stage rl-train
.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --stage controls
.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --stage rl-gate

Menagerie is a separate, conditional CLI-only stage documented after the whitebox decision receipt is written.

Results

Verdict: negative training-recipe result; DAgger gate failed. The frozen paired proxy produced:

Metric	C53 incumbent	DAgger	Delta	Registered decision
train-family episode macro	0.6048	0.3517	-0.2531	`>= +0.08` — fail
unseen-family episode macro	0.6850	0.3519	-0.3331	`>= 0`, one `>= +0.03` — fail
atom-retention macro	0.6926	0.6711	-0.0215	`>= -0.03` — pass
atom parse macro	1.0000	1.0000	0.0000	`>= -0.03` — pass

Paired bootstrap 95% intervals exclude zero by a wide margin: train macro [-0.2954, -0.2103], unseen macro [-0.3804, -0.2869]. Mean action validity fell 5.7pp on trained families and 25.2pp on unseen families, even though natural thinking closure improved 10.3pp and 12.9pp. This is not a global parser or atom-capability collapse; it is a multi-turn policy collapse.

The collection itself was valid: 400 live trajectories yielded 1,386 unique visited-state corrections, 203 expert demonstrations (12.8%, below the 20% cap), and 681 C53 replay rows. All 2,270 training rows fit the 4,096-token window. The diagnostic failure is semantic-operator imbalance: only 55/2,270 targets (2.4%) were VERIFY, including 19/1,386 visited rows. At evaluation the DAgger model emitted PATCH on all 600 loomfix turns and RULE on 599/600 unseen patchwheel turns, effectively deleting the RUN verify/commit pivot. Glyphgate fell 0.667 to 0.017 and unseen spindle 0.977 to 0.322.

Entropy/varentropy were useful only within their actual information regime: 67/200 collection groups were confident failures and 25/200 had outcome variance, while only 2/200 diverged at the coarse first semantic operator. With two siblings, outcome varentropy is degenerate; exact outcome variance still identifies forks. Do not turn either diagnostic into token pressure.

This was not classified as the preregistered mechanical-repair exception: parsing, truncation, expert selftests, merge hashes, and atom retention were all healthy. Rebalancing operators, adding a behavior/KL tether, or replacing full-sequence SFT with context-local pivot control is a new training design and belongs in a standalone follow-up. No RL, matched controls, or Menagerie seed was run.

Interpretation

Live-state labels were not empty, but broad full-sequence imitation installed the common trace/closure surface more strongly than the state-conditional decision policy. The result sharpens C52's locality lesson: supervising the right next action does not make a shared-weight update local to that state, and operator-frequency skew can erase precisely the pivot tokens needed for looping agents. A future warm start must prove semantic-pivot retention and neighboring-policy locality before execution-reward training is licensed.

Knowledgebase Update

The owning program evidence, backlog, scorecard, synthesis, practitioner brief, and result visualization record the negative. The concurrently active specialist-policy experiment owns the next integration test; this mixed-policy arm should not be rerun independently.

Artifacts

src/gym/: copied firewall-clean procedural environments.
src/curriculum.py: state-aware experts and shared process representation.
src/rollout.py: live multi-turn collection and grouped trajectory schema.
scripts/: staged orchestration, DAgger, RL, controls, analysis, merging, and aggregate-only benchmark wrapper.
configs/curriculum.yaml: frozen seed namespaces, splits, budgets, and gates.
data/, runs/, analysis/, reports/: small reproducible artifacts.
Large adapters and merged checkpoints remain under the gitignored large_artifacts/qwen35_4b_interactive_policy_curriculum/ path declared in reports/artifact_manifest.yaml.

Report

Rendered from reports/report.md

Verdict

NEGATIVE — full-sequence live-state DAgger failed its preregistered mechanism gate. Train-family macro terminal score fell 25.3 percentage points and untouched-family macro fell 33.3 points. The gate cancelled execution-reward RL, matched controls, and Menagerie before they consumed compute or benchmark seeds.

This is a training-recipe failure, not evidence that the live-state labels were wrong or that interactive consequence learning is impossible. The update installed the common trace/closure surface while damaging the semantic pivot policy that chooses when to probe, revise, verify, and commit.

Reached Pipeline

Regenerated the C53 blend from all 2,240 committed rows. The frozen 2,048-token window encoded 2,117 rows and skipped 123 exactly as receipted; 333 optimizer steps completed in 3h23m.
Explicitly merged all 128 LoRA deltas into the composite Qwen checkpoint. The merged shard hash is 9450848e9d5d....
Collected 400 live incumbent trajectories over 200 fresh train-family episodes: one greedy and one sampled sibling per episode.
Built 2,270 DAgger rows: 1,386 unique model-visited corrections, 203 expert-demo rows (12.8% of incremental data), and 681 stratified C53 replay rows. No transfer-family row entered incremental data or replay.
Trained 1.5 epochs from the merged incumbent. All 2,270 rows fit the 4,096-token window; 213 optimizer steps completed in 2h11m. The DAgger merged shard hash is 842e3aa40e5e..., distinct from the incumbent and receipt-verified back to the pinned official revision.
Evaluated both checkpoints greedily on the identical frozen vLLM protocol: 480 process episodes and 560 atom-retention items per arm.

Mechanism Gate

Metric	Incumbent	DAgger	Delta	95% paired CI	Gate
train-family episode macro	0.6048	0.3517	-0.2531	[-0.2954, -0.2103]	fail (`>=+0.08`)
transfer-family episode macro	0.6850	0.3519	-0.3331	[-0.3804, -0.2869]	fail (`>=0`)
train mean action-validity	—	—	-0.0567	—	fail (`>=-0.03`)
transfer mean action-validity	—	—	-0.2524	—	fail (`>=-0.03`)
train natural-close mean	—	—	+0.1033	—	pass
transfer natural-close mean	—	—	+0.1293	—	pass
atom family macro	0.6926	0.6711	-0.0215	—	pass (`>=-0.03`)
atom parse macro	1.0000	1.0000	0.0000	—	pass

Family episode deltas expose the damage:

Family	Role	Delta
burrowmaze	trained	+0.0167
ferrier	trained	-0.0220
glyphgate	trained	-0.6500
kilnrite	trained	-0.2658
loomfix	trained	-0.3444
gatepost	unseen	-0.0500
patchwheel	unseen	-0.2944
spindle	unseen	-0.6550

Failure Forensics

The clean guards localize the failure away from generic collapse:

zero DAgger rows truncated;
atom parsing remained 100%;
atom macro stayed inside the retention bar;
natural thinking closure improved materially;
both merged checkpoint hashes changed and their receipt chains validate;
state-aware experts pass every family/level selftest and malformed-state recovery tests.

The action distribution instead identifies semantic-pivot collapse. Across all 2,270 targets, only 55 (2.4%) are VERIFY; visited-state data contains 19 verify rows among 1,386 (1.4%). Loomfix incremental supervision contains 316 PATCH versus 31 RUN actions. At frozen evaluation:

incumbent loomfix emitted 459 PATCH and 70 RUN; DAgger emitted 600 PATCH and zero RUN, scoring 0.000;
incumbent patchwheel emitted 176 RULE and 372 RUN; DAgger emitted 599 RULE and one RUN, scoring 0.000;
DAgger kilnrite over-repeated locally invalid steps and fell 0.879 to 0.614;
unseen spindle fell 0.977 to 0.322 despite retaining its TAPE surface.

The shared process trace therefore taught a fluent decision format without preserving state-conditional decision boundaries. The update overlearned REVISE and underlearned the VERIFY pivot, then transferred that bias to an unseen repair family.

Entropy And Varentropy Read

The collection contained 67/200 confident-failure groups, 25/200 groups with nonzero terminal-score variance, and only 2/200 groups whose first coarse semantic operator differed. With only two siblings, outcome varentropy is mathematically degenerate whenever both outcome buckets have equal counts; outcome variance remained informative. These measurements support routing states, not scaling token loss or treating uncertainty as correctness.

Stopping Decision

The one allowed interface-only repair was not invoked. The failure is not target truncation, parser mismatch, invalid expert logic, or a bookkeeping error. Rebalancing semantic operators, increasing replay, adding a behavior/KL tether, lowering dose, or replacing sequence SFT with targeted pivot control changes the training design and must receive a standalone preregistration and fresh proxy seeds.

No RL collection, RL training, matched-SFT control, shuffled-reward control, or Menagerie event ran. The benchmark firewall remained sealed.

Durable Lesson

Classic DAgger's slogan—label the states the policy visits—is insufficient for a shared small-model update. The curriculum must also preserve the decision operator distribution and the incumbent policy outside corrected states. For looping agents, verification/commit pivots are scarce but causal; broad full-sequence imitation can erase them while every superficial metric (closure, parse, atom retention) looks healthy.

The next credible warm start is locality-first: use live-state expert labels to identify confident wrong semantic operators, then require a tethered push-down/pull-up intervention to preserve neighboring logits and verify/commit rates before any trajectory-reward stage. The active specialist-policy program now owns the next integration test, so this mixed arm should not be rerun in parallel.

Experiment log 3

Show the running log (3 entries, 2026-07-11)

2026-07-11 — intake, preregistration, and CPU smoke

Routed to agentic_breadth_installation; closest duplicate is C53's static gauntlet frontier, and the novelty is live visited-state supervision plus complete-trajectory reward.
Copied the C53 firewall-clean gym into a self-contained follow-up experiment.
Froze five incremental training families, three incremental transfer families, disjoint seed namespaces, terminal-only reward, DAgger/RL gates, matched controls, and conditional Menagerie rules.
Adversarial design review caught five material hazards before GPU spend: indexed imported oracles were not state-aware; validity shaping would repeat C50; raw entropy would repeat C52; transfer replay could invalidate holdouts; and injected close tokens were not policy actions. All are fixed in code or preregistration.
CPU smoke passed: 6 curriculum tests, 13 vLLM wrapper tests, and all 14 gym family selftests. No result-bearing GPU stage or Menagerie event has run.

2026-07-11 — merge validation and collection repairs

A one-step LoRA smoke produced a zero delta because the scheduler spent its only step in warmup. A two-step smoke produced nonzero LoRA-B weights, a changed merged-shard hash, and changed model output. The runner now validates local merged checkpoints recursively back to the pinned official revision.
Fixed two result-threatening harness issues before the registered run: visited states are deduplicated before the expert-demo quota is applied, and paired evaluation requires distinct named checkpoint paths rather than silently accepting one model twice.
Expanded the frozen smoke to 8 curriculum tests and 14 runner tests. These are implementation checks, not result evidence.

2026-07-11/12 — reached DAgger run and mechanism-gate stop

Regenerated the C53 incumbent from 2,240 committed rows: 2,117 encoded at the frozen 2,048-token window, 123 skipped, 333 optimizer steps, and a receipt-verified merged shard (9450848e9d5d...).
Collected 400 incumbent trajectories over 200 fresh train-family episodes. The final 2,270-row curriculum contains 1,386 unique visited-state corrections, 203 expert rows (12.8% of incremental data), and 681 C53 replay rows. No transfer-family row leaked into training or replay.
Trained the DAgger warm start for the registered 1.5 epochs: all 2,270 rows encoded, 213 optimizer steps, finite loss, and a distinct receipt-verified merged shard (842e3aa40e5e...).
The paired proxy gate failed decisively. Train-family macro changed 0.6048→0.3517 (−0.2531; paired-bootstrap 95% CI [−0.2954, −0.2103]); untouched-family macro changed 0.6850→0.3519 (−0.3331; CI [−0.3804, −0.2869]). Atom retention remained inside its guard (−0.0215), parsing stayed 1.000, and natural closure improved.
Forensics localized the damage to semantic decision pivots. Only 55/2,270 targets were VERIFY; the trained policy emitted zero RUN actions across all 600 loomfix evaluation turns and one across 600 untouched patchwheel turns. It learned the trace/closure surface while erasing verify/commit boundaries.
This is not the registered mechanical-repair case: parsing, truncation, experts, checkpoint application, and atom retention were healthy. The gate therefore cancelled RL, matched controls, and Menagerie. Zero benchmark seeds were consumed.

Data files 1

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

runs/dagger_collection/summary.json99 kB

Reproduce

Smoke test

.venv/bin/python experiments/qwen35_4b_interactive_policy_curriculum/scripts/run.py --smoke

Full run

see the staged Run section in README.md

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗