Qwen Dense-State DAgger VM Agent

Finished2026-06-24imported · line YPosttraining and Adaptation Structured Execution and Compilers GitHub ↗

The one idea you need

Instead of writing a whole program as text, the model works like a mechanic making one small fix at a time: it reads a compressed dashboard of the machine's current state, turns exactly one knob, re-inspects, and repeats until it decides to stop.

The question

Can a small model learn to build a solution by making one edit at a time and re-checking, instead of writing the whole answer out as text?

What we found

Mostly yes. Teaching the model to make one edit at a time, then correcting it on the messes it made, roughly doubled accuracy on mixed tasks (22% to 41%) and won on four of five task types. But it lost on ordinary tasks (41% versus 56% for plain prompting) and stayed far below an all-seeing expert solver reaching 81 to 97%. Useful, not a replacement.

Why it matters

The loop teaches good single moves but not good whole-plan sequences. If you build step-by-step agents, imitating one correct edit isn't enough: train pressure over complete runs, and guard the stop button, since quitting early kept capping the gains.

Mixed-task accuracy, before then after22% → 41%plain prompting versus the trained edit loop

The expert solver's ceilingup to 97%an all-seeing teacher; the learned loop topped out near 53%

Where plain prompting still won56% vs 41%ordinary tasks: untrained model beat the trained loop

Task types improved4 of 5splits where the edit loop beat plain prompting

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 7

Trained edit loop versus prompting and a flawless solver

How to read

For each of five task types, four bars: an empty-machine floor, the model answering directly, the trained step-by-step agent, and a flawless all-seeing solver. Vertical axis is share of tasks solved; taller is better.

Takeaway → The trained agent tops the direct-answer bars on four of five task types, yet every bar sits far under the flawless solver.

Data table

Evaluation split	blank VM	native Qwen	final VM agent	oracle teacher
Mixed	12.5%	21.9%	40.6%	96.9%
Standard	25%	56.2%	40.6%	96.9%
Paraphrase	12.5%	40.6%	53.1%	96.9%
Paired	6.2%	31.2%	34.4%	87.5%
Hard	15.6%	25%	31.2%	81.2%

Numbers from report table (reports/report.md); cross-checked against runs/main_joint_action_calibrated_s256_r2/results.json

Technical framing

Final VM controller vs baselines and oracle across evaluation splits — The trained VM controller beats native Qwen on 4 of 5 splits (all but Standard) yet stays far below the oracle teacher.

How accuracy changes with the number of edits allowed

How to read

One line per task type. Horizontal axis is the cap on how many edits the agent may make; vertical axis is share of tasks solved; higher is better.

Takeaway → Every line collapses near zero at two to four edits, then jumps up at eight, meaning half-finished attempts hurt while enough room to finish pays off.

Data table

Max VM steps K	Mixed	Standard	Paraphrase	Paired	Hard
0	12.5%	25%	12.5%	6.2%	15.6%
2	3.1%	3.1%	3.1%	6.2%	3.1%
4	3.1%	6.2%	6.2%	3.1%	6.2%
8	37.5%	40.6%	50%	34.4%	31.2%
12	37.5%	34.4%	53.1%	34.4%	28.1%

Numbers from report table (reports/report.md), Final Learned Accuracy by K; backed by runs/main_joint_action_calibrated_s256_r2/results.json

Technical framing

Accuracy vs rollout budget K (final checkpoint, learned stopping) — Accuracy dips at short budgets (partial edits hurt) then jumps at K=8; extra steps beyond K=8 help little and are not monotonic.

Three editing skills across the training stages

How to read

For five training stages, three bars each: picking the right edit, filling in the right value, and knowing when to stop. Vertical axis is per-step correctness; taller is better.

Takeaway → All three skills climb across stages, but the fill-in-the-value bars stay shortest throughout, marking it the persistent weak spot.

Data table

Training phase	action	argument	STOP
BC epoch 1	32.1%	26.1%	75.7%
BC epoch 2	70.1%	40.7%	87.3%
BC epoch 3	80.1%	53.9%	85.3%
DAgger round 1	76.3%	55.5%	86.2%
DAgger round 2	81.4%	63.4%	87.9%

Numbers from report table (reports/report.md), Training Dynamics

Technical framing

State-level supervision through BC and DAgger rounds — DAgger keeps improving supervised targets; argument prediction stays the weakest component even after two rounds.

Accuracy after each round of practice-on-its-own-attempts

How to read

For each task type, three bars: after basic copying of the expert, then two rounds of correcting the agent on situations it reached itself. Vertical axis is share solved; taller is better.

Takeaway → The extra correction rounds steadily lift the hardest composition tasks from about 19% to 31%, while the other task types just shuffle up and down.

Data table

Evaluation split	After behavior cloning	After DAgger round 1	After DAgger round 2
Mixed	34.4%	40.6%	37.5%
Standard	37.5%	31.2%	40.6%
Paraphrase	43.8%	56.2%	50%
Paired	34.4%	28.1%	34.4%
Hard	18.8%	25%	31.2%

Numbers from runs/main_joint_action_calibrated_s256_r2/results.json

Technical framing

Rollout accuracy after each training phase (K=8, learned stopping) — DAgger rounds lift hard composition steadily (18.8% to 31.2%) but shuffle rather than improve the other splits.

Quitting too early gets worse the more room you allow

How to read

One line per task type. Horizontal axis is the cap on edits allowed; vertical axis is how often the agent stops before the task is done; lower is better.

Takeaway → All lines climb steeply, so giving more room makes the agent quit early far more often, capping the payoff of longer runs.

Data table

Rollout budget K (steps)	Mixed	Standard	Paraphrase	Paired	Hard
0	0%	0%	0%	0%	0%
2	0%	0%	0%	6.2%	0%
4	6.2%	6.2%	6.2%	9.4%	6.2%
8	18.8%	31.2%	18.8%	21.9%	25%
12	50%	56.2%	40.6%	40.6%	50%

Numbers from runs/main_joint_action_calibrated_s256_r2/results.json

Technical framing

False STOP rate grows with rollout budget (final checkpoint) — By K=12 the learned controller stops early on 41-56% of tasks, capping the gains from longer rollouts.

Two stopping rules trade small wins across task types

How to read

For each task type, two bars: letting the agent decide when to stop, versus a confidence check before it stops. Vertical axis is share of tasks solved; taller is better.

Takeaway → The two rules trade wins, the confidence check adding a few points on some types and losing a few on the hardest, so neither clearly wins.

Data table

Evaluation split	Learned stopping	Value-gated stopping
Mixed	37.5%	40.6%
Standard	34.4%	37.5%
Paraphrase	53.1%	53.1%
Paired	34.4%	34.4%
Hard	28.1%	25%

Numbers from report table

Technical framing

Learned vs value-gated stopping at K=12 — Value gating adds about 3 points on Mixed and Standard but costs 3 points on Hard, so neither stopping rule dominates.

Full trained agent towers over two early prototypes

How to read

For each task type, three bars: two small early prototypes and the full trained agent. Vertical axis is share of tasks solved; taller is better.

Takeaway → The full agent dwarfs both prototypes, and the earliest text-only version sits near zero, showing the design change and the added training were both essential.

Data table

Evaluation split	Pilot, text-action DAgger (96 tasks)	Pilot, joint-action heads (96 tasks)	Main, joint-action calibrated (256 tasks)
Mixed	0%	18.8%	37.5%
Standard	0%	18.8%	40.6%
Paraphrase	6.2%	18.8%	50%
Paired	0%	6.2%	34.4%
Hard	0%	6.2%	31.2%

Numbers from runs/pilot_dense_state_dagger_s96_r2/results.json, runs/pilot_joint_action_s96_r2/results.json, runs/main_joint_action_calibrated_s256_r2/results.json

Technical framing

Pilot runs (96 tasks) vs main run (256 tasks), final checkpoint at K=8 — Joint action heads and 2.7x more training tasks were both needed: the pilot text-action agent barely gets off the floor.

In the author’s words from the Report · “Summary”

This standalone experiment tests whether a small posttraining adapter can turn Qwen/Qwen3-4B into one transition of a recurrent typed-bytecode VM controller. Each inference turn receives the task prompt plus dense VM-state tokens, predicts one edit action or STOP, executes that action in a fixed VM, and repeats for up to K steps. The strongest result is not oracle-close, but it is nontrivial. The final checkpoint beats direct native Qwen on four of five evaluation splits, including hard composition, while remaining far below the oracle teacher. The gap says the learned loop is useful but not yet a substitute for the privileged teacher/search process.

Overview

This standalone experiment trains Qwen/Qwen3-4B as one recurrent transition inside a typed bytecode VM loop.

The model receives a natural-language task and dense projected VM-state tokens. It predicts one VM edit action or STOP; the VM executes that action, returns a new structured state, and the same model is run again for the next step.

The main interventions are:

encoderless dense VM-state injection through inputs_embeds;
direct action and value heads instead of textual action generation;
DAgger over the model's own visited VM states;
value-gated stopping to reduce false STOP.

Large checkpoints are stored outside this directory:

large_artifacts/qwen_dense_state_dagger_vm_agent/checkpoints/

Read experiment_log.md for the iteration record. Final reports are written under reports/.

Main completed run:

runs/main_joint_action_calibrated_s256_r2/

Primary reports:

reports/report.md
reports/report.html

Report

Rendered from reports/report.md

Summary

This standalone experiment tests whether a small posttraining adapter can turn Qwen/Qwen3-4B into one transition of a recurrent typed-bytecode VM controller. Each inference turn receives the task prompt plus dense VM-state tokens, predicts one edit action or STOP, executes that action in a fixed VM, and repeats for up to K steps.

The strongest result is not oracle-close, but it is nontrivial. The final checkpoint beats direct native Qwen on four of five evaluation splits, including hard composition, while remaining far below the oracle teacher. The gap says the learned loop is useful but not yet a substitute for the privileged teacher/search process.

Final checkpoint accuracy

Split	Blank VM	Native Qwen	Final VM	Best Observed VM	Oracle Teacher
Mixed	12.5%	21.9%	40.6% (value_gated, K=12)	40.6% (dagger_r1_policy, learned, K=8)	96.9%
Standard	25.0%	56.2%	40.6% (learned, K=8)	40.6% (dagger_r2_policy, learned, K=8)	96.9%
Paraphrase	12.5%	40.6%	53.1% (value_gated, K=8)	56.2% (dagger_r1_policy, learned, K=8)	96.9%
Paired	6.2%	31.2%	34.4% (learned, K=8)	43.8% (bc_policy, value_gated, K=12)	87.5%
Hard	15.6%	25.0%	31.2% (learned, K=8)	31.2% (dagger_r2_policy, learned, K=8)	81.2%

Method

Base model: Qwen/Qwen3-4B loaded in 4-bit NF4 with LoRA rank 8.
Trainable parameters: LoRA adapters, dense VM-state encoder, direct action heads, solved head, and distance head.
State interface: prompt tokens plus 17 dense VM-state tokens through inputs_embeds.
Action interface: joint scoring over STOP, opcode edits, and argument edits.
Copy bias: argument edits are masked to constants visible in the prompt, plus VM constants 0 and 7.
Training: behavior cloning from oracle edit traces, then two DAgger rounds on policy-visited states.
Main run scale: 256 train tasks, 32 validation tasks, 32 fresh tasks per split, 32 hard-composition tasks.

K-Sweep

The final checkpoint usually benefits from additional rollout steps up to K=8 or K=12, but the curve is not cleanly monotonic. Value-gated stopping reduces false STOP in several cases, but it can also suppress correct stopping.

Final K sweep

Split	Final Learned Accuracy by K	Final Value-Gated Accuracy by K
Mixed	K0:12.5%, K2:3.1%, K4:3.1%, K8:37.5%, K12:37.5%	K0:12.5%, K2:3.1%, K4:3.1%, K8:37.5%, K12:40.6%
Standard	K0:25.0%, K2:3.1%, K4:6.2%, K8:40.6%, K12:34.4%	K0:25.0%, K2:0.0%, K4:3.1%, K8:37.5%, K12:37.5%
Paraphrase	K0:12.5%, K2:3.1%, K4:6.2%, K8:50.0%, K12:53.1%	K0:12.5%, K2:0.0%, K4:3.1%, K8:53.1%, K12:53.1%
Paired	K0:6.2%, K2:6.2%, K4:3.1%, K8:34.4%, K12:34.4%	K0:6.2%, K2:6.2%, K4:3.1%, K8:34.4%, K12:34.4%
Hard	K0:15.6%, K2:3.1%, K4:6.2%, K8:31.2%, K12:28.1%	K0:15.6%, K2:3.1%, K4:3.1%, K8:25.0%, K12:25.0%

Training Dynamics

State-level supervision becomes strong at the main scale. Argument accuracy is still the weakest supervised component, but the prompt-constant mask raises it enough for the recurrent loop to work.

Training metrics

Phase	Epoch	Action	Argument	STOP	States
bc_policy	1	32.1%	26.1%	75.7%	1855
bc_policy	2	70.1%	40.7%	87.3%	1855
bc_policy	3	80.1%	53.9%	85.3%	1855
dagger_r1_policy	1	76.3%	55.5%	86.2%	3663
dagger_r2_policy	1	81.4%	63.4%	87.9%	5468

DAgger Dynamics

DAgger collected policy-visited states without collapsing STOP calibration in the calibrated main run. Round-2 collection slightly improved rollout success and reduced false STOP states.

DAgger collection

Collection	States	False STOP States	Rollout Success	Mean Policy Steps
bc_teacher	1855	0	95.3%	6.29
dagger_r1_states	1808	56	64.5%	6.24
dagger_r2_states	1805	36	66.0%	6.26

Interpretation

The experiment supports a narrow claim: dense-state recurrent control over a fixed typed VM can improve a 4B model on some synthetic compositional tasks using a small posttraining adapter. It does not support the stronger claim that this approach closes the oracle gap.

The main limiting factors are visible:

The oracle teacher reaches 81.2% to 96.9% on the same splits, while the final learned VM tops out at 31.2% to 53.1%.
Native Qwen remains stronger on the fresh-standard split.
False STOP remains substantial on high-K learned rollouts.
Argument prediction, although improved, is still the weakest state-level target.
Evaluation and DAgger training are expensive because each VM step is a full Qwen pass.

Next Experiment

The next high-impact step is to move from one-step action imitation to oracle-like correction pressure over complete rollouts. Concretely: keep the dense-state VM loop, but train with a search-augmented teacher that labels policy states with repaired complete programs or value-ranked action sequences. This should target the observed failure directly: the model can learn many local edit actions, but it does not yet reliably choose globally useful edit trajectories.

Artifacts

Run directory: experiments/qwen_dense_state_dagger_vm_agent/runs/main_joint_action_calibrated_s256_r2/
Source: experiments/qwen_dense_state_dagger_vm_agent/src/dense_state_dagger_vm_agent.py
Report figures: experiments/qwen_dense_state_dagger_vm_agent/reports/figures/
Large checkpoints: large_artifacts/qwen_dense_state_dagger_vm_agent/checkpoints/main_joint_action_calibrated_s256_r2/

Experiment log 6

Show the running log (6 entries, 2026-06-24)

2026-06-24

Created a fresh standalone experiment directory.
Selected intervention: dense-state DAgger VM agent.
Core idea: use Qwen as one recurrent transition. At each turn, structured VM state is projected into dense tokens and fed into Qwen with the task prompt. Direct action/value heads choose the next edit or STOP.
Main training loop: oracle behavior cloning from blank programs, followed by DAgger rounds on states reached by the learned policy.
Main failure targeted: premature STOP and off-oracle rollout drift.
Large artifacts will be stored under large_artifacts/qwen_dense_state_dagger_vm_agent/checkpoints/.

Smoke: `smoke_dense_state_dagger`

Command scale: 8 train examples, 4 examples per eval split, 1 BC epoch, 1 DAgger round, K in {0,1,2}.
Outcome: completed end-to-end with Qwen load, LoRA attachment, dense inputs_embeds state tokens, BC training, DAgger state collection, evaluation, CSV/JSON outputs, and LoRA+dense-head checkpoint writes.
Smoke metric quality is not meaningful at this size; the run is only a structural validation.

Pilot: `pilot_dense_state_dagger_s96_r2`

Scale: 96 train examples, 16 examples per eval split, 2 BC epochs, 2 DAgger rounds, K in {0,2,4,8}.
Native Qwen ranged from 25.0% to 62.5% on the small eval splits.
Oracle teacher reached up to 87.5% on fresh splits and 43.8% on the hard split.
Learned policy stayed weak: best round-1 fresh-standard accuracy was 18.8%, best hard accuracy was 6.2%; round 2 regressed after STOP calibration shifted.
Diagnosis: direct 97-way argument prediction was poor, and rollouts repeated no-op structural edits. DAgger collected useful failure states but the action interface was too lossy.

Patch: joint action scoring and prompt constant masks

Added prompt-derived argument masks, with 0 and calendar constant 7 as built-in legal constants.
Replaced independent kind/slot/value decoding with joint action scoring over STOP, OP-slot, and ARG-slot choices.
Added full action cross-entropy so training optimizes the same complete edit chosen at rollout time.
Typed rollout decoding now blocks no-op edit actions.
Smoke run smoke_dense_state_joint_action completed end-to-end and removed the repeated no-op behavior seen in the first pilot.

Pilot: `pilot_joint_action_s96_r2`

Same scale and seed as the first pilot.
BC action accuracy improved from 48.6% to 57.0%; argument accuracy improved from 12.6% to 41.4%.
DAgger round-1 action accuracy improved from 68.7% to 71.6%; DAgger round-2 action accuracy improved from 56.8% to 65.3%.
DAgger round-2 rollout success improved from 25.0% to 36.5%.
Best held-out learned accuracies were still low: 31.2% on fresh paraphrase, 18.8% on fresh standard, 12.5% on the paired split by blank baseline only, and 6.2% on hard composition.
Main implication: the action interface patch is real, but not enough. The main run should keep joint actions and add stronger STOP/value calibration.

Main: `main_joint_action_calibrated_s256_r2`

Scale: 256 train tasks, 32 examples per eval split, 3 BC epochs, 2 DAgger rounds, K in {0,2,4,8,12}.
Calibration changes from pilot: stop_loss_weight=8.0, value_loss_weight=1.0, false_stop_weight=4.0, and dagger_lr=3e-5.
BC reached 80.1% action accuracy, 53.9% argument accuracy, and 85.3% STOP accuracy.
DAgger round 2 reached 81.4% action accuracy, 63.4% argument accuracy, and 87.9% STOP accuracy.
DAgger collection improved rollout success from 64.5% to 66.0% and reduced false STOP states from 56 to 36.
Final checkpoint best accuracies by split: mixed 40.6%, standard 40.6%, paraphrase 53.1%, paired 34.4%, hard 31.2%.
Native Qwen baselines by split: mixed 21.9%, standard 56.2%, paraphrase 40.6%, paired 31.2%, hard 25.0%.
Oracle teacher remained far higher: 81.2% to 96.9% depending on split.
Reports generated: experiments/qwen_dense_state_dagger_vm_agent/reports/report.md and experiments/qwen_dense_state_dagger_vm_agent/reports/report.html.

Figures 4

Data files 13

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗

Qwen Dense-State DAgger VM Agent

Results at a glance 7

Overview

Report

Summary

Method

K-Sweep

Training Dynamics

DAgger Dynamics

Interpretation

Next Experiment

Artifacts

Experiment log 6

2026-06-24

Smoke: smoke_dense_state_dagger

Pilot: pilot_dense_state_dagger_s96_r2

Patch: joint action scoring and prompt constant masks

Pilot: pilot_joint_action_s96_r2

Main: main_joint_action_calibrated_s256_r2

Figures 4

Data files 13

Reproduce

Related

Related experiments

Smoke: `smoke_dense_state_dagger`

Pilot: `pilot_dense_state_dagger_s96_r2`

Pilot: `pilot_joint_action_s96_r2`

Main: `main_joint_action_calibrated_s256_r2`