Qwen VM-Agent ECHO QLoRA

Finished2026-06-24imported · line YPosttraining and Adaptation Structured Execution and Compilers GitHub ↗

The one idea you need

Picture an apprentice mechanic fixing an engine one adjustment at a time, checking the gauge after each move. Forcing them to also predict every gauge reading before they look makes them fluent at reading dials, but no better at choosing the right adjustment.

The question

If a small model fixes a tiny program by editing it and running it over several rounds, does also teaching it to predict each result help?

What we found

Only the acting helped. Editing-and-running in a loop lifted the share of tasks solved from 10% at a blank start to 43%, beating a single one-shot guess near 37%. But adding a second job, predicting the program's output after each edit, dropped success to 36%. The model grew fluent at foreseeing results yet quit before finishing even more often.

Why it matters

When a small model acts in a loop, reward finishing the correct program, not narrating what the machine will show. Predicting the environment's responses builds fluency, not judgment, and here it made the model halt too early.

Editing in a loop vs a blank start10% → 43%share of tasks solved, action-trained model over eight rounds

Also predicting each output43% → 36%edit-only loop vs the same loop also foreseeing the result

Best reachable with a perfect guide74%an ideal step-by-step teacher's score, the room still to chase

Quit before finishingover half of runshow often the model halted too early (54-60%)

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 1

Editing in a loop beats a single guess; predicting outcomes adds nothing

How to read

Grouped bars show the share of tasks solved across five test sets plus their average; taller is better. Each cluster has four bars: a single one-shot guess, the edit-and-run loop, that loop also predicting outputs, and a flawless step-by-step teacher.

Takeaway → The loop's bars top the one-shot bars on average, the output-predicting bars are no taller on average and often shorter, and the teacher's bars tower over all, showing room still unclaimed.

Data table

condition	Native Qwen (direct)	Action-only K=8	ECHO K=8	Oracle K=8
Validation	31.2%	46.9%	50%	78.1%
Fresh std	40.6%	56.2%	37.5%	81.2%
Fresh paraphrase	37.5%	21.9%	21.9%	65.6%
Fresh paired	34.4%	34.4%	25%	62.5%
Hard comp.	40.6%	56.2%	46.9%	81.2%
Average	36.9%	43.1%	36.2%	73.8%

Numbers from

Technical framing

VM-agent loop works (10%->43%), but ECHO does not help; the oracle gap remains — The recurrent model-VM edit loop lifts a 10% blank baseline to 43.1% average and beats native Qwen by 6.2%. ECHO (also training on VM observation tokens) reached 36.2% (-6.9% vs action-only) despite far lower observation CE — modeling the textual VM state does not optimize the action policy. The oracle teacher (73.8%) marks the remaining policy gap.

In the author’s words from the Report · “Executive Summary”

The ECHO treatment also trains on the VM observation tokens with weight 0.05, so the model is explicitly trained to predict the consequences of its edits. The core result is mixed. The VM-agent loop itself worked: the action-only model improved from a 10.0% blank-program baseline to 43.1% average K=8 accuracy, and it beat native direct-answer Qwen on average by 6.2%. But ECHO did not improve the scaled result. ECHO reached 36.2% average K=8 accuracy, -6.9% relative to action-only, despite better action-token and observation-token validation losses. The main bottleneck is no longer syntax. Both trained VM-agent variants had 100% parse rate in the main run. … Read the full result →

Overview

This standalone experiment trains a Qwen-attached VM agent. The agent receives a natural-language task, an initial typed bytecode program, and VM observations. At each private turn it emits one textual edit action or STOP; the VM executes the edit and returns the updated program and execution trace as text.

The main comparison is:

action-only QLoRA: train on agent action tokens only;
ECHO QLoRA: train on both action tokens and VM observation tokens.

Main Run

The main run is:

runs/main_vm_agent_echo_blank_a512_stoprule/

It used blank-program initialization, 512 training tasks, 32 examples per eval split, K in {0, 2, 4, 8}, and a native direct-answer Qwen baseline.

Read these first:

reports/main_vm_agent_echo_blank_a512_stoprule/report.html
reports/main_vm_agent_echo_blank_a512_stoprule/report.md
reports/main_vm_agent_echo_blank_a512_stoprule/summary_k8.csv

The HTML report embeds its charts and is the most convenient single file to open. The Markdown report references the PNGs in figures/.

Directory Layout

src/        experiment, native-baseline, and report-generation scripts
runs/       per-run CSV metrics, manifests, rollout samples, and JSON results
reports/    standalone Markdown/HTML reports, summary tables, and figures

Large adapter and checkpoint files are stored under:

large_artifacts/qwen_vm_agent_echo_qlora/checkpoints/

The checkpoint manifest maps run names to large artifact directories:

checkpoint_manifest.csv

Report

Rendered from reports/main_vm_agent_echo_blank_a512_stoprule/report.md

Executive Summary

This experiment tested whether Qwen/Qwen3-4B can be post-trained into a recurrent program-editing agent over a small typed bytecode VM. At inference time, the model starts from a blank valid program, emits one edit action, receives the executed VM state as text, and repeats for up to K turns. The control trains only on edit-action tokens. The ECHO treatment also trains on the VM observation tokens with weight 0.05, so the model is explicitly trained to predict the consequences of its edits.

The core result is mixed. The VM-agent loop itself worked: the action-only model improved from a 10.0% blank-program baseline to 43.1% average K=8 accuracy, and it beat native direct-answer Qwen on average by 6.2%. But ECHO did not improve the scaled result. ECHO reached 36.2% average K=8 accuracy, -6.9% relative to action-only, despite better action-token and observation-token validation losses.

The main bottleneck is no longer syntax. Both trained VM-agent variants had 100% parse rate in the main run. The failures are policy failures: premature STOP, wrong constants, and incomplete multi-step programs. The oracle teacher reached 73.8% average K=8 accuracy, leaving a 30.6% gap for action-only and a 37.5% gap for ECHO.

Setup

Base model: Qwen/Qwen3-4B.
Training method: 4-bit QLoRA with rank 8, alpha 16, 16.5M trainable parameters.
Training data: 512 generated VM tasks.
Evaluation: five 32-example splits: validation, fresh standard wording, fresh paraphrase, paired prompts, and hard composition.
Initial program: PUSH 0; END; PAD ....
Actions: OP <slot> <opcode>, ARG <slot> <0-96>, or STOP.
Inference budgets: K in {0, 2, 4, 8} VM turns.
Baselines: blank VM at K=0, native direct-answer Qwen, and an oracle teacher that edits toward the reference bytecode.

Main K=8 Results

Split	Blank K=0	Native Qwen	Action-only K=8	ECHO K=8	Oracle K=8
Validation	12.5%	31.2%	46.9%	50.0%	78.1%
Fresh standard	9.4%	40.6%	56.2%	37.5%	81.2%
Fresh paraphrase	3.1%	37.5%	21.9%	21.9%	65.6%
Fresh paired	12.5%	34.4%	34.4%	25.0%	62.5%
Hard composition	12.5%	40.6%	56.2%	46.9%	81.2%
Average	10.0%	36.9%	43.1%	36.2%	73.8%

K=8 accuracy by split

K-Scaling

Both learned VM-agent policies benefit from more recurrent turns. The action-only model is monotonic across all five splits from K=0 to K=8. That matters: it means the model is not merely producing a one-shot answer in a different format; additional model-VM interaction is doing useful work.

Accuracy by K

ECHO Result

ECHO clearly learned the observation channel. Final validation CE:

Action-only action CE: 0.2029
ECHO action CE: 0.1895
Action-only observation CE: 0.2010
ECHO observation CE: 0.0131

That did not translate into better average rollout accuracy. ECHO improved validation K=8 accuracy from 46.9% to 50.0%, but was worse on fresh standard, paired, and hard composition. The likely interpretation is that token-level observation prediction is too easy and too local: it teaches the model to model the textual VM state, but does not directly optimize the action policy needed to close the oracle gap.

Training cross-entropy

Failure Mode

Premature STOP remains the clearest behavioral problem. At K=8, action-only had 54.4% average false STOP rate, and ECHO had 60.0%. The parse rate was 100%, so this is not a grammar problem. It is a decision problem: the model often chooses to halt before it has built the correct executable program.

False STOP rate

Interpretation

The useful signal is that a 4B Qwen model can be post-trained to act as a recurrent compiler-like policy over a typed executable substrate. From a blank program, the action-only model reached 43.1% average K=8 accuracy and beat direct-answer Qwen on validation, fresh standard, and hard composition. This supports the broad direction of using the model as one iteration of a compute loop rather than trying to place the entire latent computation inside one forward pass.

The negative signal is equally important. ECHO, as implemented here, is not the missing ingredient. Its auxiliary observation-token loss improved CE but degraded average generalization. The next experiment should move the learning signal closer to executable success: reward the whole rollout, penalize false STOP, and train a verifier or value head over VM states instead of asking the LM to predict long observation strings.

Recommended Next Experiment

Run a verifier-guided rollout optimization experiment:

Warm start from the action-only VM-agent policy.
Add a small value/verifier head over the final-token hidden state that predicts whether the current VM state solves the task.
Fine-tune with rollout-level reward: correct final VM answer, valid program, fewer edits, and a direct false-STOP penalty.
Compare supervised cloning, DPO on successful vs failed rollouts, and GRPO/REINFORCE with the VM reward.
Keep the same native Qwen, blank VM, action-only, and oracle baselines.

This attacks the observed bottleneck directly. The oracle gap shows that the substrate can solve many more examples within K=8; the current model simply does not learn the halting/action policy well enough from token imitation alone.

Experiment log 2

Show the running log (2 entries, 2026-06-24)

2026-06-24

Created a fresh standalone experiment directory.
Selected intervention: Qwen VM-agent ECHO with QLoRA.
Core idea: use Qwen itself as the recurrent transition. Each turn emits one VM edit action, the VM executes it, and the returned observation is appended to the next turn's context.
Main control: action-only QLoRA on the same transcripts, with VM observation tokens present as context but masked out of the loss.
Main treatment: ECHO QLoRA, where VM observation tokens also receive cross-entropy loss.
Large artifacts will be stored in large_artifacts/qwen_vm_agent_echo_qlora/checkpoints/.

Iteration Notes

Implemented the standalone VM-agent script with a seed compiler, textual edit-action trajectories, action-only QLoRA, ECHO QLoRA, generation-time VM rollouts, and CSV/checkpoint writing.
Smoke run smoke_vm_agent_echo completed end to end with the ECHO arm. It validated model loading, LoRA training, weighted token masks, generation, parsing, VM execution, run artifacts, and separate large checkpoint storage.
Smoke diagnostics exposed two issues before scaling: parse-rate accounting treated unparsable actions as success when zero actions were parsed, and ECHO supervision included the initial observation rather than only action-caused VM observations. Patched both and added rollout sample logging.
Patched smoke run smoke_vm_agent_echo_v2 completed. Parse-rate accounting now correctly reports zero for unparsable generations. The tiny one-epoch model mostly copied program-text fragments such as 15:PAD, so the first pilot must test whether a modest amount of action-token exposure is enough to establish the edit-action grammar before assessing answer accuracy.
Pilot run pilot_vm_agent_echo_s64_a96 trained both action-only and ECHO arms successfully, and both learned the action grammar. However, the weak seed compiler produced invalid initial programs on every eval split, so the agent mostly learned PAD cleanup actions and the oracle K-sweep had no useful headroom at the tested K values. Added a controlled blank initialization mode that starts every prompt from a valid PUSH 0; END program, making the next pilot a direct test of Qwen as an iterative VM compiler.
Pilot run pilot_vm_agent_echo_blank_a128 removed the weak compiler confound. Oracle K=8 reached 62.5% to 87.5% across the small eval splits, proving the blank VM pathway has real headroom. Action-only learned valid action syntax and occasionally compiled correct programs, but overused STOP. ECHO improved K=8 accuracy on most splits, including hard composition (43.75% vs. 18.75% action-only), while preserving 100% parse rate.
Patched eval to roll out each example once to the maximum requested K and score intermediate snapshots. This keeps K-sweeps from repeating generation work.
Added an explicit STOP rule to the prompt: stop only when the current VM state is a valid solution; otherwise edit one slot.
Pilot run pilot_vm_agent_echo_blank_exactstop_a128 tested exact-program STOP supervision. It made trajectories longer and helped some easier splits, but hard composition fell to 18.75% at K=8. Decision: do not use exact-program STOP for the main run; scale the blank ECHO setup with answer-based STOP and keep action-only as the control.
Main run main_vm_agent_echo_blank_a512_stoprule completed with 512 training tasks, 32 examples per eval split, K in {0, 2, 4, 8}, and both action-only and ECHO arms. Added a native direct-answer Qwen baseline after the run so the report can distinguish VM-loop gains from what the base language model already answers without the VM.

Figures 4

accuracy by k · reports/main_vm_agent_echo_blank_a512_stoprule/figures/

false stop k8 · reports/main_vm_agent_echo_blank_a512_stoprule/figures/

k8 accuracy by split · reports/main_vm_agent_echo_blank_a512_stoprule/figures/

train ce · reports/main_vm_agent_echo_blank_a512_stoprule/figures/

Data files 21

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗