Qwen Fuyu VM GRPO-ECHO
The one idea you need
Picture the model at the controls of a tiny computer, fixing a program one edit per full thinking-pass, never typing words, just pressing labeled buttons. It first learns by copying worked solutions, then gets rewarded for whole successful fixes so it can sharpen itself.
The question
Can a small model that fixes a tiny program step by step sharpen itself with reward feedback, beyond just copying worked solutions?
What we found
No. Copying worked solutions alone solved about 10% of tasks; one round of reward-based self-coaching dropped that to 7-8%. The coaching produced encouraging signals — it could rank good fixes over bad ones 80% of the time — yet it scrambled the model's judgment of which edit to make and when to stop. Exhaustive search on the same tasks reaches 83%.
Why it matters
Encouraging training signals — a wide reward spread, high ranking accuracy — do not predict better deployment. For small models steering tools step by step, prove reward feedback helps as a move-scorer before letting it directly rewrite the acting policy.
On this page
Results at a glance 8
How to read
Each pair of bars is one coaching method; the left bar is accuracy after copying worked solutions, the right bar after one reward round. Vertical axis is share of tasks solved; taller is better.
Takeaway → Both right bars sit below their left partner, so the reward round cost accuracy instead of adding any.
Data table
| training arm | BC policy (before update) | after one decision update |
|---|---|---|
| Shaped GRPO + ECHO | 10% | 7.1% |
| Process Preference (DPO) | 9.2% | 7.9% |
Numbers from reports/qwen_fuyu_vm_grpo_echo_report.md (Headline Metrics table)
Technical framing
Decision updates reduced deployed accuracy vs behavior cloning — Both decision-optimization arms lowered learned-mode accuracy below the behavior-cloned baseline; the oracle teacher sits at 82.5%.
How to read
Two coaching methods, each with a before-coaching and after-coaching bar; height is the share of tasks solved when the model gets eight tries. Taller is better.
Takeaway → One method's after bar drops all the way to zero, wiping out its earlier best-of-eight wins.
Data table
| training arm | BC policy (before update) | after one decision update |
|---|---|---|
| Shaped GRPO + ECHO | 13.8% | 0% |
| Process Preference (DPO) | 12.5% | 5% |
Numbers from reports/qwen_fuyu_vm_grpo_echo_report.md (Headline Metrics table)
Technical framing
K=8 split accuracy collapsed after the update — The GRPO+ECHO update drove K=8 accuracy to 0% across evaluation splits; process preference also fell sharply.
How to read
For each coaching method, before and after bars show how often the model wrongly declares the task finished. Lower is better.
Takeaway → The preference method's after bar roughly doubles, a sign the coaching scrambled its stopping decisions.
Data table
| training arm | BC policy (before update) | after one decision update |
|---|---|---|
| Shaped GRPO + ECHO | 8.3% | 4.2% |
| Process Preference (DPO) | 7.1% | 14.2% |
Numbers from reports/qwen_fuyu_vm_grpo_echo_report.md (Headline Metrics table)
Technical framing
Process-preference update doubled false STOPs — GRPO reduced false STOPs but shifted errors to invalid slot edits; DPO doubled false STOPs — both signs of broken action/STOP calibration.
How to read
Two groups — how often the model picks the right edit, and how often it correctly decides to stop — with bars for after copying and after each reward round. Taller is better.
Takeaway → Stop accuracy falls from about 85% to under a third after either round; edit-picking drops less sharply, from about half to roughly a third.
Data table
| training diagnostic | BC (final epoch) | after GRPO+ECHO round | after process-DPO round |
|---|---|---|---|
| action accuracy | 50.2% | 35% | 33% |
| STOP accuracy | 85.3% | 28.9% | 26.7% |
Numbers from reports/qwen_fuyu_vm_grpo_echo_report.md (Training Diagnostics table)
Technical framing
Training diagnostics: STOP accuracy degraded even as other signals looked healthy — Both decision updates dragged STOP accuracy from 85% to under 30%, despite DPO reaching 80.2% pairwise ranking accuracy in training.
How to read
For each task type, three bars: leaving the program blank, the copy-trained model, and exhaustive search with many tries. Height is share solved; taller is better.
Takeaway → The trained bars hug the blank-program bars while exhaustive search towers above, proving the tasks are solvable.
Data table
| Evaluation split | Blank-program baseline | BC policy, learned (K=8) | Oracle teacher (K=8) |
|---|---|---|---|
| val_mixed | 6.2% | 0% | 75% |
| fresh_standard | 12.5% | 6.2% | 93.8% |
| fresh_paraphrase | 12.5% | 12.5% | 75% |
| fresh_paired | 12.5% | 25% | 87.5% |
| hard_composition | 25% | 25% | 81.2% |
Numbers from runs/pilot_shaped_echo_s32_20260624/metrics.csv
Technical framing
Headroom per split: learned policy sits near the blank baseline while oracle search solves most tasks — On every split the behavior-cloned policy barely beats leaving the program blank, while oracle search reaches 75-94%.
How to read
Horizontal axis is how many attempts are allowed, vertical is share of tasks solved by exhaustive search, one line per task type. Up and to the right is better.
Takeaway → Every line climbs steeply toward most tasks solved, so the ceiling is the model's decisions, not the task itself.
Data table
| Rollout budget K | val_mixed | fresh_standard | fresh_paraphrase | fresh_paired | hard_composition |
|---|---|---|---|---|---|
| 0 | 6.2% | 12.5% | 12.5% | 12.5% | 25% |
| 4 | 25% | 25% | 18.8% | 25% | 43.8% |
| 8 | 75% | 93.8% | 75% | 87.5% | 81.2% |
Numbers from runs/pilot_shaped_echo_s32_20260624/metrics.csv
Technical framing
Oracle search scales with rollout budget K on all five splits — The VM plus oracle-guided search climbs from near-baseline to 75-94% as K grows, so the capability ceiling is in the policy, not the task.
How to read
For each task type, three bars: after copying solutions, after the reward round, after the preference round; height is the share of edits that are legal moves. Taller is better.
Takeaway → The reward-round bars roughly halve while the preference bars stay at full height, yet accuracy fell in both.
Data table
| Evaluation split (learned mode, K=8) | BC policy (shaped run) | After GRPO+ECHO round 1 | After process-DPO round 1 |
|---|---|---|---|
| val_mixed | 93.8% | 56.2% | 100% |
| fresh_standard | 81.2% | 50% | 100% |
| fresh_paraphrase | 75% | 56.2% | 100% |
| fresh_paired | 87.5% | 68.8% | 100% |
| hard_composition | 87.5% | 50% | 100% |
Numbers from runs/pilot_shaped_echo_s32_20260624/metrics.csv and runs/pilot_process_dpo_s32_20260624/metrics.csv
Technical framing
GRPO update broke edit validity per split; process-DPO kept edits valid but still lost accuracy — The GRPO update roughly halved valid-edit rates on every split, while process-DPO produced 100% valid edits yet still hurt accuracy.
How to read
Bars grouped by training phase; each colored bar tracks how well one piece of choosing an edit (what kind, where, and with what value) fits its training targets. Taller is better.
Takeaway → The reward-round bars stay near the copying levels, so the deployment collapse was not a training-fit failure.
Data table
| Training phase | Action kind | Slot | Opcode | Argument |
|---|---|---|---|---|
| BC epoch 1 | 82.7% | 58.1% | 36.3% | 22.8% |
| BC epoch 2 | 92.4% | 86.2% | 49.3% | 41.3% |
| GRPO+ECHO round 1 | 85.3% | 66.4% | 45% | 29.3% |
| Process-DPO round 1 | 76% | 54.9% | 28.8% | 48.4% |
Numbers from runs/pilot_shaped_echo_s32_20260624/train_log.csv and runs/pilot_process_dpo_s32_20260624/train_log.csv
Technical framing
Per-head training accuracy across phases: action sub-heads stayed near BC levels while deployment collapsed — Both decision updates left per-head training accuracy near BC epoch-1 levels, so the deployment collapse was not a per-head fitting failure.
In the author’s words from the Report · “Summary”
This standalone experiment tested whether a Qwen 4B controller can improve a typed bytecode VM policy when every VM step is one full-model pass over prompt tokens plus dense VM-state embeddings. The policy never emits natural-language action tokens. It emits direct structured action/value/ECHO heads only. The experiment did not pass the scale-up gate. Both decision-optimization arms created useful-looking process signals, but both reduced deployable accuracy relative to the behavior-cloned policy. The main failure was not lack of reward signal; it was that decision updates disturbed action/STOP calibration and produced worse VM edit trajectories.
Overview
This standalone experiment trains Qwen/Qwen3-4B as a Fuyu-style recurrent controller for a typed bytecode VM.
The model does not emit natural-language action tokens. Each VM step is one full Qwen forward pass over prompt embeddings plus dense VM-state embeddings. Structured heads choose one VM edit action or STOP; the VM executes that action; the resulting observation is projected into dense state tokens for the next full-model pass.
The key intervention is trajectory-level GRPO-style training with structured ECHO observation prediction. Rewards are assigned to complete VM rollouts, while the auxiliary ECHO loss trains the same hidden state to predict the VM observation caused by each sampled action.
Large checkpoints are stored outside this directory:
large_artifacts/qwen_fuyu_vm_grpo_echo/checkpoints/Read experiment_log.md for the iteration record. Reports are written under reports/.
Report
Rendered from reports/qwen_fuyu_vm_grpo_echo_report.md
Summary
This standalone experiment tested whether a Qwen 4B controller can improve a typed bytecode VM policy when every VM step is one full-model pass over prompt tokens plus dense VM-state embeddings. The policy never emits natural-language action tokens. It emits direct structured action/value/ECHO heads only.
The experiment did not pass the scale-up gate. Both decision-optimization arms created useful-looking process signals, but both reduced deployable accuracy relative to the behavior-cloned policy. The main failure was not lack of reward signal; it was that decision updates disturbed action/STOP calibration and produced worse VM edit trajectories.
What Ran
- Base model:
Qwen/Qwen3-4Bloaded with 4-bit QLoRA adapters. - Interface: prompt token embeddings concatenated with dense VM-state tokens via
inputs_embeds; structured heads predictedSTOP, opcode edits, argument edits, solved value, distance, and next-observation ECHO targets. - Training data per pilot: 32 synthetic train tasks, 16 examples per evaluation split, 2 BC epochs, then one decision-optimization round.
- Large checkpoints:
/workspace/large_artifacts/qwen_fuyu_vm_grpo_echo/checkpoints/.
Headline Metrics
| arm | phase | mode | mean_accuracy | k8_accuracy | false_stop_rate | mean_steps |
|---|---|---|---|---|---|---|
| Shaped GRPO + ECHO | bc_policy | forced | 8.8% | 12.5% | 0.0% | 4.00 |
| Shaped GRPO + ECHO | bc_policy | learned | 10.0% | 13.8% | 8.3% | 3.53 |
| Shaped GRPO + ECHO | bc_policy | value_gated | 8.3% | 11.2% | 4.6% | 3.89 |
| Shaped GRPO + ECHO | grpo_echo_r1_policy | forced | 7.1% | 0.0% | 0.0% | 4.00 |
| Shaped GRPO + ECHO | grpo_echo_r1_policy | learned | 7.1% | 0.0% | 4.2% | 3.92 |
| Shaped GRPO + ECHO | grpo_echo_r1_policy | value_gated | 7.1% | 0.0% | 0.0% | 4.00 |
| Shaped GRPO + ECHO | oracle_teacher | k=8 | 82.5% | 82.5% | 0.0% | 5.19 |
| Process Preference | bc_policy | forced | 9.2% | 13.8% | 0.0% | 4.00 |
| Process Preference | bc_policy | learned | 9.2% | 12.5% | 7.1% | 3.60 |
| Process Preference | process_dpo_r1_policy | forced | 5.8% | 2.5% | 0.0% | 4.00 |
| Process Preference | process_dpo_r1_policy | learned | 7.9% | 5.0% | 14.2% | 3.33 |
| Process Preference | oracle_teacher | k=8 | 82.5% | 82.5% | 0.0% | 5.19 |
Process Signals
The shaped-GRPO arm produced nonzero rollout reward variance and measurable reachability signal. The process-preference arm produced many on-policy repair labels and an 80% pairwise ranking accuracy during training. Neither translated into better deployment.
| arm | states | success/reachability | failure signal |
|---|---|---|---|
| Shaped GRPO + ECHO | 605 | 9.4% rollout success; 56.3% reachable-after | 59.4% false STOP; 12.7% destroyed reachability |
| Process Preference | 226 | 28.1% rollout success; 97/226 repair labels | 3.5% false-STOP states; 1098 candidates/state |
Training Diagnostics
| arm | phase | action_acc | stop_acc | rank_acc | next_final | train_states |
|---|---|---|---|---|---|---|
| Shaped GRPO + ECHO | bc_policy | 30.8% | 64.7% | 0.0% | 51.1% | 237 |
| Shaped GRPO + ECHO | bc_policy | 50.2% | 85.3% | 0.0% | 72.2% | 237 |
| Shaped GRPO + ECHO | grpo_echo_r1_policy | 35.0% | 28.9% | 0.0% | 77.9% | 842 |
| Process Preference | bc_policy | 30.8% | 64.7% | 0.0% | 51.1% | 237 |
| Process Preference | bc_policy | 50.2% | 85.3% | 0.0% | 72.2% | 237 |
| Process Preference | process_dpo_r1_policy | 33.0% | 26.7% | 80.2% | 65.4% | 463 |
Interpretation
The dense-state whole-network loop is mechanically viable: Qwen can consume the VM state as embeddings, predict structured actions, predict next VM observations, and be trained end to end with QLoRA. The failed gate is the decision update.
Shaped GRPO separated sampled trajectories: first-round sampled rollout success was 9.4%, reward standard deviation was 0.50, and 56.3% of non-STOP edits preserved reachability. But after one update, mean learned accuracy fell from 10.0% to 7.1%, and K=8 accuracy collapsed to 0% across the five evaluation splits in the measured modes. The update reduced false STOP but shifted the policy toward repeated invalid slot edits.
Process preference had a stronger low-variance teacher: 226 on-policy states, 97 repair-labeled states, and 28.1% rollout success before the update. The pairwise ranking term reached 80.2% training accuracy. Deployment still fell: learned mean accuracy went from 9.2% to 7.9%, forced mean accuracy went from 9.2% to 5.8%, and learned false STOP doubled from 7.1% to 14.2%.
Gate Decision
Do not scale this actor-update recipe. The next serious version should separate the model's roles:
- Use Qwen as a value/prior model inside verified beam/search rather than as the sole actor.
- Train process preferences on verifier-labeled states, but evaluate them as search heuristics before allowing them to directly mutate the policy.
- Keep ECHO as an ablation, not a core claim; it learns next-state prediction but did not protect decision quality here.
- Add a shuffled-reward/preference control only after the unshuffled signal improves deployment on this small gate.
Reproduction
Main pilot commands are recoverable from each run's dataset_manifest.json. Primary result directories:
runs/pilot_shaped_echo_s32_20260624runs/pilot_process_dpo_s32_20260624
Experiment log 1
Show the running log (1 entry, 2026-06-24)
2026-06-24
- Created a fresh standalone experiment directory.
- Selected intervention: Fuyu-style whole-network VM loop with trajectory-level GRPO-style training and structured ECHO observation prediction.
- Interface commitment: no natural-language action-token generation. Qwen receives prompt tokens plus dense VM-state tokens through
inputs_embedsand emits only structured head predictions. - Large artifacts will be stored under
large_artifacts/qwen_fuyu_vm_grpo_echo/checkpoints/. - Smoke runs validated the end-to-end dense-state loop, checkpointing, and CSV outputs.
- The first terminal-only GRPO smoke showed the expected sparse-reward failure: zero successful sampled rollouts and high false-STOP rate. I stopped the larger pilot before training on that weaker objective.
- Updated the experiment to use dense process reward: each sampled VM edit is scored by whether it preserves reachable verified completion within the remaining rollout budget, with explicit penalties for false STOP and destroying reachability.
- Added a DAgger-style teacher anchor during RL rounds, an ECHO ablation knob (
--echo_loss_weight 0), and a shuffled-reward control (--shuffle_rollout_rewards 1). - Pilot
pilot_shaped_echo_s32_20260624: shaped GRPO produced usable process signal (9.4% sampled rollout success, 56.3% reachable-after edits), but one update reduced mean learned accuracy from 10.0% to 7.1% and collapsed K=8 measured accuracy to 0.0%. - Added a process-preference arm (
--algorithm process_dpo) after the shaped GRPO gate failure. It trains on on-policy states labeled by repair/oracle actions with sampled bad actions as pairwise negatives. - Pilot
pilot_process_dpo_s32_20260624: process preference collected 226 on-policy states, found repair labels for 97, and reached 80.2% pairwise rank accuracy, but deployment still regressed (learned mean 9.2% -> 7.9%, forced mean 9.2% -> 5.8%). - Wrote standalone reports with charts:
reports/qwen_fuyu_vm_grpo_echo_report.mdandreports/qwen_fuyu_vm_grpo_echo_report.html.
Figures 4
Data files 11
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/pilot_process_dpo_s32_20260624/metrics.csv12 kBruns/pilot_process_dpo_s32_20260624/results.json45 kBruns/pilot_shaped_echo_s32_20260624/metrics.csv16 kBruns/smoke_fuyu_grpo_echo_20260624/metrics.csv7.8 kBruns/smoke_fuyu_grpo_echo_20260624/results.json31 kBruns/smoke_fuyu_grpo_echo_anchor_20260624/metrics.csv8.3 kBruns/smoke_fuyu_grpo_echo_anchor_20260624/results.json31 kBruns/smoke_fuyu_grpo_echo_shaped_20260624/metrics.csv8.3 kBruns/smoke_fuyu_grpo_echo_shaped_20260624/results.json32 kBruns/smoke_process_dpo_20260624/metrics.csv7.7 kBruns/smoke_process_dpo_20260624/results.json31 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.