Qwen Budgeted Action-Value Compiler

Finished2026-06-23imported · line YBenchmark Generalization Structured Execution and Compilers GitHub ↗

The one idea you need

For each word problem the model drafts a couple dozen tiny programs, and usually one computes the right answer. With no answer key it must guess which draft to trust — and it guesses far worse than a version allowed to peek at the answer.

The question

Can a small helper that never sees the answer learn to pick the correct program out of the many a language model drafts for a word problem?

What we found

Barely. On fresh problems the model drafts a correct program among its candidates 81% of the time but ranks it first only 67% of the time. A learned scorer that never sees the answer nudged that to just 70% — while a control that peeks at the answer reached 82%. The wall is choosing, not writing.

Why it matters

When a small model produces correct outputs but ranks them poorly, don't scale the generator. Train a judge that scores whole finished programs by actually running them, so selection catches up to what the model can already write.

Correct program present vs actually picked first81% vs 67%a right program is in the candidate pile far more often than it gets chosen (fresh problems)

Best the no-answer scorer reached67% → 70%learned value guidance barely beats plain search on fresh problems

Control that peeks at the answer82%upper bound that verifies candidates against the known answer

Textbook-matching steps vs still-salvageable steps4% vs 72%only about 1 in 23 partial steps follows the canonical program, yet most can still reach the right answer

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 8

Learned scorer versus peeking at the answer, two task sets

How to read

Bars show task accuracy (taller is better) for four ways of picking the final program — top guess, log-probability beam, best learned scorer, and a control that checks candidates against the answer — grouped as everyday paired tasks and harder multi-step tasks.

Takeaway → The learned-scorer bar sits just above plain search, but the answer-checking bar towers far above both, so the learned scorer captures little of the available gain.

Data table

decoder	fresh paired	hard composition
greedy	67.2%	56.2%
logprob beam	67.2%	56.2%
best learned-value beam	69.5%	57%
answer-verified repair	82%	78.9%

Numbers from analysis/best_family_metrics.csv

Technical framing

Top-1 accuracy: learned value guidance vs answer-verified repair — Learned budgeted action values give only small top-1 gains; the answer-verified repair oracle stays 12-22 points ahead.

The correct program is usually written but not picked

How to read

For four task sets, the short bar is how often the single top pick is right and the tall bar is how often any drafted candidate was right; taller is better, and the gap between them is missed opportunity.

Takeaway → Every tall bar sits 14 to 20 points above its short partner, meaning a correct program was drafted far more often than it was actually chosen.

Data table

evaluation split	logprob beam top-1	candidate oracle
fresh standard	64.1%	83.6%
fresh paraphrase	64.1%	78.9%
fresh paired	67.2%	81.2%
hard composition	56.2%	75.8%

Numbers from analysis/best_family_metrics.csv

Technical framing

Candidate sets already contain the answer: beam top-1 vs candidate oracle — The beam generates correct programs far more often than it ranks them first; selection, not generation, is the bottleneck.

How much to trust the learned score before it backfires

How to read

Lines track accuracy (higher is better) on fresh paired tasks as the learned score is given more weight in the final choice, for two score types; the left is light weight, the right edge is heavy weight.

Takeaway → Both lines peak at light weight then dip at the far right, showing that leaning too hard on the learned score makes choices worse, not better.

Data table

learned-value weight in beam score	exact-prefix value	advantage value
0.1	68%	68.8%
0.25	68%	69.5%
0.5	68%	68.8%
1	68.8%	68.8%
2	65.6%	62.5%

Numbers from analysis/all_final_metrics.csv

Technical framing

Value-weight sweep on fresh paired tasks: the useful range is narrow — Small value weights nudge accuracy above the 67.2% logprob baseline, but over-weighting the learned value hurts beam ranking.

How cleanly each kind of step-label can be told apart

How to read

Bars show how reliably a learned score separates good partial steps from bad ones (taller is better) for four training signals, from a strict exact-match label to richer graded scores.

Takeaway → The exact-match bar is tall and clean while the graded-score bars are much shorter, so nuance is exactly what the lightweight scorer struggles to learn.

Data table

value target	best held-out AUC
exact prefix	0.95
found (recoverable)	0.814
graded Q-value	0.628
sibling advantage	0.666

Numbers from report table (reports/qwen_budgeted_action_value_compiler_paper.md)

Technical framing

How learnable is each action-value target? Held-out AUC in the main run — Sparse exact-prefix labels are easy to discriminate; the richer graded targets are much harder for the lightweight value head.

More training examples let the program-writer actually learn

How to read

Lines show how accurately the model turns prompts into runnable programs (higher is better) as training proceeds, comparing a small-data pilot against the main run that used four times the examples.

Takeaway → The pilot line stays flat near the floor while the main-run line climbs to about 72%, showing data volume, not tuning, unlocked the program-writer.

Data table

training step	main (512 train examples)	pilot (128 train examples)
1	0%	9.4%
55	—	10.9%
110	—	7.8%
150	18.8%	—
165	—	7.8%
220	—	12.5%
300	57.8%	—
450	65.6%	—
600	71.9%	—

Numbers from experiments/qwen_budgeted_action_value_compiler/analysis/compiler_train_logs.csv

Technical framing

Compiler quick bytecode accuracy during training: main run vs pilot — The pilot compiler stalled near 12.5%; quadrupling supervised traces let the main run reach 71.9% quick bytecode accuracy.

Exactly rebuilding the intended program stays low everywhere

How to read

Bars show how often each method reproduces the exact intended program (taller is better) across four task sets, comparing plain search, the best learned scorer, and the answer-checking version.

Takeaway → Even the answer-checking bars only reach the mid-fifties, and exact recovery drops to about a third on the hardest set, so rebuilding the canonical program is hard even with answer checking.

Data table

evaluation split	greedy	best exact-prefix beam	answer-verified repair
fresh_standard	44.5%	46.9%	53.1%
fresh_paraphrase	49.2%	50%	53.1%
fresh_paired	48.4%	49.2%	54.7%
hard_composition	29.7%	29.7%	32.8%

Numbers from report table

Technical framing

Exact program match by split: value beams barely move it, repair helps modestly — Exact recovery of the canonical program stays far below answer accuracy on every split, collapsing to ~30% on hard composition.

Most steps aren't the textbook step, yet still reach the answer

How to read

For three data-collection runs, the short bar is the share of partial steps matching the single textbook program and the tall bar is the share that can still reach a correct answer; taller means more workable steps.

Takeaway → The recoverable bar towers over the textbook bar in every run — about 72% versus 4% in the main run — so textbook-only supervision misses most good steps.

Data table

run	exact-prefix positives	recoverable (found) positives
smoke (1,386 prefixes)	12.5%	54.4%
pilot (15,712 prefixes)	4.3%	68.7%
main (67,233 prefixes)	4.3%	72.1%

Numbers from report table

Technical framing

Prefix-action label rates by run: recoverable actions dwarf canonical ones — In the main run only 4.3% of actions keep the canonical trace, yet 72.1% can still be completed to the right answer.

Training longer doesn't sharpen the learned scores

How to read

Lines track how reliably each of four learned scores separates good steps from bad (higher is better) across eight training rounds; the four lines sit at clearly different heights.

Takeaway → Every line is essentially flat from the first round onward, so each score's quality is capped by its design, not by how long it trains.

Data table

training epoch	exact	found	qvalue	advantage
1	0.938	0.81	0.624	0.636
2	0.943	0.807	0.628	0.654
3	0.95	0.802	0.628	0.662
4	0.937	0.803	0.617	0.666
5	0.944	0.807	0.605	0.651
6	0.942	0.814	0.61	0.64
7	0.944	0.799	0.618	0.644
8	0.94	0.798	0.599	0.649

Numbers from experiments/qwen_budgeted_action_value_compiler/analysis/value_train_logs.csv

Technical framing

Held-out AUC across value-model training epochs: plateaus almost immediately — Each target's AUC is set by epoch 1 and drifts flat or down over 8 epochs; the head, not training time, limits graded-value quality.

In the author’s words from the Report · “Abstract”

This experiment tests whether a frozen Qwen/Qwen3-4B prompt encoder can be paired with a small posttraining head that compiles natural-language tasks into executable typed bytecode, then uses learned budgeted action values to improve search without revealing the target answer at decode time. The method trains a bytecode compiler head from supervised traces, collects prefix-action labels by bounded suffix search in a typed stack VM, and trains four action-value targets: canonical exact-prefix, binary recoverability, graded budgeted Q, and sibling-normalized advantage. The most important question is whether the learned values can close the gap between top-1 typed beam search and an answer-verified local repair oracle. … Read the full result →

Overview

Standalone experiment for action-value learning over executable bytecode prefixes.

The experiment trains a frozen-Qwen bytecode compiler head, then collects partial-program actions from typed beam search. Each action receives three offline targets:

exact: the action keeps the prefix equal to the canonical target program.
found: bounded executable completion from the post-action VM state can still find a correct completion.
qvalue: a graded return based on the rank and log-probability margin of the best correct completion under the remaining search budget.

Separate value models are trained from these targets and used to guide typed beam search without answer access at decode time.

Layout

src/        training, evaluation, and analysis code
runs/       per-run JSON/CSV logs
analysis/   aggregate tables and figures
reports/    standalone Markdown and HTML report

Large checkpoints are stored outside this directory:

large_artifacts/qwen_budgeted_action_value_compiler/checkpoints/

Reading Order

reports/qwen_budgeted_action_value_compiler_paper.md
analysis/summary.md
analysis/final_metrics.csv
experiment_log.md

Report

Rendered from reports/qwen_budgeted_action_value_compiler_paper.md

Abstract

The method trains a bytecode compiler head from supervised traces, collects prefix-action labels by bounded suffix search in a typed stack VM, and trains four action-value targets: canonical exact-prefix, binary recoverability, graded budgeted Q, and sibling-normalized advantage. The most important question is whether the learned values can close the gap between top-1 typed beam search and an answer-verified local repair oracle.

The answer is partly positive but not decisive. In the main run, the compiler reached 71.9% quick bytecode accuracy. On fresh paired tasks, greedy and logprob beam were 67.2%; the best learned budgeted value was 69.5% with beam_advantage_w0.25, and the answer-verified repair control reached 82.0%. On hard composition, greedy/logprob were 56.2%; learned value guidance reached 57.0% with beam_advantage_w1, while answer-verified repair reached 78.9%. The learned value signal is real, but the current lightweight ranker does not yet reproduce the oracle-like gains.

Experimental Setup

The task distribution emits short natural-language prompts whose answers are computed by hidden programs over a compact typed stack VM. Programs can push constants, combine stack values with arithmetic and comparisons, take modulus 97, and read from two lookup tables. Evaluation is executable: a decoded program is valid only if it is stack-safe and returns the correct value.

The trainable system has two pieces:

A frozen-Qwen feature extractor plus a small compiler head that predicts opcode logits, argument logits, and an auxiliary direct answer head.
A partial-program value model that scores typed candidate actions during beam search.

The value data is collected from the compiler itself. For each partial prefix, the collector expands candidate actions, executes them into VM states, and runs bounded suffix search. Each action receives:

exact: 1 if the action preserves the canonical supervised trace.
found: 1 if a bounded suffix search can still complete to the correct answer.
qvalue: a graded return based on the rank and margin of the best correct suffix completion.
advantage: the action Q divided by the best sibling Q from the same prefix.

The answer-verified local repair decoder is included as a diagnostic upper-bound control. It is not a deployable no-answer decoder because it selects among candidate programs by executing them against the known target answer.

Runs and Artifacts

The standalone directory contains a smoke run, two pilot runs, and the main run. Checkpoints are stored outside the experiment tree at large_artifacts/qwen_budgeted_action_value_compiler/checkpoints/.

Main run hardware: NVIDIA RTX 6000 Ada Generation.

Target Distribution

Budgeted suffix search makes many non-canonical actions recoverable; exact-prefix labels remain rare.

Budgeted suffix search makes many non-canonical actions recoverable; exact-prefix labels remain rare.

Raw Q and sibling-normalized advantage provide graded targets instead of only binary recoverability.

Raw Q and sibling-normalized advantage provide graded targets instead of only binary recoverability.

run_dir	split	prefix_samples	exact_positive_rate	found_positive_rate	mean_q_value	mean_advantage_value	mean_correct_rank
smoke_budgeted_action_value	train	1386	12.5%	54.4%	0.248	n/a	3.617
pilot_budgeted_action_value_s128	train	15712	4.3%	68.7%	0.21	n/a	8.683
pilot_budgeted_action_value_s128_advantage	train	15712	4.3%	68.7%	0.21	0.384	8.683
main_budgeted_action_value_s512	train	67233	4.3%	72.1%	0.259	0.415	7.686

In the main run, exact positives were 4.3% of train prefix actions, while recoverable found positives were 72.1%. This confirms that canonical trace supervision is too narrow to describe the action space: most actions are not canonical, but many can still be completed to the right answer.

Value Training

Exact-prefix labels are easiest to discriminate; found is learnable; graded Q and advantage are harder with this lightweight value head.

Exact-prefix labels are easiest to discriminate; found is learnable; graded Q and advantage are harder with this lightweight value head.

Graded targets have moderate regression error but weaker ranking AUC than exact-prefix supervision.

Graded targets have moderate regression error but weaker ranking AUC than exact-prefix supervision.

Best held-out AUCs in the main run were exact 0.950, found 0.814, qvalue 0.628, and advantage 0.666. The result is consistent with the target definitions: exact-prefix classification is sparse but clean, found classification is broad and noisy, and graded budgeted values are richer but difficult to calibrate from the available features.

Decoder Results

Learned value guidance gives small top-1 gains over greedy/logprob search; answer-verified repair remains much stronger.

Learned value guidance gives small top-1 gains over greedy/logprob search; answer-verified repair remains much stronger.

split	family	decoder	accuracy	program_exact	oracle	mean_completed
fresh_standard	greedy	greedy	64.1%	44.5%	64.1%	1
fresh_standard	logprob	beam_logprob	64.1%	44.5%	83.6%	26.8
fresh_standard	exact	beam_exact_w1	64.8%	46.9%	83.6%	26.3
fresh_standard	found	beam_found_w0.1	64.1%	46.1%	82.8%	27
fresh_standard	qvalue	beam_qvalue_w0.1	64.8%	46.1%	85.2%	27
fresh_standard	advantage	beam_advantage_w0.5	64.8%	45.3%	85.2%	27.1
fresh_standard	answer_repair	local_answer	86.7%	53.1%	86.7%	1
fresh_paraphrase	greedy	greedy	64.1%	49.2%	64.1%	1
fresh_paraphrase	logprob	beam_logprob	64.1%	49.2%	78.9%	25.5
fresh_paraphrase	exact	beam_exact_w2	65.6%	50.0%	80.5%	25.4
fresh_paraphrase	found	beam_found_w0.1	64.1%	49.2%	78.9%	25.8
fresh_paraphrase	qvalue	beam_qvalue_w0.1	66.4%	49.2%	78.9%	25.8
fresh_paraphrase	advantage	beam_advantage_w0.25	65.6%	49.2%	78.9%	26
fresh_paraphrase	answer_repair	local_answer	79.7%	53.1%	79.7%	1
fresh_paired	greedy	greedy	67.2%	48.4%	67.2%	1
fresh_paired	logprob	beam_logprob	67.2%	48.4%	81.2%	26.1
fresh_paired	exact	beam_exact_w1	68.8%	49.2%	80.5%	26.1
fresh_paired	found	beam_found_w0.25	68.8%	48.4%	81.2%	26.8
fresh_paired	qvalue	beam_qvalue_w0.1	68.8%	48.4%	81.2%	26.5
fresh_paired	advantage	beam_advantage_w0.25	69.5%	48.4%	81.2%	26.8
fresh_paired	answer_repair	local_answer	82.0%	54.7%	82.0%	1
hard_composition	greedy	greedy	56.2%	29.7%	56.2%	1
hard_composition	logprob	beam_logprob	56.2%	29.7%	75.8%	27.7
hard_composition	exact	beam_exact_w0.25	57.0%	29.7%	75.0%	27.5
hard_composition	found	beam_found_w0.1	56.2%	29.7%	76.6%	27.9
hard_composition	qvalue	beam_qvalue_w0.25	56.2%	29.7%	76.6%	28
hard_composition	advantage	beam_advantage_w1	57.0%	30.5%	73.4%	28.9
hard_composition	answer_repair	local_answer	78.9%	32.8%	78.9%	1

Fresh paired accuracy improved from 67.2% greedy/logprob to 69.5% with the best advantage-guided beam and 68.8% with the best exact-prefix beam. Hard composition improved only slightly, from 56.2% to 57.0% for exact-prefix value and 57.0% for advantage value.

Weight Sensitivity

The useful value-weight range is narrow; over-weighting learned values tends to hurt beam ranking.

The useful value-weight range is narrow; over-weighting learned values tends to hurt beam ranking.

The sweep shows that the learned value heads are not calibrated enough to dominate compiler log-probability. Small weights sometimes help, but larger weights frequently collapse back toward worse rankings. That matters because a scalable posttraining tweak needs a value signal that can safely override a local token or action prior when the prior is myopic.

Oracle Gap

Candidate sets often contain correct programs that the no-answer scorers fail to rank first.

Candidate sets often contain correct programs that the no-answer scorers fail to rank first.

On fresh paired tasks, logprob beam top-1 accuracy was 67.2%, but its candidate oracle was 81.2%. On hard composition, logprob beam top-1 accuracy was 56.2%, while the candidate oracle was 75.8%. The value models recover only a small part of this slack. The repair control recovers much more because it uses the answer itself as a perfect verifier.

Iteration Within This Experiment

The main run strengthened the compiler and exposed the remaining gap between learned value ranking and answer-verified repair.

The main run strengthened the compiler and exposed the remaining gap between learned value ranking and answer-verified repair.

The pilots were useful because they separated three failure modes: a weak compiler, overly broad recoverability labels, and weak calibration of graded values. The main run addressed the first issue but preserved the ranking gap. This points away from simply scaling the same value head and toward training methods that make the model imitate the answer-verified selector or learn from execution traces more directly.

Interpretation

The experiment supports three claims.

A small posttraining head can compile frozen-Qwen prompt features into executable bytecode with nontrivial generalization.
Bounded suffix search reveals a much broader recoverable action set than canonical trace supervision.
A lightweight learned value ranker is not enough to reproduce the gains of answer-verified candidate selection.

The third point is the critical one. The system already generates correct programs in the candidate set more often than it selects them. The next high-impact direction is therefore not another partial-program classifier. It is a stronger supervision loop that distills the answer-verified selector into a deployable scorer, or a policy-gradient procedure that uses execution reward to update the program policy over complete candidates.

Recommended Next Experiment

Train a selector or reranker directly on complete candidate programs sampled from the compiler beam, with positives chosen by execution and negatives chosen from near-miss candidates. The reranker should consume the prompt, bytecode, execution trace, and final stack state, then predict correctness without seeing the answer. That moves supervision much closer to the 82%/79% repair oracle while preserving a deployable no-answer inference path.

The strongest variant would combine supervised contrastive reranking with hard-negative mining:

Generate 32-64 complete candidate programs per prompt.
Execute all candidates during training only.
Mark candidates correct by target answer and retain hard incorrect candidates with high compiler score.
Train a frozen-Qwen-attached reranker, optionally with LoRA or PEFT, to score complete programs.
Decode by compiler beam plus reranker top-1, with local repair retained only as an oracle control.

This is the shortest path from the current result toward a practical Qwen-attached program executor: it attacks the observed ranking bottleneck directly instead of hoping that prefix-level value targets will indirectly learn the same selection rule.

Experiment log 1

Show the running log (1 entry, 2026-06-23)

2026-06-23

Created standalone experiment directory with src/, runs/, analysis/, reports/, and a separate large-artifact checkpoint root.
Design choice: keep the frozen-Qwen compiler and typed bytecode VM, but train an action-value model from bounded suffix search returns instead of a binary prefix verifier.
Planned conditions:
- constrained greedy decoding from the compiler;
- local answer-verified repair over complete programs;
- compiler-logprob typed beam search;
- exact-prefix value guided beam search;
- binary found-value guided beam search;
- graded budgeted-Q guided beam search.
Implemented src/qwen_budgeted_action_value_compiler_experiment.py as a standalone harness with exact, found, and qvalue target modes trained from the same prefix-action sample pool.
Ran smoke_budgeted_action_value, an end-to-end Qwen-backed smoke test with tiny data. It loaded Qwen/Qwen3-4B, extracted frozen features, trained a tiny compiler, collected budgeted action-value samples, trained exact/found/qvalue models, wrote metrics, and saved the checkpoint under the large-artifact root.
Smoke target stats: train exact positives were 12.5%; found positives were 54.4%; q-positive rate was 54.4%; mean Q target was 0.248; mean nonzero Q target was 0.456; mean correct rank was 3.6. This confirms the graded value target is materially sharper than the binary found target.
Ran pilot_budgeted_action_value_s128. The weak 128-example compiler reached 12.5% quick bytecode. Train exact positives were 4.3%; found positives were 68.7%; mean Q was 0.210; mean nonzero Q was 0.306; mean correct rank was 8.7. Fresh paired greedy/logprob were 18.8%; binary found beam reached 25.0%; qvalue beam reached 20.3%; local answer repair reached 50.0%. Hard-composition greedy/logprob were 18.8%; exact beam reached 20.3%; qvalue beam matched 18.8%; local answer repair reached 62.5%.
Interpretation: binary found-value can improve a weak compiler, but absolute Q values are too low and broad for direct decoding. Patched a sibling-normalized advantage target: each action's Q return divided by the best Q among actions from the same prefix, so the model learns the local action choice rather than only absolute recoverability.
Ran pilot_budgeted_action_value_s128_advantage with the same seed and scale. Train mean advantage target was 0.384. Fresh paired binary found beam again reached 25.0%; advantage reached 21.9%; qvalue reached 20.3%; greedy/logprob were 18.8%; local answer repair was 50.0%. Hard-composition advantage did not beat logprob. Interpretation: advantage is useful to measure but not the main bet; binary found and exact-prefix remain the strongest weak-compiler controls.
Ran main_budgeted_action_value_s512, a stronger 512-example main run. The compiler reached 71.9% quick bytecode accuracy. Train prefix labels were 4.3% exact positive, 72.1% found positive, mean Q 0.259, mean advantage 0.415, and mean correct rank 7.7.
Main value-model best held-out AUCs: exact 0.950, found 0.814, qvalue 0.628, advantage 0.666. The exact-prefix label is cleanest; the graded budgeted targets remain difficult for the lightweight scorer.
Main decoder results: fresh paired greedy/logprob were 67.2%; best learned budgeted-value beam was advantage at 69.5%; exact and found beams reached 68.8%; answer-verified repair reached 82.0%. Hard composition greedy/logprob were 56.2%; best exact and advantage beams reached 57.0%; answer-verified repair reached 78.9%.
Interpretation: bounded suffix search exposes a large recoverable action set and learned value guidance produces small no-answer gains, but the main bottleneck is ranking complete candidates. The next experiment should train a deployable complete-program reranker from answer-verified candidate sets rather than another lightweight prefix classifier.
Generated standalone analysis artifacts:
- analysis/summary.md
- analysis/final_metrics.csv
- analysis/best_family_metrics.csv
- analysis/value_train_logs.csv
- analysis/prefix_sample_stats.csv
- analysis/figures/*.png
- reports/qwen_budgeted_action_value_compiler_paper.md
- reports/qwen_budgeted_action_value_compiler_paper.html