Qwen Semantic Prefix Value Model
The one idea you need
The system writes a tiny program one instruction at a time, never allowed to see the target answer, like assembling a recipe blind and unable to taste it. A scorer rates each half-built program. Teaching it merely "this could still turn out right" proved too loose to pick the true winner.
The question
When a system builds a program step by step without seeing the answer, does scoring steps by whether the answer stays reachable beat exact-match scoring?
What we found
No. Scoring each step by whether a correct answer is still reachable pushed the top pick to about 68 percent, level with plain confidence search and short of the 71 percent from scoring steps against the known correct program. The "still reachable" signal was genuinely learnable but far too permissive: about one step in three qualifies, so it cannot single out the real winner.
Why it matters
When guiding search without the answer, don't train the scorer on yes/no "could still succeed" labels, since too many steps qualify. Instead predict how good the best reachable finish would be, a graded score, so the ranker can isolate near-certain wins.
On this page
Results at a glance 5
How to read
Bars show how often the final answer is correct (taller is better) across five ways of choosing a program, for everyday paired tasks and harder multi-step tasks. The last bar is allowed to peek at the answer.
Takeaway → Only exact-match scoring nudges the everyday bar upward; reachability scoring sits level with plain confidence search, and every method trails far below the answer-peeking bar.
Data table
| decoder | fresh paired | hard composition |
|---|---|---|
| greedy | 68.8% | 51.6% |
| logprob beam | 68.8% | 51.6% |
| exact value beam | 71.1% | 51.6% |
| semantic value beam | 68% | 51.6% |
| answer-verified repair | 82.8% | 78.9% |
Numbers from report table in reports/qwen_semantic_prefix_value_model_paper.md (best-weight rows per decoder family)
Technical framing
No-answer beam guidance: exact-prefix value helps a little, semantic value does not — Exact-prefix value gives the only top-1 gain (+2.3pp on fresh paired); semantic value never beats plain logprob beam search.
How to read
For everyday and harder tasks, three bars compare the first pick, the best program sitting anywhere in the candidate list, and an answer-verified pick. Taller is better; the gap between first pick and best-in-list is wasted potential.
Takeaway → Best-in-list towers over first-pick, so the correct program is usually generated but ranked below wrong ones — a ranking failure, not a generation one.
Data table
| evaluation split | top-1 logprob | beam oracle | answer-verified repair |
|---|---|---|---|
| fresh paired | 68.8% | 82.8% | 82.8% |
| hard composition | 51.6% | 71.9% | 78.9% |
Numbers from report table in reports/qwen_semantic_prefix_value_model_paper.md (Candidate Oracle Gap section)
Technical framing
The correct program is usually in the beam; top-1 ranking loses it — Beam oracle sits 14-20pp above top-1 selection: the gap is ranking/calibration, not candidate generation.
How to read
Bars show the share of program steps each labeling rule marks as good; a taller bar is a broader, looser target. Exact matching is strict, raw reachability is loose, and filtered reachability sits in between.
Takeaway → Raw reachability marks most steps good, about seven times as many as exact matching, giving the scorer a much looser and harder-to-use target.
Data table
| label type | positive-label rate |
|---|---|
| exact prefix | 7.7% |
| raw semantic reachability | 56.6% |
| filtered semantic (top-1) | 35.2% |
Technical framing
Label density in the main run: semantic reachability is a much broader target — Raw reachability marks most actions positive; top-1 filtering tightens it, but semantic positives stay 4-5x denser than exact ones.
How to read
Bars rate each scorer's skill at sorting good program steps from bad on fresh data, from a coin-flip (half) up to perfect (one); taller is better. Left is exact-match scoring, right is reachability scoring.
Takeaway → Both bars sit far above a coin-flip and the exact-match scorer is cleaner, yet even it converts to only a small gain on real tasks.
Data table
| value model | held-out AUC |
|---|---|
| exact-prefix value | 0.941 |
| semantic value | 0.853 |
Numbers from analysis/summary.md
Technical framing
Value model quality: both labels are learnable, exact is cleaner — Both value models learn their target well above chance, yet even 0.94 AUC exact supervision converts to only a small top-1 gain.
How to read
Across four successive trial runs, two bars each show the share of steps marked good by exact matching versus reachability. Lower reachability bars mean a tighter, less permissive target.
Takeaway → The reachability bars shrink run over run, from most steps down to about a third, while the exact bars stay low and flat — the target was progressively tightened.
Data table
| condition | Exact positives | Filtered semantic positives |
|---|---|---|
| pilot cached (raw) | 4.4% | 69.4% |
| pilot ranked | 6.1% | 49.1% |
| pilot top-1 | 7.5% | 37.5% |
| main run (top-1) | 7.7% | 35.2% |
Numbers from report table
Technical framing
Pilot iteration tightened the semantic label — Successive rank filtering cut semantic positives from 69.4% to 35.2% of actions, while exact positives stayed under 8%.
In the author’s words from the Report · “Abstract”
Exact-prefix value supervision reached AUC 0.941 and improved fresh-paired top-1 accuracy from 68.8% greedy / 68.8% logprob beam to 71.1% with beam_exact_w0.5. Semantic value supervision reached AUC 0.853, but its best fresh-paired beam was 68.0% with beam_semantic_w0.25. On hard composition, semantic value matched the logprob top-1 result at 51.6%, while answer-verified repair reached 78.9%. The gap to repair remains mostly a ranking/calibration problem rather than a candidate-generation problem.
Overview
Standalone experiment for semantic reachability supervision over executable bytecode prefixes.
The experiment trains a frozen-Qwen bytecode compiler head, then collects partial-program actions from typed beam search. Each action receives two offline labels:
exact: the action keeps the prefix equal to the canonical target program.semantic: bounded executable completion from the post-action VM state can still reach the target answer.
Separate value models are trained from these labels and used to guide typed beam search without answer access at decode time.
Layout
src/ training, evaluation, and analysis code
runs/ per-run JSON/CSV logs
analysis/ aggregate tables and figures
reports/ standalone Markdown and HTML reportLarge checkpoints are stored outside this directory:
large_artifacts/qwen_semantic_prefix_value_model/checkpoints/Reading Order
reports/qwen_semantic_prefix_value_model_paper.mdanalysis/summary.mdanalysis/final_metrics.csvexperiment_log.md
Report
Rendered from reports/qwen_semantic_prefix_value_model_paper.md
Abstract
This experiment tests whether a value model trained on semantic reachability labels can guide no-answer bytecode search better than an exact-prefix verifier. A frozen Qwen 4B model encodes natural-language prompts. A trained compiler head emits typed stack-machine bytecode distributions. A value model then scores partial bytecode actions during constrained beam search.
The central target is not whether a partial program matches a canonical trace. Instead, the semantic label asks whether bounded executable completion from the post-action VM state can still reach the target answer. Raw reachability was too broad, so the final main run trained on the top-1 reachable action per prefix by compiler prior while preserving canonical exact positives.
The main result is negative but informative. Exact-prefix value supervision reached AUC 0.941 and improved fresh-paired top-1 accuracy from 68.8% greedy / 68.8% logprob beam to 71.1% with beam_exact_w0.5. Semantic value supervision reached AUC 0.853, but its best fresh-paired beam was 68.0% with beam_semantic_w0.25. On hard composition, semantic value matched the logprob top-1 result at 51.6%, while answer-verified repair reached 78.9%. The gap to repair remains mostly a ranking/calibration problem rather than a candidate-generation problem.
Method
The task generator emits mixed natural-language tasks with executable bytecode over a compact stack VM. The opcode set includes arithmetic, comparison, min/max, modulus, and two lookup tables. Programs are normalized to a fixed length and executed invisibly for evaluation.
For each prompt, frozen Qwen hidden states are pooled by a trained compiler head. The head predicts opcode logits, argument logits, and an auxiliary answer head. Typed beam search expands only stack-valid actions. During training-data collection, every candidate action receives:
exact: 1 if the action keeps the prefix equal to the canonical target program.raw semantic: 1 if bounded suffix search can still complete to the target answer from the post-action state.filtered semantic: 1 for canonical exact positives plus the top reachable action per prefix by compiler prior.
The value models do not see the final answer at decode time. Answer-verified local repair is included only as an upper-bound diagnostic because it chooses candidates by executing them against the known target answer.
Runs
The experiment used a smoke run, three 128-example pilots, and one 512-example main run. The smoke run validated the full artifact path. The first pilot showed that raw semantic reachability was too dense. The second and third pilots introduced per-prefix rank filtering. The main run used the stricter top-1 semantic target.
Main run hardware: NVIDIA RTX 6000 Ada Generation.
Label Density
Exact labels are sparse; raw semantic reachability is broad; top-1 filtering reduces but does not eliminate semantic-only positives.
| run_dir | prefix_samples | exact_positive_rate | raw_semantic_positive_rate | semantic_positive_rate | semantic_extra_positive_rate |
|---|---|---|---|---|---|
| main_semantic_prefix_value_s512_top1 | 37946 | 7.7% | 56.6% | 35.2% | 27.5% |
| pilot_semantic_prefix_value_s128_cached | 16930 | 4.4% | n/a | 69.4% | 65.0% |
| pilot_semantic_prefix_value_s128_ranked | 12071 | 6.1% | 65.3% | 49.1% | 43.0% |
| pilot_semantic_prefix_value_s128_top1 | 9835 | 7.5% | 59.5% | 37.5% | 30.0% |
| smoke_semantic_prefix_value | 1288 | 13.9% | n/a | 47.8% | 33.9% |
Value Training
The exact-prefix value model reaches higher AUC than the semantic value model, but the semantic model is trained on a broader and noisier target.
In the main run, exact-prefix positives were 7.7% of train prefix actions. Raw semantic positives were 56.6%, and filtered semantic positives were 35.2%.
Decoder Results
Exact-prefix value improves fresh paired accuracy; semantic value does not outperform exact-prefix value in the main run.
| split | decoder | accuracy | oracle | program_exact |
|---|---|---|---|---|
| fresh_standard | greedy | 64.1% | 64.1% | 46.1% |
| fresh_standard | beam_logprob | 64.1% | 82.8% | 46.1% |
| fresh_standard | beam_exact_w0.25 | 67.2% | 83.6% | 46.9% |
| fresh_standard | beam_semantic_w4 | 67.2% | 83.6% | 44.5% |
| fresh_standard | local_answer | 83.6% | 83.6% | 49.2% |
| fresh_paraphrase | greedy | 62.5% | 62.5% | 50.0% |
| fresh_paraphrase | beam_logprob | 62.5% | 79.7% | 50.0% |
| fresh_paraphrase | beam_exact_w2 | 64.8% | 81.2% | 50.8% |
| fresh_paraphrase | beam_semantic_w0.25 | 63.3% | 79.7% | 50.0% |
| fresh_paraphrase | local_answer | 81.2% | 81.2% | 53.1% |
| fresh_paired | greedy | 68.8% | 68.8% | 50.0% |
| fresh_paired | beam_logprob | 68.8% | 82.8% | 50.0% |
| fresh_paired | beam_exact_w0.5 | 71.1% | 82.0% | 50.0% |
| fresh_paired | beam_semantic_w0.25 | 68.0% | 82.8% | 50.0% |
| fresh_paired | local_answer | 82.8% | 82.8% | 51.6% |
| hard_composition | greedy | 51.6% | 51.6% | 32.8% |
| hard_composition | beam_logprob | 51.6% | 71.9% | 32.8% |
| hard_composition | beam_exact_w0.25 | 51.6% | 71.1% | 32.8% |
| hard_composition | beam_semantic_w2 | 51.6% | 72.7% | 32.8% |
| hard_composition | local_answer | 78.9% | 78.9% | 35.2% |
Candidate Oracle Gap
Beam oracle accuracy remains much higher than top-1 no-answer selection, especially before answer-verified repair.
Fresh paired beam-logprob oracle accuracy was 82.8%, while top-1 logprob accuracy was 68.8%. Hard-composition beam-logprob oracle accuracy was 71.9%, while top-1 logprob accuracy was 51.6%. The value models did not reliably convert that oracle slack into top-1 gains.
Iteration
Pilot runs tightened semantic labels from raw reachability toward a top-1 reachable-action target before the main run.
The iteration changed the semantic label from raw reachability to rank-filtered reachability because raw reachability made too many actions positive. This improved target sharpness, but did not make semantic value dominate exact-prefix value in the final main run.
Conclusion
Bounded semantic reachability is a real, learnable signal: it creates many non-canonical positive actions and the semantic value model reaches held-out AUC above 0.85 during training. However, this form of semantic value supervision is not enough to close the no-answer beam-ranking gap. In the main run, exact-prefix value produced the best fresh-paired top-1 result, and semantic value was mostly neutral relative to logprob search.
The next useful step is not another binary reachability classifier. The result points toward a calibrated action-value target: predict the best achievable completion score or success probability under the remaining search budget, not merely whether any bounded completion exists.
Experiment log 1
Show the running log (1 entry, 2026-06-23)
2026-06-23
- Created standalone experiment directory with
src/,runs/,analysis/,reports/, and a separate large-artifact checkpoint root. - Design choice: keep the frozen-Qwen compiler and typed bytecode VM, but replace exact-prefix-only verifier supervision with semantic reachability labels generated by bounded executable suffix search.
Planned conditions:
- constrained greedy decoding from the compiler;
- local answer-verified repair over complete programs;
- compiler-logprob typed beam search;
- exact-prefix value guided beam search;
- semantic-reachability value guided beam search.
- Implemented
src/qwen_semantic_prefix_value_model_experiment.pyas a standalone harness with both exact and semantic label modes trained from the same prefix-action sample pool. - Ran
smoke_semantic_prefix_value, an end-to-end Qwen-backed smoke test with tiny data. It loadedQwen/Qwen3-4B, extracted frozen features, trained a tiny compiler, collected prefix samples, trained exact and semantic value models, wrote metrics, and saved the checkpoint under the large-artifact root. - Smoke label stats: train exact positives were 13.9%; semantic positives were 47.8%; semantic-only positives were 33.9%. This confirms that the semantic reachability target is not just a relabeled exact-prefix target.
- Started
pilot_semantic_prefix_value_s128and interrupted it during prefix collection. The initial semantic suffix labeler was CPU-bound and too slow at pilot settings because it recomputed candidate actions and suffix reachability for many repeated states. - Patched the semantic labeler to memoize reachability by
(slot, stack, ended, canonical-prefix flag)and cache per-example candidate action sets and action log-probabilities. - Ran
pilot_semantic_prefix_value_s128_cached. Collection was tractable, but raw semantic reachability was too broad: train exact positives were 4.4%, semantic positives were 69.4%, and semantic-only positives were 65.0%. Fresh paired greedy/logprob were 21.9%, local answer repair was 60.9%, and the best semantic beam was 23.4%. Hard-composition greedy/logprob were 17.2%, local answer repair was 60.9%, and the best semantic beam was 23.4%; exact-prefix beam reached 26.6% on hard. - Interpretation: raw semantic reachability is high-recall but too permissive as a binary target. Patched semantic labels to keep raw-reachable action statistics while training on only the top few reachable actions per prefix by compiler prior, preserving exact canonical positives.
- Ran
pilot_semantic_prefix_value_s128_rankedwith top-2 reachable actions per prefix retained as semantic positives. Train raw semantic positives were 65.3%, filtered semantic positives were 49.1%, and semantic-only positives were 43.0%. Fresh paired best semantic beam was 23.4% versus 21.9% greedy/logprob and 60.9% local answer repair. Hard-composition best semantic beam was 23.4% versus 17.2% greedy/logprob and 60.9% local answer repair. - Ran
pilot_semantic_prefix_value_s128_top1with top-1 reachable action per prefix retained as semantic positive. Train raw semantic positives were 59.5%, filtered semantic positives were 37.5%, and semantic-only positives were 30.0%. Fresh paired best exact beam was 26.6%, best semantic beam was 23.4%, greedy/logprob were 21.9%, and local answer repair was 60.9%. Hard-composition best exact and semantic beams were both 21.9%, greedy/logprob were 17.2%, and local answer repair was 60.9%. - Ran
main_semantic_prefix_value_s512_top1. Compiler quick validation bytecode accuracy reached 64.1% at the final logged checkpoint. Main train prefix labels were 7.7% exact positive, 56.6% raw semantic positive, 35.2% filtered semantic positive, and 27.5% semantic-only positive. Exact-prefix value AUC reached 0.941; semantic value AUC reached 0.853. - Main decoder result: fresh paired greedy/logprob were both 68.8%; best exact value beam reached 71.1%; best semantic value beam reached 68.0%; local answer repair reached 82.8%. Hard-composition greedy/logprob, best exact value beam, and best semantic value beam were all 51.6%; local answer repair reached 78.9%.
- Interpretation: semantic reachability labels are learnable and identify many non-canonical live actions, but this binary/rank-filtered target did not outperform exact-prefix supervision for no-answer top-1 beam selection. The remaining gap points toward calibrated action-value prediction rather than another binary reachability classifier.
- Generated aggregate analysis tables, five figures,
analysis/summary.md,reports/qwen_semantic_prefix_value_model_paper.md, andreports/qwen_semantic_prefix_value_model_paper.html.
Figures 5
Data files 12
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv47 kBanalysis/final_metrics.csv47 kBruns/main_semantic_prefix_value_s512_top1/metrics.csv9.7 kBruns/main_semantic_prefix_value_s512_top1/results.json30 kBruns/pilot_semantic_prefix_value_s128_cached/metrics.csv8.0 kBruns/pilot_semantic_prefix_value_s128_cached/results.json25 kBruns/pilot_semantic_prefix_value_s128_ranked/metrics.csv8.0 kBruns/pilot_semantic_prefix_value_s128_ranked/results.json26 kBruns/pilot_semantic_prefix_value_s128_top1/metrics.csv7.9 kBruns/pilot_semantic_prefix_value_s128_top1/results.json26 kBruns/smoke_semantic_prefix_value/metrics.csv4.2 kBruns/smoke_semantic_prefix_value/results.json16 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).