Qwen3.5-4B Prefix Value Guided Search
The one idea you need
Instead of writing eight complete programs and hoping one passes, the model writes four half-finished openings, secretly grades each by running the hidden answer tests, then finishes only the top-graded opening. Does a well-chosen head start beat brute-force full attempts?
The question
When a small model writes code, does grading half-finished attempts and finishing only the most promising one solve more problems than just writing full solutions?
What we found
No. Even when the half-finished drafts were graded with perfect knowledge of the hidden answer tests, finishing only the best-graded one solved the same share of problems as plain full-solution writing — 75% either way. Ungraded random drafts did worse, at 58%. The lone upside: the graded winner used about a quarter of the compute, but only because a flawless grader hand-picked it.
Why it matters
Don't build machinery to score and rank half-written code by promise: the good drafts reach no solutions plain writing misses. Spend compute finishing full attempts, not grading fragments, unless you already own a flawless grader.
On this page
Results at a glance 3
How to read
Five methods along the bottom: plain full writing and four graded-draft variants. Darker bars show the share of twelve problems solved at all; lighter bars the first-pick success rate. Taller is better.
Takeaway → Every draft variant's solved share tops out level with plain writing and none rises above it, so the graded drafts add no new solutions even as first-pick success climbs.
Data table
| search arm | coverage | pass@1 proxy |
|---|---|---|
| full sampling | 75% | 41.7% |
| prefix union | 75% | 66.7% |
| oracle-selected prefix | 75% | 75% |
| lexical-selected prefix | 66.7% | 66.7% |
| random-selected prefix | 58.3% | 50% |
Numbers from reports/report_summary.json
Technical framing
Hidden-test coverage by search arm (matched completion budget) — Gate failed: no prefix arm beats plain full-code sampling on coverage (all top out at 75%), so a prefix value model is not justified.
How to read
Each bar is one method; height is the total compute it burned, so shorter is cheaper. Plain full writing sits on the left, the four graded-draft variants follow.
Takeaway → The best graded-draft bar is about four times shorter than plain writing, so it is cheaper, but only because the hidden answers were already known when picking that draft.
Data table
| search arm | forward tokens |
|---|---|
| full sampling | 20.91k |
| prefix union | 18.39k |
| oracle-selected prefix | 5541 |
| lexical-selected prefix | 5592 |
| random-selected prefix | 5614 |
Numbers from reports/report_summary.json
Technical framing
Compute cost of each arm (forward tokens) — The oracle-selected prefix matches full-sampling coverage at ~4x fewer tokens, but finding that prefix still required the full sweep.
How to read
Each method has two bars: how differently the programs behave when run, and how differently they are written. Taller means more varied. Neither direction is inherently better here.
Takeaway → Graded-draft methods behave far more variously, yet that extra behavioral variety never converted into more problems solved.
Data table
| search arm | functional diversity | program diversity |
|---|---|---|
| full sampling | 26% | 87.5% |
| prefix union | 33.3% | 97.6% |
| oracle-selected prefix | 70.8% | 95.8% |
| lexical-selected prefix | 75% | 95.8% |
| random-selected prefix | 79.2% | 100% |
Numbers from reports/report_summary.json
Technical framing
Sample diversity: prefix-conditioned completions vary more — Prefix-selected arms produce far more functionally distinct programs per completion, but the extra diversity did not translate into coverage gains.
In the author’s words from the Overview · “Summary”
Can hidden-test-valued code prefixes expose a search state that is better than ordinary full-code sampling at matched completion budget?
Overview
This top-level README was generated during repository normalization because the imported experiment did not include one.
- Source track:
track-z - Primary report: reports/final_report.md
- Metadata: metadata.yaml
How To Read
Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.
Summary
Can hidden-test-valued code prefixes expose a search state that is better than ordinary full-code sampling at matched completion budget?
Report
Rendered from reports/final_report.md
Question
Can hidden-test-valued code prefixes expose a search state that is better than ordinary full-code sampling at matched completion budget?
Setup
- Dataset split:
test. - Task count: 12.
- Full-code samples per task: 8.
- Prefix proposals per task: 4.
- Completions per prefix: 2.
- Matched completion budget: True.
- Mean valid prefixes per task: 3.17.
Results
| arm | coverage | pass@1 proxy | visible coverage | functional diversity | program diversity | forward tokens |
|---|---|---|---|---|---|---|
| full_sample_base | 75.0% | 41.7% | 91.7% | 26.0% | 87.5% | 20906 |
| prefix_lexical_selected | 66.7% | 66.7% | 75.0% | 75.0% | 95.8% | 5592 |
| prefix_oracle_selected | 75.0% | 75.0% | 91.7% | 70.8% | 95.8% | 5541 |
| prefix_random_selected | 58.3% | 50.0% | 66.7% | 79.2% | 100.0% | 5614 |
| prefix_union | 75.0% | 66.7% | 91.7% | 33.3% | 97.6% | 18387 |
Gate Decision
Oracle-prefix gate: failed. Full sampling coverage was 75.0%; prefix union coverage was 75.0%; oracle-selected prefix coverage was 75.0%; lexical-selected prefix coverage was 66.7%. Do not train a prefix value model from this result.
Interpretation
This is an oracle-ceiling experiment. A positive result would mean the prefix state space contains selectable states whose completions solve tasks more efficiently than ordinary full-code sampling.
There is an efficiency hint: the oracle-selected prefix arm matches full-sampling coverage with only one selected prefix's completion set per task. But that is not enough to justify training here, because discovering that prefix still required the full prefix-completion sweep, and the prefix union did not improve coverage over ordinary full-code sampling at matched completion count. The strict gate therefore fails: this proposed prefix action space is not yet a useful enough MDP state representation for a learned value model.
Experiment log 3
Show the running log (3 entries)
Purpose
This standalone experiment tests whether partial code prefixes expose useful search states. The first gate is an oracle ceiling: if hidden-test oracle prefix selection cannot beat ordinary full-code sampling at matched completion budget, then training a prefix value model is not justified.
Protocol
- Implement MBPP execution, full-code sampling, prefix proposal, prefix completion, and prefix-search metrics.
- Run a smoke pass on a tiny held-out slice.
- Run a pilot oracle-prefix ceiling against matched full-code sampling.
- Only train a prefix value selector if the oracle ceiling clears.
- Generate a report with charts and a gate decision.
Running Notes
- 2026-06-26: Created fresh standalone experiment and large-artifact directories.
- 2026-06-26: Smoke run on 2 held-out test tasks completed. A summary bug initially reported zeros because per-task metrics were nested; fixed the aggregator and rebuilt the smoke summary.
- 2026-06-26: Pilot run on 12 held-out test tasks completed with matched completion budget: 8 full samples versus 4 prefixes x 2 completions. Full sampling, prefix union, and oracle-selected prefix all covered 9/12. Lexical-selected prefix covered 8/12; random-selected prefix covered 7/12.
- 2026-06-26: Gate decision: strict oracle-prefix coverage gate failed because prefix search did not improve coverage over full sampling at matched completion count. No value model was trained.
Figures 3
Data files 3
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/pilot_prefix_gate_summary.json2.2 kBreports/report_summary.json2.4 kBreports/smoke_prefix_gate_summary.json1.9 kB
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.