Qwen Complete-Program Trace Reranker
The one idea you need
A fixed model writes a tiny program to solve each problem. The system then spins off about 111 slightly edited variant programs and trains a small judge to pick the best one purely by watching each program run, never seeing the correct answer.
The question
If a model generates many candidate programs and a correct one is almost always among them, can a small trained judge reliably pick that winner out?
What we found
Barely, and it backfires on hard cases. A correct program sat in the candidate pile 91 to 100 percent of the time, yet the trained picker nudged easy prompts only from 69 to 70 percent and actively hurt longer ones, dropping them from 41 to 35 percent. The ceiling was right there; the picker just could not reach it. Selection, not availability, is the bottleneck.
Why it matters
When you generate many candidates and rerank, the hard part is not producing a correct one; it is supervision. Grading a picker only on the final answer is too weak, since many wrongly reasoned programs land on it. Reward correct intermediate steps instead.
On this page
Results at a glance 6
How to read
Grouped bars, one cluster per test set. Each cluster shows three heights: the model's plain first pick, the trained picker, and the best candidate that actually exists. Taller means more answers correct; the best-possible bar towers over the other two everywhere.
Takeaway → The picker's bar sits almost on top of the plain first pick in every cluster, while the best-possible bar stays near the top, so nearly all reachable gain goes untouched.
Data table
| evaluation split | frozen compiler (base) | learned verifier | oracle (best candidate) |
|---|---|---|---|
| validation mixed L6 | 59.4% | 60.2% | 98.4% |
| fresh standard L6 | 68.8% | 70.3% | 98.4% |
| fresh paraphrase L6 | 57.8% | 60.2% | 100% |
| hard standard L8 | 50% | 46.9% | 98.4% |
| harder standard L10 | 40.6% | 35.2% | 98.4% |
| harder paraphrase L10 | 17.2% | 14.1% | 91.4% |
Numbers from report table (reports/qwen_complete_program_trace_reranker_report.md, Main Results)
Technical framing
Program selection accuracy: learned verifier vs frozen compiler vs oracle — Correct candidates exist for ~91-100% of prompts, but the learned verifier gains little in-distribution and loses on longer chains.
How to read
Lines track how often the answer is correct as problems grow from six to ten steps, left to right. One line each for the plain first pick, the trained picker, and the best available candidate. Higher is better; the best-possible line stays flat near the top.
Takeaway → Past the shortest problems the picker line dips under the plain first pick, even though a correct candidate is nearly always available, showing the picker actively hurts on harder problems.
Data table
| program length (VM steps) | frozen compiler (base) | learned verifier | oracle (best candidate) |
|---|---|---|---|
| 6 | 68.8% | 70.3% | 98.4% |
| 8 | 50% | 46.9% | 98.4% |
| 10 | 40.6% | 35.2% | 98.4% |
Numbers from report table (reports/qwen_complete_program_trace_reranker_report.md, Main Results)
Technical framing
Accuracy vs program length on standard splits — The oracle stays near-perfect as programs get longer, but the learned verifier falls below the frozen compiler beyond length 6.
How to read
One bar per test set showing how much of the gap between the plain first pick and the best available candidate the picker closed. Above zero is good; below zero means it did worse than picking nothing.
Takeaway → Bars reach at most a thin positive sliver on the shortest splits and turn negative on the paired split and every longer problem, so the picker captured almost none of the available headroom.
Data table
| evaluation split | learned verifier |
|---|---|
| validation mixed L6 | 2% |
| fresh standard L6 | 5.3% |
| fresh paraphrase L6 | 5.6% |
| fresh paired L6 | -5.1% |
| hard standard L8 | -6.5% |
| hard paraphrase L8 | -4% |
| harder standard L10 | -9.5% |
| harder paraphrase L10 | -4.2% |
Numbers from report table (reports/qwen_complete_program_trace_reranker_report.md, Main Results)
Technical framing
Share of the base-to-oracle gap recovered by the learned verifier — The verifier recovers at most 5.6% of the reachable headroom and goes negative on paired and longer-chain splits.
How to read
Grouped bars per task type: the plain first pick, the trained picker, and the best available candidate. Higher is better. The best-possible bar sits near the top for every type.
Takeaway → The picker nudges four task types up slightly, leaves one unchanged, and knocks the yes-or-no type down about nine points, where answers collapse to few values.
Data table
| task domain | frozen compiler (base) | learned verifier | oracle (best candidate) |
|---|---|---|---|
| arithmetic | 31.2% | 34.4% | 96.9% |
| calendar | 37.5% | 40.6% | 100% |
| unit | 40.6% | 43.8% | 96.9% |
| list | 81.2% | 84.4% | 100% |
| boolean | 87.5% | 78.1% | 100% |
| lookup | 65.6% | 65.6% | 100% |
Numbers from report table (reports/qwen_complete_program_trace_reranker_report.md, Domain Breakdown)
Technical framing
Per-domain accuracy on length-6 prompts — Modest verifier gains in four domains, no change on lookup, and a 9.4-point loss on boolean, where answers collapse to few values.
How to read
Grouped bars per test set: the plain first pick, a shortcut that scores candidates by simulating them run, and the trained picker. Higher is better.
Takeaway → The simulation shortcut's bar sits below the plain first pick on every test set except the hardest reworded one, so scoring by simulation alone underperforms doing nothing.
Data table
| Evaluation split | Frozen compiler (base) | Soft-trace heuristic | Learned verifier |
|---|---|---|---|
| Val mixed L6 | 59.4% | 43% | 60.2% |
| Fresh std L6 | 68.8% | 46.1% | 70.3% |
| Fresh para L6 | 57.8% | 35.2% | 60.2% |
| Hard std L8 | 50% | 32% | 46.9% |
| Hard para L8 | 38.3% | 27.3% | 35.9% |
| Harder std L10 | 40.6% | 23.4% | 35.2% |
| Harder para L10 | 17.2% | 23.4% | 14.1% |
Technical framing
Differentiable soft-executor scorer trails the frozen compiler on every split — The differentiable soft-executor scoring heuristic loses to just keeping the compiler argmax on all splits except the hardest paraphrase one.
How to read
Lines track accuracy as problems grow from six to ten steps, split into plainly worded and reworded versions, for both the plain system and the trained picker. Higher is better; reworded lines sit lower.
Takeaway → Reworded lines fall well below plainly worded ones at every length, and the picker fails to close that reworded gap on the longer problems, dragging accuracy lower on both.
Data table
| Program length (VM steps) | Base, standard | Base, paraphrase | Learned, standard | Learned, paraphrase |
|---|---|---|---|---|
| 6 | 68.8% | 57.8% | 70.3% | 60.2% |
| 8 | 50% | 38.3% | 46.9% | 35.9% |
| 10 | 40.6% | 17.2% | 35.2% | 14.1% |
Technical framing
Paraphrased prompts widen the accuracy drop as programs get longer — Paraphrasing costs ~11-23 points at every length, and the learned verifier makes the paraphrase drop worse, not better, at lengths 8-10.
In the author’s words from the Report · “Abstract”
The candidate search contains answer-correct programs for 91.4% to 100.0% of the main evaluation prompts, so the executable candidate set has substantial reachable headroom. The learned verifier captures only small in-distribution gains and loses accuracy on paired and longer-chain splits. The bottleneck is therefore candidate selection and credit assignment, not candidate availability.
Overview
This standalone experiment tests whether a learned verifier can better select hidden-VM candidate traces when it receives prompt-conditioned Qwen hidden-state context in addition to candidate execution features.
Large checkpoints are stored outside this directory:
/workspace/large_artifacts/qwen_complete_program_trace_reranker/checkpoints
Local structure:
src/qwen_complete_program_trace_reranker_experiment.py: run generator, training, and evaluationsrc/analyze_qwen_complete_program_trace_reranker.py: regenerate aggregate CSVs, figures, reports, and checkpoint manifestruns/: per-run logs and metricsanalysis/: aggregate metrics and figuresreports/: standalone Markdown and HTML reportcheckpoint_manifest.csv: checkpoint and artifact index
Regenerate analysis artifacts from completed runs:
python experiments/qwen_complete_program_trace_reranker/src/analyze_qwen_complete_program_trace_reranker.pyReport
Rendered from reports/qwen_complete_program_trace_reranker_report.md
Abstract
This experiment tests a complete-program posttraining route for improving a frozen local Qwen compiler. A Qwen3-4B hidden-state adapter emits a compact virtual-machine program. Around that program, the system enumerates local executable edits and trains a small context-conditioned verifier to select the best candidate using only prompt hidden-state context, candidate features, and execution traces. The verifier never sees the target answer at inference time.
The result is diagnostic rather than successful. The candidate search contains answer-correct programs for 91.4% to 100.0% of the main evaluation prompts, so the executable candidate set has substantial reachable headroom. The learned verifier captures only small in-distribution gains and loses accuracy on paired and longer-chain splits. The bottleneck is therefore candidate selection and credit assignment, not candidate availability.
Question
Can a small posttraining module make a frozen Qwen-attached compiler select better complete programs by inspecting executable traces, without forcing the model to generate every intermediate reasoning step as text?
Method
- Base model:
Qwen/Qwen3-4B. - Value space: arithmetic over modulus
97with up to10VM steps. - Frozen compiler: Qwen hidden states feed a trained hidden-VM compiler checkpoint.
- Candidate generator: local edits around the compiler argmax program, with top-k alternatives and up to
2edits. - Verifier input: candidate execution trace, scalar candidate features, and compact prompt hidden-state summaries.
- Main verifier:
3trace-transformer layers, width128,4heads. - Main training set:
384prompts, length range1to6, positive labelanswer. - Main checkpoint selection: best validation learned accuracy; selected epoch
14.
The key baselines are:
base: the frozen compiler argmax program.soft trace: a differentiable executor scoring heuristic.learned: the trained verifier top-1 selection.pair rerank: pair-level consistency reranking for paraphrase pairs.oracle: any answer-correct candidate in the generated local neighborhood.
Main Results
| split | base | soft trace | learned | pair rerank | oracle | learned gap recovered | avg candidates | answer positives |
|---|---|---|---|---|---|---|---|---|
| validation mixed L6 | 59.4% | 43.0% | 60.2% | n/a | 98.4% | 2.0% | 111.0 | 30.4 |
| fresh standard L6 | 68.8% | 46.1% | 70.3% | n/a | 98.4% | 5.3% | 111.0 | 30.3 |
| fresh paraphrase L6 | 57.8% | 35.2% | 60.2% | n/a | 100.0% | 5.6% | 111.0 | 27.9 |
| fresh paired L6 | 57.3% | 43.2% | 55.2% | 54.7% | 98.4% | -5.1% | 111.0 | 30.2 |
| hard standard L8 | 50.0% | 32.0% | 46.9% | n/a | 98.4% | -6.5% | 179.0 | 44.7 |
| hard paraphrase L8 | 38.3% | 27.3% | 35.9% | n/a | 96.9% | -4.0% | 179.0 | 43.2 |
| harder standard L10 | 40.6% | 23.4% | 35.2% | n/a | 98.4% | -9.5% | 263.0 | 47.6 |
| harder paraphrase L10 | 17.2% | 23.4% | 14.1% | n/a | 91.4% | -4.2% | 263.0 | 36.2 |
Fresh length-6 standard prompts improved from 68.8% to 70.3%, a 1.6% absolute gain. Fresh length-6 paraphrase prompts improved from 57.8% to 60.2%. These gains are real but small relative to the oracle.
The same selector did not extrapolate. Fresh paired length-6 accuracy moved from 57.3% to 55.2%, a -2.1% absolute change. Harder standard length-10 moved from 40.6% to 35.2%, a -5.5% absolute change. The oracle stayed high on these splits, so the selector failed to locate available correct candidates.
Candidate Geometry
The candidate generator is broad. On length-6 fresh splits it produces about 111 candidates per prompt, with roughly 28 to 30 answer-correct candidates. On length-10 splits it produces 263 candidates per prompt, with many answer-correct candidates but far fewer state-exact or program-exact candidates. This explains why answer-only labels are easy to satisfy but weakly identify the best computational trace.
Objective Pilots
Several label objectives were tested before the main run:
- Answer-correct labels learned a nontrivial selector on validation but were noisy because many candidates share the final answer.
- State-exact labels were too conservative and mostly preserved the base program.
- Single-oracle labels were sharper but still mostly preserved the base program.
- Repair-focused weighting did not overcome the base-preservation tendency.
The main run eventually recovered the best validation learned accuracy at epoch 14, but validation gains were small while train accuracy rose strongly. That is the signature of a selector that can fit candidate artifacts without learning a robust preference rule for unseen prompts.
Domain Breakdown
| domain | base | learned | oracle | learned delta |
|---|---|---|---|---|
| arithmetic | 31.2% | 34.4% | 96.9% | 3.1% |
| calendar | 37.5% | 40.6% | 100.0% | 3.1% |
| unit | 40.6% | 43.8% | 96.9% | 3.1% |
| list | 81.2% | 84.4% | 100.0% | 3.1% |
| boolean | 87.5% | 78.1% | 100.0% | -9.4% |
| lookup | 65.6% | 65.6% | 100.0% | 0.0% |
The verifier helped arithmetic, calendar, unit, and list prompts modestly, left lookup unchanged, and hurt boolean prompts. Boolean has many answer-correct candidates but relatively little trace-identifying signal because final answers collapse to few values.
Interpretation
This experiment answers one useful subquestion: local executable candidate search is not the limiting factor. The oracle remains high even when the base compiler is weak. The limiting factor is how to train a verifier that identifies the right complete program from a dense equivalence class of answer-correct candidates.
The current learned verifier is too passive. It sees traces and context, but its supervision says many different programs are equally good whenever they hit the final answer. Sharper labels alone did not fix this because the base program dominates many training groups and the correct non-base candidates are sparse. The result argues against scaling this exact reranker unchanged.
Most Impactful Next Options
Train the selector from executable teacher traces, not just final answers. Use the oracle candidate to produce dense supervision over every intermediate state and operation, then train the verifier or compiler with a margin that explicitly ranks state-consistent candidates above answer-only candidates. This directly targets the observed dense-label failure.
Convert reranking into preference learning on repairable failures. Filter training to groups where the base program is wrong and at least one non-base candidate is right, then train pairwise preferences with hard negatives that share the answer but diverge in state. This removes the base-preservation shortcut.
Let Qwen read serialized candidate traces. Instead of only a small verifier, serialize a small shortlist of candidate programs and traces back into Qwen hidden states or text tokens, then train a LoRA selector head. This tests whether the frozen model already has the semantic machinery needed to choose among executable candidates when the candidates are made legible.
- Train a differentiable interpreter objective upstream. Backpropagate through the soft executor into the adapter with auxiliary losses on intermediate states, then use the discrete candidate oracle only for evaluation. This attacks the compiler's crystallized program quality rather than relying on post-hoc selection.
The first option is the cleanest next experiment because it directly matches the failure mode found here: answer-correct candidate availability is high, but answer-only selection is underdetermined.
Artifacts
- Aggregate metrics:
experiments/qwen_complete_program_trace_reranker/analysis/all_final_metrics.csv - Main metrics:
experiments/qwen_complete_program_trace_reranker/analysis/final_metrics.csv - Training logs:
experiments/qwen_complete_program_trace_reranker/analysis/verifier_train_logs.csv - Checkpoint manifest:
experiments/qwen_complete_program_trace_reranker/checkpoint_manifest.csv - Large checkpoints:
large_artifacts/qwen_complete_program_trace_reranker/checkpoints
Experiment log 1
Show the running log (1 entry, 2026-06-23)
2026-06-23
Objective: train and evaluate a context-conditioned learned verifier for local candidate traces emitted by a frozen Qwen-attached hidden VM compiler. The verifier uses candidate execution traces, compiler-local features, and compact Qwen hidden-state summaries from the prompt. Reports must be standalone.
Initial setup:
- Created standalone experiment directory.
- Copied the frozen mixed-domain trace compiler into this experiment's own large-artifact area:
/workspace/large_artifacts/qwen_complete_program_trace_reranker/checkpoints/fixed_mixed_vm_trace_compiler_s512. - Seeded local source from the mixed-domain verifier implementation and prepared to add context features, hard-negative contrastive loss, and optional paired training.
- Patched the context-conditioned reranker path so context normalization statistics are returned, saved, and used during evaluation.
- Switched the primary positive label from full trajectory exactness to final answer correctness, keeping state-exact and program-exact metrics as diagnostics. This aligns the experiment with complete-program candidate selection from executable answer labels.
- Ran
smoke_complete_program_trace_reranker, a tiny Qwen-backed end-to-end smoke test. It loaded the fixedQwen/Qwen3-4Bhidden-VM compiler checkpoint, generated candidate groups, trained the reranker for two epochs, wrote run artifacts, and saved a small verifier checkpoint. Fresh paired length-3 base, learned, and pair-rerank accuracy were all 66.7%; oracle was 100.0%. This validates the pipeline and confirms there is answer-correct candidate headroom even at smoke scale. - Ran
pilot_complete_program_trace_reranker_s96with answer-correct positives, length-6 fresh splits, length-8/10 hard splits, top-k 3, and two-edit candidate neighborhoods. Validation improved from 60.4% base to 64.6% learned, and hard standard improved from 54.2% to 56.2%, but fresh paired fell from 65.6% base to 57.8% learned; pair reranking recovered to 64.1%. Interpretation: answer-correct labels are dense, averaging 29.4 positives per 111 candidates on fresh paired, and are not sharp enough by themselves for robust top-1 selection. - Ran matched state-exact and oracle-selector pilots. State-exact training and single-oracle training mostly selected the base program everywhere, matching base accuracy on fresh and hard splits. Interpretation: sharper labels alone are insufficient when base-correct groups dominate the training signal.
- Ran
pilot_complete_program_trace_reranker_s96_repairfocus, which used single-oracle positives with base-positive groups downweighted and repairable groups upweighted. This also mostly selected the base program. Interpretation: weighting alone did not expose a strong repair preference under the current feature and trace representation. - Ran
main_complete_program_trace_reranker_s384_answer, the main answer-label run with 384 training prompts, 128 validation prompts, 128 fresh/hard evaluation prompts, 96 paraphrase pairs, length-6 in-distribution evaluation, and length-8/10 extrapolation. Best validation epoch was 14. The learned selector improved validation from 59.4% to 60.2%, fresh standard length-6 from 68.8% to 70.3%, and fresh paraphrase length-6 from 57.8% to 60.2%. It degraded fresh paired length-6 from 57.3% to 55.2%, hard standard length-8 from 50.0% to 46.9%, hard paraphrase length-8 from 38.3% to 35.9%, harder standard length-10 from 40.6% to 35.2%, and harder paraphrase length-10 from 17.2% to 14.1%. Oracle answer-correct candidate availability stayed high, ranging from 91.4% to 100.0% across the main evaluation splits. - Generated aggregate CSVs, six figures, a standalone Markdown report, and a standalone HTML report with
src/analyze_qwen_complete_program_trace_reranker.py. The core conclusion is diagnostic: local executable candidate search has enough headroom, but answer-only verifier training is underdetermined and does not learn robust program selection for paired or longer-chain prompts.
Figures 6
Data files 14
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv64 kBanalysis/final_metrics.csv20 kBruns/main_complete_program_trace_reranker_s384_answer/metrics.csv21 kBruns/main_complete_program_trace_reranker_s384_answer/results.json151 kBruns/pilot_complete_program_trace_reranker_s96/metrics.csv14 kBruns/pilot_complete_program_trace_reranker_s96/results.json96 kBruns/pilot_complete_program_trace_reranker_s96_oracle/metrics.csv14 kBruns/pilot_complete_program_trace_reranker_s96_oracle/results.json96 kBruns/pilot_complete_program_trace_reranker_s96_repairfocus/metrics.csv14 kBruns/pilot_complete_program_trace_reranker_s96_repairfocus/results.json96 kBruns/pilot_complete_program_trace_reranker_s96_state/metrics.csv14 kBruns/pilot_complete_program_trace_reranker_s96_state/results.json96 kBruns/smoke_complete_program_trace_reranker/metrics.csv11 kBruns/smoke_complete_program_trace_reranker/results.json52 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).