Qwen Candidate-Trace Verifier
The one idea you need
A word problem becomes a 24-step arithmetic recipe, and the machine often fumbles a step. A search spins off about 1,300 near-miss versions, each showing its own worked-out running totals. A checker reads only those totals — never the answer — and picks the version that hangs together best.
The question
When a small model garbles a short arithmetic program, can another model pick the right fix from many candidates without ever seeing the answer?
What we found
Yes. A small checker that reads each candidate's worked-out steps — never the true answer — lifted correctly-running programs from 30% to 54%, and to 56% when it cross-checks two wordings of one task. The twist: a picker handed the answer key hits 88%, so the right fix is usually in the pile; the checker just fails to spot it. Selection, not search, is the wall.
Why it matters
When repairing shaky small-model outputs, spend on the selector, not just on generating more candidates — the correct fix usually already exists. First measure your answer-key ceiling: a big gap means selection, not coverage, is where to invest.
On this page
Results at a glance 3
How to read
Three fresh test sets along the bottom (standard wording, reworded, paired wordings). Each has three bars: no-repair base (lowest), the trace-reading checker's pick (middle), and an answer-key picker (highest). Taller means more programs run exactly right.
Takeaway → The checker's bar roughly doubles the base on every set yet sits far below the answer-key bar — real gains, but most of the reachable headroom stays unclaimed.
Data table
| evaluation split | base (no repair) | trace verifier | oracle ceiling |
|---|---|---|---|
| fresh standard | 28.5% | 50.4% | 90.6% |
| fresh paraphrase | 28.5% | 55.5% | 86.7% |
| fresh paired | 30.3% | 53.7% | 88.1% |
Numbers from analysis/final_metrics.csv (run main_trace_verifier_s512)
Technical framing
Exact execution on fresh length-24 splits: verifier recovers 35-46% of the oracle gap — A learned trace verifier picks repairs without seeing true answers, lifting accuracy from ~29% to 50-56% (35-46% of the oracle gap).
How to read
Four measures along the bottom (runs exactly right, matching step prefix, both wordings correct, agreement between the two wordings), each with four bars: base, single-guess checker, pair-aware reranker, and answer-key ceiling. Higher is better everywhere.
Takeaway → The single-guess checker lifts most bars but drops wording-agreement below base (71% to 58%); reranking the pair jointly restores agreement to 87% and lifts both-correct to 52%.
Data table
| metric | base | trace verifier | pair rerank | oracle |
|---|---|---|---|---|
| exact execution | 30.3% | 53.7% | 56.2% | 88.1% |
| state-prefix match | 58.6% | 76.4% | 78% | 90.8% |
| both of pair correct | 28.1% | 39.8% | 52.3% | 85.9% |
| pair state consistency | 71.1% | 58.2% | 87.1% | 92.6% |
Numbers from analysis/final_metrics.csv (run main_trace_verifier_s512, fresh_paired_len24)
Technical framing
Fresh paired split: pair reranking mainly buys cross-paraphrase consistency — Independent verification hurts pair consistency (71% to 58%); reranking the pair jointly restores it to 87% and lifts both-correct to 52%.
How to read
Two rising lines — checker accuracy on the fresh standard set and the fresh paired set — plotted against the number of training programs (128, then 192, then 512 across the bottom). Up and to the right is better.
Takeaway → Both lines climb steadily, from about 41% at 128 programs to 50-54% at 512, so accuracy was still rising with more training data — not yet plateaued.
Data table
| verifier training examples | fresh standard | fresh paired |
|---|---|---|
| 128 | 42.2% | 40.6% |
| 192 | 43.8% | 44.5% |
| 512 | 50.4% | 53.7% |
Numbers from analysis/all_final_metrics.csv
Technical framing
Verifier accuracy on fresh splits grows with training-set size — Across the s128/s192 pilots and the s512 main run (configs differ slightly), fresh-split accuracy rises steadily with more training programs.
In the author’s words from the Report · “Abstract”
On fresh paired length-24 programs, the base compiler reaches 30.3% exact execution. The candidate-trace verifier reaches 53.7%. A paired consistency reranker reaches 56.2%. The oracle trajectory-verifier ceiling is 88.1%. The trace verifier therefore recovers 40.5% of the base-to-oracle gap on the paired split.
Overview
This experiment trains a transformer verifier over candidate execution traces for local repairs of Qwen-compiled modular-arithmetic programs.
The fixed compiler reads a prompt, copies an initial value plus 24 operation and argument slots, and executes the copied program modulo 97. Candidate repair search enumerates 1,299 local programs for each length-24 prompt. The trace verifier chooses among those candidates without access to the true answer or true state trajectory at test time.
Main Result
Primary run: main_trace_verifier_s512
| Split | Base | Trace verifier | Pair rerank | Oracle ceiling |
|---|---|---|---|---|
| Fresh standard L24 | 28.5% | 50.4% | n/a | 90.6% |
| Fresh paraphrase L24 | 28.5% | 55.5% | n/a | 86.7% |
| Fresh paired L24 | 30.3% | 53.7% | 56.2% | 88.1% |
The trace verifier recovers 40.5% of the base-to-oracle gap on the fresh paired split. Pair reranking improves paired exact execution to 56.2% and paired both-correct accuracy to 52.3%.
Layout
experiments/qwen_candidate_trace_verifier/
src/ experiment and analysis scripts
reports/ standalone paper, HTML report, and experiment log
analysis/ regenerated CSVs, summary, and figures
runs/ per-run metrics and train logs, without large checkpoints
checkpoint_manifest.csv
large_artifacts/qwen_candidate_trace_verifier/checkpoints/
fixed_compiler_step00800/
main_trace_verifier_s512/candidate_trace_verifier.ptReproduction
Run the main experiment:
python experiments/qwen_candidate_trace_verifier/src/qwen_candidate_trace_verifier_experiment.py \
--run_name main_trace_verifier_s512 \
--train_examples 512 \
--val_examples 128 \
--eval_examples 256 \
--eval_pairs 256 \
--repair_topk 3 \
--repair_max_edits 2 \
--verifier_epochs 18 \
--qwen_batch_size 8 \
--trace_d_model 128 \
--trace_layers 3 \
--trace_heads 4Regenerate analysis:
python experiments/qwen_candidate_trace_verifier/src/analyze_qwen_candidate_trace_verifier.pyKey Files
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/executor_accuracy.pngreports/qwen_candidate_trace_verifier_experiment_log.mdreports/qwen_candidate_trace_verifier_paper.md
Report
Rendered from reports/qwen_candidate_trace_verifier_paper.md
Abstract
This experiment tests whether a transformer verifier over candidate execution traces can choose useful local repairs of programs compiled by a Qwen-attached numeric compiler. Each prompt describes 24 modular arithmetic updates. A fixed compiler copies an initial value, operation sequence, and argument sequence from the prompt, and a deterministic runtime executes the copied program modulo 97.
For each prompt, the experiment enumerates 1,299 local repair candidates. Each candidate contains a full executable program and a predicted 24-step state trajectory. Offline labels identify candidates whose full state trajectory matches the true trajectory. At test time, the verifier does not receive the true answer or true states.
The result is positive. On fresh paired length-24 programs, the base compiler reaches 30.3% exact execution. The candidate-trace verifier reaches 53.7%. A paired consistency reranker reaches 56.2%. The oracle trajectory-verifier ceiling is 88.1%. The trace verifier therefore recovers 40.5% of the base-to-oracle gap on the paired split.
Question
Can a learned verifier select the right local repair by reading the candidate program and its execution trace, rather than relying on the true state trajectory at test time?
This is a direct test of whether local repair headroom can be converted into a non-oracle runtime mechanism. The verifier must decide among many plausible nearby programs using only candidate-local signals.
Setup
The fixed compiler reads a modular-arithmetic prompt and predicts:
- the initial value modulo 97;
- one operation per step from add, subtract, and multiply;
- one numeric argument per step.
The candidate search uses the compiler's top local alternatives:
| Candidate class | Budget |
|---|---|
| Base compiled program | 1 |
| Alternate initial values | top-3 |
| Alternate operations | top-3 per active step |
| Alternate arguments | top-3 per active step |
| Same-step operation and argument edits | top-3 by slot |
| Two-argument edits | top-3 by slot, up to 24 slots |
For length 24, this produces 1,299 candidates per prompt.
Verifier Architecture
Each candidate is represented as a 25-token trace:
- one global token containing candidate prior, edit count, initial value, final answer, and final soft-executor support;
- 24 step tokens containing candidate operation, argument, predicted state, base-program operation, base argument, base predicted state, edit flags, compiler log-probabilities, margins, entropies, and state soft support.
The model is a three-layer transformer encoder with width 128 and four attention heads. It also receives aggregate candidate features through a small projection layer. A groupwise ranking loss trains it to score correct trajectory candidates above incorrect candidates:
loss = logsumexp(all candidate scores) - logsumexp(correct candidate scores)When no correct candidate exists in a group, a small auxiliary term selects the base program.
Results
Main Fresh Results
| Split | Base | Trace verifier | Pair rerank | Oracle ceiling |
|---|---|---|---|---|
| Fresh standard L24 | 28.5% | 50.4% | n/a | 90.6% |
| Fresh paraphrase L24 | 28.5% | 55.5% | n/a | 86.7% |
| Fresh paired L24 | 30.3% | 53.7% | 56.2% | 88.1% |
The trace verifier improves every fresh split. On the paired split, it recovers:
(53.7 - 30.3) / (88.1 - 30.3) = 40.5%of the measured base-to-oracle gap.
Fresh Paired Details
| Metric | Base | Trace verifier | Pair rerank | Oracle ceiling |
|---|---|---|---|---|
| Executor accuracy | 30.3% | 53.7% | 56.2% | 88.1% |
| Program exact | 30.3% | 53.7% | 56.2% | 87.7% |
| State prefix fraction | 58.6% | 76.4% | 78.0% | 90.7% |
| Pair both-correct | 28.1% | 39.8% | 52.3% | 85.9% |
| Pair state consistency | 71.1% | 58.2% | 87.1% | 92.6% |
The single-prompt trace verifier improves exact execution but reduces paired state consistency: two prompt renderings may choose different local repairs. A paired reranker that rewards agreement among the top trace-verifier candidates improves executor accuracy to 56.2%, paired both-correct accuracy to 52.3%, and paired state consistency to 87.1%.
Training Dynamics
The main run used 512 training examples and 128 validation examples. The best validation epoch was epoch 15.
| Split | Base | Trace verifier | Oracle |
|---|---|---|---|
| Train L24 | 28.7% | 58.6% | 86.7% |
| Validation L24 | 32.0% | 56.2% | 85.2% |
The fresh held-out paired result, 53.7%, is close to validation, suggesting that the learned signal transfers to new length-24 programs.
Iterations
The smoke run used a tiny top-2/one-edit candidate space to validate loading, trace construction, training, checkpointing, and metric writing.
The first full-neighborhood pilot used a two-layer trace verifier with 128 training examples. It reached 53.1% validation accuracy but only 40.6% fresh paired accuracy.
The second pilot used a deeper three-layer trace verifier and 192 training examples. It reached 54.2% validation accuracy and 44.5% fresh paired accuracy. That configuration was selected for the main run.
The main run scaled the selected configuration to 512 training examples and larger fresh held-out splits. It reached 53.7% fresh paired accuracy and 56.2% with paired reranking.
Interpretation
The experiment supports the central hypothesis: candidate execution traces contain useful non-oracle information for repair selection. A small transformer can read those traces and choose better repairs than the base compiler's top program.
The result also shows that selection remains the bottleneck. The oracle ceiling is 88.1% on fresh paired programs, while the trace verifier reaches 53.7%. The candidate neighborhood contains correct programs far more often than the learned verifier can identify them.
Pair consistency is a powerful additional signal when multiple renderings of the same latent program are available. It substantially improves paired both-correct accuracy and state consistency. It is not, however, a single-prompt method.
Limitations
- The task is synthetic modular arithmetic.
- The compiler and executor are specialized.
- Offline labels use true state trajectories.
- The verifier still uses engineered trace fields, not raw prompt tokens.
- The main result is one fixed compiler checkpoint and one main verifier seed.
- Paired reranking requires two renderings of the same latent program.
Conclusion
A candidate-trace verifier improves fresh paired length-24 exact execution from 30.3% to 53.7%, and paired consistency reranking raises it to 56.2%. This is a substantial conversion of local repair headroom into a non-oracle mechanism, but the 88.1% oracle ceiling shows that much better selection is still possible.
The next step should train the verifier and compiler together: either distill the trace verifier's successful choices back into the compiler, or train a verifier that consumes compact prompt-conditioned representations in addition to the candidate trace.
Artifacts
Small files live in:
experiments/qwen_candidate_trace_verifier/Large checkpoints live in:
large_artifacts/qwen_candidate_trace_verifier/checkpoints/Primary files:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/executor_accuracy.pngruns/main_trace_verifier_s512/metrics.csvruns/main_trace_verifier_s512/verifier_train_log.csvcheckpoint_manifest.csv
Experiment log 9
Show the running log (9 entries)
Objective
Train a non-oracle verifier that reads each candidate program's execution trace and chooses a local repair of a fixed Qwen-compiled modular-arithmetic program.
The verifier is trained from offline trajectory labels, but at test time it sees only candidate-local information: copied slots, edit locations, compiler confidence, candidate states, base states, and differentiable-executor support.
Success Criteria
- Create a standalone experiment directory with source, run metadata, analysis, reports, and a checkpoint manifest.
- Keep large artifacts under
large_artifacts/. - Evaluate base compiler selection, trace-verifier selection, paired consistency reranking, and oracle trajectory selection.
- Run smoke, pilot, and main configurations rather than one-shotting the final experiment.
Runs
Smoke
smoke_trace_verifier
- Used a top-2/one-edit candidate neighborhood, tiny data, and a one-layer trace transformer.
- Verified checkpoint loading, trace construction, training, metric writing, and checkpoint writing.
Pilot: Two-Layer Trace Verifier
pilot_trace_verifier_s128
- Full top-3/two-edit neighborhood.
- 128 verifier-training examples and 64 validation examples.
- Two-layer trace transformer.
- Validation improved from 34.4% base to 53.1%, with an 84.4% oracle ceiling.
- Fresh paired improved from 30.5% base to 40.6%; paired reranking reached 44.5%.
Pilot: Deeper Trace Verifier
pilot_trace_deep_s192
- 192 verifier-training examples and 96 validation examples.
- Three-layer trace transformer.
- Validation improved from 32.3% base to 54.2%, with an 84.4% oracle ceiling.
- Fresh paired improved from 30.5% base to 44.5%.
- This configuration was selected for the main run because it gave the strongest single-prompt trace-verifier result.
Main
main_trace_verifier_s512
- 512 verifier-training examples and 128 validation examples.
- 256 fresh standard examples, 256 fresh paraphrase examples, and 256 paired latent programs rendered twice.
- Top-3/two-edit candidate neighborhood: 1,299 candidates per length-24 example.
- Three-layer trace transformer, model width 128, four attention heads, 18 epochs, selected by validation trace-verifier executor accuracy.
Main fresh results:
| Split | Base | Trace verifier | Pair rerank | Oracle |
|---|---|---|---|---|
| Standard L24 | 28.5% | 50.4% | n/a | 90.6% |
| Paraphrase L24 | 28.5% | 55.5% | n/a | 86.7% |
| Paired L24 | 30.3% | 53.7% | 56.2% | 88.1% |
Fresh paired details:
| Metric | Base | Trace verifier | Pair rerank | Oracle |
|---|---|---|---|---|
| Executor accuracy | 30.3% | 53.7% | 56.2% | 88.1% |
| Program exact | 30.3% | 53.7% | 56.2% | 87.7% |
| State prefix fraction | 58.6% | 76.4% | 78.0% | 90.7% |
| Pair both-correct | 28.1% | 39.8% | 52.3% | 85.9% |
| Pair state consistency | 71.1% | 58.2% | 87.1% | 92.6% |
Interpretation
The trace verifier materially improves non-oracle repair selection. It recovers 40.5% of the base-to-oracle gap on fresh paired length-24 programs.
Single-prompt trace verification increases exact accuracy but lowers paired state consistency, because the two prompt renderings can choose different local repairs. Pair reranking restores much of that consistency and improves paired both-correct accuracy.
The oracle ceiling remains far above the learned verifier, so candidate search is not the limiting factor. Selection quality is still the main bottleneck.
Artifacts
- Small files:
experiments/qwen_candidate_trace_verifier/ - Large checkpoints:
large_artifacts/qwen_candidate_trace_verifier/checkpoints/
Figures 1
Data files 10
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv8.7 kBanalysis/final_metrics.csv3.4 kBruns/main_trace_verifier_s512/metrics.csv3.4 kBruns/main_trace_verifier_s512/results.json38 kBruns/pilot_trace_deep_s192/metrics.csv3.4 kBruns/pilot_trace_deep_s192/results.json30 kBruns/pilot_trace_verifier_s128/metrics.csv3.2 kBruns/pilot_trace_verifier_s128/results.json26 kBruns/smoke_trace_verifier/metrics.csv2.6 kBruns/smoke_trace_verifier/results.json13 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).