Qwen Mixed-Domain Trace Verifier
The one idea you need
A frozen model answers each task by writing a hidden step-by-step recipe. A small proofreader that never sees the answer key rereads several near-copies of that recipe, each with one step tweaked, and bets on the version it trusts most.
The question
Can a small second model, without ever seeing the correct answer, rerank a frozen model's step-by-step solutions and reliably pick the better ones?
What we found
Yes, partly. On fresh tasks the frozen model alone got 46% right; the proofreader lifted that to 57%, and a cross-check that compares reworded versions of the same task reached 61%. But a correct recipe was already among the candidates 92% of the time, so the proofreader captured only about a quarter of the reachable gain, leaving most of it untouched.
Why it matters
When you can cheaply generate variant solutions, a lightweight reranker reading only prompt and execution clues buys real accuracy with no retraining of your main model. But the ceiling is the reranker's judgment, not the size of the candidate pool.
On this page
Results at a glance 7
How to read
Each bar is how often a way of choosing a recipe gets the task right on fresh prompts: the frozen model's own pick, the learned proofreader, the proofreader plus cross-check, and a perfect picker. Taller is better.
Takeaway → The proofreader and cross-check bars rise well above the model's own pick, but the perfect-picker bar towers over all, showing most of the room stays unused.
Data table
| selector | Len6-8 verifier arm |
|---|---|
| base compiler | 46.1% |
| learned verifier | 57.4% |
| pair rerank | 61.3% |
| oracle selector | 92.2% |
Numbers from report table (reports/qwen_mixed_domain_trace_verifier_paper.md, Length-6-to-8 Verifier)
Technical framing
Verifier reranking lifts frozen compiler accuracy on fresh paired length-6 prompts — The learned verifier adds +11.3 pp and pair reranking +15.2 pp over the frozen compiler, but the oracle shows most headroom remains.
How to read
Three bars per task type — the frozen model alone, the learned proofreader, and a perfect picker — with task types ordered easy to hard. Higher is better; the perfect-picker bar marks each ceiling.
Takeaway → The proofreader bar sits above the frozen-model bar in every group, including longer tasks it never trained on, yet stays far under the ceiling.
Data table
| evaluation split | base compiler | learned verifier | oracle selector |
|---|---|---|---|
| fresh standard len6 | 60.4% | 67.7% | 95.8% |
| fresh paraphrase len6 | 47.9% | 53.1% | 92.2% |
| fresh paired len6 | 46.1% | 57.4% | 92.2% |
| hard standard len8 | 39.6% | 41.1% | 83.9% |
| hard paraphrase len8 | 22.9% | 30.7% | 69.3% |
| harder standard len10 | 18.8% | 24.5% | 58.9% |
Numbers from report table (reports/qwen_mixed_domain_trace_verifier_paper.md, Length-6-to-8 Verifier)
Technical framing
Length-6-to-8 verifier gains generalize across held-out splits — The verifier improves every split, including length-10 prompts beyond its training lengths, yet stays far below the oracle candidate ceiling.
How to read
Each bar shows how often a task and its reworded twin are both solved at once: the frozen model, the cross-check proofreader, and the best-possible ceiling. Taller is better.
Takeaway → The cross-check bar rises clearly above the frozen model, closing part of the gap to the ceiling for getting both wordings right together.
Data table
| selector | both prompts correct |
|---|---|
| base compiler | 37.5% |
| pair rerank | 55.5% |
| oracle pair ceiling | 86.7% |
Numbers from report table (reports/qwen_mixed_domain_trace_verifier_paper.md, Paired Prompt Behavior)
Technical framing
Pair reranking makes paired standard+paraphrase prompts jointly correct more often — Agreement-based reranking is the cleanest inference-time gain: joint correctness rises from 37.5% to 55.5% on the Len6-8 arm.
How to read
Two bars per task type showing how much of the gap to a perfect picker each proofreader closes: one trained only on short recipes, one trained on mixed-length recipes. Higher means more room recovered.
Takeaway → The mixed-length proofreader's bars are taller on the harder task types, but even the best bars recover only a minority of the room.
Data table
| evaluation split | Len6 verifier | Len6-8 verifier |
|---|---|---|
| fresh standard len6 | 16.7% | 20.6% |
| fresh paraphrase len6 | 14.4% | 11.8% |
| fresh paired len6 | 12.6% | 24.6% |
| hard paraphrase len8 | 4.5% | 16.9% |
| harder standard len10 | 2.8% | 14.3% |
Numbers from report table (reports/qwen_mixed_domain_trace_verifier_paper.md, Main Results)
Technical framing
Share of the oracle selection gap the learned verifier recovers — Training on longer length-6-to-8 traces recovers more of the oracle gap on the hardest splits, but most of the selection problem is unsolved.
How to read
Bars for five ways to pick a recipe — no edit, a most-likely pick, a support-based pick, the learned proofreader, and a perfect picker — shown for two task types. Taller is better.
Takeaway → The most-likely pick just ties the frozen model and the support-based pick drops below it; only the learned proofreader rises above, though well short of the ceiling.
Data table
| Candidate selector | Fresh paired length-6 | Hard paraphrase length-8 |
|---|---|---|
| Base (no edit) | 46.1% | 22.9% |
| Compiler prior | 46.1% | 22.9% |
| Soft-trace support | 25.8% | 7.3% |
| Learned verifier | 57.4% | 30.7% |
| Oracle | 92.2% | 69.3% |
Numbers from experiments/qwen_mixed_domain_trace_verifier/analysis/main_final_metrics.csv
Technical framing
Selector ablation: only the learned verifier beats the base compiler — Compiler-prior selection ties the base and soft-trace support actively hurts; only the learned scorer converts candidates into gains.
How to read
Wiggly lines track each proofreader's accuracy on a practice set across training rounds, with a flat line for the untrained frozen model. Higher is better; the peak is the round worth keeping.
Takeaway → The learned lines bounce up and down and can dip near the flat baseline, so the single best round must be chosen rather than just training longer.
Data table
| Training epoch | Len-6 verifier (learned) | Len-6 verifier base (frozen) | Len-6-to-8 verifier (learned) | Len-6-to-8 verifier base (frozen) |
|---|---|---|---|---|
| 1 | 52.3% | 52.3% | 27.3% | 28.1% |
| 2 | 53.9% | — | 28.1% | — |
| 3 | 60.9% | — | 31.2% | — |
| 4 | 58.6% | — | 28.9% | — |
| 5 | 50.8% | — | 33.6% | — |
| 6 | 59.4% | — | 32% | — |
| 7 | 49.2% | — | 32% | — |
| 8 | 54.7% | — | 29.7% | — |
| 9 | 57% | — | 32% | — |
| 10 | 57.8% | — | 31.2% | — |
| 11 | 60.9% | — | 32% | — |
| 12 | 60.9% | — | 35.9% | — |
| 13 | 57.8% | — | 34.4% | — |
| 14 | 57.8% | — | 31.2% | — |
| 15 | 60.2% | — | 33.6% | — |
| 16 | 64.8% | — | 32.8% | — |
| 17 | 58.6% | — | 35.9% | — |
| 18 | 53.1% | 52.3% | 34.4% | 28.1% |
Numbers from experiments/qwen_mixed_domain_trace_verifier/analysis/main_verifier_train_logs.csv
Technical framing
Verifier validation accuracy over training epochs is noisy, so checkpoint selection matters — Validation gains fluctuate and can dip below base; the best checkpoints (epochs 16 and 12) must be picked by validation, not by training longer.
How to read
Two bars per task type: a proofreader trained only on short recipes versus one trained on mixed lengths, across easy-to-hard task types. Taller is better.
Takeaway → The mixed-length bar is taller on five of seven task types, notably the hard reworded one, at a small cost on the other two.
Data table
| Held-out evaluation split | Len-6-only verifier | Len-6-to-8 verifier |
|---|---|---|
| Fresh standard len-6 | 57.8% | 67.7% |
| Fresh paraphrase len-6 | 50% | 53.1% |
| Fresh paired len-6 | 53.1% | 57.4% |
| Hard standard len-8 | 44.8% | 41.1% |
| Hard paraphrase len-8 | 20.3% | 30.7% |
| Harder standard len-10 | 21.4% | 24.5% |
| Harder paraphrase len-10 | 8.9% | 5.7% |
Numbers from report table (experiments/qwen_mixed_domain_trace_verifier/reports/qwen_mixed_domain_trace_verifier_paper.md)
Technical framing
Training on mixed lengths 6-8 beats length-6-only training on most held-out splits — The mixed-length curriculum wins on 5 of 7 splits, including +10.4 pp on hard paraphrase len-8, at a small cost on two splits.
In the author’s words from the Report · “Abstract”
On fresh paired length-6 prompts, the base compiler achieved 46.1%, the learned verifier achieved 57.4%, and paired reranking achieved 61.3%. That is a learned gain of +11.3 pp and a paired-rerank gain of +15.2 pp. The same arm also improved hard paraphrase length-8 from 22.9% to 30.7% (+7.8 pp) and harder standard length-10 from 18.8% to 24.5% (+5.7 pp). The oracle selector remains far higher than the learned verifier. This means the candidate set often contains a correct executable trace, but the learned scoring function recovers only a minority of the available headroom.
Overview
This standalone experiment tests whether a learned candidate-trace verifier can select better hidden-VM programs from a frozen Qwen-attached mixed-domain compiler. The compiler proposes an executable VM trace; the verifier reranks local candidate traces using only prompt/compiler/execution-derived features at test time.
Large checkpoints are stored outside this directory:
/workspace/large_artifacts/qwen_mixed_domain_trace_verifier/checkpoints
Expected local structure:
src/: experiment and analysis coderuns/: per-run metrics and logsanalysis/: aggregate CSVs, summaries, and figuresreports/: standalone Markdown and HTML reportscheckpoint_manifest.csv: checkpoint/artifact index
Report
Rendered from reports/qwen_mixed_domain_trace_verifier_paper.md
Abstract
This experiment tests whether a small learned verifier can improve a frozen Qwen-attached hidden-VM compiler by selecting among local executable candidate traces. The model under test emits a hidden program consisting of an initial value plus a sequence of VM operations and arguments. The verifier never sees gold answers or gold states at test time; it scores candidate traces using compiler priors, local edit metadata, soft-executor support, and the candidate's own executed trajectory.
The strongest arm trained the verifier on length-6-to-8 traces and selected the checkpoint on length-8 validation. On fresh paired length-6 prompts, the base compiler achieved 46.1%, the learned verifier achieved 57.4%, and paired reranking achieved 61.3%. That is a learned gain of +11.3 pp and a paired-rerank gain of +15.2 pp. The same arm also improved hard paraphrase length-8 from 22.9% to 30.7% (+7.8 pp) and harder standard length-10 from 18.8% to 24.5% (+5.7 pp).
The oracle selector remains far higher than the learned verifier. This means the candidate set often contains a correct executable trace, but the learned scoring function recovers only a minority of the available headroom.
Setup
- Backbone/compiler input: a frozen Qwen-attached hidden-VM compiler checkpoint localized under this experiment's large-artifact directory.
- VM value modulus: 97.
- VM operations: ADD, SUB, MUL, ADD7, SUB7, SET, MAX, MIN, XOR, GT.
- Domains: arithmetic, calendar, unit conversion, list aggregation, boolean thresholding, and table lookup.
- Candidate neighborhood: top-3 compiler values for init/op/arg slots, one-edit candidates, same-step op+arg two-edit candidates, and two-argument edits over up to the first 10 slots.
- Candidate labels for verifier training: exact answer and exact state trajectory match. These labels are offline supervision only.
Test-time selectors:
base: unedited compiler argmax trace.prior: highest compiler-prior candidate.soft_trace: highest soft-executor state-support candidate.learned: learned trace-verifier argmax.pair_rerank: learned verifier plus agreement bonus across paired standard
and paraphrased prompts.
oracle: highest-prior candidate with exact gold state trajectory.
Main Results
Length-6 Verifier
| split | base | learned | pair-rerank | oracle | learned delta | oracle gap recovered |
|---|---|---|---|---|---|---|
fresh_standard_len6 | 50.5% | 57.8% | n/a | 94.3% | +7.3 pp | 16.7% |
fresh_paraphrase_len6 | 43.2% | 50.0% | n/a | 90.1% | +6.8 pp | 14.4% |
fresh_paired_len6 | 47.3% | 53.1% | 54.7% | 93.8% | +5.9 pp | 12.6% |
hard_standard_len8 | 41.7% | 44.8% | n/a | 79.2% | +3.1 pp | 8.3% |
hard_paraphrase_len8 | 18.2% | 20.3% | n/a | 64.1% | +2.1 pp | 4.5% |
harder_standard_len10 | 20.3% | 21.4% | n/a | 57.8% | +1.0 pp | 2.8% |
harder_paraphrase_len10 | 7.3% | 8.9% | n/a | 30.2% | +1.6 pp | 6.8% |
Length-6-to-8 Verifier
| split | base | learned | pair-rerank | oracle | learned delta | oracle gap recovered |
|---|---|---|---|---|---|---|
fresh_standard_len6 | 60.4% | 67.7% | n/a | 95.8% | +7.3 pp | 20.6% |
fresh_paraphrase_len6 | 47.9% | 53.1% | n/a | 92.2% | +5.2 pp | 11.8% |
fresh_paired_len6 | 46.1% | 57.4% | 61.3% | 92.2% | +11.3 pp | 24.6% |
hard_standard_len8 | 39.6% | 41.1% | n/a | 83.9% | +1.6 pp | 3.5% |
hard_paraphrase_len8 | 22.9% | 30.7% | n/a | 69.3% | +7.8 pp | 16.9% |
harder_standard_len10 | 18.8% | 24.5% | n/a | 58.9% | +5.7 pp | 14.3% |
harder_paraphrase_len10 | 3.6% | 5.7% | n/a | 31.8% | +2.1 pp | 7.4% |
Oracle Gap
The learned verifier is real but not yet close to the oracle. The length-6-to-8 arm recovers about 24.6% of the fresh paired length-6 oracle gap and 16.9% of the hard paraphrase length-8 oracle gap. This is enough to be useful, but it also shows that most of the selection problem remains unsolved.
Paired Prompt Behavior
Paired reranking is the cleanest inference-time gain. For the length-6-to-8 arm, paired prompts both correct improved from 37.5% under the base compiler to 55.5% with pair reranking, while the oracle pair ceiling was 86.7%.
Domain Breakdown
The strongest domain gains in the length-6-to-8 arm were list aggregation, boolean thresholding, lookup, and unit conversion. Arithmetic and calendar remain harder for the verifier despite large oracle headroom, suggesting that the trace features are not yet identifying the right local arithmetic repairs.
Training Dynamics
| verifier | best epoch | base val | learned val | oracle val |
|---|---|---|---|---|
| Len6-8 verifier | 12 | 28.1% | 35.9% | 73.4% |
| Len6 verifier | 16 | 52.3% | 64.8% | 91.4% |
The verifier has a narrow useful checkpoint window. Later epochs can over-edit, so validation-based checkpoint selection is required.
Interpretation
The experiment supports the narrow claim that hidden executable traces are repairable by a small learned selection layer. The best arm gives a meaningful inference-time lift without changing the frozen Qwen compiler. It does not support a broad claim of large universal intelligence gain: the learned verifier recovers only part of a much larger oracle candidate-selection ceiling, and the hardest paraphrase length-10 split remains very weak.
The next technical bottleneck is verifier quality, not candidate availability. Useful next steps are stronger trace encoders, explicit contrastive training on base-wrong/repairable groups, and candidate generation that proposes structured multi-slot fixes without relying on a small local edit budget.
Experiment log 1
Show the running log (1 entry, 2026-06-23)
2026-06-23
Objective: train and evaluate a learned verifier/reranker for local candidate traces emitted by a frozen Qwen-attached hidden VM compiler on mixed deterministic domains. The report must be standalone and must not depend on earlier experiment narrative.
Initial setup:
- Created standalone experiment directory.
- Heavy artifacts will live under
/workspace/large_artifacts/qwen_mixed_domain_trace_verifier/checkpoints. - Planned primary run: freeze the existing Qwen-attached mixed-domain compiler, enumerate local candidates, train a small trace verifier, evaluate base/prior, soft-executor support, learned verifier, paired reranker, and oracle ceiling.
- Copied the frozen mixed-domain trace compiler into this experiment's own large-artifact area:
/workspace/large_artifacts/qwen_mixed_domain_trace_verifier/checkpoints/fixed_mixed_vm_trace_compiler_s512. Added local standalone source:
src/mixed_vm_core.pycontains the mixed-domain VM generator, compiler, and
executor code used by this experiment.
src/qwen_mixed_domain_trace_verifier_experiment.pyloads the frozen
compiler, builds local candidate neighborhoods, trains a candidate-trace verifier, evaluates base/prior/soft-trace/learned/pair-rerank/oracle selectors, and writes run metrics plus checkpoint manifests.
Smoke run:
- Command:
python src/qwen_mixed_domain_trace_verifier_experiment.py --run_name smoke_mixed_domain_trace_verifier --train_examples 12 --val_examples 6 --eval_examples 6 --eval_pairs 4 --domain_eval_examples 0 --verifier_epochs 1 --qwen_batch_size 4 --repair_topk 2 --repair_max_edits 1 --repair_max_pair_arg_slots 4 --trace_d_model 64 --trace_layers 1 --trace_heads 4 - Result: completed end to end. The run wrote
runs/smoke_mixed_domain_trace_verifier/metrics.csv,runs/smoke_mixed_domain_trace_verifier/results.json, and a verifier checkpoint underlarge_artifacts/qwen_mixed_domain_trace_verifier/checkpoints/smoke_mixed_domain_trace_verifier/. - Takeaway: loader, local candidate construction, feature tensor dimensions, verifier training, pair reranking path, and artifact writing are functional.
Pilot 1:
- Command:
python src/qwen_mixed_domain_trace_verifier_experiment.py --run_name pilot_mixed_domain_trace_verifier_s96 --train_examples 96 --val_examples 32 --eval_examples 48 --eval_pairs 32 --domain_eval_examples 12 --verifier_epochs 6 --qwen_batch_size 8 --repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 10 --trace_d_model 96 --trace_layers 2 --trace_heads 4 - Result: oracle headroom was large but the learned verifier selected the base candidate on every split (
learned_changed_fraction=0.0on final metrics). - Diagnosis: verifier training was dominated by base-correct candidate groups; the model learned preservation but not repair selection.
- Iteration: added group-level loss weights so base-correct groups retain a preservation objective, while wrong-but-repairable groups carry more gradient.
Pilot 2:
- Command:
python src/qwen_mixed_domain_trace_verifier_experiment.py --run_name pilot_weighted_len6_trace_verifier_s128 --train_examples 128 --val_examples 48 --eval_examples 64 --eval_pairs 40 --domain_eval_examples 12 --train_min_len 6 --train_max_len 6 --verifier_epochs 12 --qwen_batch_size 8 --repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 10 --trace_d_model 128 --trace_layers 2 --trace_heads 4 --base_positive_group_weight 0.25 --repairable_group_weight 8.0 --no_positive_group_weight 1.0 - Result: learned selection improved length-6 fresh paired from 40.0% to 43.8%; pair reranking improved it to 46.25%. Hard length-8/10 did not improve.
- Takeaway: weighted repair training can move off the base candidate, but a length-6-only verifier does not learn enough about longer traces.
Pilot 3:
- Command:
python src/qwen_mixed_domain_trace_verifier_experiment.py --run_name pilot_weighted_len6_8_val8_trace_verifier_s160 --train_examples 160 --val_examples 48 --eval_examples 64 --eval_pairs 40 --domain_eval_examples 12 --train_min_len 6 --train_max_len 8 --val_length 8 --verifier_epochs 12 --qwen_batch_size 8 --repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 10 --trace_d_model 128 --trace_layers 2 --trace_heads 4 --base_positive_group_weight 0.25 --repairable_group_weight 8.0 --no_positive_group_weight 1.0 - Result: length-8 validation improved from 33.3% to 41.7%, hard standard length 8 improved from 40.6% to 43.8%, hard paraphrase length 8 from 25.0% to 32.8%, and harder standard length 10 from 14.1% to 17.2%. Easy length-6 standard degraded, showing an over-editing tradeoff.
- Final plan: run two final arms, one optimized for length-6 repair and one optimized for length-8/generalization, then compare both against the oracle candidate ceiling.
Final arm A: length-6 verifier
- Command:
python src/qwen_mixed_domain_trace_verifier_experiment.py --run_name main_len6_weighted_trace_verifier_s512 --train_examples 512 --val_examples 128 --eval_examples 192 --eval_pairs 128 --domain_eval_examples 48 --train_min_len 6 --train_max_len 6 --verifier_epochs 18 --qwen_batch_size 8 --repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 10 --trace_d_model 128 --trace_layers 2 --trace_heads 4 --base_positive_group_weight 0.25 --repairable_group_weight 8.0 --no_positive_group_weight 1.0 - Result: fresh paired length 6 improved from 47.3% base to 53.1% learned. Hard length 8 and harder length 10 improved slightly. Best validation epoch: 16.
Final arm B: length-6-to-8 verifier selected on length 8
- Command:
python src/qwen_mixed_domain_trace_verifier_experiment.py --run_name main_len6_8_val8_weighted_trace_verifier_s512 --train_examples 512 --val_examples 128 --eval_examples 192 --eval_pairs 128 --domain_eval_examples 48 --train_min_len 6 --train_max_len 8 --val_length 8 --verifier_epochs 18 --qwen_batch_size 8 --repair_topk 3 --repair_max_edits 2 --repair_max_pair_arg_slots 10 --trace_d_model 128 --trace_layers 2 --trace_heads 4 --base_positive_group_weight 0.25 --repairable_group_weight 8.0 --no_positive_group_weight 1.0 - Result: fresh paired length 6 improved from 46.1% base to 57.4% learned. Hard paraphrase length 8 improved from 22.9% to 30.7%, harder standard length 10 from 18.8% to 24.5%, and harder paraphrase length 10 from 3.6% to 5.7%. Best validation epoch: 12/17 tied at 35.9% length-8 validation.
Analysis and report:
- Added
src/analyze_qwen_mixed_domain_trace_verifier.py. Generated aggregate metrics:
analysis/final_metrics.csvanalysis/main_final_metrics.csvanalysis/all_final_metrics.csvanalysis/main_verifier_train_logs.csv
Generated figures:
Generated standalone reports:
reports/qwen_mixed_domain_trace_verifier_paper.mdreports/qwen_mixed_domain_trace_verifier_paper.html
- Rebuilt
checkpoint_manifest.csvwith fixed compiler and verifier checkpoint paths plus byte sizes.
Figures 5
Data files 15
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv69 kBanalysis/final_metrics.csv29 kBanalysis/main_final_metrics.csv29 kBruns/main_len6_8_val8_weighted_trace_verifier_s512/metrics.csv16 kBruns/main_len6_8_val8_weighted_trace_verifier_s512/results.json131 kBruns/main_len6_weighted_trace_verifier_s512/metrics.csv16 kBruns/main_len6_weighted_trace_verifier_s512/results.json130 kBruns/pilot_mixed_domain_trace_verifier_s96/metrics.csv14 kBruns/pilot_mixed_domain_trace_verifier_s96/results.json74 kBruns/pilot_weighted_len6_8_val8_trace_verifier_s160/metrics.csv15 kBruns/pilot_weighted_len6_8_val8_trace_verifier_s160/results.json100 kBruns/pilot_weighted_len6_trace_verifier_s128/metrics.csv15 kBruns/pilot_weighted_len6_trace_verifier_s128/results.json101 kBruns/smoke_mixed_domain_trace_verifier/metrics.csv5.0 kBruns/smoke_mixed_domain_trace_verifier/results.json19 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.