Qwen Context-Conditioned Trace Verifier
The one idea you need
A small model turns each question into a tiny step-by-step calculator program, then spawns about a hundred edited variants of it. A separate judge watches each one run and tries to pick the best, but never sees the correct answer, like a talent scout choosing without the scoreboard.
The question
If a model rewrites a question as a tiny program, can a separate judge pick the best version just by watching each one run, without ever seeing the answer?
What we found
Mostly no. A correct program was almost always sitting in the candidate pile, for 91 to 100 percent of questions, yet the judge barely helped: it nudged easy questions from about 69 to 70 percent and actually made the hardest ones worse, dropping them from 41 to 35 percent. The wall is choosing the right program, not finding one.
Why it matters
When dozens of candidate programs land on the same final answer, "is the answer right" is too weak a signal to train a picker. Supervise on the intermediate calculation steps, not just the final number.
On this page
Results at a glance 5
How to read
Bars grouped by difficulty, easier on the left to harder on the right. Height is the share of questions answered correctly, so taller is better. Three bars per group: the model's own first choice, the trained judge, and a best-possible bar showing what perfect selection would reach.
Takeaway → The best-possible bar sits near 98 to 100 percent everywhere while the judge's bar hugs the model's own bar, so correct answers exist but the judge rarely selects them.
Data table
| evaluation split | base (frozen compiler) | learned verifier | oracle (any correct candidate) |
|---|---|---|---|
| fresh standard L6 | 68.8% | 70.3% | 98.4% |
| fresh paraphrase L6 | 57.8% | 60.2% | 100% |
| fresh paired L6 | 57.3% | 55.2% | 98.4% |
| hard standard L8 | 50% | 46.9% | 98.4% |
| harder standard L10 | 40.6% | 35.2% | 98.4% |
Numbers from report table (reports/qwen_context_trace_verifier_report.md), matches analysis/final_metrics.csv
Technical framing
Program selection accuracy: learned verifier barely moves the base compiler while oracle headroom stays near 100% — Answer-correct candidates exist for ~98% of prompts, but the learned verifier gains only 1-2 points in-distribution and loses on harder splits.
How to read
Each bar is one difficulty level. Height shows how much of the gap between the model's own accuracy and best-possible the judge captured; the zero line means no change, above zero is better, below zero means it made things worse.
Takeaway → Bars are only slightly positive on easy questions and drop below zero on paired and longer questions, so the judge helps a little then backfires.
Data table
| evaluation split | gap recovered |
|---|---|
| val mixed L6 | 2% |
| fresh standard L6 | 5.3% |
| fresh paraphrase L6 | 5.6% |
| fresh paired L6 | -5.1% |
| hard standard L8 | -6.5% |
| hard paraphrase L8 | -4% |
| harder standard L10 | -9.5% |
| harder paraphrase L10 | -4.2% |
Technical framing
Fraction of the base-to-oracle gap the verifier recovers, by split — The verifier recovers at most 5.6% of available headroom in-distribution and goes negative on paired and longer-chain splits.
How to read
Bars grouped by task type, from math and dates to true-or-false. Height is the share answered correctly, taller is better, with bars for the model's own choice, the trained judge, and best-possible selection.
Takeaway → The judge's bar edges above the model for most tasks but drops sharply on true-or-false, where few possible answers make competing programs hard to tell apart.
Data table
| task domain | base (frozen compiler) | learned verifier | oracle (any correct candidate) |
|---|---|---|---|
| arithmetic | 31.2% | 34.4% | 96.9% |
| calendar | 37.5% | 40.6% | 100% |
| unit | 40.6% | 43.8% | 96.9% |
| list | 81.2% | 84.4% | 100% |
| boolean | 87.5% | 78.1% | 100% |
| lookup | 65.6% | 65.6% | 100% |
Numbers from report table (reports/qwen_context_trace_verifier_report.md), matches analysis/final_metrics.csv domain_* columns
Technical framing
Per-domain accuracy on length-6 prompts: modest gains except boolean, which the verifier hurts — Boolean drops 9.4 points: many candidates share the few possible boolean answers, so answer-only labels give little trace-identifying signal.
How to read
Bars grouped by the rule used to teach the judge what counts as a good program. Height is the share of mixed questions answered correctly, taller is better, with the model's own choice next to the trained judge.
Takeaway → Three of five training rules leave the judge identical to the model; only rewarding a correct final answer nudged it above, and only slightly.
Data table
| Training label objective | Frozen compiler (base) | Learned verifier |
|---|---|---|
| Answer labels (pilot, 96 prompts) | 60.4% | 64.6% |
| State-exact labels (pilot) | 60.4% | 60.4% |
| Single-oracle labels (pilot) | 60.4% | 60.4% |
| Repair-focused labels (pilot) | 60.4% | 60.4% |
| Answer labels (main, 384 prompts) | 59.4% | 60.2% |
Numbers from experiments/qwen_context_trace_verifier/analysis/all_final_metrics.csv
Technical framing
Label-objective ablation: only answer-correct labels move the verifier at all — State-exact, single-oracle, and repair-focused labels all collapse to copying the base program; only answer labels change selections.
How to read
The horizontal axis is how many steps the program takes, longer to the right. Height is the share answered correctly, taller is better. Three lines track a rule-based scorer, the model's own choice, and the trained judge.
Takeaway → The rule-based scorer line sits well below the other two at every length, ruling it out as a standalone way to pick the best program.
Data table
| Program length (VM steps) | Soft-trace heuristic | Frozen compiler (base) | Learned verifier |
|---|---|---|---|
| 6 | 46.1% | 68.8% | 70.3% |
| 8 | 32% | 50% | 46.9% |
| 10 | 23.4% | 40.6% | 35.2% |
Numbers from experiments/qwen_context_trace_verifier/analysis/all_final_metrics.csv
Technical framing
Soft-trace executor heuristic underperforms even the base compiler at every program length — The differentiable-executor scoring baseline trails the base by 17-23 points everywhere, ruling it out as a selection signal on its own.
In the author’s words from the Report · “Abstract”
The candidate search contains answer-correct programs for 91.4% to 100.0% of the main evaluation prompts, so the executable candidate set has substantial reachable headroom. The learned verifier captures only small in-distribution gains and loses accuracy on paired and longer-chain splits. The bottleneck is therefore candidate selection and credit assignment, not candidate availability.
Overview
This standalone experiment tests whether a learned verifier can better select hidden-VM candidate traces when it receives prompt-conditioned Qwen hidden-state context in addition to candidate execution features.
Large checkpoints are stored outside this directory:
/workspace/large_artifacts/qwen_context_trace_verifier/checkpoints
Local structure:
src/qwen_context_trace_verifier_experiment.py: run generator, training, and evaluationsrc/analyze_qwen_context_trace_verifier.py: regenerate aggregate CSVs, figures, reports, and checkpoint manifestruns/: per-run logs and metricsanalysis/: aggregate metrics and figuresreports/: standalone Markdown and HTML reportcheckpoint_manifest.csv: checkpoint and artifact index
Regenerate analysis artifacts from completed runs:
python experiments/qwen_context_trace_verifier/src/analyze_qwen_context_trace_verifier.pyReport
Rendered from reports/qwen_context_trace_verifier_report.md
Abstract
This experiment tests a complete-program posttraining route for improving a frozen local Qwen compiler. A Qwen3-4B hidden-state adapter emits a compact virtual-machine program. Around that program, the system enumerates local executable edits and trains a small context-conditioned verifier to select the best candidate using only prompt hidden-state context, candidate features, and execution traces. The verifier never sees the target answer at inference time.
The result is diagnostic rather than successful. The candidate search contains answer-correct programs for 91.4% to 100.0% of the main evaluation prompts, so the executable candidate set has substantial reachable headroom. The learned verifier captures only small in-distribution gains and loses accuracy on paired and longer-chain splits. The bottleneck is therefore candidate selection and credit assignment, not candidate availability.
Question
Can a small posttraining module make a frozen Qwen-attached compiler select better complete programs by inspecting executable traces, without forcing the model to generate every intermediate reasoning step as text?
Method
- Base model:
Qwen/Qwen3-4B. - Value space: arithmetic over modulus
97with up to10VM steps. - Frozen compiler: Qwen hidden states feed a trained hidden-VM compiler checkpoint.
- Candidate generator: local edits around the compiler argmax program, with top-k alternatives and up to
2edits. - Verifier input: candidate execution trace, scalar candidate features, and compact prompt hidden-state summaries.
- Main verifier:
3trace-transformer layers, width128,4heads. - Main training set:
384prompts, length range1to6, positive labelanswer. - Main checkpoint selection: best validation learned accuracy; selected epoch
14.
The key baselines are:
base: the frozen compiler argmax program.soft trace: a differentiable executor scoring heuristic.learned: the trained verifier top-1 selection.pair rerank: pair-level consistency reranking for paraphrase pairs.oracle: any answer-correct candidate in the generated local neighborhood.
Main Results
| split | base | soft trace | learned | pair rerank | oracle | learned gap recovered | avg candidates | answer positives |
|---|---|---|---|---|---|---|---|---|
| validation mixed L6 | 59.4% | 43.0% | 60.2% | n/a | 98.4% | 2.0% | 111.0 | 30.4 |
| fresh standard L6 | 68.8% | 46.1% | 70.3% | n/a | 98.4% | 5.3% | 111.0 | 30.3 |
| fresh paraphrase L6 | 57.8% | 35.2% | 60.2% | n/a | 100.0% | 5.6% | 111.0 | 27.9 |
| fresh paired L6 | 57.3% | 43.2% | 55.2% | 54.7% | 98.4% | -5.1% | 111.0 | 30.2 |
| hard standard L8 | 50.0% | 32.0% | 46.9% | n/a | 98.4% | -6.5% | 179.0 | 44.7 |
| hard paraphrase L8 | 38.3% | 27.3% | 35.9% | n/a | 96.9% | -4.0% | 179.0 | 43.2 |
| harder standard L10 | 40.6% | 23.4% | 35.2% | n/a | 98.4% | -9.5% | 263.0 | 47.6 |
| harder paraphrase L10 | 17.2% | 23.4% | 14.1% | n/a | 91.4% | -4.2% | 263.0 | 36.2 |
Fresh length-6 standard prompts improved from 68.8% to 70.3%, a 1.6% absolute gain. Fresh length-6 paraphrase prompts improved from 57.8% to 60.2%. These gains are real but small relative to the oracle.
The same selector did not extrapolate. Fresh paired length-6 accuracy moved from 57.3% to 55.2%, a -2.1% absolute change. Harder standard length-10 moved from 40.6% to 35.2%, a -5.5% absolute change. The oracle stayed high on these splits, so the selector failed to locate available correct candidates.
Candidate Geometry
The candidate generator is broad. On length-6 fresh splits it produces about 111 candidates per prompt, with roughly 28 to 30 answer-correct candidates. On length-10 splits it produces 263 candidates per prompt, with many answer-correct candidates but far fewer state-exact or program-exact candidates. This explains why answer-only labels are easy to satisfy but weakly identify the best computational trace.
Objective Pilots
Several label objectives were tested before the main run:
- Answer-correct labels learned a nontrivial selector on validation but were noisy because many candidates share the final answer.
- State-exact labels were too conservative and mostly preserved the base program.
- Single-oracle labels were sharper but still mostly preserved the base program.
- Repair-focused weighting did not overcome the base-preservation tendency.
The main run eventually recovered the best validation learned accuracy at epoch 14, but validation gains were small while train accuracy rose strongly. That is the signature of a selector that can fit candidate artifacts without learning a robust preference rule for unseen prompts.
Domain Breakdown
| domain | base | learned | oracle | learned delta |
|---|---|---|---|---|
| arithmetic | 31.2% | 34.4% | 96.9% | 3.1% |
| calendar | 37.5% | 40.6% | 100.0% | 3.1% |
| unit | 40.6% | 43.8% | 96.9% | 3.1% |
| list | 81.2% | 84.4% | 100.0% | 3.1% |
| boolean | 87.5% | 78.1% | 100.0% | -9.4% |
| lookup | 65.6% | 65.6% | 100.0% | 0.0% |
The verifier helped arithmetic, calendar, unit, and list prompts modestly, left lookup unchanged, and hurt boolean prompts. Boolean has many answer-correct candidates but relatively little trace-identifying signal because final answers collapse to few values.
Interpretation
This experiment answers one useful subquestion: local executable candidate search is not the limiting factor. The oracle remains high even when the base compiler is weak. The limiting factor is how to train a verifier that identifies the right complete program from a dense equivalence class of answer-correct candidates.
The current learned verifier is too passive. It sees traces and context, but its supervision says many different programs are equally good whenever they hit the final answer. Sharper labels alone did not fix this because the base program dominates many training groups and the correct non-base candidates are sparse. The result argues against scaling this exact reranker unchanged.
Most Impactful Next Options
Train the selector from executable teacher traces, not just final answers. Use the oracle candidate to produce dense supervision over every intermediate state and operation, then train the verifier or compiler with a margin that explicitly ranks state-consistent candidates above answer-only candidates. This directly targets the observed dense-label failure.
Convert reranking into preference learning on repairable failures. Filter training to groups where the base program is wrong and at least one non-base candidate is right, then train pairwise preferences with hard negatives that share the answer but diverge in state. This removes the base-preservation shortcut.
Let Qwen read serialized candidate traces. Instead of only a small verifier, serialize a small shortlist of candidate programs and traces back into Qwen hidden states or text tokens, then train a LoRA selector head. This tests whether the frozen model already has the semantic machinery needed to choose among executable candidates when the candidates are made legible.
- Train a differentiable interpreter objective upstream. Backpropagate through the soft executor into the adapter with auxiliary losses on intermediate states, then use the discrete candidate oracle only for evaluation. This attacks the compiler's crystallized program quality rather than relying on post-hoc selection.
The first option is the cleanest next experiment because it directly matches the failure mode found here: answer-correct candidate availability is high, but answer-only selection is underdetermined.
Artifacts
- Aggregate metrics:
experiments/qwen_context_trace_verifier/analysis/all_final_metrics.csv - Main metrics:
experiments/qwen_context_trace_verifier/analysis/final_metrics.csv - Training logs:
experiments/qwen_context_trace_verifier/analysis/verifier_train_logs.csv - Checkpoint manifest:
experiments/qwen_context_trace_verifier/checkpoint_manifest.csv - Large checkpoints:
large_artifacts/qwen_context_trace_verifier/checkpoints
Experiment log 1
Show the running log (1 entry, 2026-06-23)
2026-06-23
Objective: train and evaluate a context-conditioned learned verifier for local candidate traces emitted by a frozen Qwen-attached hidden VM compiler. The verifier uses candidate execution traces, compiler-local features, and compact Qwen hidden-state summaries from the prompt. Reports must be standalone.
Initial setup:
- Created standalone experiment directory.
- Copied the frozen mixed-domain trace compiler into this experiment's own large-artifact area:
/workspace/large_artifacts/qwen_context_trace_verifier/checkpoints/fixed_mixed_vm_trace_compiler_s512. - Seeded local source from the mixed-domain verifier implementation and prepared to add context features, hard-negative contrastive loss, and optional paired training.
- Patched the context-conditioned reranker path so context normalization statistics are returned, saved, and used during evaluation.
- Switched the primary positive label from full trajectory exactness to final answer correctness, keeping state-exact and program-exact metrics as diagnostics. This aligns the experiment with complete-program candidate selection from executable answer labels.
- Ran
smoke_context_trace_reranker, a tiny Qwen-backed end-to-end smoke test. It loaded the fixedQwen/Qwen3-4Bhidden-VM compiler checkpoint, generated candidate groups, trained the reranker for two epochs, wrote run artifacts, and saved a small verifier checkpoint. Fresh paired length-3 base, learned, and pair-rerank accuracy were all 66.7%; oracle was 100.0%. This validates the pipeline and confirms there is answer-correct candidate headroom even at smoke scale. - Ran
pilot_context_trace_reranker_s96with answer-correct positives, length-6 fresh splits, length-8/10 hard splits, top-k 3, and two-edit candidate neighborhoods. Validation improved from 60.4% base to 64.6% learned, and hard standard improved from 54.2% to 56.2%, but fresh paired fell from 65.6% base to 57.8% learned; pair reranking recovered to 64.1%. Interpretation: answer-correct labels are dense, averaging 29.4 positives per 111 candidates on fresh paired, and are not sharp enough by themselves for robust top-1 selection. - Ran matched state-exact and oracle-selector pilots. State-exact training and single-oracle training mostly selected the base program everywhere, matching base accuracy on fresh and hard splits. Interpretation: sharper labels alone are insufficient when base-correct groups dominate the training signal.
- Ran
pilot_context_trace_reranker_s96_repairfocus, which used single-oracle positives with base-positive groups downweighted and repairable groups upweighted. This also mostly selected the base program. Interpretation: weighting alone did not expose a strong repair preference under the current feature and trace representation. - Ran
main_context_trace_reranker_s384_answer, the main answer-label run with 384 training prompts, 128 validation prompts, 128 fresh/hard evaluation prompts, 96 paraphrase pairs, length-6 in-distribution evaluation, and length-8/10 extrapolation. Best validation epoch was 14. The learned selector improved validation from 59.4% to 60.2%, fresh standard length-6 from 68.8% to 70.3%, and fresh paraphrase length-6 from 57.8% to 60.2%. It degraded fresh paired length-6 from 57.3% to 55.2%, hard standard length-8 from 50.0% to 46.9%, hard paraphrase length-8 from 38.3% to 35.9%, harder standard length-10 from 40.6% to 35.2%, and harder paraphrase length-10 from 17.2% to 14.1%. Oracle answer-correct candidate availability stayed high, ranging from 91.4% to 100.0% across the main evaluation splits. - Generated aggregate CSVs, six figures, a standalone Markdown report, and a standalone HTML report with
src/analyze_qwen_context_trace_verifier.py. The core conclusion is diagnostic: local executable candidate search has enough headroom, but answer-only verifier training is underdetermined and does not learn robust program selection for paired or longer-chain prompts.
Figures 6
Data files 14
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv64 kBanalysis/final_metrics.csv19 kBruns/main_context_trace_reranker_s384_answer/metrics.csv21 kBruns/main_context_trace_reranker_s384_answer/results.json151 kBruns/pilot_context_trace_reranker_s96/metrics.csv14 kBruns/pilot_context_trace_reranker_s96/results.json96 kBruns/pilot_context_trace_reranker_s96_oracle/metrics.csv14 kBruns/pilot_context_trace_reranker_s96_oracle/results.json96 kBruns/pilot_context_trace_reranker_s96_repairfocus/metrics.csv14 kBruns/pilot_context_trace_reranker_s96_repairfocus/results.json96 kBruns/pilot_context_trace_reranker_s96_state/metrics.csv14 kBruns/pilot_context_trace_reranker_s96_state/results.json96 kBruns/smoke_context_trace_reranker/metrics.csv11 kBruns/smoke_context_trace_reranker/results.json52 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).