Qwen Candidate-Conditioned Trace Verifier
The one idea you need
Picture a program that might have a bug, plus 64 suggested rewrites, each with a receipt showing exactly what it did when run. The test: can a small model read those receipts and reliably pick the rewrite that actually works?
The question
If you show a small model each candidate fix for a program plus what it actually did when run, can it reliably pick the one that works?
What we found
No. Doing nothing already solved about 44% of tasks, and always choosing a correct candidate could reach 90%. Yet no trained picker captured any of that headroom; the best merely tied doing nothing. Left unchecked, the model rewrote almost every program, including ones already correct, breaking up to 97% and crashing accuracy to about 1%.
Why it matters
A model reading candidate code and its run trace still can't score which fix is correct. Before wiring a model fix-picker into a repair loop, gate it hard: unchecked, it breaks far more code than it repairs.
On this page
Results at a glance 7
How to read
Each bar is accuracy (taller is better) for one way of choosing a fix: doing nothing, the best trained picker, two free-editing pickers, then two cheat pickers that already know the right answer.
Takeaway → The best trained picker sits exactly level with doing nothing, the free-editing pickers collapse to the floor, and only the cheat pickers rise far above.
Data table
| selector | accuracy |
|---|---|
| no repair (base) | 44.4% |
| best non-oracle selector (tied) | 44.4% |
| Qwen ranking head (ungated) | 6.9% |
| frozen Qwen (ungated) | 1.4% |
| oracle over shortlist | 75% |
| oracle over full pool | 90.3% |
Numbers from report table (Standard L24 Gate, reports/qwen_candidate_conditioned_trace_verifier_report.md)
Technical framing
Standard split: no trained selector beats the no-repair base policy — Oracles show 31-46pp of reachable repair headroom, but every deployable selector ties base at best; ungated Qwen selectors collapse.
How to read
For each picker variant, one bar is accuracy (taller better) and one is how often it broke a working program (shorter better). Each pair shows a free-editing version beside a fallback version that keeps the original when unsure.
Takeaway → Free-editing versions show near-maxed breakage bars; the fallback versions zero out damage but their accuracy only climbs back to the do-nothing level, never past it.
Data table
| Qwen selector variant | accuracy | damage rate (broke a correct base program) |
|---|---|---|
| frozen Qwen | 1.4% | 96.9% |
| frozen Qwen + gate | 44.4% | 0% |
| Qwen rank | 6.9% | 87.5% |
| Qwen rank + gate | 41.7% | 6.2% |
| Qwen echo | 7.6% | 85.9% |
| Qwen echo + gate | 44.4% | 0% |
Numbers from report table (Standard L24 Gate, reports/qwen_candidate_conditioned_trace_verifier_report.md)
Technical framing
Validation gating stops the damage but recovers no gains (standard split) — Ungated Qwen selectors over-edit and break ~86-97% of correct base programs; the base-fallback gate removes the damage only by barely editing.
How to read
Grouped bars per test set: doing nothing, a cheat picker limited to the shortlist, and a cheat picker over the full pile. Taller cheat bars mean more available improvement. Higher is better.
Takeaway → On every test set the cheat-picker bars tower over doing-nothing, proving correct fixes are present and the problem is choosing them, not lacking them.
Data table
| evaluation split (L24) | no repair (base) | oracle over shortlist | oracle over full pool |
|---|---|---|---|
| standard | 44.4% | 75% | 90.3% |
| held-out source | 37.5% | 65.3% | 84.7% |
| paired | 62.5% | 81.9% | 94.4% |
| paired held-out | 38.9% | 65.3% | 88.9% |
| paraphrase | 84.7% | 94.4% | 98.6% |
Numbers from report table (Candidate Coverage, reports/qwen_candidate_conditioned_trace_verifier_report.md)
Technical framing
Repair headroom exists on every evaluation split — The candidate pool contains correct repairs on every split (full-pool oracle 14-50pp above base), so the failure is selection, not coverage.
How to read
Per approach, one bar is how often it changed the original program and one is how often that change broke a working one. Both lower is safer. Cheat pickers and fallback variants sit low; free-editing variants sit high.
Takeaway → Free-editing bars nearly max out on both measures — they touch almost everything and wreck most working programs — unlike the cheat pickers, which edit only where a fix helps.
Data table
| Selector arm | Changed base program | Damaged a correct program |
|---|---|---|
| base | 0% | 0% |
| oracle_full | 45.8% | 0% |
| oracle_shortlist | 30.6% | 0% |
| qwen_rank | 92.4% | 87.5% |
| qwen_rank_gated | 10.4% | 6.2% |
| qwen_echo | 92.4% | 85.9% |
| qwen_echo_gated | 3.5% | 0% |
| zero_qwen | 97.2% | 96.9% |
| qwen_shuffled | 97.9% | 96.9% |
| qwen_trace_corrupt | 97.9% | 98.4% |
Numbers from reports/qwen_candidate_conditioned_trace_verifier_report.md, Standard L24 Gate table
Technical framing
Ungated Qwen selectors rewrite almost everything on the standard split — Ungated Qwen arms edit ~92-98% of groups and damage most correct programs; oracles edit only where a repair exists.
How to read
Bars show each training approach's final usefulness score, one bar per repeat run; higher is better. Simple baselines sit high; the real fix-reading models sit low, near controls fed scrambled labels or corrupted run traces.
Takeaway → The real fix-reading bars land down near the deliberately-broken control bars on both runs, showing they learned nothing the run traces should have provided.
Data table
| Training arm | Training seed 101 | Training seed 202 |
|---|---|---|
| feature_mlp | 0.75 | 0.75 |
| trace_transformer | 0.75 | 0.75 |
| qwen_prompt_only | 0.75 | 0.75 |
| qwen_rank | -0.5 | -0.583 |
| qwen_echo | -0.5 | -0.583 |
| qwen_shuffled | -0.667 | -0.75 |
| qwen_candidate_only | -0.667 | -0.75 |
| qwen_trace_corrupt | -0.75 | -0.75 |
Numbers from reports/qwen_candidate_conditioned_trace_verifier_report.md, Training Dynamics table
Technical framing
Trace corruption and label shuffling barely change the failure — Real candidate-reading Qwen heads end training near the shuffled-label and corrupted-trace controls, on both seeds.
How to read
Grouped bars show how often the untouched program is already correct, split by test set. Each colored series is one batch of source problems; higher just means fewer things left to fix.
Takeaway → One batch sits near the top on every test set while another drops to zero, so the room for repair concentrates in particular batches of problems.
Data table
| Evaluation split | Source seed 123 | Source seed 456 | Source seed 789 |
|---|---|---|---|
| heldout_L24 | 25% | 8.3% | 79.2% |
| paired_L24 | 37.5% | 58.3% | 91.7% |
| paired_heldout_L24 | 16.7% | 29.2% | 70.8% |
| paraphrase_L24 | 75% | 79.2% | 100% |
| standard_L24 | 0% | 41.7% | 91.7% |
Numbers from reports/candidate_summary.csv
Technical framing
Base-policy accuracy swings wildly across source seeds — Seed 789 is near ceiling while seed 123 hits 0% on standard, so repair headroom concentrates in specific source seeds.
How to read
Bars show accuracy on a batch of problems the model never trained on; taller is better. They span doing nothing, a cheat picker, and free-editing versus fallback versions of each trained picker.
Takeaway → With doing-nothing already near the top, the free-editing bars crash to the floor while the fallback bars climb back near do-nothing but never pass it.
Data table
| Selector arm | Accuracy (training seed 101; frozen arms seed -1) |
|---|---|
| base | 91.7% |
| oracle_full | 95.8% |
| zero_qwen | 0% |
| zero_qwen_gated | 91.7% |
| qwen_rank | 8.3% |
| qwen_rank_gated | 87.5% |
| qwen_echo | 12.5% |
| qwen_echo_gated | 91.7% |
Numbers from reports/qwen_candidate_conditioned_trace_verifier_report.md, Held-Out Source Readout table
Technical framing
On the held-out source, gating is the only thing standing between Qwen and collapse — With base already at 91.7%, ungated Qwen selectors fall to 0-12.5%; gated variants recover 87.5-91.7% but never beat base.
In the author’s words from the Report · “Summary”
On standard_L24, no non-oracle selector improved over the no-repair base policy: base was 44.4%, the best tied selector accuracy was 44.4% (feature_mlp, qwen_echo_gated, trace_transformer, zero_qwen_gated), deployable-shortlist oracle accuracy was 75.0%, and full-pool oracle accuracy was 90.3%. Ungated candidate-conditioned Qwen selectors usually over-edited and damaged correct base programs; validation-gated variants avoided most damage by falling back to the base candidate, but did not recover the reachable repair gap.
Overview
Standalone experiment for testing whether Qwen can select executable repair candidates by reading the prompt, candidate program, and candidate execution trace.
Large embeddings and checkpoints live outside this directory:
/workspace/large_artifacts/qwen_candidate_conditioned_trace_verifier/
Reports
- Markdown:
reports/qwen_candidate_conditioned_trace_verifier_report.md - HTML:
reports/qwen_candidate_conditioned_trace_verifier_report.html - Figures:
reports/figures/
Report
Rendered from reports/qwen_candidate_conditioned_trace_verifier_report.md
Summary
This standalone experiment tests whether a Qwen reader can score concrete executable repair candidates. Each candidate contains a program and execution trace. Learned selectors are trained only from offline labels; at inference they do not receive the target answer or target state.
On standard_L24, no non-oracle selector improved over the no-repair base policy: base was 44.4%, the best tied selector accuracy was 44.4% (feature_mlp, qwen_echo_gated, trace_transformer, zero_qwen_gated), deployable-shortlist oracle accuracy was 75.0%, and full-pool oracle accuracy was 90.3%.
Ungated candidate-conditioned Qwen selectors usually over-edited and damaged correct base programs; validation-gated variants avoided most damage by falling back to the base candidate, but did not recover the reachable repair gap.
Setup
Base reader:
Qwen/Qwen3-4B.Candidate shortlist size:
64.Held-out source seed:
789.- Train groups per non-held-out source:
32; validation groups per non-held-out source:12; eval groups per source/split:24.
Selectors:
base: no repair.oracle_full: best answer-correct candidate in the full candidate pool.oracle_shortlist: best answer-correct candidate in the deployable shortlist.zero_qwen: frozen Qwen yes/no score with no training.zero_qwen_gated: frozen Qwen score with validation-tuned base fallback.feature_mlp: feature-only learned baseline.trace_transformer: small trace-only learned baseline.qwen_rank: Qwen embedding plus groupwise ranking head.qwen_rank_gated: Qwen ranking head with validation-tuned base fallback.qwen_echo: same ranking head with auxiliary candidate trace prediction.qwen_echo_gated: ECHO-ablation head with validation-tuned base fallback.- Control arms: shuffled labels, prompt-only, candidate-only, and trace-corrupted candidate reading.
Candidate Coverage
| split | groups | avg_full_candidates | avg_shortlist | base_accuracy | shortlist_oracle_accuracy | full_oracle_accuracy | shortlist_oracle_capture |
|---|---|---|---|---|---|---|---|
| heldout_L24 | 72 | 177 | 64 | 37.5% | 65.3% | 84.7% | 77.0% |
| paired_L24 | 72 | 177 | 64 | 62.5% | 81.9% | 94.4% | 86.8% |
| paired_heldout_L24 | 72 | 177 | 64 | 38.9% | 65.3% | 88.9% | 73.4% |
| paraphrase_L24 | 72 | 177 | 64 | 84.7% | 94.4% | 98.6% | 95.8% |
| standard_L24 | 72 | 177 | 64 | 44.4% | 75.0% | 90.3% | 83.1% |
| train_mixed_L24 | 64 | 177 | 64 | 81.2% | 96.9% | 100.0% | 96.9% |
| val_mixed_L24 | 24 | 177 | 64 | 75.0% | 83.3% | 91.7% | 90.9% |
Standard L24 Gate
| arm | mean_accuracy | mean_gap_capture | mean_changed | mean_damage | mean_recovery |
|---|---|---|---|---|---|
| base | 44.4% | 0.0% | 0 | 0 | 0 |
| feature_mlp | 44.4% | 0.0% | 0 | 0 | 0 |
| oracle_full | 90.3% | 100.0% | 0.458 | 0 | 0.825 |
| oracle_shortlist | 75.0% | 66.7% | 0.306 | 0 | 0.55 |
| qwen_candidate_only | 3.5% | -89.4% | 0.993 | 0.922 | 0 |
| qwen_echo | 7.6% | -80.3% | 0.924 | 0.859 | 0.025 |
| qwen_echo_gated | 44.4% | 0.0% | 0.035 | 0 | 0 |
| qwen_prompt_only | 44.4% | 0.0% | 0 | 0 | 0 |
| qwen_rank | 6.9% | -81.8% | 0.924 | 0.875 | 0.025 |
| qwen_rank_gated | 41.7% | -6.1% | 0.104 | 0.062 | 0 |
| qwen_shuffled | 2.1% | -92.4% | 0.979 | 0.969 | 0.013 |
| qwen_trace_corrupt | 1.4% | -93.9% | 0.979 | 0.984 | 0.013 |
| trace_transformer | 44.4% | 0.0% | 0 | 0 | 0 |
| zero_qwen | 1.4% | -93.9% | 0.972 | 0.969 | 0 |
| zero_qwen_gated | 44.4% | 0.0% | 0 | 0 | 0 |
Split Results
| split | arm | seed | accuracy | gap_capture | changed_fraction | damage_rate | recovery_rate |
|---|---|---|---|---|---|---|---|
| heldout_L24 | base | -1 | 37.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_L24 | base | -1 | 62.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_heldout_L24 | base | -1 | 38.9% | 0.0% | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | base | -1 | 84.7% | 0.0% | 0.0% | 0.0% | 0.0% |
| standard_L24 | base | -1 | 44.4% | 0.0% | 0.0% | 0.0% | 0.0% |
| heldout_L24 | oracle_full | -1 | 84.7% | 100.0% | 47.2% | 0.0% | 75.6% |
| paired_L24 | oracle_full | -1 | 94.4% | 100.0% | 31.9% | 0.0% | 85.2% |
| paired_heldout_L24 | oracle_full | -1 | 88.9% | 100.0% | 50.0% | 0.0% | 81.8% |
| paraphrase_L24 | oracle_full | -1 | 98.6% | 100.0% | 13.9% | 0.0% | 90.9% |
| standard_L24 | oracle_full | -1 | 90.3% | 100.0% | 45.8% | 0.0% | 82.5% |
| heldout_L24 | oracle_shortlist | -1 | 65.3% | 58.8% | 27.8% | 0.0% | 44.4% |
| paired_L24 | oracle_shortlist | -1 | 81.9% | 60.9% | 19.4% | 0.0% | 51.9% |
| paired_heldout_L24 | oracle_shortlist | -1 | 65.3% | 52.8% | 26.4% | 0.0% | 43.2% |
| paraphrase_L24 | oracle_shortlist | -1 | 94.4% | 70.0% | 9.7% | 0.0% | 63.6% |
| standard_L24 | oracle_shortlist | -1 | 75.0% | 66.7% | 30.6% | 0.0% | 55.0% |
| heldout_L24 | zero_qwen | -1 | 34.7% | -5.9% | 4.2% | 7.4% | 0.0% |
| paired_L24 | zero_qwen | -1 | 1.4% | -191.3% | 98.6% | 100.0% | 3.7% |
| paired_heldout_L24 | zero_qwen | -1 | 15.3% | -47.2% | 54.2% | 60.7% | 0.0% |
| paraphrase_L24 | zero_qwen | -1 | 12.5% | -520.0% | 87.5% | 85.2% | 0.0% |
| standard_L24 | zero_qwen | -1 | 1.4% | -93.9% | 97.2% | 96.9% | 0.0% |
| heldout_L24 | zero_qwen_gated | -1 | 37.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_L24 | zero_qwen_gated | -1 | 61.1% | -4.3% | 1.4% | 2.2% | 0.0% |
| paired_heldout_L24 | zero_qwen_gated | -1 | 38.9% | 0.0% | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | zero_qwen_gated | -1 | 72.2% | -90.0% | 12.5% | 14.8% | 0.0% |
| standard_L24 | zero_qwen_gated | -1 | 44.4% | 0.0% | 0.0% | 0.0% | 0.0% |
| heldout_L24 | feature_mlp | 101 | 37.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_L24 | feature_mlp | 101 | 62.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_heldout_L24 | feature_mlp | 101 | 38.9% | 0.0% | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | feature_mlp | 101 | 84.7% | 0.0% | 0.0% | 0.0% | 0.0% |
| standard_L24 | feature_mlp | 101 | 44.4% | 0.0% | 0.0% | 0.0% | 0.0% |
| heldout_L24 | trace_transformer | 101 | 37.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_L24 | trace_transformer | 101 | 62.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_heldout_L24 | trace_transformer | 101 | 38.9% | 0.0% | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | trace_transformer | 101 | 84.7% | 0.0% | 0.0% | 0.0% | 0.0% |
| standard_L24 | trace_transformer | 101 | 44.4% | 0.0% | 0.0% | 0.0% | 0.0% |
| heldout_L24 | qwen_rank | 101 | 31.9% | -11.8% | 13.9% | 14.8% | 0.0% |
| paired_L24 | qwen_rank | 101 | 16.7% | -143.5% | 86.1% | 77.8% | 7.4% |
| paired_heldout_L24 | qwen_rank | 101 | 19.4% | -38.9% | 61.1% | 57.1% | 4.5% |
| paraphrase_L24 | qwen_rank | 101 | 6.9% | -560.0% | 94.4% | 93.4% | 9.1% |
| standard_L24 | qwen_rank | 101 | 6.9% | -81.8% | 91.7% | 87.5% | 2.5% |
| heldout_L24 | qwen_rank_gated | 101 | 37.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_L24 | qwen_rank_gated | 101 | 59.7% | -8.7% | 9.7% | 6.7% | 3.7% |
| paired_heldout_L24 | qwen_rank_gated | 101 | 38.9% | 0.0% | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | qwen_rank_gated | 101 | 73.6% | -80.0% | 13.9% | 14.8% | 9.1% |
| standard_L24 | qwen_rank_gated | 101 | 41.7% | -6.1% | 9.7% | 6.2% | 0.0% |
| heldout_L24 | qwen_echo | 101 | 31.9% | -11.8% | 15.3% | 14.8% | 0.0% |
| paired_L24 | qwen_echo | 101 | 16.7% | -143.5% | 86.1% | 77.8% | 7.4% |
| paired_heldout_L24 | qwen_echo | 101 | 22.2% | -33.3% | 56.9% | 50.0% | 4.5% |
| paraphrase_L24 | qwen_echo | 101 | 5.6% | -570.0% | 94.4% | 93.4% | 0.0% |
| standard_L24 | qwen_echo | 101 | 9.7% | -75.8% | 90.3% | 81.2% | 2.5% |
| heldout_L24 | qwen_echo_gated | 101 | 37.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_L24 | qwen_echo_gated | 101 | 62.5% | 0.0% | 1.4% | 0.0% | 0.0% |
| paired_heldout_L24 | qwen_echo_gated | 101 | 38.9% | 0.0% | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | qwen_echo_gated | 101 | 80.6% | -30.0% | 4.2% | 4.9% | 0.0% |
| standard_L24 | qwen_echo_gated | 101 | 44.4% | 0.0% | 6.9% | 0.0% | 0.0% |
| heldout_L24 | feature_mlp | 202 | 37.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_L24 | feature_mlp | 202 | 62.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_heldout_L24 | feature_mlp | 202 | 38.9% | 0.0% | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | feature_mlp | 202 | 84.7% | 0.0% | 0.0% | 0.0% | 0.0% |
| standard_L24 | feature_mlp | 202 | 44.4% | 0.0% | 0.0% | 0.0% | 0.0% |
| heldout_L24 | trace_transformer | 202 | 37.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_L24 | trace_transformer | 202 | 62.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_heldout_L24 | trace_transformer | 202 | 38.9% | 0.0% | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | trace_transformer | 202 | 84.7% | 0.0% | 0.0% | 0.0% | 0.0% |
| standard_L24 | trace_transformer | 202 | 44.4% | 0.0% | 0.0% | 0.0% | 0.0% |
| heldout_L24 | qwen_rank | 202 | 31.9% | -11.8% | 15.3% | 14.8% | 0.0% |
| paired_L24 | qwen_rank | 202 | 12.5% | -156.5% | 88.9% | 84.4% | 7.4% |
| paired_heldout_L24 | qwen_rank | 202 | 19.4% | -38.9% | 61.1% | 53.6% | 2.3% |
| paraphrase_L24 | qwen_rank | 202 | 4.2% | -580.0% | 95.8% | 95.1% | 0.0% |
| standard_L24 | qwen_rank | 202 | 6.9% | -81.8% | 93.1% | 87.5% | 2.5% |
| heldout_L24 | qwen_rank_gated | 202 | 37.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| paired_L24 | qwen_rank_gated | 202 | 58.3% | -13.0% | 9.7% | 6.7% | 0.0% |
| paired_heldout_L24 | qwen_rank_gated | 202 | 38.9% | 0.0% | 1.4% | 0.0% | 0.0% |
| paraphrase_L24 | qwen_rank_gated | 202 | 75.0% | -70.0% | 13.9% | 11.5% | 0.0% |
| standard_L24 | qwen_rank_gated | 202 | 41.7% | -6.1% | 11.1% | 6.2% | 0.0% |
| heldout_L24 | qwen_echo | 202 | 33.3% | -8.8% | 11.1% | 11.1% | 0.0% |
| paired_L24 | qwen_echo | 202 | 15.3% | -147.8% | 87.5% | 80.0% | 7.4% |
| paired_heldout_L24 | qwen_echo | 202 | 16.7% | -44.4% | 62.5% | 60.7% | 2.3% |
| paraphrase_L24 | qwen_echo | 202 | 6.9% | -560.0% | 93.1% | 91.8% | 0.0% |
| standard_L24 | qwen_echo | 202 | 5.6% | -84.8% | 94.4% | 90.6% | 2.5% |
Held-Out Source Readout
| arm | seed | accuracy | base_accuracy | oracle_accuracy | gap_capture | changed_fraction | damage_rate | recovery_rate |
|---|---|---|---|---|---|---|---|---|
| base | -1 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| oracle_full | -1 | 95.8% | 91.7% | 95.8% | 100.0% | 4.2% | 0.0% | 50.0% |
| oracle_shortlist | -1 | 95.8% | 91.7% | 95.8% | 100.0% | 4.2% | 0.0% | 50.0% |
| zero_qwen | -1 | 0.0% | 91.7% | 95.8% | -2200.0% | 100.0% | 100.0% | 0.0% |
| zero_qwen_gated | -1 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| feature_mlp | 101 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| trace_transformer | 101 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| qwen_rank | 101 | 8.3% | 91.7% | 95.8% | -2000.0% | 91.7% | 90.9% | 0.0% |
| qwen_rank_gated | 101 | 87.5% | 91.7% | 95.8% | -100.0% | 4.2% | 4.5% | 0.0% |
| qwen_echo | 101 | 12.5% | 91.7% | 95.8% | -1900.0% | 87.5% | 86.4% | 0.0% |
| qwen_echo_gated | 101 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| qwen_shuffled | 101 | 4.2% | 91.7% | 95.8% | -2100.0% | 95.8% | 95.5% | 0.0% |
| qwen_prompt_only | 101 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| qwen_candidate_only | 101 | 4.2% | 91.7% | 95.8% | -2100.0% | 100.0% | 95.5% | 0.0% |
| qwen_trace_corrupt | 101 | 4.2% | 91.7% | 95.8% | -2100.0% | 100.0% | 95.5% | 0.0% |
| feature_mlp | 202 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| trace_transformer | 202 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| qwen_rank | 202 | 8.3% | 91.7% | 95.8% | -2000.0% | 91.7% | 90.9% | 0.0% |
| qwen_rank_gated | 202 | 87.5% | 91.7% | 95.8% | -100.0% | 4.2% | 4.5% | 0.0% |
| qwen_echo | 202 | 8.3% | 91.7% | 95.8% | -2000.0% | 91.7% | 90.9% | 0.0% |
| qwen_echo_gated | 202 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| qwen_shuffled | 202 | 4.2% | 91.7% | 95.8% | -2100.0% | 95.8% | 95.5% | 0.0% |
| qwen_prompt_only | 202 | 91.7% | 91.7% | 95.8% | 0.0% | 0.0% | 0.0% | 0.0% |
| qwen_candidate_only | 202 | 8.3% | 91.7% | 95.8% | -2000.0% | 100.0% | 90.9% | 0.0% |
| qwen_trace_corrupt | 202 | 0.0% | 91.7% | 95.8% | -2200.0% | 100.0% | 100.0% | 0.0% |
Training Dynamics
| arm | seed | epoch | train_loss | val_accuracy | val_damage_rate | val_recovery_rate | val_utility |
|---|---|---|---|---|---|---|---|
| qwen_trace_corrupt | 101 | 8 | 3.935 | 0.0% | 100.0% | 0.0% | -0.75 |
| qwen_candidate_only | 101 | 8 | 3.559 | 4.2% | 94.4% | 0.0% | -0.667 |
| qwen_shuffled | 101 | 8 | 3.833 | 4.2% | 94.4% | 0.0% | -0.667 |
| qwen_prompt_only | 101 | 8 | 3.728 | 75.0% | 0.0% | 0.0% | 0.75 |
| trace_transformer | 101 | 8 | 1.435 | 75.0% | 0.0% | 0.0% | 0.75 |
| feature_mlp | 101 | 8 | 0.062 | 75.0% | 0.0% | 0.0% | 0.75 |
| qwen_rank | 101 | 8 | 3.273 | 12.5% | 88.9% | 16.7% | -0.5 |
| qwen_echo | 101 | 8 | 4.684 | 12.5% | 88.9% | 16.7% | -0.5 |
| qwen_echo | 202 | 8 | 4.633 | 8.3% | 88.9% | 0.0% | -0.583 |
| qwen_rank | 202 | 8 | 3.295 | 8.3% | 88.9% | 0.0% | -0.583 |
| feature_mlp | 202 | 8 | 0.067 | 75.0% | 0.0% | 0.0% | 0.75 |
| trace_transformer | 202 | 8 | 1.372 | 75.0% | 0.0% | 0.0% | 0.75 |
| qwen_prompt_only | 202 | 8 | 3.679 | 75.0% | 0.0% | 0.0% | 0.75 |
| qwen_shuffled | 202 | 8 | 3.673 | 0.0% | 100.0% | 0.0% | -0.75 |
| qwen_candidate_only | 202 | 8 | 3.714 | 0.0% | 100.0% | 0.0% | -0.75 |
| qwen_trace_corrupt | 202 | 8 | 3.562 | 0.0% | 100.0% | 0.0% | -0.75 |
Interpretation
The decisive measurement is not raw accuracy alone. The report separates candidate coverage from selector capture, because a selector cannot choose a correct program that is absent from its candidate shortlist. A useful selector should capture a stable fraction of the available oracle gap, make nontrivial repairs, and avoid damaging already-correct base programs.
Artifacts
Run directory:
/workspace/experiments/qwen_candidate_conditioned_trace_verifier/runs/main_candidate_conditioned_qwen_trace_verifier_v2Large embeddings:
/workspace/large_artifacts/qwen_candidate_conditioned_trace_verifier/embeddings/main_candidate_conditioned_qwen_trace_verifier_v1Large checkpoints:
/workspace/large_artifacts/qwen_candidate_conditioned_trace_verifier/checkpoints/main_candidate_conditioned_qwen_trace_verifier_v2Metrics CSV:
/workspace/experiments/qwen_candidate_conditioned_trace_verifier/reports/metrics.csv- Candidate summary CSV:
/workspace/experiments/qwen_candidate_conditioned_trace_verifier/reports/candidate_summary.csv
Experiment log 8
Show the running log (8 entries, 2026-06-25)
Objective
Test whether candidate-conditioned Qwen reading can select correct executable repair candidates from a fixed candidate pool without using the target answer or target state at inference.
Design Commitments
- Fresh experiment directory:
experiments/qwen_candidate_conditioned_trace_verifier/. - Large artifacts separated under:
large_artifacts/qwen_candidate_conditioned_trace_verifier/. - Candidate pool and prompt reconstruction are deterministic.
- Main report is standalone and does not require external experiment context.
- Raw accuracy is reported, but source-normalized oracle-gap capture is the primary selector-quality metric.
- Include no-repair, oracle, feature, trace-only, frozen Qwen, trained Qwen, ECHO-ablation, shuffled-label, prompt-only, candidate-only, trace-corrupted, and held-out-source diagnostics.
Iterations
2026-06-25 Scaffold
Created the standalone directory, large-artifact directories, README, and experiment log.
2026-06-25 Smoke Iteration 1
Ran the full smoke path with the initial text serialization. It completed, but inspection showed that full prompt-plus-candidate examples were about 955 Qwen tokens, so the configured reader context would have truncated the verifier question and candidate tail.
Patched serialization before the main run:
- Strip internal register placeholders from the task text.
- Compact the natural task to semicolon-separated operations.
- Compact candidate traces to short
step:op=valuelines.
The corrected full prompt-plus-candidate serialization is about 367 tokens on a length-24 example, fitting the configured reader context without cutting off the verdict question.
2026-06-25 Smoke Iteration 2
Ran smoke_candidate_conditioned_qwen_trace_verifier_v2 with compact serialization and all controls enabled. Cache loading, prompt reconstruction, Qwen embedding, feature baseline, trace baseline, Qwen ranking head, ECHO-ablation, shuffled-label control, prompt-only, candidate-only, trace-corrupted controls, metrics, charts, Markdown report, and HTML report all completed.
2026-06-25 Main Iteration 1
Ran main_candidate_conditioned_qwen_trace_verifier_v1 with three source seeds, two critic seeds, shortlist size 64, and full candidate-conditioned Qwen embeddings rebuilt from Qwen/Qwen3-4B.
Important results:
- Candidate coverage was high. On
standard_L24, no repair was 44.4%, deployable-shortlist oracle was 75.0%, and full-pool oracle was 90.3%. - Frozen Qwen yes/no scoring was destructive on
standard_L24: 1.4% accuracy, 97.2% changed fraction, and 96.9% damage rate. - Trained candidate-conditioned Qwen heads also over-edited.
qwen_rankreached 6.9%;qwen_echoreached 5.6-9.7%. - Feature-only and trace-only baselines selected no repair and therefore tied the 44.4% base accuracy.
This showed that concrete candidate reading did not produce a safe selector in the ungated setup; the dominant failure was high-confidence destructive edits.
2026-06-25 Main Iteration 2
Patched the runner to add validation-tuned base fallback gates for frozen Qwen, qwen_rank, and qwen_echo, plus an --embedding_run_name flag so the second main run could reuse the large Qwen embedding cache without loading the model again.
Ran main_candidate_conditioned_qwen_trace_verifier_v2 using the v1 embedding cache. Runtime was 291.979 seconds and all trained/control heads completed.
Main result:
- Gating prevented most destructive edits but did not recover the reachable repair gap.
- On
standard_L24,qwen_echo_gatedtied the base at 44.4% with 0.0% oracle-gap capture;qwen_rank_gatedfell to 41.7%. - The validation-tuned gates mostly selected no-op. For
qwen_rank_gatedandqwen_echo_gated, validation accuracy equaled the base validation accuracy and validation recovery was 0.0%. - The main report and HTML report were regenerated from v2 metrics.
Interpretation: the candidate set contains many answer-correct repairs, but the tested Qwen-conditioned selectors did not learn to identify them safely without target leakage. Safe gating converted the method into a conservative no-op policy rather than a useful repair selector.
Figures 4
Data files 14
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/candidate_summary.csv2.4 kBreports/metrics.csv112 kBruns/main_candidate_conditioned_qwen_trace_verifier_v1/candidate_summary.csv2.4 kBruns/main_candidate_conditioned_qwen_trace_verifier_v1/metrics.csv91 kBruns/main_candidate_conditioned_qwen_trace_verifier_v1/run_summary.json103 Bruns/main_candidate_conditioned_qwen_trace_verifier_v2/candidate_summary.csv2.4 kBruns/main_candidate_conditioned_qwen_trace_verifier_v2/metrics.csv114 kBruns/main_candidate_conditioned_qwen_trace_verifier_v2/run_summary.json102 Bruns/smoke_candidate_conditioned_qwen_trace_verifier/candidate_summary.csv856 Bruns/smoke_candidate_conditioned_qwen_trace_verifier/metrics.csv14 kBruns/smoke_candidate_conditioned_qwen_trace_verifier/run_summary.json98 Bruns/smoke_candidate_conditioned_qwen_trace_verifier_v2/candidate_summary.csv856 Bruns/smoke_candidate_conditioned_qwen_trace_verifier_v2/metrics.csv15 kBruns/smoke_candidate_conditioned_qwen_trace_verifier_v2/run_summary.json99 B
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.