Qwen Readable Candidate Verifier
The one idea you need
A repair tool hands you 64 candidate fixes for one broken program, each written as plain pseudocode with the answer it claims to produce. The test: can a small model, never told the real answer, spot the fix that actually works?
The question
If you show a small model several candidate code fixes written as readable pseudocode with their claimed outputs, can it reliably pick the one that actually works?
What we found
Barely. Training a picker on the model's reading of each fix's pseudocode plus its claimed output raised the share of tasks fixed correctly from 44% to 51%, closing only about 15% of the distance to a perfect picker's 90%. And piling on detail backfired: handing the model a step-by-step execution trace crashed accuracy from 50% down to 8%, far worse than doing nothing.
Why it matters
When ranking code fixes with a small model, keep what it reads lean: plain pseudocode plus the claimed output works best, while full execution traces flood its attention and mislead. Add a safety fallback so weak picks default to leaving the program unchanged.
On this page
Results at a glance 7
How to read
Bars show the share of broken programs each method fixes correctly; taller is better. Left to right: doing nothing, majority vote on claimed answers, plain reading of the two programs, the trained picker, then two perfect-choice ceilings (best in the shortlist, best in the full pool).
Takeaway → The trained-picker bar sits just above the do-nothing bar and far below both ceilings, so reading the fixes helps only a sliver.
Data table
| selector | accuracy |
|---|---|
| no repair (base) | 44.4% |
| majority vote on claimed answers | 9.7% |
| frozen equivalence reading | 50% |
| trained head + gate (best) | 51.4% |
| shortlist oracle | 75% |
| full-pool oracle | 90.3% |
Numbers from report table (Standard L24 Gate), reports/qwen_readable_candidate_verifier_report.md
Technical framing
Can a frozen-Qwen reader pick the right repair candidate? (standard split) — The best non-oracle selector beats no-repair by only 7pp (51.4% vs 44.4%), capturing 15% of the oracle gap — reading works, but weakly.
How to read
Each bar is the model's accuracy as it is fed more, from just the two programs on the left, through adding the claimed answer, a full execution trace, a scrambled trace, the answer alone, or the fix alone; taller is better.
Takeaway → Bars fall off a cliff from about half correct to near zero, so extra detail distracts the model rather than informing it.
Data table
| information given to the frozen reader | frozen-Qwen selector accuracy |
|---|---|
| task + candidate programs (equivalence) | 50% |
| + claimed final value | 29.2% |
| + full execution trace | 8.3% |
| + corrupted trace | 1.4% |
| claimed value only (no programs) | 2.8% |
| candidate only | 0% |
Numbers from report table (Standard L24 Gate), reports/qwen_readable_candidate_verifier_report.md
Technical framing
More readable context hurts: what the frozen reader sees vs accuracy — Adding claimed values and execution traces makes the frozen reader worse, collapsing from 50% to near 0% — extra text distracts, not informs.
How to read
Grouped bars for each test set: doing nothing, the trained picker, and the best-possible pick from the shortlist; taller is better. Compare the middle bar against the left one within each group.
Takeaway → The trained picker meets or beats doing nothing in every set yet never reaches the perfect-choice bar, a small but consistent gain.
Data table
| evaluation split | no repair (base) | trained head + gate | shortlist oracle |
|---|---|---|---|
| standard | 44.4% | 51.4% | 75% |
| held-out | 37.5% | 44.4% | 65.3% |
| paired | 62.5% | 68.1% | 81.9% |
| paired held-out | 38.9% | 47.2% | 65.3% |
| paraphrase | 84.7% | 84.7% | 94.4% |
Numbers from report table (Split Results, seed 101), reports/qwen_readable_candidate_verifier_report.md
Technical framing
Gated trained selector vs baseline and oracle across evaluation splits — The gated selector improves or matches base on every split without damaging correct programs, but stays far below the shortlist oracle.
How to read
Lines track training error over repeated passes through the data; lower is better. Separate lines compare the reader trained on real labels against one trained on scrambled, meaningless labels.
Takeaway → Real-label lines slope downward while the scrambled line stays flat and high, showing the picker learns genuine signal rather than noise.
Data table
| Epoch | Feature MLP | Qwen head (real labels) | Trace transformer | Qwen head (shuffled labels) |
|---|---|---|---|---|
| 1 | 2.14 | 2.05 | 1.56 | 3.79 |
| 2 | 0.867 | 1.25 | 1.5 | 3.81 |
| 3 | 0.627 | 1.19 | 1.39 | 3.67 |
| 4 | 0.448 | 0.872 | 1.46 | 3.81 |
| 5 | 0.319 | 1.28 | 1.49 | 3.66 |
| 6 | 0.206 | 1.04 | 1.4 | 3.81 |
| 7 | 0.107 | 1.07 | 1.36 | 3.76 |
| 8 | 0.0617 | 0.71 | 1.44 | 3.71 |
Numbers from experiments/qwen_readable_candidate_verifier/reports/training_log.csv (seed 101)
Technical framing
Training loss: real labels learn, shuffled labels never do — Loss drops steadily with real labels but stays flat near 3.7 when labels are shuffled: the head is learning genuine candidate signal.
How to read
Grouped bars for three task-generator batches: doing nothing, the best pick from the shortlist, and the best pick from the full pool; taller is better.
Takeaway → Do-nothing accuracy swings from zero to over 90% across batches, so nearly all the room to improve is concentrated in one batch.
Data table
| Task generator seed (standard split) | No repair (base) | Oracle within shortlist | Oracle over full pool |
|---|---|---|---|
| source seed 123 | 0% | 41.7% | 79.2% |
| source seed 456 | 41.7% | 87.5% | 95.8% |
| source seed 789 | 91.7% | 95.8% | 95.8% |
Numbers from experiments/qwen_readable_candidate_verifier/reports/metrics.csv (standard_L24, difficulty=all rows)
Technical framing
Which generator seed you test on matters more than the selector — Per-seed spread is huge: base accuracy runs 0% to 92% across generator seeds, so headroom for repair is concentrated in seed 123.
How to read
Paired bars for six pickers: one bar for how often it breaks an already-correct program, one for how often it fixes a broken one. Lower breakage and higher fixing are better.
Takeaway → Adding the fallback slashes breakage toward zero while the trained picker keeps most of its fixing, so gating buys safety cheaply.
Data table
| Selector arm (standard split) | Damage rate (breaks correct programs) | Recovery rate (fixes wrong programs) |
|---|---|---|
| Task-value (frozen) | 93.8% | 0% |
| Task-value gated | 21.9% | 0% |
| Program+final (frozen) | 40.6% | 5% |
| Program+final gated | 3.1% | 2.5% |
| Trained Qwen head | 9.4% | 12.5% |
| Trained head gated | 0% | 12.5% |
Numbers from report table (reports/qwen_readable_candidate_verifier_report.md, Standard L24 Gate)
Technical framing
Gating buys safety: damage collapses while recovery mostly survives — Validation-tuned gating cuts damage from 94% to 22% (and 9% to 0% for the trained head) while the trained head keeps its 12.5% recovery.
How to read
Bars for doing nothing, the trained picker, and the same picker trained on scrambled answers, shown for two test sets; taller is better.
Takeaway → Training on scrambled answers drops accuracy well below doing nothing on both sets, proving the picker relies on real signal.
Data table
| Arm | Standard split | Held-out source |
|---|---|---|
| No repair (base) | 44.4% | 91.7% |
| Trained Qwen head | 47.2% | 87.5% |
| Same head, shuffled labels | 26.4% | 37.5% |
Technical framing
Shuffled-label control: the trained head uses real signal, not artifacts — Training on shuffled labels drops accuracy far below the no-repair baseline on both evaluations, confirming the learned signal is real.
In the author’s words from the Report · “Summary”
On standard_L24, the best non-oracle selector was qwen_program_final_gated at 51.4% accuracy and 15.2% oracle-gap capture versus 44.4% for no repair. The best mean ranking AUC on standard_L24 was feature_mlp at 0.799. Frozen program-equivalence reading reached 50.0% accuracy with AUC 0.677; adding full trace text dropped to 8.3% with AUC 0.620, while corrupted trace dropped to 1.4% with AUC 0.486.
Overview
Standalone experiment for testing whether Qwen can select executable repair candidates when the task and candidate are rendered as readable pseudocode and execution text.
Large embeddings, checkpoints, and run artifacts live outside this directory:
/workspace/large_artifacts/qwen_readable_candidate_verifier/
Reports
- Markdown:
reports/qwen_readable_candidate_verifier_report.md - HTML:
reports/qwen_readable_candidate_verifier_report.html - Figures:
reports/figures/
Report
Rendered from reports/qwen_readable_candidate_verifier_report.md
Summary
This standalone experiment tests whether a Qwen reader can select executable repair candidates when the task and candidate are rendered as readable pseudocode, claimed final values, and optional execution traces. Learned selectors are trained only from offline labels; at inference they do not receive the target answer or target state.
On standard_L24, the best non-oracle selector was qwen_program_final_gated at 51.4% accuracy and 15.2% oracle-gap capture versus 44.4% for no repair.
The best mean ranking AUC on standard_L24 was feature_mlp at 0.799.
Frozen program-equivalence reading reached 50.0% accuracy with AUC 0.677; adding full trace text dropped to 8.3% with AUC 0.620, while corrupted trace dropped to 1.4% with AUC 0.486.
Setup
Base reader:
Qwen/Qwen3-4B.Candidate shortlist size:
64.Held-out source seed:
789.Train groups per non-held-out source:
32; validation groups per non-held-out source:12; eval groups per source/split:24.Frozen readable modes:
task_value,program_equiv,program_final,program_trace,trace_corrupt,candidate_only,prompt_only.- Trained Qwen head modes:
program_final.
Selectors:
base: no repair.oracle_full: best answer-correct candidate in the full candidate pool.oracle_shortlist: best answer-correct candidate in the deployable shortlist.majority_answer: chooses the most common claimed final value in the shortlist.zero_task_value: frozen Qwen reads the task program and scores candidates by the logit of each candidate's claimed final value.zero_program_equiv: frozen Qwen reads task program and candidate program, then answers whether they return the same value.zero_program_final: frozen Qwen reads task program, candidate program, and candidate claimed final value.zero_program_trace: frozen Qwen also receives a readable execution trace.zero_trace_corrupt: same as trace mode but with corrupted state order.qwen_<mode>: learned groupwise ranking head over frozen Qwen embeddings for that readable mode._gated: validation-tuned base fallback for the corresponding score.- Additional controls: prompt-only, candidate-only, feature-only, trace-only, and shuffled-label arms.
Candidate Coverage
| split | groups | avg_full_candidates | avg_shortlist | avg_target_answer_count | avg_target_answer_margin | base_accuracy | shortlist_oracle_accuracy | full_oracle_accuracy | shortlist_oracle_capture |
|---|---|---|---|---|---|---|---|---|---|
| heldout_L24 | 72 | 177 | 64 | 1.208 | -2.667 | 37.5% | 65.3% | 84.7% | 77.0% |
| paired_L24 | 72 | 177 | 64 | 1.222 | -2.222 | 62.5% | 81.9% | 94.4% | 86.8% |
| paired_heldout_L24 | 72 | 177 | 64 | 1.042 | -2.444 | 38.9% | 65.3% | 88.9% | 73.4% |
| paraphrase_L24 | 72 | 177 | 64 | 1.75 | -1.861 | 84.7% | 94.4% | 98.6% | 95.8% |
| standard_L24 | 72 | 177 | 64 | 1.431 | -2.472 | 44.4% | 75.0% | 90.3% | 83.1% |
| train_mixed_L24 | 64 | 177 | 64 | 1.75 | -1.75 | 81.2% | 96.9% | 100.0% | 96.9% |
| val_mixed_L24 | 24 | 177 | 64 | 1.167 | -2.583 | 75.0% | 83.3% | 91.7% | 90.9% |
Standard L24 Gate
| arm | mean_accuracy | mean_gap_capture | mean_auc | mean_changed | mean_damage | mean_recovery |
|---|---|---|---|---|---|---|
| base | 44.4% | 0.0% | n/a | 0 | 0 | 0 |
| feature_mlp | 44.4% | 0.0% | 0.799 | 0 | 0 | 0 |
| majority_answer | 9.7% | -75.8% | n/a | 0.875 | 0.781 | 0 |
| oracle_full | 90.3% | 100.0% | n/a | 0.458 | 0 | 0.825 |
| oracle_shortlist | 75.0% | 66.7% | n/a | 0.306 | 0 | 0.55 |
| qwen_program_final | 47.2% | 6.1% | 0.669 | 0.569 | 0.094 | 0.125 |
| qwen_program_final_gated | 51.4% | 15.2% | 0.669 | 0.299 | 0 | 0.125 |
| qwen_program_final_shuffled | 26.4% | -39.4% | 0.662 | 0.771 | 0.516 | 0.087 |
| trace_transformer | 44.4% | 0.0% | 0.689 | 0 | 0 | 0 |
| zero_candidate_only | 0.0% | -97.0% | 0.477 | 1 | 1 | 0 |
| zero_candidate_only_gated | 36.1% | -18.2% | 0.477 | 0.139 | 0.188 | 0 |
| zero_program_equiv | 50.0% | 12.1% | 0.677 | 0.486 | 0 | 0.1 |
| zero_program_equiv_gated | 50.0% | 12.1% | 0.677 | 0.486 | 0 | 0.1 |
| zero_program_final | 29.2% | -33.3% | 0.648 | 0.694 | 0.406 | 0.05 |
| zero_program_final_gated | 44.4% | 0.0% | 0.648 | 0.056 | 0.031 | 0.025 |
| zero_program_trace | 8.3% | -78.8% | 0.620 | 0.917 | 0.906 | 0.075 |
| zero_program_trace_gated | 44.4% | 0.0% | 0.620 | 0 | 0 | 0 |
| zero_prompt_only | 44.4% | 0.0% | 0.500 | 0 | 0 | 0 |
| zero_prompt_only_gated | 44.4% | 0.0% | 0.500 | 0 | 0 | 0 |
| zero_task_value | 2.8% | -90.9% | 0.462 | 0.972 | 0.938 | 0 |
| zero_task_value_gated | 34.7% | -21.2% | 0.462 | 0.139 | 0.219 | 0 |
| zero_trace_corrupt | 1.4% | -93.9% | 0.486 | 1 | 0.969 | 0 |
| zero_trace_corrupt_gated | 44.4% | 0.0% | 0.486 | 0.028 | 0 | 0 |
Easy/Hard Standard L24 Readout
| difficulty | arm | seed | n | accuracy | gap_capture | ranking_auc | changed_fraction | damage_rate | recovery_rate |
|---|---|---|---|---|---|---|---|---|---|
| easy | base | -1 | 1 | 100.0% | n/a | n/a | 0.0% | 0.0% | n/a |
| hard | base | -1 | 71 | 43.7% | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| easy | oracle_shortlist | -1 | 1 | 100.0% | n/a | n/a | 0.0% | 0.0% | n/a |
| hard | oracle_shortlist | -1 | 71 | 74.6% | 66.7% | n/a | 31.0% | 0.0% | 55.0% |
| easy | majority_answer | -1 | 1 | 100.0% | n/a | n/a | 0.0% | 0.0% | n/a |
| hard | majority_answer | -1 | 71 | 8.5% | -75.8% | n/a | 88.7% | 80.6% | 0.0% |
| easy | zero_task_value | -1 | 1 | 0.0% | n/a | 0.220 | 100.0% | 100.0% | n/a |
| hard | zero_task_value | -1 | 71 | 2.8% | -87.9% | 0.467 | 97.2% | 93.5% | 0.0% |
| easy | zero_program_final | -1 | 1 | 100.0% | n/a | 0.763 | 0.0% | 0.0% | n/a |
| hard | zero_program_final | -1 | 71 | 28.2% | -33.3% | 0.646 | 70.4% | 41.9% | 5.0% |
| easy | zero_program_trace | -1 | 1 | 0.0% | n/a | 0.780 | 100.0% | 100.0% | n/a |
| hard | zero_program_trace | -1 | 71 | 8.5% | -75.8% | 0.617 | 91.5% | 90.3% | 7.5% |
| easy | zero_trace_corrupt | -1 | 1 | 0.0% | n/a | 0.819 | 100.0% | 100.0% | n/a |
| hard | zero_trace_corrupt | -1 | 71 | 1.4% | -90.9% | 0.480 | 100.0% | 96.8% | 0.0% |
| easy | qwen_program_final | 101 | 1 | 100.0% | n/a | 0.715 | 0.0% | 0.0% | n/a |
| hard | qwen_program_final | 101 | 71 | 46.5% | 6.1% | 0.670 | 57.7% | 9.7% | 12.5% |
| easy | qwen_program_final_gated | 101 | 1 | 100.0% | n/a | 0.715 | 0.0% | 0.0% | n/a |
| hard | qwen_program_final_gated | 101 | 71 | 50.7% | 15.2% | 0.670 | 29.6% | 0.0% | 12.5% |
| easy | qwen_program_final | 202 | 1 | 100.0% | n/a | 0.722 | 0.0% | 0.0% | n/a |
| hard | qwen_program_final | 202 | 71 | 46.5% | 6.1% | 0.666 | 57.7% | 9.7% | 12.5% |
| easy | qwen_program_final_gated | 202 | 1 | 100.0% | n/a | 0.722 | 0.0% | 0.0% | n/a |
| hard | qwen_program_final_gated | 202 | 71 | 50.7% | 15.2% | 0.666 | 31.0% | 0.0% | 12.5% |
Split Results
| split | arm | seed | accuracy | gap_capture | ranking_auc | changed_fraction | damage_rate | recovery_rate |
|---|---|---|---|---|---|---|---|---|
| heldout_L24 | base | -1 | 37.5% | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| paired_L24 | base | -1 | 62.5% | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| paired_heldout_L24 | base | -1 | 38.9% | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | base | -1 | 84.7% | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| standard_L24 | base | -1 | 44.4% | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| heldout_L24 | oracle_full | -1 | 84.7% | 100.0% | n/a | 47.2% | 0.0% | 75.6% |
| paired_L24 | oracle_full | -1 | 94.4% | 100.0% | n/a | 31.9% | 0.0% | 85.2% |
| paired_heldout_L24 | oracle_full | -1 | 88.9% | 100.0% | n/a | 50.0% | 0.0% | 81.8% |
| paraphrase_L24 | oracle_full | -1 | 98.6% | 100.0% | n/a | 13.9% | 0.0% | 90.9% |
| standard_L24 | oracle_full | -1 | 90.3% | 100.0% | n/a | 45.8% | 0.0% | 82.5% |
| heldout_L24 | oracle_shortlist | -1 | 65.3% | 58.8% | n/a | 27.8% | 0.0% | 44.4% |
| paired_L24 | oracle_shortlist | -1 | 81.9% | 60.9% | n/a | 19.4% | 0.0% | 51.9% |
| paired_heldout_L24 | oracle_shortlist | -1 | 65.3% | 52.8% | n/a | 26.4% | 0.0% | 43.2% |
| paraphrase_L24 | oracle_shortlist | -1 | 94.4% | 70.0% | n/a | 9.7% | 0.0% | 63.6% |
| standard_L24 | oracle_shortlist | -1 | 75.0% | 66.7% | n/a | 30.6% | 0.0% | 55.0% |
| heldout_L24 | majority_answer | -1 | 9.7% | -58.8% | n/a | 80.6% | 74.1% | 0.0% |
| paired_L24 | majority_answer | -1 | 9.7% | -165.2% | n/a | 87.5% | 84.4% | 0.0% |
| paired_heldout_L24 | majority_answer | -1 | 9.7% | -58.3% | n/a | 86.1% | 82.1% | 4.5% |
| paraphrase_L24 | majority_answer | -1 | 9.7% | -540.0% | n/a | 90.3% | 88.5% | 0.0% |
| standard_L24 | majority_answer | -1 | 9.7% | -75.8% | n/a | 87.5% | 78.1% | 0.0% |
| heldout_L24 | zero_task_value | -1 | 0.0% | -79.4% | 0.581 | 98.6% | 100.0% | 0.0% |
| paired_L24 | zero_task_value | -1 | 5.6% | -178.3% | 0.527 | 95.8% | 97.8% | 11.1% |
| paired_heldout_L24 | zero_task_value | -1 | 0.0% | -77.8% | 0.588 | 98.6% | 100.0% | 0.0% |
| paraphrase_L24 | zero_task_value | -1 | 0.0% | -610.0% | 0.533 | 100.0% | 100.0% | 0.0% |
| standard_L24 | zero_task_value | -1 | 2.8% | -90.9% | 0.462 | 97.2% | 93.8% | 0.0% |
| heldout_L24 | zero_task_value_gated | -1 | 36.1% | -2.9% | 0.581 | 6.9% | 3.7% | 0.0% |
| paired_L24 | zero_task_value_gated | -1 | 59.7% | -8.7% | 0.527 | 6.9% | 4.4% | 0.0% |
| paired_heldout_L24 | zero_task_value_gated | -1 | 36.1% | -5.6% | 0.588 | 11.1% | 7.1% | 0.0% |
| paraphrase_L24 | zero_task_value_gated | -1 | 70.8% | -100.0% | 0.533 | 20.8% | 16.4% | 0.0% |
| standard_L24 | zero_task_value_gated | -1 | 34.7% | -21.2% | 0.462 | 13.9% | 21.9% | 0.0% |
| heldout_L24 | zero_program_equiv | -1 | 44.4% | 14.7% | 0.751 | 58.3% | 3.7% | 13.3% |
| paired_L24 | zero_program_equiv | -1 | 62.5% | 0.0% | 0.824 | 37.5% | 8.9% | 14.8% |
| paired_heldout_L24 | zero_program_equiv | -1 | 45.8% | 13.9% | 0.745 | 56.9% | 3.6% | 13.6% |
| paraphrase_L24 | zero_program_equiv | -1 | 84.7% | 0.0% | 0.733 | 9.7% | 0.0% | 0.0% |
| standard_L24 | zero_program_equiv | -1 | 50.0% | 12.1% | 0.677 | 48.6% | 0.0% | 10.0% |
| heldout_L24 | zero_program_final | -1 | 36.1% | -2.9% | 0.738 | 65.3% | 22.2% | 11.1% |
| paired_L24 | zero_program_final | -1 | 52.8% | -30.4% | 0.786 | 47.2% | 20.0% | 7.4% |
| paired_heldout_L24 | zero_program_final | -1 | 22.2% | -33.3% | 0.698 | 76.4% | 50.0% | 4.5% |
| paraphrase_L24 | zero_program_final | -1 | 36.1% | -350.0% | 0.663 | 59.7% | 57.4% | 0.0% |
| standard_L24 | zero_program_final | -1 | 29.2% | -33.3% | 0.648 | 69.4% | 40.6% | 5.0% |
| heldout_L24 | zero_program_final_gated | -1 | 40.3% | 5.9% | 0.738 | 5.6% | 0.0% | 4.4% |
| paired_L24 | zero_program_final_gated | -1 | 63.9% | 4.3% | 0.786 | 2.8% | 0.0% | 3.7% |
| paired_heldout_L24 | zero_program_final_gated | -1 | 40.3% | 2.8% | 0.698 | 6.9% | 0.0% | 2.3% |
| paraphrase_L24 | zero_program_final_gated | -1 | 81.9% | -20.0% | 0.663 | 2.8% | 3.3% | 0.0% |
| standard_L24 | zero_program_final_gated | -1 | 44.4% | 0.0% | 0.648 | 5.6% | 3.1% | 2.5% |
| heldout_L24 | zero_program_trace | -1 | 9.7% | -58.8% | 0.712 | 88.9% | 81.5% | 4.4% |
| paired_L24 | zero_program_trace | -1 | 12.5% | -156.5% | 0.652 | 87.5% | 82.2% | 3.7% |
| paired_heldout_L24 | zero_program_trace | -1 | 5.6% | -66.7% | 0.640 | 91.7% | 85.7% | 0.0% |
| paraphrase_L24 | zero_program_trace | -1 | 11.1% | -530.0% | 0.655 | 87.5% | 86.9% | 0.0% |
| standard_L24 | zero_program_trace | -1 | 8.3% | -78.8% | 0.620 | 91.7% | 90.6% | 7.5% |
| heldout_L24 | zero_program_trace_gated | -1 | 37.5% | 0.0% | 0.712 | 0.0% | 0.0% | 0.0% |
| paired_L24 | zero_program_trace_gated | -1 | 62.5% | 0.0% | 0.652 | 0.0% | 0.0% | 0.0% |
| paired_heldout_L24 | zero_program_trace_gated | -1 | 38.9% | 0.0% | 0.640 | 0.0% | 0.0% | 0.0% |
| paraphrase_L24 | zero_program_trace_gated | -1 | 84.7% | 0.0% | 0.655 | 0.0% | 0.0% | 0.0% |
| standard_L24 | zero_program_trace_gated | -1 | 44.4% | 0.0% | 0.620 | 0.0% | 0.0% | 0.0% |
| heldout_L24 | zero_trace_corrupt | -1 | 5.6% | -67.6% | 0.582 | 94.4% | 88.9% | 2.2% |
| paired_L24 | zero_trace_corrupt | -1 | 0.0% | -195.7% | 0.444 | 100.0% | 100.0% | 0.0% |
| paired_heldout_L24 | zero_trace_corrupt | -1 | 1.4% | -75.0% | 0.520 | 97.2% | 96.4% | 0.0% |
| paraphrase_L24 | zero_trace_corrupt | -1 | 1.4% | -600.0% | 0.471 | 98.6% | 98.4% | 0.0% |
| standard_L24 | zero_trace_corrupt | -1 | 1.4% | -93.9% | 0.486 | 100.0% | 96.9% | 0.0% |
| heldout_L24 | qwen_program_final | 101 | 43.1% | 11.8% | 0.752 | 51.4% | 3.7% | 11.1% |
| paired_L24 | qwen_program_final | 101 | 66.7% | 13.0% | 0.802 | 30.6% | 2.2% | 14.8% |
| paired_heldout_L24 | qwen_program_final | 101 | 44.4% | 11.1% | 0.742 | 58.3% | 7.1% | 13.6% |
| paraphrase_L24 | qwen_program_final | 101 | 79.2% | -40.0% | 0.741 | 18.1% | 6.6% | 0.0% |
| standard_L24 | qwen_program_final | 101 | 47.2% | 6.1% | 0.671 | 56.9% | 9.4% | 12.5% |
| heldout_L24 | qwen_program_final_gated | 101 | 44.4% | 14.7% | 0.752 | 20.8% | 0.0% | 11.1% |
| paired_L24 | qwen_program_final_gated | 101 | 68.1% | 17.4% | 0.802 | 16.7% | 0.0% | 14.8% |
| paired_heldout_L24 | qwen_program_final_gated | 101 | 47.2% | 16.7% | 0.742 | 31.9% | 0.0% | 13.6% |
| paraphrase_L24 | qwen_program_final_gated | 101 | 84.7% | 0.0% | 0.741 | 2.8% | 0.0% | 0.0% |
| standard_L24 | qwen_program_final_gated | 101 | 51.4% | 15.2% | 0.671 | 29.2% | 0.0% | 12.5% |
| heldout_L24 | qwen_program_final | 202 | 43.1% | 11.8% | 0.756 | 51.4% | 3.7% | 11.1% |
| paired_L24 | qwen_program_final | 202 | 66.7% | 13.0% | 0.797 | 29.2% | 2.2% | 14.8% |
| paired_heldout_L24 | qwen_program_final | 202 | 43.1% | 8.3% | 0.746 | 59.7% | 7.1% | 11.4% |
| paraphrase_L24 | qwen_program_final | 202 | 79.2% | -40.0% | 0.739 | 18.1% | 6.6% | 0.0% |
| standard_L24 | qwen_program_final | 202 | 47.2% | 6.1% | 0.667 | 56.9% | 9.4% | 12.5% |
| heldout_L24 | qwen_program_final_gated | 202 | 44.4% | 14.7% | 0.756 | 23.6% | 0.0% | 11.1% |
| paired_L24 | qwen_program_final_gated | 202 | 68.1% | 17.4% | 0.797 | 18.1% | 0.0% | 14.8% |
| paired_heldout_L24 | qwen_program_final_gated | 202 | 45.8% | 13.9% | 0.746 | 30.6% | 0.0% | 11.4% |
| paraphrase_L24 | qwen_program_final_gated | 202 | 84.7% | 0.0% | 0.739 | 2.8% | 0.0% | 0.0% |
| standard_L24 | qwen_program_final_gated | 202 | 51.4% | 15.2% | 0.667 | 30.6% | 0.0% | 12.5% |
Held-Out Source Readout
| arm | seed | accuracy | base_accuracy | oracle_accuracy | gap_capture | ranking_auc | changed_fraction | damage_rate | recovery_rate |
|---|---|---|---|---|---|---|---|---|---|
| base | -1 | 91.7% | 91.7% | 95.8% | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| oracle_full | -1 | 95.8% | 91.7% | 95.8% | 100.0% | n/a | 4.2% | 0.0% | 50.0% |
| oracle_shortlist | -1 | 95.8% | 91.7% | 95.8% | 100.0% | n/a | 4.2% | 0.0% | 50.0% |
| majority_answer | -1 | 12.5% | 91.7% | 95.8% | -1900.0% | n/a | 87.5% | 86.4% | 0.0% |
| zero_task_value | -1 | 4.2% | 91.7% | 95.8% | -2100.0% | 0.448 | 95.8% | 95.5% | 0.0% |
| zero_task_value_gated | -1 | 75.0% | 91.7% | 95.8% | -400.0% | 0.448 | 16.7% | 18.2% | 0.0% |
| zero_program_equiv | -1 | 95.8% | 91.7% | 95.8% | 100.0% | 0.770 | 8.3% | 0.0% | 50.0% |
| zero_program_equiv_gated | -1 | 95.8% | 91.7% | 95.8% | 100.0% | 0.770 | 8.3% | 0.0% | 50.0% |
| zero_program_final | -1 | 54.2% | 91.7% | 95.8% | -900.0% | 0.718 | 45.8% | 40.9% | 0.0% |
| zero_program_final_gated | -1 | 87.5% | 91.7% | 95.8% | -100.0% | 0.718 | 4.2% | 4.5% | 0.0% |
| zero_program_trace | -1 | 12.5% | 91.7% | 95.8% | -1900.0% | 0.634 | 87.5% | 86.4% | 0.0% |
| zero_program_trace_gated | -1 | 91.7% | 91.7% | 95.8% | 0.0% | 0.634 | 0.0% | 0.0% | 0.0% |
| zero_trace_corrupt | -1 | 0.0% | 91.7% | 95.8% | -2200.0% | 0.513 | 100.0% | 100.0% | 0.0% |
| zero_trace_corrupt_gated | -1 | 91.7% | 91.7% | 95.8% | 0.0% | 0.513 | 4.2% | 0.0% | 0.0% |
| zero_candidate_only | -1 | 0.0% | 91.7% | 95.8% | -2200.0% | 0.448 | 100.0% | 100.0% | 0.0% |
| zero_candidate_only_gated | -1 | 75.0% | 91.7% | 95.8% | -400.0% | 0.448 | 16.7% | 18.2% | 0.0% |
| zero_prompt_only | -1 | 91.7% | 91.7% | 95.8% | 0.0% | 0.500 | 0.0% | 0.0% | 0.0% |
| zero_prompt_only_gated | -1 | 91.7% | 91.7% | 95.8% | 0.0% | 0.500 | 0.0% | 0.0% | 0.0% |
| feature_mlp | 101 | 91.7% | 91.7% | 95.8% | 0.0% | 0.989 | 0.0% | 0.0% | 0.0% |
| trace_transformer | 101 | 91.7% | 91.7% | 95.8% | 0.0% | 0.794 | 0.0% | 0.0% | 0.0% |
| qwen_program_final | 101 | 87.5% | 91.7% | 95.8% | -100.0% | 0.769 | 16.7% | 9.1% | 50.0% |
| qwen_program_final_gated | 101 | 95.8% | 91.7% | 95.8% | 100.0% | 0.769 | 8.3% | 0.0% | 50.0% |
| qwen_program_final_shuffled | 101 | 37.5% | 91.7% | 95.8% | -1300.0% | 0.720 | 62.5% | 59.1% | 0.0% |
| feature_mlp | 202 | 91.7% | 91.7% | 95.8% | 0.0% | 0.993 | 0.0% | 0.0% | 0.0% |
| trace_transformer | 202 | 91.7% | 91.7% | 95.8% | 0.0% | 0.828 | 0.0% | 0.0% | 0.0% |
| qwen_program_final | 202 | 87.5% | 91.7% | 95.8% | -100.0% | 0.769 | 16.7% | 9.1% | 50.0% |
| qwen_program_final_gated | 202 | 95.8% | 91.7% | 95.8% | 100.0% | 0.769 | 8.3% | 0.0% | 50.0% |
| qwen_program_final_shuffled | 202 | 62.5% | 91.7% | 95.8% | -700.0% | 0.766 | 37.5% | 31.8% | 0.0% |
Training Dynamics
| arm | seed | epoch | train_loss | val_accuracy | val_damage_rate | val_recovery_rate | val_utility |
|---|---|---|---|---|---|---|---|
| qwen_program_final_shuffled | 101 | 8 | 3.711 | 0.0% | 100.0% | 0.0% | -0.75 |
| qwen_program_final | 101 | 8 | 0.71 | 75.0% | 5.6% | 16.7% | 0.75 |
| feature_mlp | 101 | 8 | 0.062 | 75.0% | 0.0% | 0.0% | 0.75 |
| trace_transformer | 101 | 8 | 1.435 | 75.0% | 0.0% | 0.0% | 0.75 |
| trace_transformer | 202 | 8 | 1.372 | 75.0% | 0.0% | 0.0% | 0.75 |
| feature_mlp | 202 | 8 | 0.067 | 75.0% | 0.0% | 0.0% | 0.75 |
| qwen_program_final | 202 | 8 | 0.918 | 75.0% | 5.6% | 16.7% | 0.75 |
| qwen_program_final_shuffled | 202 | 8 | 3.657 | 0.0% | 100.0% | 0.0% | -0.75 |
Interpretation
The decisive measurement is not raw accuracy alone. The report separates candidate coverage from selector capture, because a selector cannot choose a correct program that is absent from its candidate shortlist. A useful selector should capture a stable fraction of the available oracle gap, rank correct candidates above wrong candidates, make nontrivial repairs, and avoid damaging already-correct base programs. Easy and hard rows test whether failures are caused by representation readability or by candidate sets where the target answer is not distinguishable from simple shortlist statistics. In this run the easy/hard split is itself diagnostic: only one standard_L24 group was easy under the claimed-output frequency criterion, so most of the evaluated repair decisions sit in the hard regime.
Artifacts
Run directory:
/workspace/experiments/qwen_readable_candidate_verifier/runs/main_qwen_readable_candidate_verifier_v5Large embeddings:
/workspace/large_artifacts/qwen_readable_candidate_verifier/embeddings/main_qwen_readable_candidate_verifier_v1Large checkpoints:
/workspace/large_artifacts/qwen_readable_candidate_verifier/checkpoints/main_qwen_readable_candidate_verifier_v5Metrics CSV:
/workspace/experiments/qwen_readable_candidate_verifier/reports/metrics.csv- Candidate summary CSV:
/workspace/experiments/qwen_readable_candidate_verifier/reports/candidate_summary.csv
Experiment log 11
Show the running log (11 entries, 2026-06-25)
Objective
Test whether a frozen or lightly trained Qwen reader can select correct repair candidates when candidates are rendered as readable task code, candidate code, claimed final values, and optional execution traces. The target answer and target state are never provided to the verifier at inference.
Design Commitments
- Fresh experiment directory:
experiments/qwen_readable_candidate_verifier/. - Large artifacts separated under:
large_artifacts/qwen_readable_candidate_verifier/. - Candidate pool and task reconstruction are deterministic.
- Main report is standalone and does not require external experiment context.
- Report both ungated and validation-gated decisions.
- Report selector accuracy, oracle-gap capture, shortlist-oracle capture, ranking AUC, changed fraction, damage rate, and recovery rate.
- Include readable trace-field ablations: task-value, program equivalence, program plus claimed final, program plus full trace, trace-corrupted, candidate-only, prompt-only, and majority-by-claimed-output controls.
- Split selector quality by easy versus hard candidate groups, where difficulty is determined by how many shortlist candidates share the target answer.
Iterations
2026-06-25 Scaffold
Created the standalone directory, large-artifact directories, README, and experiment log.
2026-06-25 Smoke Iteration 1
Ran smoke_qwen_readable_candidate_verifier_v1 with readable task/candidate rendering, frozen Qwen scoring, one trained Qwen head, feature/trace controls, charts, Markdown, and HTML report generation.
The smoke path completed, but the primary value-recompute arm used a first-token numeric approximation. Qwen's tokenizer split all residue strings in this setup, which collapsed zero_task_value into an uninformative score. I patched the runner to score exact numeric continuation log-likelihood over all residues instead.
2026-06-25 Smoke Iteration 2
Ran smoke_qwen_readable_candidate_verifier_v2 with rebuilt embeddings and exact continuation scoring for task_value.
Smoke result on standard_L24:
- Base no-repair: 41.7%.
- Shortlist oracle: 50.0%; full-pool oracle: 100.0%.
zero_task_value: 0.0%, ranking AUC 0.381, fully destructive on the tiny sample.zero_program_final,zero_program_trace,zero_trace_corrupt, andqwen_program_final: tied base with ranking AUC 0.500.- Feature and trace controls showed high AUC on the tiny sample but did not improve commit accuracy.
Smoke verified the data path, exact value scoring, trained-head path, easy/hard metrics, charts, and report generation.
2026-06-25 Main Iteration 1
Started main_qwen_readable_candidate_verifier_v1 with all readable modes, shortlist size 64, two critic seeds, and max_length=1024.
The exact task_value mode completed and cached, but prompt-length inspection showed that full-trace prompts are about 1,400 tokens. A 1,024-token context would truncate the trace arms, so I interrupted during program_equiv and reran with a larger context.
Measured readable token lengths on the main selected set:
task_value: max about 506 tokens.program_equiv: max about 990 tokens.program_final: max about 1,000 tokens.program_trace: max about 1,402 tokens.trace_corrupt: max about 1,406 tokens.candidate_only: max about 515 tokens.prompt_only: max about 505 tokens.
2026-06-25 Main Iteration 2
Started main_qwen_readable_candidate_verifier_v2 with max_length=1536, loading the completed task_value cache. program_equiv and program_final completed and cached. The full-trace hidden-state embedding path was too slow for a practical complete run, so I interrupted before finishing program_trace.
2026-06-25 Main Iteration 3
Patched the runner so modes not listed in --train_qwen_modes are scored as zero-only Qwen yes/no arms instead of full hidden-state embedding arms. This keeps the trace and corrupted-trace controls while avoiding unnecessary learned embedding caches. I first set train_qwen_modes=program_final.
The first v3 attempt revealed a control-flow bug: zero-only stores were being overwritten by the old hidden-state branch. I fixed the branch and also changed final-token yes/no scoring to call the base model and apply lm_head only to the final hidden state, avoiding full-sequence vocabulary logits.
2026-06-25 Main Iteration 4
Reran after the scorer optimization. The optimized scorer reduced memory use substantially, so I interrupted early and restarted with a larger Qwen batch size.
2026-06-25 Main Iteration 5
Completed main_qwen_readable_candidate_verifier_v5 with:
embedding_run_name=main_qwen_readable_candidate_verifier_v1max_length=1536qwen_batch_size=64train_qwen_modes=program_final- two critic seeds: 101 and 202
Runtime for the completed v5 run was 9,380.221 seconds. Large caches were written under large_artifacts/qwen_readable_candidate_verifier/embeddings.
Main result on standard_L24:
- Base no-repair: 44.4%.
- Shortlist oracle: 75.0%; full-pool oracle: 90.3%.
- Majority-by-claimed-output: 9.7%, destructive.
- Frozen
zero_program_equiv: 50.0%, 12.1% oracle-gap capture, AUC 0.677. - Frozen
zero_program_trace: 8.3%, destructive, AUC 0.620. - Frozen
zero_trace_corrupt: 1.4%, destructive, AUC 0.486. - Trained
qwen_program_final: 47.2%, 6.1% oracle-gap capture. - Validation-gated
qwen_program_final: 51.4%, 15.2% oracle-gap capture, zero damage rate, 12.5% recovery rate. - Shuffled-label trained control was destructive on average.
Interpretation: readable program text does expose usable signal that was absent from the dense/compact candidate channel, but the gain is modest. Full readable trace text did not help; it made the frozen selector highly destructive, and corrupting the trace made it worse. The easy/hard split was mostly degenerate under the claimed-output frequency criterion: only one standard_L24 group was easy, so nearly all measured decisions were in the hard regime.
Figures 5
Data files 15
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/candidate_summary.csv7.2 kBreports/metrics.csv191 kBruns/main_qwen_readable_candidate_verifier_v1/candidate_summary.csv7.2 kBruns/main_qwen_readable_candidate_verifier_v2/candidate_summary.csv7.2 kBruns/main_qwen_readable_candidate_verifier_v3/candidate_summary.csv7.2 kBruns/main_qwen_readable_candidate_verifier_v4/candidate_summary.csv7.2 kBruns/main_qwen_readable_candidate_verifier_v5/candidate_summary.csv7.2 kBruns/main_qwen_readable_candidate_verifier_v5/metrics.csv193 kBruns/main_qwen_readable_candidate_verifier_v5/run_summary.json102 Bruns/smoke_qwen_readable_candidate_verifier_v1/candidate_summary.csv2.1 kBruns/smoke_qwen_readable_candidate_verifier_v1/metrics.csv34 kBruns/smoke_qwen_readable_candidate_verifier_v1/run_summary.json99 Bruns/smoke_qwen_readable_candidate_verifier_v2/candidate_summary.csv2.1 kBruns/smoke_qwen_readable_candidate_verifier_v2/metrics.csv35 kBruns/smoke_qwen_readable_candidate_verifier_v2/run_summary.json99 B
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.