Qwen Tail Repair Stability Critic
The one idea you need
A model writes 24-step number puzzles that usually run right until the final steps break. The system generates many rewrites of those last steps, then trains a judge to pick the winning rewrite from summary statistics alone, like a spellchecker choosing a correction without rereading the sentence.
The question
If a model's step-by-step answers are usually right until the last few steps, can a trained judge pick a corrected version without ever seeing the answer?
What we found
No. A correct fix sat among the candidate rewrites for about nine in ten programs, but the trained judge could not tell which rewrite was right from summary statistics alone, so it played safe and edited nothing, staying at 45% correct, exactly the no-repair mark. A picker allowed to peek at the answer hit 91% from the very same fixes.
Why it matters
To repair a model's reasoning, the judge must actually read each candidate rewrite against the original prompt, not score it from aggregate confidence statistics about the generator. Those summary features simply cannot spot the correct fix, so the picker safely does nothing.
On this page
Results at a glance 5
How to read
Each bar is one way of choosing a fix; height is the share of 24-step programs that run correctly start to finish, so taller is better. Bars: no repair, two learned pickers, a scrambled-label control, and two pickers allowed to peek at the answer.
Takeaway → The two learned-picker bars sit exactly on the no-repair bar while the peek-allowed answer bar towers far above, so a good fix exists but is never chosen.
Data table
| selection arm | mean exact accuracy |
|---|---|
| no repair | 44.8% |
| critic v1 | 44.8% |
| critic v2 (focused) | 44.8% |
| shuffled-label control | 30.4% |
| state oracle | 50.5% |
| answer oracle | 91.1% |
Numbers from report table (reports/qwen_tail_repair_stability_critic_report.md, Stability Gate and Split Results tables)
Technical framing
Tail-repair critic never beats no-repair on standard length-24 programs — Both learned critics collapse to no-repair (44.8%) while an answer oracle over the same candidates reaches 91.1% — selection, not coverage, fails.
How to read
Four batches of programs, three bars each: leave the program as-is, the learned picker, and a picker allowed to peek at the answer. Height is the share that run correctly, so taller is better.
Takeaway → The learned-picker bar is identical to leave-as-is on every batch, always sitting far below the peek-allowed bar it could have matched.
Data table
| length-24 evaluation split | no repair | focused critic (v2) | answer oracle |
|---|---|---|---|
| standard | 44.8% | 44.8% | 91.1% |
| held-out | 40.1% | 40.1% | 88% |
| paired | 66.7% | 66.7% | 94.3% |
| paired held-out | 35.4% | 35.4% | 90.1% |
Numbers from report table (reports/qwen_tail_repair_stability_critic_report.md, Split Results table)
Technical framing
Oracle repair gap persists on every evaluation split — The learned critic matches no-repair exactly on all splits; the answer oracle shows 28-55 point headroom it fails to capture.
How to read
For each batch, one bar is the share of programs where a correct ending-fix already exists among the tried edits; the other is the share already correct with no fix. Taller means more fixable.
Takeaway → The fix-exists bar sits near ninety percent everywhere, roughly double the already-correct bar, so the raw material for repair is abundant.
Data table
| length-24 evaluation split | groups with a correct-answer candidate | base program already correct |
|---|---|---|
| standard | 91.1% | 44.8% |
| held-out | 88% | 40.1% |
| paired | 94.3% | 66.7% |
| paired held-out | 90.1% | 35.4% |
| paraphrase | 99% | 85.4% |
Numbers from report table (reports/qwen_tail_repair_stability_critic_report.md, Candidate Coverage table)
Technical framing
Candidate coverage is not the bottleneck: repairable groups far exceed base accuracy — Local tail edits contain a correct-answer fix for about 90% or more of groups on every split — roughly double base accuracy on all but the paraphrase split.
How to read
Three source generators, two bars each: leaving programs as-is versus a picker allowed to peek at the answer. Height is the share that run correctly, so taller is better.
Takeaway → The peek-allowed picker lifts the weakest source from near zero to about 78%, so nearly all the unclaimed gains sit on that one weak source.
Data table
| Frozen source-compiler seed (standard_L24) | No repair | Answer-oracle repair |
|---|---|---|
| Seed 123 | 1.6% | 78.1% |
| Seed 456 | 42.2% | 96.9% |
| Seed 789 | 90.6% | 98.4% |
Numbers from experiments/qwen_tail_repair_stability_critic/reports/selection_metrics.csv
Technical framing
Oracle repair would mostly rescue the weakest source seed on standard programs — Oracle candidate selection lifts the worst seed from 1.6% to 78.1%, so nearly all the unclaimed repair headroom sits on weak seeds.
How to read
Four batches of programs. Bars show how often a scrambled-label control edits programs and how often that harms them, next to how often the real picker edits. Lower editing by the real picker means it stayed hands-off.
Takeaway → The scrambled control changes and damages roughly a third of programs while the real picker edits zero, showing its stillness is a deliberate retreat, not a broken pipeline.
Data table
| Evaluation split (v2 focused run) | Shuffled control: programs changed | Shuffled control: damage rate | True critic: programs changed |
|---|---|---|---|
| standard_L24 | 29.5% | 27.6% | 0% |
| heldout_L24 | 43.2% | 36.6% | 0% |
| paired_L24 | 36.6% | 42.2% | 0% |
| paired_heldout_L24 | 38.7% | 37.9% | 0% |
Numbers from report table
Technical framing
Shuffled-label control actively edits and damages programs; the true critic edits nothing — The pipeline can make edits (shuffled control damages 28-42% of programs); the true critic's zero changes is a learned retreat, not a dead path.
In the author’s words from the Overview · “Result”
The repair candidate set had high oracle coverage, but the learned feature critic did not improve the deployable selector. On standard length-24 examples, no repair, the iteration-1 critic, and the iteration-2 focused critic all ended at 44.8% mean exact accuracy with 44.6% source-seed standard deviation. The answer-oracle candidate selector reached 91.1%, so candidate coverage was not the limiting factor. The stability gate failed because mean accuracy and source-seed variance were unchanged.
Overview
Standalone experiment for testing whether a learned tail-repair critic can turn mostly-correct length-24 modular programs into stable exact executable programs.
The experiment uses frozen source compilers as generators, enumerates local tail edits around each generated program, labels candidates with exact execution trace checks, and trains a small critic to select repair candidates without using target answers or target states at inference.
Large checkpoints and cached candidate groups live under:
/workspace/large_artifacts/qwen_tail_repair_stability_critic/
Reports
- Markdown:
reports/qwen_tail_repair_stability_critic_report.md - HTML:
reports/qwen_tail_repair_stability_critic_report.html - Figures:
reports/figures/
Result
The repair candidate set had high oracle coverage, but the learned feature critic did not improve the deployable selector. On standard length-24 examples, no repair, the iteration-1 critic, and the iteration-2 focused critic all ended at 44.8% mean exact accuracy with 44.6% source-seed standard deviation. The answer-oracle candidate selector reached 91.1%, so candidate coverage was not the limiting factor. The stability gate failed because mean accuracy and source-seed variance were unchanged.
Report
Rendered from reports/qwen_tail_repair_stability_critic_report.md
Summary
This standalone experiment tested whether a learned tail-repair critic can stabilize length-24 executable modular programs. The result is a clear negative for the tested critic class.
The candidate set had a large oracle ceiling: on standard_L24, no repair achieved 44.8%, while the best answer-oracle candidate reached 91.1%. However, both learned critic variants selected the no-repair base program on every standard example selected by their best validation checkpoint. The stability gate failed: mean accuracy stayed at 44.8% and source-seed standard deviation stayed at 44.6%.
Experimental Setup
Three frozen source compiler snapshots generated length-24 modular programs under modulus 97. For each generated program, the experiment enumerated local edits in the last eight slots. Candidate labels were computed offline by exact execution, but learned critics could only use non-target features at selection time.
Two learned critic iterations were run:
v1_unweighted: aggregate candidate features plus frozen Qwen context, trained on all groups with answer and state labels.v2_focused: adds explicit tail-slot candidate details and trains only the answer critic on a recovery-balanced subset, with checkpoint selection penalizing damage.
Controls:
base: no repair.oracle_answer: highest-prior candidate that executes to the correct final answer.oracle_state: highest-prior candidate with exact state trajectory.shuffled_answer/shuffled_state: same training pipeline with randomized labels.
Candidate Coverage
| split | groups | avg_candidates | answer_positive_fraction | state_positive_fraction | base_executor_accuracy |
|---|---|---|---|---|---|
| heldout_L24 | 192 | 177 | 88.0% | 46.9% | 40.1% |
| paired_L24 | 192 | 177 | 94.3% | 67.7% | 66.7% |
| paired_heldout_L24 | 192 | 177 | 90.1% | 42.2% | 35.4% |
| paraphrase_L24 | 192 | 177 | 99.0% | 85.4% | 85.4% |
| standard_L24 | 192 | 177 | 91.1% | 50.0% | 44.8% |
| train_mixed_L24 | 480 | 177 | 98.8% | 89.2% | 88.5% |
| val_mixed_L24 | 144 | 177 | 97.2% | 86.1% | 86.1% |
The key fact is that coverage is not the bottleneck for final-answer repair: standard_L24 has answer-positive candidates for 91.1% of groups. State-exact repair is harder at 50.0%, but still nontrivial.
Stability Gate
| run_key | arm | critic_seed | source_seed_count | mean_executor_accuracy | std_executor_accuracy | min_executor_accuracy | max_executor_accuracy | mean_changed_fraction | mean_damage_rate | mean_recovery_rate |
|---|---|---|---|---|---|---|---|---|---|---|
| v1_unweighted | base | -1 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v1_unweighted | critic_answer | 101 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v1_unweighted | critic_answer | 202 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v1_unweighted | critic_answer | 303 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v1_unweighted | critic_state | 101 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v1_unweighted | critic_state | 202 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v1_unweighted | critic_state | 303 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v1_unweighted | oracle_answer | -1 | 3 | 91.1% | 11.3% | 78.1% | 98.4% | 46.4% | 0.0% | 85.2% |
| v1_unweighted | oracle_state | -1 | 3 | 50.5% | 47.7% | 1.6% | 96.9% | 5.7% | 0.0% | 28.5% |
| v2_focused | critic_answer | 101 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v2_focused | critic_answer | 202 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v2_focused | critic_answer | 303 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v2_focused | shuffled_answer | 101 | 3 | 29.2% | 49.2% | 0.0% | 85.9% | 38.5% | 35.1% | 0.0% |
| v2_focused | shuffled_answer | 202 | 3 | 44.8% | 44.6% | 1.6% | 90.6% | 0.0% | 0.0% | 0.0% |
| v2_focused | shuffled_answer | 303 | 3 | 17.2% | 27.1% | 1.6% | 48.4% | 50.0% | 47.6% | 0.0% |
The learned true-label critics did not improve mean accuracy or reduce spread. In v1, the selected true critics made zero changes. In v2, focused training learned to make edits during training, but the best validation checkpoint was still the no-change checkpoint for all three true critic seeds.
Split Results
| split | run_key | arm | mean_executor_accuracy | std_executor_accuracy | mean_changed_fraction | mean_damage_rate | mean_recovery_rate |
|---|---|---|---|---|---|---|---|
| standard_L24 | v1_unweighted | base | 44.8% | 44.6% | 0.0% | 0.0% | 0.0% |
| standard_L24 | v2_focused | critic_answer | 44.8% | 44.6% | 0.0% | 0.0% | 0.0% |
| standard_L24 | v2_focused | shuffled_answer | 30.4% | 40.3% | 29.5% | 27.6% | 0.0% |
| standard_L24 | v1_unweighted | oracle_answer | 91.1% | 11.3% | 46.4% | 0.0% | 85.2% |
| heldout_L24 | v1_unweighted | base | 40.1% | 38.5% | 0.0% | 0.0% | 0.0% |
| heldout_L24 | v2_focused | critic_answer | 40.1% | 38.5% | 0.0% | 0.0% | 0.0% |
| heldout_L24 | v2_focused | shuffled_answer | 31.2% | 35.5% | 43.2% | 36.6% | 0.4% |
| heldout_L24 | v1_unweighted | oracle_answer | 88.0% | 10.2% | 47.9% | 0.0% | 82.5% |
| paired_L24 | v1_unweighted | base | 66.7% | 28.2% | 0.0% | 0.0% | 0.0% |
| paired_L24 | v2_focused | critic_answer | 66.7% | 28.2% | 0.0% | 0.0% | 0.0% |
| paired_L24 | v2_focused | shuffled_answer | 41.3% | 37.7% | 36.6% | 42.2% | 0.0% |
| paired_L24 | v1_unweighted | oracle_answer | 94.3% | 7.4% | 27.6% | 0.0% | 89.2% |
| paired_heldout_L24 | v1_unweighted | base | 35.4% | 34.0% | 0.0% | 0.0% | 0.0% |
| paired_heldout_L24 | v2_focused | critic_answer | 35.4% | 34.0% | 0.0% | 0.0% | 0.0% |
| paired_heldout_L24 | v2_focused | shuffled_answer | 25.0% | 31.9% | 38.7% | 37.9% | 0.2% |
| paired_heldout_L24 | v1_unweighted | oracle_answer | 90.1% | 7.4% | 54.7% | 0.0% | 86.6% |
The oracle gap is large on every split. The learned focused critic does not close it.
Repair Behavior
The v2 shuffled control is destructive, which confirms that the control is active. The true focused critic avoids that collapse, but it does so by falling back to no repair rather than by learning safe recovery. Later epochs make more edits, but validation accuracy falls because damage outpaces recovery.
Interpretation
The experiment separates reachability from selection. Tail-local candidate enumeration can often recover the right final answer, especially on the weak source seeds. The learned critic tested here cannot reliably identify those recoveries from frozen context plus candidate features.
The failure is not that the tail-repair idea has no ceiling. The failure is that this critic is not grounded enough in the prompt-program relation. Aggregate features were insufficient, and even explicit tail-slot details plus recovery-focused training did not produce a deployable selector.
Next Diagnostic
The next version should score each candidate with a real candidate-conditioned forward pass: prompt plus dense candidate program/trace representation into Qwen, then a scalar value head. The scorer needs to compare the concrete candidate program against the prompt, not infer correctness from summary statistics around the source compiler logits.
Artifacts
- Experiment root:
/workspace/experiments/qwen_tail_repair_stability_critic/ - Large artifacts:
/workspace/large_artifacts/qwen_tail_repair_stability_critic/ - Main v1 run:
/workspace/experiments/qwen_tail_repair_stability_critic/runs/main_tail_repair_critic_v1/ - Main v2 run:
/workspace/experiments/qwen_tail_repair_stability_critic/runs/main_tail_repair_critic_focus_v2/ - Combined summary CSV:
/workspace/experiments/qwen_tail_repair_stability_critic/reports/combined_summary_by_source_seed.csv
Experiment log 8
Show the running log (8 entries, 2026-06-25)
Objective
Test whether a learned tail-repair critic can improve exact length-24 executable accuracy while reducing seed-to-seed variance. The experiment is standalone: its report describes only this experiment's setup, results, and conclusions.
Design Commitments
- Fresh experiment directory:
experiments/qwen_tail_repair_stability_critic/. - Large artifacts separated under
large_artifacts/qwen_tail_repair_stability_critic/. - Multi-seed result reporting from the start.
- Stability is a co-equal gate: mean accuracy alone is not enough.
- Candidate selection cannot use target answers or target states at inference.
- Include shuffled-label control.
- Produce Markdown and HTML reports with figures.
Iterations
2026-06-25 Initial Scaffold
Created standalone experiment directories and recorded the intended design.
2026-06-25 Smoke Iteration 1
Ran a one-source smoke suite. Candidate export, critic training, control training, chart generation, Markdown report generation, and HTML report generation completed.
Findings from the smoke:
- The true critic path can train and evaluate.
- The shuffled-label path is active and can damage performance, which is useful as a control signal.
- The initial report aggregation duplicated base/prior rows once per label mode.
- The first accuracy chart selected the global minimum critic seed, which hid learned-critic bars when base rows used critic seed
-1.
Patched aggregation and chart filtering before any main run.
2026-06-25 Main Iteration 1
Ran main_tail_repair_critic_v1 across three frozen source compiler seeds.
Candidate coverage was high enough to make the repair task meaningful:
standard_L24base executable accuracy: 44.8%.standard_L24answer-positive candidate fraction: 91.1%.standard_L24state-exact candidate fraction: 50.0%.- Average candidate count: 177 per example.
The learned unweighted critics did not move off the base candidate. Answer and state critics across seeds all matched the no-repair baseline on standard L24 with zero changed candidates. The candidate ceiling was high, but the critic input was too aggregate: it did not expose enough concrete tail-slot program content to identify the correct edit.
Patched the runner for iteration 2:
- Added explicit fixed-window tail-slot features for each candidate.
- Added
train_focus=recoverable_balanced, which emphasizes base-wrong groups with at least one positive candidate while keeping preserve/impossible examples for damage control. - Added utility checkpoint selection: validation accuracy plus recovery bonus minus damage penalty.
- Added
cache_run_nameso iteration 2 can reuse the expensive v1 candidate caches without re-exporting Qwen outputs.
2026-06-25 Main Iteration 2
Ran main_tail_repair_critic_focus_v2 from the cached candidate groups with answer labels only, explicit tail-slot features, recoverable-balanced training, and utility-selected checkpoints.
The focused critics learned to make edits in later epochs, but those edits damaged already-correct programs more often than they recovered wrong programs. The validation utility therefore selected the initial no-change checkpoint for all three true-label critic seeds.
Final standard-L24 gate:
- No repair: 44.8% mean exact accuracy, 44.6% source-seed std.
- Iteration-1 critic: 44.8% mean exact accuracy, 44.6% source-seed std.
- Iteration-2 focused critic: 44.8% mean exact accuracy, 44.6% source-seed std.
- Shuffled-label focused critic: 30.4% mean exact accuracy.
- Answer-oracle candidate selector: 91.1% mean exact accuracy.
Conclusion: candidate coverage is not the bottleneck in this task, but this small feature-based critic did not learn a safe selector. The stability gate failed because neither mean exact accuracy nor source-seed variance improved.
Final Artifacts
- Markdown report:
reports/qwen_tail_repair_stability_critic_report.md - HTML report:
reports/qwen_tail_repair_stability_critic_report.html - Figures:
reports/figures/ - Large candidate caches and critic checkpoints:
/workspace/large_artifacts/qwen_tail_repair_stability_critic/
Figures 8
Data files 13
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/candidate_group_summary.csv621 Breports/combined_summary_by_source_seed.csv15 kBreports/selection_metrics.csv39 kBreports/summary_by_source_seed.csv4.3 kBruns/main_tail_repair_critic_focus_v2/candidate_group_summary.csv629 Bruns/main_tail_repair_critic_focus_v2/run_summary.json70 Bruns/main_tail_repair_critic_focus_v2/selection_metrics.csv40 kBruns/main_tail_repair_critic_v1/candidate_group_summary.csv629 Bruns/main_tail_repair_critic_v1/run_summary.json70 Bruns/main_tail_repair_critic_v1/selection_metrics.csv87 kBruns/smoke_tail_repair_critic/candidate_group_summary.csv367 Bruns/smoke_tail_repair_critic/run_summary.json65 Bruns/smoke_tail_repair_critic/selection_metrics.csv18 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).