Qwen Verifier-Guided Slot Repair

Finished2026-06-22imported · line YEvidence-Conditioned Selection Interpretability and Diagnostics Posttraining and Adaptation Reliability and Safety+1 moreGitHub ↗

The one idea you need

Think of a 24-step calculation where each step updates a running number. The model copies the recipe from memory, but a step or two comes out wrong and wrecks the final total. A checker that knows the correct running number at each step can pinpoint and swap the bad steps.

The question

When a small model builds a long 24-step calculation and gets it almost right, can checking each step let a few small edits fix the final answer?

What we found

Yes, mostly, but with a catch. A small model copying 24-step calculations got only about 27% exactly right on its own. A checker that knows the correct running number after every step, allowed to swap one or two bad steps, lifted that to 91%; a single swap already reached about 70%. The catch: that step-by-step answer key isn't available in real use.

Why it matters

When a small model nearly solves a long task, its errors are usually a couple of local slips, not gibberish. A cheap search over small edits plus a strong step-by-step checker recovers most of it, so building that checker is the high-value move.

Exactly-correct 24-step runs, before vs after fixing27% → 91%share run perfectly, allowing one or two step swaps

Fixed by a single swapabout 70%changing just one step already recovers most failures

True calculation rebuilt90%genuine fixes, not a lucky matching final answer

Accuracy as length grows to 24 steps (before fixing)86% → 25%why long chains fail: one bad step ruins the whole answer

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 3

Fixing one or two steps recovers most failed calculations

How to read

Bars are grouped by how many steps the checker may swap: none, one, or two. In each group one bar counts runs with the right final answer, the other counts runs whose whole calculation matches the true one. Taller is better.

Takeaway → Bars climb from about 27% with no fixing to 91% with two swaps, and the two heights stay matched, so it rebuilds the real calculation, not just a lucky answer.

Data table

repair budget	exact final answer	exact program
no repair	27.5%	27.5%
one-edit repair	70.5%	69.7%
two-edit repair	91%	90%

Numbers from analysis/selected_retest_metrics.csv, analysis/selected_retest_metrics_one_edit.csv (fresh_paired_len24)

Technical framing

Verifier-guided repair on fresh length-24 programs — State-verifier repair lifts fresh length-24 exact execution from 27.5% to 91.0%, and usually recovers the true program, not just the answer.

Longer calculations fail far more, but fixing recovers most

How to read

The horizontal axis is how many steps the calculation has; the vertical axis is the share of runs that come out exactly right. One line is before fixing, one after. Higher is better.

Takeaway → Before fixing, accuracy slides from about 86% at four steps to 25% at 24 steps, while after fixing it stays near 82% even at the longest length.

Data table

program length	before repair	after repair
4	85.9%	100%
8	60.9%	97.7%
12	58.6%	97.7%
24	25%	82%

Numbers from analysis/summary.md final-metrics table (paired_len4-24, run main_state_w025_repair_s900)

Technical framing

Repair headroom grows with program length (paired eval) — Unrepaired accuracy collapses as programs get longer, but two-edit verifier repair keeps most of it recoverable even at length 24.

The repair gain holds across different prompt wordings

How to read

Bars are grouped by prompt style: plain wording, reworded, and matched pairs shown both ways. In each group one bar is before fixing, one after. Higher is better.

Takeaway → Every wording jumps from roughly a quarter of runs correct before fixing to about 86 to 91% after, so the gain is not a quirk of one phrasing.

Data table

prompt split	before repair	after repair
standard	27.3%	88.3%
paraphrase	23.8%	86.3%
paired	27.5%	91%

Numbers from analysis/selected_retest_metrics.csv

Technical framing

Fresh length-24 retest by prompt style: before vs after repair — The 27.5% -> 91% repair gain holds across standard and paraphrased prompts, so it is not an artifact of one phrasing.

In the author’s words from the Report · “Abstract”

On fresh paired length-24 standard/paraphrase programs, the selected checkpoint scored 27.5% exact execution before repair and 91.0% after top-3/two-edit state-verifier repair. Repaired program exact was 90.0%, and repaired paired state consistency was 92.2%. A one-edit ablation reached 70.5%, showing that many failures are one local slot edit away and more are recoverable with two-slot edits. This is not a deployable inference method yet. The primary verifier uses the true intermediate state trajectory. The result shows that the compiler's errors are often locally repairable, and that a learned or task-native verifier is a high-value next target.

Overview

Standalone experiment testing whether exact-answer verifier search can repair near-miss compiled programs from a QLoRA-adapted Qwen/Qwen3-4B numeric-copy compiler.

The compiler reads prompt hidden states, copies an initial value, operation sequence, and operation arguments from token-level maps, and executes the copied program modulo 97. The repair evaluator searches small local edits around the copied program and keeps the highest-prior candidate whose intermediate state trajectory satisfies the verifier.

Main Question

How much length-24 exact execution is recoverable if the compiler is almost right and a state-trajectory verifier can select locally repaired candidates?

Main Result

On fresh paired length-24 programs, the selected Qwen compiler scored 27.5% exact execution before repair and 91.0% after top-3/two-edit state-verifier repair. Repaired program exact was 90.0%, so the search usually recovered the true program rather than only an equivalent final answer. A one-edit ablation reached 70.5%.

The result is a headroom result, not a deployable inference recipe: the primary verifier uses the true intermediate state trajectory.

Layout

src/qwen_verifier_guided_slot_repair_experiment.py  training, evaluation, and repair search
src/analyze_qwen_verifier_guided_slot_repair.py     aggregation, selected checkpoint table, and plots
src/evaluate_selected_qwen_verifier_guided_slot_repair.py
                                                     fresh retest for selected checkpoints
runs/                                                lightweight JSON and CSV outputs
analysis/                                            aggregate CSVs and generated figures
reports/                                             experiment log and standalone write-up
checkpoint_manifest.csv                              generated list of large checkpoint files

Large checkpoints are stored outside the experiment directory:

large_artifacts/qwen_verifier_guided_slot_repair/checkpoints/

Main Metrics

executor_accuracy: exact final answer from the unrepaired compiled program.
repair_executor_accuracy: exact final answer after verifier-guided local repair.
program_exact: exact unrepaired compiled program.
repair_program_exact: exact repaired compiled program.
repair_found_fraction: fraction of examples with at least one state-verifier-satisfying local candidate.
repair_changed_fraction: fraction where repair selected a different program.
repair_pair_state_consistency: paired standard/paraphrase consistency after repair.

Reading Order

reports/qwen_verifier_guided_slot_repair_experiment_log.md
analysis/summary.md
analysis/selected_checkpoints.csv
analysis/selected_retest_metrics.csv
analysis/selected_retest_metrics_one_edit.csv
analysis/final_metrics.csv
reports/qwen_verifier_guided_slot_repair_paper.md

Report

Rendered from reports/qwen_verifier_guided_slot_repair_paper.md

Abstract

This experiment tests whether local verifier-guided search can recover exact long-chain execution from near-miss programs compiled by a QLoRA-adapted Qwen/Qwen3-4B model. Each prompt describes modular arithmetic updates. The compiler copies an initial value, operation sequence, and operation arguments from Qwen hidden states, then an invisible executor runs the copied program modulo 97.

The main result is positive as a headroom result. On fresh paired length-24 standard/paraphrase programs, the selected checkpoint scored 27.5% exact execution before repair and 91.0% after top-3/two-edit state-verifier repair. Repaired program exact was 90.0%, and repaired paired state consistency was 92.2%. A one-edit ablation reached 70.5%, showing that many failures are one local slot edit away and more are recoverable with two-slot edits.

This is not a deployable inference method yet. The primary verifier uses the true intermediate state trajectory. The result shows that the compiler's errors are often locally repairable, and that a learned or task-native verifier is a high-value next target.

Question

If a Qwen-attached compiler is mostly right but brittle over long chains, how much exact execution can be recovered by searching a small neighborhood of locally plausible slot edits?

The tested repair space includes:

alternate initial values from the compiler logits;
alternate operations from the compiler logits;
alternate arguments from the compiler logits;
same-step operation and argument edits;
pairs of argument edits.

The primary verifier requires the full intermediate state trajectory to match the true trajectory. This avoids a failure mode of final-answer-only verification: with many candidates and only 97 possible final answers, spurious final-answer matches are common.

Method

The main run used Qwen/Qwen3-4B with 4-bit QLoRA adapters and a numeric-copy compiler. Training used a four-stage curriculum:

Stage	Length range	Steps
short	1-4	200
medium	1-8	200
train	1-12	200
long	8-24	300

The compiler was trained with trace supervision, executor loss, token-position selection losses, paired standard/paraphrase batches, and light intermediate-state supervision with weight 0.25. Checkpoints were saved at evaluation points and selected by validation paired_len24_repair_executor_accuracy.

Repair search used:

Parameter	Value
verifier	full state trajectory
candidate top-k per slot	3
maximum edits	2
max argument-pair slots	24

The fresh retest used 256 standard length-24 programs, 256 paraphrase length-24 programs, and 256 paired length-24 latent programs rendered in both forms.

Results

Selected Validation Checkpoint

The selected checkpoint was step 800.

Split	Unrepaired exact	Repaired exact	Repaired program exact	Repair found	Repair changed
Standard L24	37.5%	85.9%	85.9%	85.9%	48.4%
Paraphrase L24	20.3%	81.2%	79.7%	81.2%	60.9%
Paired L24	30.5%	88.3%	88.3%	88.3%	57.8%

Step 900 was worse on the primary repaired paired metric: 82.0%. Checkpoint selection therefore mattered.

Fresh Retest

Split	Unrepaired exact	Repaired exact	Repaired program exact	Repaired prefix	Repair found
Fresh standard L24	27.3%	88.3%	87.9%	90.7%	88.3%
Fresh paraphrase L24	23.8%	86.3%	85.5%	88.6%	86.3%
Fresh paired L24	27.5%	91.0%	90.0%	93.4%	91.0%

On the paired split, unrepaired paired state consistency was 69.1%; repaired paired state consistency was 92.2%. Repaired pair both-correct was 89.5%.

One-Edit Ablation

The same selected checkpoint was retested with the same top-3 candidates but only one allowed edit.

Split	Unrepaired exact	One-edit repaired exact	Two-edit repaired exact
Fresh standard L24	27.3%	64.1%	88.3%
Fresh paraphrase L24	23.8%	59.8%	86.3%
Fresh paired L24	27.5%	70.5%	91.0%

One-edit repair recovers a large fraction of failures, but two-edit repair is much stronger. This suggests the compiler often makes one or two local slot errors rather than globally incoherent programs.

Interpretation

The repair result is much larger than the training-objective changes tested in this harness. The reason is straightforward: exact long-chain execution is an all-or-nothing metric, while the compiler's per-slot predictions are already close. At length 24, a small number of wrong arguments or operations can destroy the final answer. Local search converts the compiler's near-miss distribution into exact programs when the verifier can identify the correct state trajectory.

The repaired program-exact numbers are important. On fresh paired L24, repaired exact execution is 91.0% and repaired program exact is 90.0%. The search is usually recovering the true compiled program, not merely exploiting final-answer collisions.

The answer-only verifier pilot failed immediately and was stopped. With roughly 1,299 candidates and only 97 final answers, even weak candidates often include a spurious final-answer match. The state-trajectory verifier fixes that measurement problem by requiring the whole execution path to be correct.

Limitations

The primary verifier is oracle-like. It uses the true intermediate state trajectory, which is available to the synthetic training and evaluation harness but would not be available for ordinary inference. Therefore this experiment should be read as a repair headroom test, not as a completed posttraining recipe.

The task is synthetic modular arithmetic. The compiler, token maps, operation set, and executor are all specialized. The result does not demonstrate broad intelligence improvement. It does show a concrete fact about this Qwen-attached runtime: most long-chain failures are locally repairable if a strong verifier is available.

Next Step

The next experiment should replace the oracle state verifier with a learned verifier or repair policy. The most direct version is:

Generate candidate repair sets from the trained compiler.
Label candidates by whether their state trajectory is correct.
Train a verifier/reranker from Qwen hidden states, compiled slot logits, candidate edits, and execution features.
Evaluate whether the learned verifier can recover a meaningful fraction of the 91.0% oracle-repair ceiling without access to oracle states.

The success criterion should be fresh paired L24 exact execution substantially above the unrepaired 27.5% baseline, while preserving paired consistency.

Artifacts

Small files live in:

experiments/qwen_verifier_guided_slot_repair/

Large checkpoints live in:

large_artifacts/qwen_verifier_guided_slot_repair/checkpoints/

Primary result files:

analysis/selected_checkpoints.csv
analysis/selected_retest_metrics.csv
analysis/selected_retest_metrics_one_edit.csv
reports/qwen_verifier_guided_slot_repair_experiment_log.md

Experiment log 9

Show the running log (9 entries, 2026-06-22)

Objective

Test whether verifier-guided local search over compiled slots can recover exact long-chain execution from near-miss Qwen numeric-copy programs.

Experiment Question

If a compiler produces high-accuracy but imperfect modular programs, can a small repair search over likely slot alternatives substantially improve length-24 exact execution when selected by a state-trajectory verifier?

Planned Conditions

Tiny-model smoke test for repair metrics and checkpoint writing.
Qwen smoke test for 4-bit QLoRA training with repair evaluation enabled.
Main Qwen light-state compiler with verifier-guided repair search and checkpoint selection by repaired paired length-24 accuracy.
Fresh selected-checkpoint retest on length-24 programs.
One-edit repair-budget ablation on the selected checkpoint.

Primary Selection Rule

Select the saved checkpoint with the highest paired_len24_repair_executor_accuracy.

Primary Metrics

paired_len24_executor_accuracy
paired_len24_repair_executor_accuracy
fresh_paired_len24_executor_accuracy
fresh_paired_len24_repair_executor_accuracy
repair_found_fraction
repair_changed_fraction
repair_pair_state_consistency

Artifact Policy

Lightweight outputs stay in:

experiments/qwen_verifier_guided_slot_repair/runs/
experiments/qwen_verifier_guided_slot_repair/analysis/
experiments/qwen_verifier_guided_slot_repair/reports/

Large adapters and head checkpoints stay in:

large_artifacts/qwen_verifier_guided_slot_repair/checkpoints/

Log

2026-06-22

Created standalone experiment directory.
Forked the checkpoint-selected Qwen numeric-copy compiler harness.
Added verifier-guided local repair search over initial value, operation, argument, same-step operation/argument, and two-argument edits.
Added repair metrics to ordinary evaluation, selected-checkpoint tracking, analysis, and fresh retesting.
Stopped the first main run after the step-1 checkpoint showed that answer-only verification was degenerate: with many local candidates in a 97-way answer space, even an untrained compiler could find spurious final-answer matches. Changed the primary verifier to require the full intermediate state trajectory.
Reran smoke_tiny_repair with the state-trajectory verifier. The smoke completed and no longer produced fake repair gains on length-3 paired examples.
Reran smoke_qwen3_4b_repair with Qwen/Qwen3-4B, 4-bit QLoRA, and the state-trajectory verifier. The smoke completed and saved checkpoints.
Ran main_state_w025_repair_s900 with light state supervision, paired training, top-3 repair candidates, up to two edits, and checkpoint selection by paired_len24_repair_executor_accuracy.
Main validation selected step 800. At that checkpoint, paired length-24 exact execution was 30.5% unrepaired and 88.3% repaired. Repaired paired state consistency was 90.6%, repaired changed fraction was 57.8%, and repair found fraction was 88.3%. Step 900 dropped to 82.0% repaired paired length-24.
Ran a fresh selected-checkpoint retest with 256 fresh standard length-24 programs, 256 fresh paraphrase length-24 programs, and 256 paired length-24 programs. Fresh paired length-24 exact execution was 27.5% unrepaired and 91.0% repaired. Fresh repaired program exact was 90.0%, repaired pair state consistency was 92.2%, and repaired pair both-correct was 89.5%.
Ran a one-edit ablation on the same selected checkpoint and fresh seed. Fresh paired length-24 exact execution improved from 27.5% unrepaired to 70.5% with one-edit repair, versus 91.0% with two-edit repair.

Result

State-trajectory verifier repair reveals large local-search headroom. With a top-3/two-edit repair budget, the selected Qwen compiler improves from 27.5% to 91.0% fresh paired length-24 exact execution. The one-edit ablation reaches 70.5%, so many failures are one local slot edit away, while the remaining lift depends on two-slot repair.

This is not yet a deployable verifier, because the state trajectory is an oracle training/evaluation signal. It is a strong headroom result: the compiled programs usually contain enough local evidence that a small verifier-guided search can recover the true long-chain execution.