Counterexample Rule Repair Experiment

Finished2026-06-20imported · line ZEvidence-Conditioned Selection Reliability and Safety GitHub ↗

The one idea you need

Picture a repair worker handed broken code and a defect log: "for input 7 we wanted LOW but got BADL." From a handful of such failure lines the worker must deduce the hidden rule behind them and rewrite the code so it also handles inputs the log never showed.

The question

If you show a code-fixing model concrete examples of how its code got the wrong answer, can it deduce the hidden rule and fix the bug for good?

What we found

Yes, but only for rule shapes it has practiced. Given the failing examples, a small fine-tuned model fixed 91% of bugs on familiar tasks and got unseen inputs right too, versus 0% untrained and near-zero for every version fed no examples or scrambled ones. Strip the real examples out and the same model drops to 0%, proving it truly reads them. Hand it a rule shape never trained on, and nothing works.

Why it matters

When your tests can print concrete input/expected/actual cases, train the model to actually read them, not just the error text: that habit fixes the vast majority of bugs, while its absence fixes almost none. But expect failure on rule shapes absent from training.

Bugs fixed when the model reads the failing examples0% → 91%untrained model vs. trained to read the examples, on familiar tasks

Same trained model with the failing examples removed91% → 0%proof it was truly reading them, not replaying memorized fixes

Familiar rules shown in an unfamiliar format53% fixedpartial transfer to a new presentation of the same rules

A brand-new kind of rule never trained on0 of 45nothing the model tried fixed it

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 5

Only the model trained to read failing examples fixes the bug

How to read

Bars grouped by test set: familiar tasks, familiar rules shown in a new format, and a brand-new rule type. Each colored bar is one training setup; bar height is the share of bugs fully fixed, taller is better.

Takeaway → Only the model trained on real failing examples rises high on familiar tasks and holds about half on the new format; every other setup hugs the floor, and all sit at or near zero on the brand-new rule.

Data table

Validation split	frozen base + trace	final-patch SFT	no-trace SFT	shuffled-trace SFT + trace	trace SFT + trace
IID	0%	0%	8.9%	4.4%	91.1%
Format holdout	0%	0%	0%	0%	53.3%
Rule-family holdout	0%	0%	2.2%	0%	0%

Numbers from report table (reports/counterexample_rule_repair_paper.md, Final Results); matches reports/final_core_results.csv

Technical framing

Repair success: only trace-trained SFT with real traces repairs the bug — Trace SFT with real counterexample traces hits 91% IID and 53% on format holdout; every control stays near zero, and no condition transfers to the withheld rule family.

Take away the failing examples and the same model fails

How to read

Bars grouped by test set; the three bars are the same trained model given the real failing examples, those examples removed, or those examples scrambled. Height is the share of bugs fixed, taller is better.

Takeaway → Only the real-examples bar has height; removing or scrambling the examples flattens it to zero, so the model is genuinely reading the evidence rather than reciting fixes.

Data table

Validation split	real trace	trace removed	shuffled trace
IID	91.1%	0%	0%
Format holdout	53.3%	0%	0%
Rule-family holdout	0%	0%	0%

Numbers from report table (reports/counterexample_rule_repair_paper.md, Trace Adapter Ablations); matches reports/final_ablation_results.csv

Technical framing

Trace adapter ablation: the trace input, not the adapter alone, carries the signal — The same trace-trained adapter drops from 91% to 0% when the failed-test trace is removed or shuffled: it is reading the counterexamples, not memorizing patches.

Some rule types survive a new format, one collapses

How to read

Bars grouped by rule type: arithmetic, text-formatting, threshold-labeling. Two colors compare familiar tasks against the same rules shown in a new format; height is the share of bugs fixed, taller is better.

Takeaway → Threshold-labeling rules stay fully fixed in both formats and text-formatting mostly holds, but the arithmetic family collapses from largely fixed to zero once the format changes.

Data table

Rule family	IID	Format holdout
affine_int	73.3%	0%
slug_affix	100%	60%
threshold_label	100%	100%

Numbers from report table (reports/counterexample_rule_repair_paper.md, Trace Adapter Family Breakdown); matches reports/final_trace_by_family.csv

Technical framing

Trace SFT repair rate by rule family — Format-holdout transfer varies by family (threshold rules fully transfer, affine ones collapse); the withheld parity_offset family is 0/45 everywhere.

Every trained version writes valid patches; only one fixes the logic

How to read

Paired bars for each training setup on familiar tasks; one color is the share of patches well-formed enough to apply, the other the share that actually fixed the bug. Taller is better for both.

Takeaway → Nearly every trained setup produces applicable patches, but only the model trained on real failing examples turns them into correct fixes; the rest apply cleanly yet fix almost nothing.

Data table

Condition	Patch apply rate	Repair@1
Frozen base + trace	4.4%	0%
Final-patch SFT	100%	0%
No-trace SFT	100%	8.9%
Shuffled-trace SFT	100%	4.4%
Trace SFT + trace	100%	91.1%

Numbers from experiments/counterexample_rule_repair/reports/final_core_results.csv

Technical framing

Patch format vs actual repair: SFT fixes syntax, only traces fix semantics — Every SFT variant learns to emit perfectly applicable patches, but only trace-trained SFT turns applicable patches into correct repairs.

More training examples and passes roughly double the fix rate

How to read

One bar per model on a small check set: no training, a small quick training run, then a larger, longer one. Height is the share of bugs fixed, taller is better.

Takeaway → The bar climbs from zero with no training, to just under half with the small run, to most bugs fixed with the larger run, so more data and passes roughly double the rate.

Data table

Model	Repair@1
Frozen base (6 records)	0%
Pilot trace adapter, rank 16 (20 records)	45%
Full trace adapter, rank 32 (20 records)	85%

Numbers from experiments/counterexample_rule_repair/reports/pilot_results.csv

Technical framing

Training budget matters: 90 examples x 1 epoch vs 240 x 3 epochs — A 90-example one-epoch pilot already reaches 45% repair; scaling to 240 examples, 3 epochs, rank 32 nearly doubles it to 85%.

In the author’s words from the Report · “Abstract”

This experiment tests whether a code-repair model can use failed-test counterexamples as evidence for a compact behavioral rule. Each task presents a wrong-patched implementation and visible pytest failures that print concrete input, expected output, and actual output triples. The desired rule is not stated in the issue text. A correct patch must infer the rule from visible counterexamples and pass hidden tests on unseen inputs.

Overview

This directory contains the small, download-friendly artifacts for a standalone counterexample-to-rule repair experiment.

Large generated artifacts such as model adapters and checkpoints are stored separately in:

/workspace/large_artifacts/counterexample_rule_repair/

configs/: experiment configuration.
data/: generated JSONL datasets and dataset manifest.
figures/: report figures generated by scripts/make_report.py.
logs/: detailed experiment log.
reports/: pilot results, final evaluation JSON, CSV summaries, and final paper.
scripts/: dataset builder, evaluator, final evaluation runner, and report generator.

Main Report

Read reports/counterexample_rule_repair_paper.md after final report generation.

Large Artifacts

Adapters and checkpoints live in:

/workspace/large_artifacts/counterexample_rule_repair/models/

Do not include that directory when downloading only the small package.

Report

Rendered from reports/counterexample_rule_repair_paper.md

Generated: 2026-06-20 20:16:10 UTC.

Abstract

Artifact Layout

Small, download-friendly experiment package: /workspace/experiments/counterexample_rule_repair.
Large adapters and checkpoints: /workspace/large_artifacts/counterexample_rule_repair.
The small package contains configs, data JSONL files, reports, figures, scripts, and logs.
The large artifact directory contains LoRA adapters and is excluded from the small package.

Dataset

Train records: 240.
IID validation records: 45.
Format-holdout validation records: 45.
Rule-family-holdout validation records: 45.
Train families: affine_int, slug_affix, threshold_label.
Withheld rule families: parity_offset_holdout.
Dataset seed: 20260620.
Invariants: wrong-patched implementation fails visible counterexamples; target corrective diff applies to the wrong-patched implementation; target implementation passes visible and hidden tests; hidden test inputs do not overlap visible trace inputs; visible expected outputs appear in the failed execution trace.

Each record contains src/repair_target.py, visible tests, hidden tests, the failed trace from the wrong-patched implementation, and the target corrective diff. The visible trace emits COUNTEREXAMPLE input=... expected=... actual=... lines. Hidden cases are disjoint from visible inputs, so copying only the visible cases is insufficient.

Model and Training

Base model: Qwen/Qwen2.5-Coder-3B-Instruct.
Revision: 488639f1ff808d1d3d0ba301aef8c11461451ec5.
Training method: QLoRA adapters.
Final training recipe: 3 epochs, rank 32, alpha 64, dropout 0.05, learning rate 1.5e-4, max length 3072.
Decoding: deterministic generation with max_new_tokens=256 for final evaluations.

Adapter	Mode	Shuffled	Rank	Alpha	Dropout	Epochs	LR	Max length	Train records	Eval records
final_patch_lora	final_patch	False	32	64	0.05	3.0	0.00015	3072	240	45
no_trace_lora	no_trace	False	32	64	0.05	3.0	0.00015	3072	240	45
pilot_trace_lora	trace	False	16	32	0.05	1.0	0.0002	3072	90	45
shuffled_trace_lora	trace	True	32	64	0.05	3.0	0.00015	3072	240	45
trace_lora	trace	False	32	64	0.05	3.0	0.00015	3072	240	45

Conditions

Frozen base + trace: base model with the wrong-patched file and failed trace, no fine-tuning.
Trace SFT + trace: adapter trained and evaluated with failed counterexample traces.
No-trace SFT + no trace: adapter trained and evaluated without failed trace text.
Shuffled-trace SFT + trace: adapter trained on mismatched trace evidence, evaluated with the real trace.
Final-patch SFT + final patch: adapter trained to reproduce final diffs from the original buggy state rather than repair from the wrong-patched state.
Trace SFT + no trace and Trace SFT + shuffled trace: input ablations for the trace adapter.

Metrics

Repair@1: the generated diff applies and repaired files pass both visible and hidden tests.
Visible pass: repaired files pass the visible counterexample tests.
Hidden pass: repaired files pass hidden tests on unseen inputs.
Patch apply: the generated unified diff applies to the intended file state.
Marker match: the diff contains all target rule markers recorded by the dataset builder.
Input literal: the diff contains at least one visible input literal, a diagnostic for hardcoding visible cases.

Iteration Log Summary

Condition	Repair@1	Visible pass	Hidden pass	Patch apply	Successes
Frozen base + trace, 6 IID	0.0%	0.0%	missing	16.7%	0/6
Pilot trace adapter + trace, 20 IID	45.0%	45.0%	65.0%	100.0%	9/20
Pilot trace adapter + no trace, 20 IID	0.0%	0.0%	0.0%	100.0%	0/20
Pilot trace adapter + shuffled trace, 20 IID	0.0%	0.0%	0.0%	100.0%	0/20
Full trace adapter + trace, 20 IID check	85.0%	85.0%	85.0%	100.0%	17/20

The initial frozen pilot showed near-zero patch application and no successful repairs. A small trace adapter made the task learnable, while no-trace and shuffled-trace prompts remained at zero repair. The full trace adapter then improved the 20-record IID check enough to justify training the full control adapters.

Final Results

Split	Condition	Repair@1	Visible pass	Hidden pass	Patch apply	Marker match	Input literal	Successes
IID	Frozen base + trace	0.0%	0.0%	0.0%	4.4%	2.2%	48.9%	0/45
IID	Final-patch SFT + final patch	0.0%	0.0%	0.0%	100.0%	17.8%	64.4%	0/45
IID	No-trace SFT + no trace	8.9%	8.9%	8.9%	100.0%	22.2%	53.3%	4/45
IID	Shuffled-trace SFT + trace	4.4%	4.4%	4.4%	100.0%	17.8%	53.3%	2/45
IID	Trace SFT + trace	91.1%	91.1%	91.1%	100.0%	95.6%	55.6%	41/45
Format holdout	Frozen base + trace	0.0%	0.0%	0.0%	0.0%	0.0%	48.9%	0/45
Format holdout	Final-patch SFT + final patch	0.0%	0.0%	0.0%	100.0%	0.0%	42.2%	0/45
Format holdout	No-trace SFT + no trace	0.0%	0.0%	0.0%	100.0%	0.0%	62.2%	0/45
Format holdout	Shuffled-trace SFT + trace	0.0%	0.0%	0.0%	100.0%	0.0%	60.0%	0/45
Format holdout	Trace SFT + trace	53.3%	53.3%	53.3%	100.0%	68.9%	66.7%	24/45
Rule-family holdout	Frozen base + trace	0.0%	0.0%	0.0%	0.0%	0.0%	100.0%	0/45
Rule-family holdout	Final-patch SFT + final patch	0.0%	0.0%	0.0%	100.0%	0.0%	100.0%	0/45
Rule-family holdout	No-trace SFT + no trace	2.2%	2.2%	2.2%	100.0%	2.2%	100.0%	1/45
Rule-family holdout	Shuffled-trace SFT + trace	0.0%	0.0%	0.0%	100.0%	2.2%	100.0%	0/45
Rule-family holdout	Trace SFT + trace	0.0%	0.0%	0.0%	100.0%	6.7%	100.0%	0/45

Trace Adapter Ablations

Split	Condition	Repair@1	Visible pass	Hidden pass	Patch apply	Marker match	Input literal	Successes
IID	Trace SFT + no trace	0.0%	0.0%	0.0%	100.0%	11.1%	53.3%	0/45
IID	Trace SFT + shuffled trace	0.0%	0.0%	0.0%	100.0%	8.9%	53.3%	0/45
Format holdout	Trace SFT + no trace	0.0%	0.0%	0.0%	100.0%	0.0%	57.8%	0/45
Format holdout	Trace SFT + shuffled trace	0.0%	0.0%	0.0%	100.0%	0.0%	55.6%	0/45
Rule-family holdout	Trace SFT + no trace	0.0%	0.0%	0.0%	100.0%	0.0%	100.0%	0/45
Rule-family holdout	Trace SFT + shuffled trace	0.0%	0.0%	0.0%	100.0%	2.2%	100.0%	0/45

Trace Adapter Family Breakdown

Condition	Family	Repair@1	Visible pass	Hidden pass	Successes
Trace SFT + trace / IID	affine_int	73.3%	73.3%	73.3%	11/15
Trace SFT + trace / IID	slug_affix	100.0%	100.0%	100.0%	15/15
Trace SFT + trace / IID	threshold_label	100.0%	100.0%	100.0%	15/15
Trace SFT + trace / Format holdout	affine_int	0.0%	0.0%	0.0%	0/15
Trace SFT + trace / Format holdout	slug_affix	60.0%	60.0%	60.0%	9/15
Trace SFT + trace / Format holdout	threshold_label	100.0%	100.0%	100.0%	15/15
Trace SFT + trace / Rule-family holdout	parity_offset_holdout	0.0%	0.0%	0.0%	0/45

Figures

Qualitative Examples

Trace-conditioned IID success

Episode: val_iid_threshold_0002::counterexample_rule.
Family: threshold_label.
Outcome: patch_applied=True, visible_passed=True, hidden_passed=True.

Visible counterexamples:

f"COUNTEREXAMPLE input={value!r} expected={expected!r} actual={actual!r}"
E       AssertionError: COUNTEREXAMPLE input=7 expected='LOW_DYNF' actual='BADL_1KL8'
E         COUNTEREXAMPLE input=8 expected='LOW_DYNF' actual='BADL_1KL8'
E         COUNTEREXAMPLE input=9 expected='HIGH_C68M' actual='BADL_1KL8'
E         COUNTEREXAMPLE input=11 expected='HIGH_C68M' actual='BADH_BTX1'
E       assert not ["COUNTEREXAMPLE input=7 expected='LOW_DYNF' actual='BADL_1KL8'", "COUNTEREXAMPLE input=8 expected='LOW_DYNF' actual='...LE input=9 expected='HIGH_C68M' actual='BADL_1KL8'", "COUNTEREXAMPLE input=11 expected='HIGH_C68M' actual='BADH_BTX1'"]
FAILED tests/test_visible.py::test_visible_counterexamples - AssertionError: COUNTEREXAMPLE input=7 expected='LOW_DYNF' actual='BADL_1KL8'

Generated diff:

--- a/src/repair_target.py
+++ b/src/repair_target.py
@@ -2,6 +2,6 @@
 
 
 def apply_rule(value):
-    if value < 11:
-        return "BADL_1KL8"
-    return "BADH_BTX1"
+    if value < 9:
+        return "LOW_DYNF"
+    return "HIGH_C68M"

Trace-conditioned format-holdout success

Episode: val_format_holdout_slug_0003::counterexample_rule.
Family: slug_affix.
Outcome: patch_applied=True, visible_passed=True, hidden_passed=True.

Visible counterexamples:

f"COUNTEREXAMPLE input={value!r} expected={expected!r} actual={actual!r}"
E       AssertionError: COUNTEREXAMPLE input='  Alpha Beta  ' expected='pre:ckmalpha.beta:doneje' actual='WP_8SSCalpha/beta_WSW7JH'
E         COUNTEREXAMPLE input='MIXED_case Word' expected='pre:ckmmixed.case.word:doneje' actual='WP_8SSCmixed/case/word_WSW7JH'
E         COUNTEREXAMPLE input='two   spaces' expected='pre:ckmtwo.spaces:doneje' actual='WP_8SSCtwo/spaces_WSW7JH'
E       assert not ["COUNTEREXAMPLE input='  Alpha Beta  ' expected='pre:ckmalpha.beta:doneje' actual='WP_8SSCalpha/beta_WSW7JH'", "COUNT..._WSW7JH'", "COUNTEREXAMPLE input='two   spaces' expected='pre:ckmtwo.spaces:doneje' actual='WP_8SSCtwo/spaces_WSW7JH'"]
FAILED tests/test_visible.py::test_visible_counterexamples - AssertionError: COUNTEREXAMPLE input='  Alpha Beta  ' expected='pre:ckmalpha.beta:doneje' actual='WP_8SSCalpha/beta_WSW7JH'

Generated diff:

--- a/src/repair_target.py
+++ b/src/repair_target.py
@@ -4,5 +4,5 @@
 def apply_rule(value):
     text = str(value).strip().lower().replace("_", " ")
     pieces = [piece for piece in text.split() if piece]
-    body = "/".join(pieces)
-    return "WP_8SSC" + body + "_WSW7JH"
+    body = ".".join(pieces)
+    return "pre:ckm" + body + ":doneje"

Withheld-rule-family trace-conditioned example

Episode: val_rule_holdout_parity_0027::counterexample_rule.
Family: parity_offset_holdout.
Outcome: patch_applied=True, visible_passed=False, hidden_passed=False.

Visible counterexamples:

f"COUNTEREXAMPLE input={value!r} expected={expected!r} actual={actual!r}"
E       AssertionError: COUNTEREXAMPLE input=0 expected=4 actual=2
E         COUNTEREXAMPLE input=1 expected=12 actual=9
E         COUNTEREXAMPLE input=4 expected=8 actual=6
E         COUNTEREXAMPLE input=7 expected=18 actual=15
E       assert not ['COUNTEREXAMPLE input=0 expected=4 actual=2', 'COUNTEREXAMPLE input=1 expected=12 actual=9', 'COUNTEREXAMPLE input=4 expected=8 actual=6', 'COUNTEREXAMPLE input=7 expected=18 actual=15']
FAILED tests/test_visible.py::test_visible_counterexamples - AssertionError: COUNTEREXAMPLE input=0 expected=4 actual=2

Generated diff:

--- a/src/repair_target.py
+++ b/src/repair_target.py
@@ -2,6 +2,6 @@
 
 
 def apply_rule(value):
-    if value % 2 == 0:
-        return value + 2
-    return value + 8
+    if value % 2 == 0:
+        return value + 4
+    return value + 6

Trace adapter with trace removed

Episode: val_iid_threshold_0002::counterexample_rule.
Family: threshold_label.
Outcome: patch_applied=True, visible_passed=False, hidden_passed=False.

Visible counterexamples:

f"COUNTEREXAMPLE input={value!r} expected={expected!r} actual={actual!r}"
E       AssertionError: COUNTEREXAMPLE input=7 expected='LOW_DYNF' actual='BADL_1KL8'
E         COUNTEREXAMPLE input=8 expected='LOW_DYNF' actual='BADL_1KL8'
E         COUNTEREXAMPLE input=9 expected='HIGH_C68M' actual='BADL_1KL8'
E         COUNTEREXAMPLE input=11 expected='HIGH_C68M' actual='BADH_BTX1'
E       assert not ["COUNTEREXAMPLE input=7 expected='LOW_DYNF' actual='BADL_1KL8'", "COUNTEREXAMPLE input=8 expected='LOW_DYNF' actual='...LE input=9 expected='HIGH_C68M' actual='BADL_1KL8'", "COUNTEREXAMPLE input=11 expected='HIGH_C68M' actual='BADH_BTX1'"]
FAILED tests/test_visible.py::test_visible_counterexamples - AssertionError: COUNTEREXAMPLE input=7 expected='LOW_DYNF' actual='BADL_1KL8'

Generated diff:

--- a/src/repair_target.py
+++ b/src/repair_target.py
@@ -2,6 +2,6 @@
 
 
 def apply_rule(value):
-    if value < 11:
-        return "BADL_1KL8"
-    return "BADH_BTX1"
+    if value < 7:
+        return "BADL_1KL8"
+    return "BADH_BTX1"

Trace adapter with shuffled trace

Episode: val_iid_threshold_0002::counterexample_rule.
Family: threshold_label.
Outcome: patch_applied=True, visible_passed=False, hidden_passed=False.

Visible counterexamples:

f"COUNTEREXAMPLE input={value!r} expected={expected!r} actual={actual!r}"
E       AssertionError: COUNTEREXAMPLE input=7 expected='LOW_DYNF' actual='BADL_1KL8'
E         COUNTEREXAMPLE input=8 expected='LOW_DYNF' actual='BADL_1KL8'
E         COUNTEREXAMPLE input=9 expected='HIGH_C68M' actual='BADL_1KL8'
E         COUNTEREXAMPLE input=11 expected='HIGH_C68M' actual='BADH_BTX1'
E       assert not ["COUNTEREXAMPLE input=7 expected='LOW_DYNF' actual='BADL_1KL8'", "COUNTEREXAMPLE input=8 expected='LOW_DYNF' actual='...LE input=9 expected='HIGH_C68M' actual='BADL_1KL8'", "COUNTEREXAMPLE input=11 expected='HIGH_C68M' actual='BADH_BTX1'"]
FAILED tests/test_visible.py::test_visible_counterexamples - AssertionError: COUNTEREXAMPLE input=7 expected='LOW_DYNF' actual='BADL_1KL8'

Generated diff:

--- a/src/repair_target.py
+++ b/src/repair_target.py
@@ -2,6 +2,6 @@
 
 
 def apply_rule(value):
-    if value < 11:
-        return "BADL_1KL8"
-    return "BADH_BTX1"
+    if value == '  Alpha Beta  ':
+        return "P_512Talpha+beta_SX05E"
+    return "P_512Ttwo+spaces_SX05E"

Discussion

The core contrast is whether the model can transform failed counterexample traces into a general rule rather than merely producing syntactically plausible diffs. The visible and hidden pass split is important: a patch can sometimes satisfy hidden cases while violating visible counterexamples, so the primary metric requires both. The rule-family holdout is a harder extrapolation test because the parity-offset structure is absent from training.

Limitations

The tasks are synthetic and intentionally focused on one-file rule repair.
The experiment measures greedy single-sample repair, not sampling-based pass rates.
Hidden tests are generated from known templates, so they are controlled probes rather than open-ended software behavior.
The withheld rule family tests structural transfer to one unseen family only.

Reproducibility

Dataset build:

python experiments/counterexample_rule_repair/scripts/build_counterexample_dataset.py --output-dir experiments/counterexample_rule_repair/data --train-per-family 80 --iid-per-family 15 --format-per-family 15 --rule-holdout 45 --seed 20260620

Final evaluations:

python experiments/counterexample_rule_repair/scripts/run_final_evaluations.py --suite all --max-new-tokens 256

Report generation:

python experiments/counterexample_rule_repair/scripts/make_report.py

Experiment log 11

Show the running log (11 entries, 2026-06-20)

2026-06-20 Setup

Objective: design and run a standalone counterexample-to-rule repair experiment where the failed execution trace contains concrete examples of the desired behavior, and hidden tests require applying the inferred rule to new inputs.

Directory policy:

Small, downloadable experiment package: /workspace/experiments/counterexample_rule_repair/.
Large artifacts excluded from the small package: /workspace/large_artifacts/counterexample_rule_repair/.
Model adapters and checkpoints go under large_artifacts/counterexample_rule_repair/models/.
Reports, logs, configs, figures, and compact JSON/JSONL result summaries go under experiments/counterexample_rule_repair/.

Initial hypothesis:

A repair model trained on wrong patched state + failed execution counterexamples -> corrective diff will outperform no-trace and shuffled-trace controls when the correct fix requires inferring a compact rule from visible counterexamples and then passing hidden tests on unseen inputs.

Design constraints:

The paper and artifacts must be standalone.
The expected rule parameters must not appear in the issue text.
The visible trace must contain enough counterexamples to infer the rule.
Hidden tests must require generalization beyond the visible counterexamples.

2026-06-20 Dataset Design

Primary task shape:

Each record contains one production file, src/repair_target.py, with an apply_rule(value) function.
The issue text says the validator rule is not stated and must be inferred from failed-test counterexamples.
The wrong-patched implementation is a compact but incorrect rule.
The failed visible test emits COUNTEREXAMPLE input=... expected=... actual=... lines.
The target corrective diff patches the implementation into a compact rule.
Hidden tests use inputs that are not present in the visible trace, so a patch that hardcodes only visible counterexamples should fail.

Train and IID/format-holdout families:

affine_int: infer a linear integer rule from numeric counterexamples.
threshold_label: infer a threshold and two output labels from boundary counterexamples.
slug_affix: infer string affixes and separator while preserving slug normalization behavior.

Rule-family holdout:

parity_offset_holdout: infer separate offsets for even and odd inputs. This family is withheld from training.

Validation performed by the builder:

The wrong patch fails visible tests.
The target corrective diff applies to the wrong-patched implementation.
The repaired implementation passes visible and hidden tests.
The repaired implementation compiles.
Hidden test inputs do not overlap visible trace inputs.
Visible expected outputs appear in the failed execution trace.

Smoke build:

A 14-record smoke dataset passed all builder invariants.
Inspected affine and threshold examples confirmed that visible counterexamples are present in trace output and hidden examples require unseen inputs.

2026-06-20 Full Dataset Build

Command:

python experiments/counterexample_rule_repair/scripts/build_counterexample_dataset.py --output-dir experiments/counterexample_rule_repair/data --train-per-family 80 --iid-per-family 15 --format-per-family 15 --rule-holdout 45 --seed 20260620

Result:

Train: 240 records.
IID validation: 45 records.
Format-holdout validation: 45 records.
Rule-family-holdout validation: 45 records.
All records: 375.
Data directory size: about 4.3 MB.

Validation split composition:

IID: 15 affine_int, 15 threshold_label, 15 slug_affix.
Format holdout: 15 affine_int, 15 threshold_label, 15 slug_affix, with shifted numeric ranges and different string-token formats.
Rule-family holdout: 45 parity_offset_holdout records.

All builder invariants passed.

2026-06-20 Frozen Pilot

Command:

python experiments/counterexample_rule_repair/scripts/eval_counterexample_rule.py --data experiments/counterexample_rule_repair/data/repair_val_iid.jsonl --output experiments/counterexample_rule_repair/reports/frozen_trace_iid_pilot6.json --condition trace --max-records 6 --max-new-tokens 256

Result:

Model: Qwen/Qwen2.5-Coder-3B-Instruct, revision 488639f1ff808d1d3d0ba301aef8c11461451ec5.
Records: 6 IID validation examples.
Repair@1: 0/6.
Visible pass rate: 0/6.
Patch apply rate: 1/6.
Target-added-line match rate: 0/6.

Observed failure mode:

The frozen model often generated diffs against the original placeholder implementation rather than the wrong-patched state.
When a patch applied, it preserved wrong constants rather than deriving the target rule from counterexamples.

Next step:

Train a small trace-conditioned pilot adapter on 90 training records for one epoch, then evaluate trace, no-trace, and shuffled-trace behavior.

2026-06-20 Trace Pilot Training and Controls

Training command:

python scripts/train_repair_lora.py --train experiments/counterexample_rule_repair/data/repair_train.jsonl --eval experiments/counterexample_rule_repair/data/repair_val_iid.jsonl --mode trace --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/counterexample_rule_repair/models/pilot_trace_lora --max-length 3072 --epochs 1 --lr 2e-4 --rank 16 --alpha 32 --dropout 0.05 --grad-accum 8 --save-steps 20 --eval-steps 20 --max-train-records 90

Training observations:

Trainable parameters: 29,933,568, about 0.96% of the model.
Training took about 57 seconds.
Train loss: 0.09427.
Eval loss on all 45 IID validation records: 0.03152.

Evaluation metric correction:

Initial pilot inspection showed that hidden-only pass can overstate success when a patch passes unseen hidden inputs but fails visible counterexamples.
Updated evaluator so repair@1 requires both visible and hidden tests to pass.
Added hidden_pass_rate as a separate diagnostic.

Pilot evaluations, all on the first 20 IID validation records:

Frozen base + trace: 0/6 in the earlier smoke pilot.
Pilot trace adapter + trace: 9/20 repair@1, 20/20 patch apply, 20/20 syntax valid.
Pilot trace adapter + no trace: 0/20 repair@1, 20/20 patch apply, 20/20 syntax valid.
Pilot trace adapter + shuffled trace: 0/20 repair@1, 20/20 patch apply, 20/20 syntax valid.

Per-family trace pilot notes:

slug_affix: 4/5 repair@1.
threshold_label: 4/8 repair@1; several failures were off-by-one thresholds that passed hidden examples but failed visible counterexamples.
affine_int: 1/7 repair@1; arithmetic inference from counterexamples is the hardest family.

Decision:

The pilot establishes a causal trace contrast but is not saturated.
Proceed to a stronger full trace adapter first: all 240 train records, 3 epochs, rank 32, alpha 64, max length 3072.
Train the full control adapters after confirming the stronger trace adapter improves.

2026-06-20 Full Trace Adapter

Training command:

python scripts/train_repair_lora.py --train experiments/counterexample_rule_repair/data/repair_train.jsonl --eval experiments/counterexample_rule_repair/data/repair_val_iid.jsonl --mode trace --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/counterexample_rule_repair/models/trace_lora --max-length 3072 --epochs 3 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30

Training observations:

Trainable parameters: 59,867,136, about 1.90% of the model.
Training took about 422 seconds.
Final train loss: 0.01636.
Final IID eval loss: 0.004355.

Sanity evaluation command:

python experiments/counterexample_rule_repair/scripts/eval_counterexample_rule.py --data experiments/counterexample_rule_repair/data/repair_val_iid.jsonl --output experiments/counterexample_rule_repair/reports/full_trace_iid20_check.json --condition trace --adapter large_artifacts/counterexample_rule_repair/models/trace_lora --max-records 20 --max-new-tokens 256

Result:

Records: 20 IID validation examples.
Repair@1: 17/20.
Visible pass rate: 17/20.
Hidden pass rate: 17/20.
Patch apply rate: 20/20.
Syntax valid rate: 20/20.

Per-family sanity notes:

slug_affix: 5/5 repair@1.
threshold_label: 8/8 repair@1.
affine_int: 4/7 repair@1.

Decision:

The full trace adapter is strong enough to justify the full control suite.
Use the same model, revision, LoRA rank, alpha, dropout, epochs, learning rate, and max length for the no-trace, shuffled-trace, and final-patch controls.

2026-06-20 Full Control Adapter Training

No-trace training command:

python scripts/train_repair_lora.py --train experiments/counterexample_rule_repair/data/repair_train.jsonl --eval experiments/counterexample_rule_repair/data/repair_val_iid.jsonl --mode no_trace --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/counterexample_rule_repair/models/no_trace_lora --max-length 3072 --epochs 3 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30

No-trace result:

Trainable parameters: 59,867,136, about 1.90% of the model.
Training took about 422 seconds.
Final train loss: 0.2545.
Final IID eval loss: 0.2151.

Shuffled-trace training command:

python scripts/train_repair_lora.py --train experiments/counterexample_rule_repair/data/repair_train.jsonl --eval experiments/counterexample_rule_repair/data/repair_val_iid.jsonl --mode trace --shuffle-traces --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/counterexample_rule_repair/models/shuffled_trace_lora --max-length 3072 --epochs 3 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30

Shuffled-trace result:

Trainable parameters: 59,867,136, about 1.90% of the model.
Training took about 417 seconds.
Final train loss: 0.2538.
Final IID eval loss: 0.2106.

Final-patch training command:

python scripts/train_repair_lora.py --train experiments/counterexample_rule_repair/data/repair_train.jsonl --eval experiments/counterexample_rule_repair/data/repair_val_iid.jsonl --mode final_patch --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/counterexample_rule_repair/models/final_patch_lora --max-length 3072 --epochs 3 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30

Final-patch result:

Trainable parameters: 59,867,136, about 1.90% of the model.
Training took about 426 seconds.
Final train loss: 0.3666.
Final IID eval loss: 0.2827.

Interpretation before final evaluations:

The trace adapter fit the supervised target much more closely than the no-trace, shuffled-trace, and final-patch controls.
This does not prove repair success by itself, so the final comparison must use generated patches run against visible and hidden tests.

2026-06-20 Final Evaluation Plan

Evaluation runner:

python experiments/counterexample_rule_repair/scripts/run_final_evaluations.py --suite all --max-new-tokens 256

Planned splits:

IID validation: 45 records from the training rule families.
Format holdout: 45 records from the training rule families with shifted numeric ranges and token formats.
Rule-family holdout: 45 parity_offset_holdout records, with the family absent from training.

Core conditions:

Frozen base + trace.
Final-patch SFT + final patch.
No-trace SFT + no trace.
Shuffled-trace SFT + trace.
Trace SFT + trace.

Trace-adapter input ablations:

Trace SFT + no trace.
Trace SFT + shuffled trace.

2026-06-20 Final Evaluation Results

Final evaluation command:

python experiments/counterexample_rule_repair/scripts/run_final_evaluations.py --suite all --max-new-tokens 256

Result files:

21 final JSON result files were written to experiments/counterexample_rule_repair/reports/final_*.json.
Each final JSON contains aggregate metrics plus per-record completions, extracted patches, patch-application status, syntax status, visible-test output, and hidden-test output.

Core results:

Split	Frozen + trace	Final-patch SFT	No-trace SFT	Shuffled-trace SFT	Trace SFT + trace
IID	0/45	0/45	4/45	2/45	41/45
Format holdout	0/45	0/45	0/45	0/45	24/45
Rule-family holdout	0/45	0/45	1/45	0/45	0/45

Trace adapter input ablations:

Split	Trace SFT + no trace	Trace SFT + shuffled trace
IID	0/45	0/45
Format holdout	0/45	0/45
Rule-family holdout	0/45	0/45

Family breakdown for Trace SFT + trace:

IID: affine_int 11/15, slug_affix 15/15, threshold_label 15/15.
Format holdout: affine_int 0/15, slug_affix 9/15, threshold_label 15/15.
Rule-family holdout: parity_offset_holdout 0/45.

Interpretation:

The aligned trace condition is the only condition with strong IID and format-holdout repair.
Valid diff generation alone is not enough: final-patch, no-trace, shuffled-trace, and input-ablation conditions often reached 100% patch application while still failing visible and hidden tests.
The trace adapter remained dependent on aligned trace evidence at inference time: removing or shuffling the trace reduced repair@1 to 0/45 on every split.
The withheld parity_offset_holdout family did not transfer. The trace adapter generated applicable, syntactically valid diffs, but failed every visible and hidden test in that split.

2026-06-20 Report Generation

Report command:

python experiments/counterexample_rule_repair/scripts/make_report.py

Generated small-package report artifacts:

reports/counterexample_rule_repair_paper.md
reports/counterexample_rule_repair_summary.md
reports/final_core_results.csv
reports/final_ablation_results.csv
reports/final_trace_by_family.csv
reports/pilot_results.csv
figures/core_repair_rates.png
figures/trace_ablation_repair_rates.png
figures/visible_pass_rates.png
large_artifacts_manifest.md

Artifact split:

Small, download-friendly package: experiments/counterexample_rule_repair/.
Large adapters/checkpoints: large_artifacts/counterexample_rule_repair/models/.
The small package intentionally contains no model adapter files.

2026-06-20 Final Verification

Verification commands and outcomes:

Script compilation passed: python -m py_compile experiments/counterexample_rule_repair/scripts/build_counterexample_dataset.py experiments/counterexample_rule_repair/scripts/eval_counterexample_rule.py experiments/counterexample_rule_repair/scripts/run_final_evaluations.py experiments/counterexample_rule_repair/scripts/make_report.py scripts/train_repair_lora.py
Final result count: 21 reports/final_*.json files.
Figure files are nonempty:
Small-package size: about 9.9 MB.
Large-artifact directory size: about 5.5 GB.
No files larger than 50 MB were found under experiments/counterexample_rule_repair/.
No .safetensors, .bin, .pt, or .pth model-weight files were found under experiments/counterexample_rule_repair/.
No active train_repair_lora, run_final_evaluations, or eval_counterexample_rule processes remained after the final sweep.
Removed generated __pycache__ and .ipynb_checkpoints directories from the small package.
Reference hygiene check found no stale references to unrelated experiment names in the counterexample-rule package.

Figures 3

Data files 3

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

reports/final_ablation_results.csv8.4 kB
reports/final_core_results.csv21 kB
reports/pilot_results.csv7.9 kB

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Counterexample Rule Repair Experiment

Results at a glance 5

Overview

Contents

Main Report

Large Artifacts

Report

Abstract

Artifact Layout

Dataset

Model and Training

Conditions

Metrics

Iteration Log Summary

Final Results

Trace Adapter Ablations

Trace Adapter Family Breakdown

Figures

Qualitative Examples

Trace-conditioned IID success

Trace-conditioned format-holdout success

Withheld-rule-family trace-conditioned example

Trace adapter with trace removed

Trace adapter with shuffled trace

Discussion

Limitations

Reproducibility

Experiment log 11

2026-06-20 Setup

2026-06-20 Dataset Design

2026-06-20 Full Dataset Build

2026-06-20 Frozen Pilot

2026-06-20 Trace Pilot Training and Controls

2026-06-20 Full Trace Adapter

2026-06-20 Full Control Adapter Training

2026-06-20 Final Evaluation Plan

2026-06-20 Final Evaluation Results

2026-06-20 Report Generation

2026-06-20 Final Verification

Figures 3

Data files 3

Reproduce

Related

Related experiments