Trace-Keyed Symbol Repair Experiment

Finished2026-06-20imported · line Zprogram review needed · proposedGitHub ↗

The one idea you need

Give the code a wrong constant and its tests fail with a message that literally spells out the value it expected — a value found nowhere else, not in the source files, not in the bug report. The only way to fix the code is to read that failure message.

The question

When a program's tests fail and print the exact value they expected, can a small coding model learn to lift that value out and paste it into the fix?

What we found

Yes. After light training, a small coding model fixed all 60 test cases — including value styles it never saw during training — by copying the correct value straight out of the failure message. Take that message away or scramble it and repair drops to none. An untrained model pulls the right value out only about 18 percent of the time, so this skill genuinely had to be trained in.

Why it matters

When building an auto-repair loop, feed the model the raw failure output, not a tidy summary — the answer often hides there. And test honestly: if scrambling that output does not hurt results, your model is pattern-matching, not reading the evidence.

Fixes before vs after training on the failure message0 of 60 → 60 of 60same model, familiar value styles

Same trained model once the failure message is removed or scrambled60 of 60 → 0 of 60repair collapses without the real message

Fixes on value formats never seen during training60 out of 60the skill carried over to brand-new styles

Untrained model pulling the right value out on its ownabout 18%rarely does it without training

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 6

Only the model trained to read the failure message fixes the code

How to read

Each group is one training-and-inference setup; within it, two bars give the share of tasks fixed on familiar value styles and on brand-new styles. Taller is better. Only the rightmost setup — trained on the failure message and given it — rises off the floor.

Takeaway → Four setups sit flat at zero and the failure-message setup hits full on both styles, so reading the message is what does the work.

Data table

training/eval condition	IID split	format holdout
frozen base + trace	0%	0%
final-patch SFT	0%	0%
no-trace SFT	0%	0%
shuffled-trace SFT	0%	0%
trace SFT + trace	100%	100%

Numbers from experiments/trace_keyed_symbol_repair/reports/final_core_results.csv

Technical framing

Repair success: only trace-trained model with trace at inference repairs the code — Every control sits at 0% while trace-conditioned SFT evaluated with the trace repairs 60/60 on both splits.

Take the failure message away and the same model fixes nothing

How to read

Bars show the share of tasks the one trained model fixes under three conditions: given the correct failure message, given none, and given a mismatched one — split into familiar and new value styles. Taller is better.

Takeaway → Only the correct-message bars reach the top; removing or swapping the message drops repair to zero, so the answer lives in the message content itself.

Data table

trace given at inference	IID split	format holdout
with trace	100%	100%
no trace	0%	0%
shuffled trace	0%	0%

Numbers from experiments/trace_keyed_symbol_repair/reports/final_core_results.csv; experiments/trace_keyed_symbol_repair/reports/final_ablation_results.csv

Technical framing

Ablation: the same trace-trained model fails without a correct trace — Removing or shuffling the trace at inference drops the trace-SFT model from 100% to 0% — the trace content, not the format, carries the answer.

How often each setup copies the correct value out of the message

How to read

Bars show how often each setup pastes the exact correct value into its fix, split into familiar and new value styles. Taller is better. The untrained base shows a short bar, the trained model reaches the top, and the rest stay at zero.

Takeaway → The untrained model copies the right value only about 18 percent of the time and almost never on new styles, while training makes copying reliable every time.

Data table

training/eval condition	IID split	format holdout
frozen base + trace	18.3%	3.3%
final-patch SFT	0%	0%
no-trace SFT	0%	0%
shuffled-trace SFT	0%	0%
trace SFT + trace	100%	100%

Numbers from experiments/trace_keyed_symbol_repair/reports/final_core_results.csv

Technical framing

Copying the expected token from the trace — The frozen base copies the right token from the trace only 18% (IID) / 3% (holdout) of the time; trace SFT makes copying reliable at 100%.

Deleting the wrong value is easy; writing the right one is not

How to read

For each setup, one bar is how often it removes the wrong value and another is how often the code is actually fixed, on familiar-style tasks. Taller is better. Most setups clear the wrong-value bar but leave the fix bar flat.

Takeaway → Nearly every trained setup deletes the wrong value, but only the one reading the correct failure message writes the right replacement and passes.

Data table

Model + inference condition	Wrong token removed	Repair success (repair@1)
Frozen base + trace	0%	0%
Trace SFT + trace	100%	100%
Trace SFT + no trace	95%	0%
Trace SFT + shuffled trace	100%	0%
No-trace SFT	100%	0%

Numbers from experiments/trace_keyed_symbol_repair/reports/final_core_results.csv and final_ablation_results.csv

Technical framing

Edit localization vs actual repair (IID validation) — Every fine-tuned variant deletes the wrong token, but only the trace-trained model with a correct trace writes the right replacement.

Well-formed edits are cheap; correct edits need the failure message

How to read

For each setup, bars show how often the edit applies cleanly, the code still runs, and the tests actually pass, on familiar-style tasks. Taller is better. The first two fill up for every trained setup; only the last separates them.

Takeaway → All trained setups produce clean, runnable edits, but only the model that reads the failure message produces edits that pass the tests.

Data table

Model + inference condition	Patch applies	Repaired code is syntactically valid	Visible tests pass
Frozen base + trace	0%	0%	0%
Final-patch SFT	100%	100%	0%
No-trace SFT	100%	100%	0%
Shuffled-trace SFT	100%	100%	0%
Trace SFT + trace	100%	100%	100%

Numbers from experiments/trace_keyed_symbol_repair/reports/final_core_results.csv

Technical framing

Format compliance is free with SFT; correctness is not — All SFT variants emit well-formed, syntactically valid patches; only the trace-conditioned model produces patches that pass tests.

Give the model enough room to write the whole fix

How to read

Two runs of the trained model differ only in how much output length it may produce. Bars show whether the edit applies and whether it fixes the code, on a small five-task pilot. Taller is better. The tight-limit run is flat; the roomier one fills up.

Takeaway → With too little output room the model cannot finish a usable edit at all, but doubling the room flips it to fully fixing the pilot cases.

Data table

Max new tokens (pilot trace adapter + trace)	Patch applies	Repair success (repair@1)
64 tokens	0%	0%
128 tokens	100%	100%

Numbers from experiments/trace_keyed_symbol_repair/reports/pilot_results.csv

Technical framing

Pilot generation-budget sweep: 64 tokens is too few — With a 64-token cap the trace-trained adapter cannot even finish an applicable patch; 128 tokens flips repair from 0 to 1.

In the author’s words from the Report · “Abstract”

This experiment tests whether a repair model can use a failed execution trace as a data-bearing input, not only as a generic failure signal. Each synthetic task requires replacing a wrong canonical token with an expected token that is absent from the issue text and repository files but present in the pytest failure output. The primary comparison is a trace-conditioned LoRA against frozen, no-trace, shuffled-trace, and final-patch controls.

Overview

This directory contains the small, download-friendly artifacts for a standalone trace-conditioned repair experiment.

Large generated artifacts such as model adapters and checkpoints are stored separately in:

/workspace/large_artifacts/trace_keyed_symbol_repair/

configs/: experiment configuration.
data/: generated JSONL datasets and dataset manifest.
figures/: report figures generated by scripts/make_report.py.
logs/: detailed experiment log.
reports/: pilot results, final evaluation JSON, CSV summaries, and the standalone paper.
scripts/: dataset builder, evaluator, final evaluation runner, and report generator.
requirements.txt: Python dependencies used by the experiment.

Main Report

Read reports/trace_keyed_symbol_repair_paper.md after final report generation.

Large Artifacts

Adapters and checkpoints live in:

/workspace/large_artifacts/trace_keyed_symbol_repair/models/

Do not include that directory when downloading only the small package.

Report

Rendered from reports/trace_keyed_symbol_repair_paper.md

Generated: 2026-06-20 10:09:05 UTC.

Abstract

Artifact Layout

Small, download-friendly experiment package: /workspace/experiments/trace_keyed_symbol_repair.
Large adapters and checkpoints: /workspace/large_artifacts/trace_keyed_symbol_repair.
The small package contains configs, data JSONL files, reports, figures, scripts, and logs.
The large artifact directory contains model adapters and is excluded from the small package.

Dataset

Train records: 240.
IID validation records: 60.
Format-holdout validation records: 60.
Train token styles: dash_upper, underscore_upper, mixed_hex.
Holdout token styles: colon_upper, dot_lower.
Dataset seed: 20260620.
Invariant: expected token is absent from current files and present in failing trace.

Each record contains a wrong-patched src/repair_target.py, visible and hidden pytest tests, a failing trace from the wrong-patched state, and a target corrective diff. The builder validates that the expected token is absent from current_files, present in the failing trace, and that the target diff passes visible and hidden tests.

Model and Training

Base model: Qwen/Qwen2.5-Coder-3B-Instruct.
Revision: 488639f1ff808d1d3d0ba301aef8c11461451ec5.
Training method: one-epoch QLoRA adapters.
Decoding: deterministic generation with max_new_tokens=128 for final evaluations.

Adapter	Mode	Shuffled	Rank	Alpha	Dropout	Epochs	LR	Max length	Train records
final_patch_lora	final_patch	False	16	32	0.05	1.0	0.0002	2048	240
no_trace_lora	no_trace	False	16	32	0.05	1.0	0.0002	2048	240
pilot_trace_lora	trace	False	16	32	0.05	1.0	0.0002	2048	80
shuffled_trace_lora	trace	True	16	32	0.05	1.0	0.0002	2048	240
trace_lora	trace	False	16	32	0.05	1.0	0.0002	2048	240

Metrics

Repair@1: the generated diff applies and the repaired files pass hidden tests.
Visible pass: the generated diff applies and the repaired files pass visible tests.
Patch apply: the generated unified diff applies to the intended file state.
Expected-token copy: the generated diff contains the record-specific expected token.
Wrong-token removed: the generated diff removes the wrong token without reintroducing it.

Pilot Results

Condition	Repair@1	Patch apply	Expected-token copy	Max new tokens	Successes
Frozen base + trace, 10 IID	0.0%	0.0%	10.0%	192	0/10
Pilot trace adapter + trace, 20 IID	100.0%	100.0%	100.0%	192	20/20
Pilot trace adapter + no trace, 20 IID	0.0%	100.0%	0.0%	192	0/20
Pilot trace adapter + shuffled trace, 20 IID	0.0%	100.0%	0.0%	192	0/20
Pilot trace adapter + trace, 5 IID, 64 tokens	0.0%	0.0%	0.0%	64	0/5
Pilot trace adapter + trace, 5 IID, 128 tokens	100.0%	100.0%	100.0%	128	5/5

The pilot established that the task is learnable from traces and that removing or shuffling trace evidence breaks repair even when the adapter can still emit syntactically valid diffs.

Final Results

Split	Condition	Repair@1	Visible pass	Patch apply	Expected-token copy	Wrong-token removed	Successes
IID	Frozen base + trace	0.0%	0.0%	0.0%	18.3%	0.0%	0/60
Format holdout	Frozen base + trace	0.0%	0.0%	0.0%	3.3%	0.0%	0/60
IID	Final-patch SFT + final patch	0.0%	0.0%	100.0%	0.0%	0.0%	0/60
Format holdout	Final-patch SFT + final patch	0.0%	0.0%	100.0%	0.0%	0.0%	0/60
IID	No-trace SFT + no trace	0.0%	0.0%	100.0%	0.0%	100.0%	0/60
Format holdout	No-trace SFT + no trace	0.0%	0.0%	100.0%	0.0%	100.0%	0/60
IID	Shuffled-trace SFT + trace	0.0%	0.0%	100.0%	0.0%	100.0%	0/60
Format holdout	Shuffled-trace SFT + trace	0.0%	0.0%	100.0%	0.0%	100.0%	0/60
IID	Trace SFT + trace	100.0%	100.0%	100.0%	100.0%	100.0%	60/60
Format holdout	Trace SFT + trace	100.0%	100.0%	100.0%	100.0%	100.0%	60/60

Trace Adapter Ablations

Split	Condition	Repair@1	Visible pass	Patch apply	Expected-token copy	Wrong-token removed	Successes
IID	Trace SFT + no trace	0.0%	0.0%	100.0%	0.0%	95.0%	0/60
Format holdout	Trace SFT + no trace	0.0%	0.0%	100.0%	0.0%	90.0%	0/60
IID	Trace SFT + shuffled trace	0.0%	0.0%	100.0%	0.0%	100.0%	0/60
Format holdout	Trace SFT + shuffled trace	0.0%	0.0%	100.0%	0.0%	95.0%	0/60

Figures

Qualitative Examples

Trace-conditioned success

Episode: val_iid_dash_upper_0000::wrong_token.
Token style: dash_upper.
Expected token: CANON-031H-7L04.
Wrong token: CANON-EJDA-00IY.
Outcome: patch_applied=True, visible_passed=True, hidden_passed=True.

Trace evidence:

___________________________________________________________________________________________________________________________________________________ test_trace_reveals_expected_token ___________________________________________________________________________________________________________________________________________________
    def test_trace_reveals_expected_token():
>       assert actual == "CANON-031H-7L04", (
            "TRACE_KEY expected_token=CANON-031H-7L04 actual_token=" + actual
E       AssertionError: TRACE_KEY expected_token=CANON-031H-7L04 actual_token=CANON-EJDA-00IY
FAILED tests/test_visible.py::test_trace_reveals_expected_token - AssertionError: TRACE_KEY expected_token=CANON-031H-7L04 actual_token=CANON-EJDA-00IY

Generated diff:

--- a/src/repair_target.py
+++ b/src/repair_target.py
@@ -1,6 +1,6 @@
 """Tiny target module for trace-keyed repair."""
 
-CANONICAL_TOKEN = "CANON-EJDA-00IY"
+CANONICAL_TOKEN = "CANON-031H-7L04"
 
 
 def canonical_token(case_id):

Format-holdout trace-conditioned success

Episode: val_format_holdout_colon_upper_0000::wrong_token.
Token style: colon_upper.
Expected token: TRACE:TW:GV:EX.
Wrong token: TRACE:8E:ZD:7S.
Outcome: patch_applied=True, visible_passed=True, hidden_passed=True.

Trace evidence:

___________________________________________________________________________________________________________________________________________________ test_trace_reveals_expected_token ___________________________________________________________________________________________________________________________________________________
    def test_trace_reveals_expected_token():
>       assert actual == "TRACE:TW:GV:EX", (
            "TRACE_KEY expected_token=TRACE:TW:GV:EX actual_token=" + actual
E       AssertionError: TRACE_KEY expected_token=TRACE:TW:GV:EX actual_token=TRACE:8E:ZD:7S
FAILED tests/test_visible.py::test_trace_reveals_expected_token - AssertionError: TRACE_KEY expected_token=TRACE:TW:GV:EX actual_token=TRACE:8E:ZD:7S

Generated diff:

--- a/src/repair_target.py
+++ b/src/repair_target.py
@@ -1,6 +1,6 @@
 """Tiny target module for trace-keyed repair."""
 
-CANONICAL_TOKEN = "TRACE:8E:ZD:7S"
+CANONICAL_TOKEN = "TRACE:TW:GV:EX"
 
 
 def canonical_token(case_id):

No-trace failure

Episode: val_iid_dash_upper_0000::wrong_token.
Token style: dash_upper.
Expected token: CANON-031H-7L04.
Wrong token: CANON-EJDA-00IY.
Outcome: patch_applied=True, visible_passed=False, hidden_passed=False.

Correct trace withheld from the prompt, shown here for reference:

___________________________________________________________________________________________________________________________________________________ test_trace_reveals_expected_token ___________________________________________________________________________________________________________________________________________________
    def test_trace_reveals_expected_token():
>       assert actual == "CANON-031H-7L04", (
            "TRACE_KEY expected_token=CANON-031H-7L04 actual_token=" + actual
E       AssertionError: TRACE_KEY expected_token=CANON-031H-7L04 actual_token=CANON-EJDA-00IY
FAILED tests/test_visible.py::test_trace_reveals_expected_token - AssertionError: TRACE_KEY expected_token=CANON-031H-7L04 actual_token=CANON-EJDA-00IY

Generated diff:

--- a/src/repair_target.py
+++ b/src/repair_target.py
@@ -1,6 +1,6 @@
 """Tiny target module for trace-keyed repair."""
 
-CANONICAL_TOKEN = "CANON-EJDA-00IY"
+CANONICAL_TOKEN = "CANON-EJDA-00IY-0000"
 
 
 def canonical_token(case_id):

Shuffled-trace failure

Episode: val_iid_dash_upper_0000::wrong_token.
Token style: dash_upper.
Expected token: CANON-031H-7L04.
Wrong token: CANON-EJDA-00IY.
Outcome: patch_applied=True, visible_passed=False, hidden_passed=False.

Record's correct trace was replaced by another record's trace; correct trace shown here for reference:

___________________________________________________________________________________________________________________________________________________ test_trace_reveals_expected_token ___________________________________________________________________________________________________________________________________________________
    def test_trace_reveals_expected_token():
>       assert actual == "CANON-031H-7L04", (
            "TRACE_KEY expected_token=CANON-031H-7L04 actual_token=" + actual
E       AssertionError: TRACE_KEY expected_token=CANON-031H-7L04 actual_token=CANON-EJDA-00IY
FAILED tests/test_visible.py::test_trace_reveals_expected_token - AssertionError: TRACE_KEY expected_token=CANON-031H-7L04 actual_token=CANON-EJDA-00IY

Generated diff:

--- a/src/repair_target.py
+++ b/src/repair_target.py
@@ -1,6 +1,6 @@
 """Tiny target module for trace-keyed repair."""
 
-CANONICAL_TOKEN = "CANON-EJDA-00IY"
+CANONICAL_TOKEN = "CANON-F5I7-B4DW"
 
 
 def canonical_token(case_id):

Discussion

The controlled task isolates whether the trace supplies information needed for repair. A successful trace-conditioned adapter must both localize the wrong constant and copy a token that is not available in the repository context. The no-trace and shuffled-trace controls test whether performance can be explained by format memorization or generic patch syntax alone.

Limitations

The task family is synthetic and intentionally narrow.
Results measure controlled trace-conditioned token recovery, not broad real-world software maintenance ability.
All final evaluations use greedy decoding; sampling-based pass rates were not measured.
The format-holdout split changes token surface form but not program structure.

Reproducibility

Dataset build:

python experiments/trace_keyed_symbol_repair/scripts/build_trace_keyed_dataset.py --output-dir experiments/trace_keyed_symbol_repair/data --train 240 --iid 60 --holdout 60 --seed 20260620

Final evaluations:

python experiments/trace_keyed_symbol_repair/scripts/run_final_evaluations.py --suite core --max-new-tokens 128
python experiments/trace_keyed_symbol_repair/scripts/run_final_evaluations.py --suite ablation --max-new-tokens 128

Report generation:

python experiments/trace_keyed_symbol_repair/scripts/make_report.py

Experiment log 10

Show the running log (10 entries, 2026-06-20)

2026-06-20 Setup

Objective: design and run a standalone experiment in execution-conditioned program repair where the execution trace carries task-specific information that is not recoverable from repository context alone.

Directory policy:

Small, downloadable experiment package: /workspace/experiments/trace_keyed_symbol_repair/.
Large artifacts excluded from the small package: /workspace/large_artifacts/trace_keyed_symbol_repair/.
Model adapters and checkpoints go under large_artifacts/trace_keyed_symbol_repair/models/.
Reports, logs, configs, figures, and compact JSON/JSONL result summaries go under experiments/trace_keyed_symbol_repair/.

Initial hypothesis:

A repair model trained on wrong patched state + failed execution trace -> corrective diff will outperform no-trace and shuffled-trace controls when the correct fix requires a literal symbol/value revealed only by the failed test output.

Design constraint:

The paper and artifacts must be standalone. They should not depend on or cite earlier experiments.

2026-06-20 Dataset Design and First Build

Design:

Each task has one production file, src/repair_target.py.
The wrong-patched file contains a random wrong CANONICAL_TOKEN.
The correct patch replaces that wrong token with a random expected token.
The expected token is not present in repository context or the issue text.
The expected token is present in the failed pytest output, both in the assertion source and in a custom TRACE_KEY expected_token=... actual_token=... message.
A no-trace model cannot infer the token except by guessing.
A shuffled-trace model sees a plausible trace but for the wrong record, so copying from it should produce an incorrect token.

Splits:

Train: 240 records.
IID validation: 60 records using the same token styles as training.
Format-holdout validation: 60 records using unseen token formats.

Token styles:

Train styles: dash_upper, underscore_upper, mixed_hex.
Format-holdout styles: colon_upper, dot_lower.

Validation performed by the builder:

The wrong patch fails visible tests.
The target corrective diff applies to the wrong-patched file.
The repaired file passes visible and hidden tests.
The repaired file compiles.
The expected token is absent from current files.
The expected token is present in the failing trace.
The wrong token is present in current files.

Generated files:

data/repair_train.jsonl
data/repair_val_iid.jsonl
data/repair_val_format_holdout.jsonl
data/repair_all.jsonl
data/dataset_manifest.json

First build result:

All 360 records passed the builder invariants.
Data directory size: about 3.6 MB.
Full validation was slow because it runs pytest multiple times per generated record; this is acceptable for the final dataset and worth recording as a reproducibility cost.

2026-06-20 Frozen Pilot

Command:

python experiments/trace_keyed_symbol_repair/scripts/eval_trace_keyed.py --data experiments/trace_keyed_symbol_repair/data/repair_val_iid.jsonl --output experiments/trace_keyed_symbol_repair/reports/frozen_trace_iid_pilot10.json --condition trace --max-records 10 --max-new-tokens 192

Result:

Model: Qwen/Qwen2.5-Coder-3B-Instruct, revision 488639f1ff808d1d3d0ba301aef8c11461451ec5.
Records: 10 IID validation examples.
Repair@1: 0/10.
Patch apply rate: 0/10.
Expected-token copy rate: 1/10.

Observed failure mode:

The frozen model usually generated a diff from the original placeholder token to the current wrong token.
That patch does not apply to the wrong-patched tree.
This confirms the task is not solved by the frozen model and is suitable for a training pilot.

Next step:

Train a small trace-conditioned pilot adapter on 80 records for one epoch, then evaluate on 20 IID records.

2026-06-20 Trace Pilot Training

Command:

python scripts/train_repair_lora.py --train experiments/trace_keyed_symbol_repair/data/repair_train.jsonl --eval experiments/trace_keyed_symbol_repair/data/repair_val_iid.jsonl --mode trace --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/trace_keyed_symbol_repair/models/pilot_trace_lora --max-length 2048 --epochs 1 --lr 2e-4 --rank 16 --alpha 32 --dropout 0.05 --grad-accum 8 --save-steps 20 --eval-steps 20 --max-train-records 80

Training observations:

Trainable parameters: 29,933,568, about 0.96% of the model.
Training took about 50 seconds.
Training loss dropped from about 0.13 to 0.0006.
Eval loss on all 60 IID records after the epoch: 0.0005632.

Pilot trace evaluation command:

python experiments/trace_keyed_symbol_repair/scripts/eval_trace_keyed.py --data experiments/trace_keyed_symbol_repair/data/repair_val_iid.jsonl --output experiments/trace_keyed_symbol_repair/reports/pilot_trace_iid20.json --condition trace --adapter large_artifacts/trace_keyed_symbol_repair/models/pilot_trace_lora --max-records 20 --max-new-tokens 192

Pilot trace evaluation result:

Records: 20 IID validation examples.
Repair@1: 20/20.
Patch apply rate: 20/20.
Syntax-valid rate: 20/20.
Expected-token copy rate: 20/20.

Interpretation:

The trace-conditioned task is learnable with a small LoRA and a small training subset.
Next control check: evaluate the same pilot adapter with no trace and shuffled traces.

2026-06-20 Pilot Trace Controls

No-trace control:

Command output: reports/pilot_trace_adapter_no_trace_iid20.json.
Records: 20 IID validation examples.
Repair@1: 0/20.
Patch apply rate: 20/20.
Expected-token copy rate: 0/20.
Interpretation: the adapter can produce syntactically valid patches without the trace, but cannot identify the hidden expected token.

Shuffled-trace control:

Command output: reports/pilot_trace_adapter_shuffled_trace_iid20.json.
Records: 20 IID validation examples.
Repair@1: 0/20.
Patch apply rate: 20/20.
Expected-token copy rate: 0/20.
Interpretation: when given another record's trace, the adapter copies or uses a wrong token-shaped value and fails the hidden test.

Token-budget check:

max_new_tokens=64 truncates long token strings and causes 0/5 repairs.
max_new_tokens=128 preserves 5/5 repairs in a spot check.
Final evaluation will use max_new_tokens=128.

Decision:

The pilot establishes the intended causal contrast: normal trace succeeds, no trace fails, shuffled trace fails.
Proceed to full one-epoch training for trace, no-trace, shuffled-trace, and final-patch conditions on all 240 train records.

2026-06-20 Full Adapter Training

Shared hyperparameters:

Base model: Qwen/Qwen2.5-Coder-3B-Instruct.
Revision: 488639f1ff808d1d3d0ba301aef8c11461451ec5.
QLoRA rank: 16.
LoRA alpha: 32.
Dropout: 0.05.
Epochs: 1.
Learning rate: 2e-4.
Max length: 2048.
Gradient accumulation: 8.
Train records: 240.
Eval records during training: 60 IID validation records.

Adapters:

Trace SFT: large_artifacts/trace_keyed_symbol_repair/models/trace_lora.
No-trace SFT: large_artifacts/trace_keyed_symbol_repair/models/no_trace_lora.
Shuffled-trace SFT: large_artifacts/trace_keyed_symbol_repair/models/shuffled_trace_lora.
Final-patch SFT: large_artifacts/trace_keyed_symbol_repair/models/final_patch_lora.

Training observations:

Trace SFT final train loss: about 0.0105; final eval loss: 4.971e-05.
No-trace SFT final train loss: about 0.4329; final eval loss: 0.4074.
Shuffled-trace SFT final train loss: about 0.4332; final eval loss: 0.3953.
Final-patch SFT final train loss: about 0.5052; final eval loss: 0.4231.

Interpretation before executable evaluation:

Only the normal trace condition fits the target distribution well.
The other conditions remain high-loss because the random expected token is not present in the model input or is present only in an unrelated shuffled trace.
Next step: executable diff evaluation on IID and format-holdout splits.

2026-06-20 Packaging and Reporting Setup

Package structure:

Small experiment package: /workspace/experiments/trace_keyed_symbol_repair/.
Large artifact package: /workspace/large_artifacts/trace_keyed_symbol_repair/.
Adapter directories are excluded from the small package and listed in large_artifacts_manifest.md.
Added PACKAGE_README.md, updated README.md, and copied dependency pins into requirements.txt.

Report generator:

Added scripts/make_report.py.
The script reads pilot and final evaluation JSON files, writes CSV summary tables, generates figures in figures/, writes reports/trace_keyed_symbol_repair_summary.md, writes reports/trace_keyed_symbol_repair_paper.md, and refreshes the large-artifact manifest.
The paper is written as a standalone document and does not depend on prior experiment context.

Current final evaluation status at setup time:

Core suite is running with max_new_tokens=128.
Completed full frozen trace IID and format-holdout evaluations.
Completed full final-patch IID evaluation.
Final-patch format-holdout evaluation is in progress.

2026-06-20 Final Core Evaluation

Command:

python experiments/trace_keyed_symbol_repair/scripts/run_final_evaluations.py --suite core --max-new-tokens 128

Core results:

Frozen base + trace, IID: 0/60 repair@1, patch apply 0/60, expected-token copy 11/60.
Frozen base + trace, format holdout: 0/60 repair@1, patch apply 0/60, expected-token copy 2/60.
Final-patch SFT + final patch, IID: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60.
Final-patch SFT + final patch, format holdout: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60.
No-trace SFT + no trace, IID: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60.
No-trace SFT + no trace, format holdout: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60.
Shuffled-trace SFT + real trace, IID: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60.
Shuffled-trace SFT + real trace, format holdout: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60.
Trace SFT + real trace, IID: 60/60 repair@1, patch apply 60/60, expected-token copy 60/60.
Trace SFT + real trace, format holdout: 60/60 repair@1, patch apply 60/60, expected-token copy 60/60.

Interpretation:

The trace-trained adapter learned to copy the trace-revealed expected token and repair both IID and held-out token formats.
Controls that lacked the correct trace, or were trained with shuffled trace evidence, emitted syntactically valid diffs but did not recover the expected token.
The frozen base sometimes copied a token-shaped string, but generated patches against the wrong source state, so none applied.

Next step:

Run ablations that keep the trace-trained adapter fixed but remove the trace or replace it with shuffled traces at evaluation time.

2026-06-20 Trace Adapter Input Ablations

Command:

python experiments/trace_keyed_symbol_repair/scripts/run_final_evaluations.py --suite ablation --max-new-tokens 128

Ablation results:

Trace SFT + no trace, IID: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60, wrong-token removed 57/60.
Trace SFT + no trace, format holdout: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60, wrong-token removed 54/60.
Trace SFT + shuffled trace, IID: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60, wrong-token removed 60/60.
Trace SFT + shuffled trace, format holdout: 0/60 repair@1, patch apply 60/60, expected-token copy 0/60, wrong-token removed 57/60.

Interpretation:

The trained trace adapter depends on the correct trace at inference time.
With no trace, it often emits valid patches that remove the wrong token but cannot recover the expected token.
With shuffled traces, it emits valid patches keyed to incorrect evidence and fails the hidden tests.

Next step:

Generate final CSV summaries, figures, standalone paper, and package manifest.

2026-06-20 Final Report and Package Verification

Report generation command:

python experiments/trace_keyed_symbol_repair/scripts/make_report.py

Generated report artifacts:

reports/trace_keyed_symbol_repair_paper.md
reports/trace_keyed_symbol_repair_summary.md
reports/final_core_results.csv
reports/final_ablation_results.csv
reports/pilot_results.csv
figures/core_repair_rates.png
figures/expected_token_copy_rates.png
figures/trace_ablation_repair_rates.png
large_artifacts_manifest.md

Verification:

All experiment scripts compiled successfully with python -m py_compile.
The paper and summary contain no references to prior experiment package names.
Required report, figure, README, log, manifest, and CSV artifacts are present.
Final evaluation JSON count: 14.
No files larger than 50 MB remain inside /workspace/experiments/trace_keyed_symbol_repair/.
Removed transient __pycache__ and .ipynb_checkpoints directories from the small package.
No trace-keyed experiment evaluation processes remain running.

Final artifact sizes:

Small package: /workspace/experiments/trace_keyed_symbol_repair/, about 7.6 MB.
Large artifacts: /workspace/large_artifacts/trace_keyed_symbol_repair/, about 1.6 GB.

Previous experiment packaging status:

Small package: /workspace/experiments/execution_conditioned_repair/, about 13 MB.
Large artifacts: /workspace/large_artifacts/execution_conditioned_repair/, about 13 GB.

Figures 3

Data files 3

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

reports/final_ablation_results.csv1.9 kB
reports/final_core_results.csv4.2 kB
reports/pilot_results.csv2.6 kB

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Trace-Keyed Symbol Repair Experiment

Results at a glance 6

Overview

Contents

Main Report

Large Artifacts

Report

Abstract

Artifact Layout

Dataset

Model and Training

Metrics

Pilot Results

Final Results

Trace Adapter Ablations

Figures

Qualitative Examples

Trace-conditioned success

Format-holdout trace-conditioned success

No-trace failure

Shuffled-trace failure

Discussion

Limitations

Reproducibility

Experiment log 10

2026-06-20 Setup

2026-06-20 Dataset Design and First Build

2026-06-20 Frozen Pilot

2026-06-20 Trace Pilot Training

2026-06-20 Pilot Trace Controls

2026-06-20 Full Adapter Training

2026-06-20 Packaging and Reporting Setup

2026-06-20 Final Core Evaluation

2026-06-20 Trace Adapter Input Ablations

2026-06-20 Final Report and Package Verification

Figures 3

Data files 3

Reproduce

Related

Related experiments