Qwen 3.5 4B Counterexample-Directed DSL
The one idea you need
A small AI fixes a broken expression in a tiny custom programming language, guided by worked input-and-output examples. Here the examples are hand-picked to expose the gap between the right rule and tempting wrong ones, like a driving examiner choosing exactly the turns that reveal whether you truly understand the rules.
The question
If you give a small AI examples specially chosen to expose the difference between the right rule and tempting wrong ones, does it repair broken code better?
What we found
It depends. Hand-picked examples lifted the model's single-best-guess repair rate from 51% to 58% over random examples. But when it generated several candidates and kept the best, the edge vanished (64% slipped to 61%). One task family reached a perfect score and another was rescued, yet a third collapsed to zero, giving the same valid-but-wrong answer on all 24 hidden tests.
Why it matters
Curated examples reinforce a rule the model already half-knows but cannot install one it never grasped. If it locks onto a plausible wrong reading, pick examples that separate its own mistakes, not just hand-authored distractors, and watch for total collapse.
On this page
Results at a glance 3
How to read
Two situations along the bottom: fixing a program in a single attempt, and keeping the best of several attempts. Paired bars compare training on random worked examples versus targeted ones. Taller means more programs fully fixed on hidden tests.
Takeaway → The targeted-training bar stands taller for the single attempt but shorter for best-of-several, so the extra care pays off only without retries.
Data table
| decoding / selection | random-trace adapter | counterexample-trace adapter |
|---|---|---|
| greedy hidden all-pass | 51.4% | 58.3% |
| reranked hidden all-pass | 63.9% | 61.1% |
Numbers from report table (reports/qwen35_4b_counterexample_directed_dsl_report.md, Main Results)
Technical framing
Counterexample-trained traces help greedy repair but not reranking — On counterexample traces, training on them lifts greedy repair 51.4% to 58.3% but slightly hurts rerank-selected accuracy (63.9% to 61.1%).
How to read
Three kinds of program-fixing task along the bottom. Paired bars compare training on random worked examples versus targeted ones, showing the share of programs fully fixed on unseen cases; taller is better.
Takeaway → On one task the targeted bar drops flat to nothing while hitting full marks on another, showing the method's help is wildly uneven.
Data table
| held-out task family | random-trace adapter | counterexample-trace adapter |
|---|---|---|
| modulo_sum_label | 75% | 100% |
| length_contains_code | 45.8% | 0% |
| tuple_branch_label | 33.3% | 75% |
Numbers from report table (reports/qwen35_4b_counterexample_directed_dsl_report.md, Per-Condition Details)
Technical framing
Effect is family-specific: rescue on two families, total collapse on one — Counterexample training reaches 100% on modulo and 75% on tuple but collapses to 0% on the length family via a stable count_eq alias.
How to read
Bottom axis shows what worked examples the model saw while fixing: proper matched examples, none at all, or scrambled ones. Each bar is the share of programs fully fixed on unseen cases; taller is better.
Takeaway → The scrambled-examples bar is by far the shortest, well below the no-examples bar, proving the model genuinely reads the examples rather than ignoring them.
Data table
| prompt trace condition | counterexample-trace adapter |
|---|---|
| counterexample traces | 58.3% |
| no traces | 52.8% |
| shuffled traces | 22.2% |
Numbers from report table (reports/qwen35_4b_counterexample_directed_dsl_report.md, Main Results)
Technical framing
Coherent traces matter: shuffled traces are worse than no traces — The adapter reads traces rather than ignoring them: removing traces costs 5.5pp, but corrupting them cuts accuracy from 58.3% to 22.2%.
In the author’s words from the Report · “Interpretation”
This experiment gives a mixed but useful answer. Counterexample-directed visible traces are strong supervision when the model has already learned the right primitive composition, as shown by the tuple-family rescue from sampled reranking. They are not sufficient by themselves to force the model to learn the correct latent primitive binding: the length family collapsed into a stable count_eq alias even though the selected traces were intended to distinguish plausible wrong programs. The next iteration should make the counterexamples adaptive to the model's actual wrong program, not just to hand-authored distractors. In this run the selector distinguished the target from planned distractors, but it did not anticipate the learned count_eq alias. … Read the full result →
Overview
This standalone experiment tests whether visible execution traces become more useful when their examples are chosen to distinguish the target executable program from plausible wrong programs.
The model receives:
- an input schema,
- a current wrong DSL program,
- visible execution cases,
- and must output one corrected DSL expression.
The evaluator parses and executes generated DSL candidates on visible cases, optionally reranks candidates by visible pass count, and scores hidden cases.
Layout
configs/experiment.json: fixed design and hyperparameters.src/: standalone DSL, data, prompt, and model utilities.scripts/: dataset generation, training, evaluation, and report entry points.data/: generated JSONL datasets and manifests.reports/: evaluation JSON files and final report.logs/andrun_logs/: experiment notebook and command output.
Large adapters and checkpoints are intentionally outside this directory:
/workspace/large_artifacts/qwen35_4b_counterexample_directed_dsl/
Report
Rendered from reports/qwen35_4b_counterexample_directed_dsl_report.md
Question
Can visible traces chosen as counterexamples to plausible wrong programs improve executable DSL repair compared with random visible traces?
Design
- Base model:
Qwen/Qwen3.5-4B. - Model output: one executable DSL expression.
- Training: 4-bit NF4 QLoRA adapters.
- Evaluation: parse and execute generated programs on visible and hidden cases.
- Candidate selection: choose the valid candidate with the most visible-case passes.
- Main held-out families:
modulo_sum_label,length_contains_code, andtuple_branch_label. - Large adapter/checkpoint files are stored outside the compact experiment directory.
Dataset
- Random trace train records: 240.
- Counterexample trace train records: 240.
- Holdout records per trace regime: 72.
- Visible cases per record: 6.
- Hidden cases per record: 18.
Main Results
| Condition | Prompt | Samples | Greedy Hidden | Rerank Hidden | modulo_sum_label | length_contains_code | tuple_branch_label |
|---|---|---|---|---|---|---|---|
| Random-trace adapter on random traces | trace | 3 | 58.3% (42/72) | 72.2% (52/72) | 95.8% (23/24) | 66.7% (16/24) | 54.2% (13/24) |
| Random-trace adapter on counterexample traces | trace | 3 | 51.4% (37/72) | 63.9% (46/72) | 91.7% (22/24) | 58.3% (14/24) | 41.7% (10/24) |
| Counterexample-trace adapter on counterexample traces | trace | 3 | 58.3% (42/72) | 61.1% (44/72) | 100.0% (24/24) | 0.0% (0/24) | 83.3% (20/24) |
| Counterexample-trace adapter, no trace | no_trace | 0 | 52.8% (38/72) | 52.8% (38/72) | 100.0% (24/24) | 0.0% (0/24) | 58.3% (14/24) |
| Counterexample-trace adapter, shuffled trace | shuffled_trace | 0 | 22.2% (16/72) | 22.2% (16/72) | 45.8% (11/24) | 0.0% (0/24) | 20.8% (5/24) |
Readout
- Counterexample-directed training improved greedy hidden all-pass on the counterexample holdout from 51.4% (37/72) to 58.3% (42/72), but did not improve reranked hidden all-pass: 63.9% (46/72) to 61.1% (44/72).
- Coherent traces mattered for the counterexample-trained adapter: no-trace greedy hidden all-pass was 52.8% (38/72), while shuffled traces fell to 22.2% (16/72).
- The effect was family-specific. The counterexample-trained adapter reached 100.0% (24/24) on
modulo_sum_labeland 83.3% (20/24) ontuple_branch_label, but stayed at 0.0% (0/24) onlength_contains_code. - The main failure was not syntax. On
length_contains_code, the counterexample-trained adapter generated(if (and (contains text needle) (gt (count_eq text needle) threshold)) "MATCH_LONG" "MISS")on 24/24 holdout records. This valid program confuses text length with needle count; sampling produced no useful diversity for reranking. - The random-trace adapter was less collapsed on the same
length_contains_codeholdout. Its top greedy programs were: 9/24(if (and (contains text needle) (gt (len text) threshold)) "MATCH_LONG" "MISS"); 9/24(if (contains text needle) "MATCH_LONG" (if (gt (len text) threshold) "MATCH_LONG" "MISS")); 3/24(if (contains text needle) (if (len text) gt threshold "MATCH_LONG" "MISS") "MISS").
Interpretation
This experiment gives a mixed but useful answer. Counterexample-directed visible traces are strong supervision when the model has already learned the right primitive composition, as shown by the tuple-family rescue from sampled reranking. They are not sufficient by themselves to force the model to learn the correct latent primitive binding: the length family collapsed into a stable count_eq alias even though the selected traces were intended to distinguish plausible wrong programs.
The next iteration should make the counterexamples adaptive to the model's actual wrong program, not just to hand-authored distractors. In this run the selector distinguished the target from planned distractors, but it did not anticipate the learned count_eq alias. A stronger loop would sample candidate model programs during training-data construction, execute them, add traces that separate those candidates from the target, and then retrain or continue training on those model-specific counterexamples.
Per-Condition Details
random_lora_random_holdout
- Adapter:
/workspace/large_artifacts/qwen35_4b_counterexample_directed_dsl/models/random_trace_lora. - Data:
data/random/dsl_eval_holdout.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 58.3% (42/72).
- Rerank hidden all-pass: 72.2% (52/72).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| length_contains_code | 54.2% (13/24) | 66.7% (16/24) | 54.2% (13/24) | 79.2% (19/24) |
| modulo_sum_label | 87.5% (21/24) | 95.8% (23/24) | 87.5% (21/24) | 95.8% (23/24) |
| tuple_branch_label | 33.3% (8/24) | 54.2% (13/24) | 33.3% (8/24) | 54.2% (13/24) |
random_lora_counterexample_holdout
- Adapter:
/workspace/large_artifacts/qwen35_4b_counterexample_directed_dsl/models/random_trace_lora. - Data:
data/counterexample/dsl_eval_holdout.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 51.4% (37/72).
- Rerank hidden all-pass: 63.9% (46/72).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| length_contains_code | 45.8% (11/24) | 58.3% (14/24) | 45.8% (11/24) | 58.3% (14/24) |
| modulo_sum_label | 75.0% (18/24) | 91.7% (22/24) | 75.0% (18/24) | 91.7% (22/24) |
| tuple_branch_label | 33.3% (8/24) | 41.7% (10/24) | 33.3% (8/24) | 41.7% (10/24) |
counterexample_lora_counterexample_holdout
- Adapter:
/workspace/large_artifacts/qwen35_4b_counterexample_directed_dsl/models/counterexample_trace_lora. - Data:
data/counterexample/dsl_eval_holdout.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 58.3% (42/72).
- Rerank hidden all-pass: 61.1% (44/72).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| length_contains_code | 0.0% (0/24) | 0.0% (0/24) | 0.0% (0/24) | 0.0% (0/24) |
| modulo_sum_label | 100.0% (24/24) | 100.0% (24/24) | 100.0% (24/24) | 100.0% (24/24) |
| tuple_branch_label | 75.0% (18/24) | 83.3% (20/24) | 79.2% (19/24) | 83.3% (20/24) |
counterexample_lora_no_trace_holdout
- Adapter:
/workspace/large_artifacts/qwen35_4b_counterexample_directed_dsl/models/counterexample_trace_lora. - Data:
data/counterexample/dsl_eval_holdout.jsonl. - Prompt mode:
no_trace. - Samples: 0.
- Greedy hidden all-pass: 52.8% (38/72).
- Rerank hidden all-pass: 52.8% (38/72).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| length_contains_code | 0.0% (0/24) | 0.0% (0/24) | 0.0% (0/24) | 0.0% (0/24) |
| modulo_sum_label | 100.0% (24/24) | 100.0% (24/24) | 100.0% (24/24) | 100.0% (24/24) |
| tuple_branch_label | 58.3% (14/24) | 58.3% (14/24) | 58.3% (14/24) | 58.3% (14/24) |
counterexample_lora_shuffled_trace_holdout
- Adapter:
/workspace/large_artifacts/qwen35_4b_counterexample_directed_dsl/models/counterexample_trace_lora. - Data:
data/counterexample/dsl_eval_holdout.jsonl. - Prompt mode:
shuffled_trace. - Samples: 0.
- Greedy hidden all-pass: 22.2% (16/72).
- Rerank hidden all-pass: 22.2% (16/72).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| length_contains_code | 0.0% (0/24) | 0.0% (0/24) | 0.0% (0/24) | 0.0% (0/24) |
| modulo_sum_label | 45.8% (11/24) | 45.8% (11/24) | 45.8% (11/24) | 45.8% (11/24) |
| tuple_branch_label | 20.8% (5/24) | 20.8% (5/24) | 20.8% (5/24) | 20.8% (5/24) |
Artifact Layout
- Compact artifacts:
/workspace/experiments/qwen35_4b_counterexample_directed_dsl/. - Large artifacts:
/workspace/large_artifacts/qwen35_4b_counterexample_directed_dsl/. - Dataset manifest:
data/dataset_manifest.json. - Evaluation JSON files:
reports/eval/.
Experiment log 5
Show the running log (5 entries, 2026-06-22)
2026-06-22
- Created standalone experiment directory.
- Selected
Qwen/Qwen3.5-4Brevision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. Designed the data generator to emit two trace regimes:
- random visible cases,
- counterexample-directed visible cases selected to distinguish the target program from plausible wrong programs.
Dataset Build
- Generated standalone datasets with
python scripts/build_dataset.py. Random trace regime:
- 240 train records,
- 60 IID eval records,
- 72 compositional holdout records.
Counterexample trace regime:
- 240 train records,
- 60 IID eval records,
- 72 compositional holdout records.
- Each record has 6 visible cases and 18 hidden cases.
- Heldout families:
modulo_sum_label,length_contains_code,tuple_branch_label. - The counterexample selector used a 160-case candidate pool.
Training
- Ran a smoke adapter first on 8 random-trace records to validate QLoRA loading, training, saving, and evaluation.
Trained a random-trace LoRA adapter on
data/random/dsl_train.jsonl.- Final IID eval loss: 0.0005662.
- Training runtime: about 875 seconds.
- Output directory:
/workspace/large_artifacts/qwen35_4b_counterexample_directed_dsl/models/random_trace_lora.
Trained a counterexample-trace LoRA adapter on
data/counterexample/dsl_train.jsonl.- Final IID eval loss: 0.00005084.
- Training runtime: about 840 seconds.
- Output directory:
/workspace/large_artifacts/qwen35_4b_counterexample_directed_dsl/models/counterexample_trace_lora.
Evaluation
Random-trace adapter on random-trace holdout:
- Greedy hidden all-pass: 42/72.
- Reranked hidden all-pass: 52/72.
Random-trace adapter on counterexample-trace holdout:
- Greedy hidden all-pass: 37/72.
- Reranked hidden all-pass: 46/72.
Counterexample-trace adapter on counterexample-trace holdout:
- Greedy hidden all-pass: 42/72.
- Reranked hidden all-pass: 44/72.
modulo_sum_label: 24/24 reranked hidden all-pass.length_contains_code: 0/24 reranked hidden all-pass.tuple_branch_label: 20/24 reranked hidden all-pass.
Counterexample-trace adapter with no visible trace:
- Greedy hidden all-pass: 38/72.
Counterexample-trace adapter with shuffled visible traces:
- Greedy hidden all-pass: 16/72.
Iteration Notes
- Coherent traces help the counterexample-trained adapter: removing traces drops the aggregate from 42/72 greedy hidden all-pass to 38/72, and shuffling traces drops it to 16/72.
- The main negative result is the
length_contains_codecollapse. The counterexample-trained adapter emitted(if (and (contains text needle) (gt (count_eq text needle) threshold)) "MATCH_LONG" "MISS")on all 24 length-family holdout records. - The length-family failure is semantic, not syntactic: the generated program is valid but binds the threshold comparison to needle count instead of text length.
- Sampling did not recover the target on that family because all sampled candidates repeated the same wrong program.
- The next experiment should make counterexamples adaptive to model-generated wrong programs instead of only to hand-authored distractors.
Reproduce
Run steps are documented inside the experiment folder (README and scripts).