Research log Small Model Experimentation
GitHub

Qwen 3.5 4B Model-In-Loop Counterexamples

Finished2026-06-22imported · line ZBenchmark GeneralizationEvidence-Conditioned SelectionReliability and SafetyStructured Execution and CompilersGitHub ↗
Hunting the model's own errors added nothing extra

The one idea you need

Picture a driving instructor fixing a student's habits. One practice course drills tricky maneuvers chosen in advance; another watches the student's own bad turns and drills those exact spots. Here both courses produced flawless drivers, so the personalized one added nothing extra.

The question

When teaching a small model to repair broken programs, does building practice around its own real mistakes beat simply hand-picking the hard categories in advance?

What we found

No. Building practice cases from the model's actual wrong answers matched, but never beat, simply hand-picking the tricky categories in advance. Both lifted the hardest problems from 64% to a perfect 100% passing every hidden test, and the hand-picked set was even a touch cleaner on a single guess (100% versus 99%). The extra error-hunting loop earned nothing here.

Why it matters

Before investing in a loop that mines a model's own failures, first curate targeted hard examples and pair them with a checker that actually runs the output. On a narrow task that simpler recipe can fully solve the benchmark, making the fancier loop wasted effort.

Hard programs solved: generic vs targeted practice64% → 100%share of 72 tricky repair problems passing every hidden test
The one weak spot both practice sets repaired25% → 100%the hardest case type, before vs after adding targeted practice
Error-hunting vs hand-picked, on a single best guess99% vs 100%the hand-picked set was marginally cleaner
When the worked examples are scrambled99% → 36%accuracy collapses, showing the model truly reads them
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Question
    2. Design
    3. Dataset
    4. Mining Summary
    5. Main Results
    6. Readout
    7. Failure Signatures
    8. Interpretation
    9. Per-Condition Details
    10. Artifact Layout
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 3

Both targeted practice sets fully fixed the model's weak spot

How to read

Bars show how often each trained version gets a whole repaired program right on unseen inputs; taller is better. Three versions—no targeted practice, hand-picked practice, and error-hunting practice—each with a single-guess bar and a best-of-a-few-tries bar.

0%25%50%75%100%seed adapterseed adapter62.5%63.9%static bridgestatic bridge100%100%model-loop bridgemodel-loop bridge98.6%100%

Takeaway → The no-targeting bars sit near two-thirds while both targeted versions reach the top, so having targeted practice, not which kind, drives the fix.

Data table
training conditiongreedyreranked (3 samples)
seed adapter62.5%63.9%
static bridge100%100%
model-loop bridge98.6%100%

Numbers from experiments/qwen35_4b_model_in_loop_counterexamples/reports/eval/seed_lora_challenge.json, static_bridge_lora_challenge.json, model_loop_lora_challenge.json (report table)

Technical framing

Challenge-set repair accuracy: counterexample bridge data fixes the seed adapter — Both bridge recipes reach 72/72 reranked vs 46/72 for the seed adapter; static bridge is also perfect under greedy decoding.

Where practice helps: it repairs the one failing case type

How to read

Bars group by three tricky case types (a text-length rule, a paired-number rule, a remainder rule); taller is better. Within each group, the no-targeting version stands beside the two targeted versions, showing how often each fully solves that type on unseen inputs.

0%25%50%75%100%length_contains_codelength_contains_code25%100%100%tuple_branch_labeltuple_branch_label66.7%100%100%modulo_sum_labelmodulo_sum_label100%100%100%

Takeaway → The no-targeting version clears the text-length rule only a quarter of the time while both targeted versions clear every type, pinpointing where the fix lands.

Data table
challenge familyseed adapterstatic bridgemodel-loop bridge
length_contains_code25%100%100%
tuple_branch_label66.7%100%100%
modulo_sum_label100%100%100%

Numbers from experiments/qwen35_4b_model_in_loop_counterexamples/reports/eval/seed_lora_challenge.json, static_bridge_lora_challenge.json, model_loop_lora_challenge.json (report table)

Technical framing

Where counterexamples help: per-family challenge accuracy — The seed adapter's failure is concentrated in length_contains_code (6/24); both bridge conditions repair every family to 24/24.

The worked examples genuinely guide the answer

How to read

Bars show how often the model gets a whole program right when its prompt's worked examples are correct, removed, or scrambled; taller is better.

0%25%50%75%100%aligned tracealigned trace98.6%no traceno trace86.1%shuffled traceshuffled trace36.1%

Takeaway → Accuracy falls from near-perfect to about a third once the examples are scrambled, proving the model reads them rather than treating them as decoration.

Data table
prompt conditionmodel-loop bridge, greedy
aligned trace98.6%
no trace86.1%
shuffled trace36.1%

Numbers from experiments/qwen35_4b_model_in_loop_counterexamples/reports/eval/model_loop_lora_challenge.json, model_loop_lora_no_trace_challenge.json, model_loop_lora_shuffled_trace_challenge.json (report table)

Technical framing

The execution trace is semantically active, not decoration — Removing the trace costs 12.5 points; shuffling it collapses accuracy to 36%, showing the model reads the trace, not just its presence.

In the author’s words from the Report · “Interpretation”

Static bridge and model-loop bridge both solved the challenge set under reranking, while static bridge was cleaner under greedy decoding. Model-loop mining was still useful diagnostically: it exposed the seed adapter's stable wrong hypotheses, especially the count_eq substitute for string length. Trace ablations show the symbolic trace is semantically active. Removing it mainly hurts length_contains_code; shuffling it collapses all challenge families. For this task shape, the strongest training recipe is not yet the extra active-mining loop. It is targeted bridge coverage with an execution-based verifier. … Read the full result →

Overview

This standalone experiment tests whether executable DSL traces become more useful when their visible cases are selected against wrong programs generated by the current model.

The model receives:

  • an input schema,
  • a current wrong DSL program,
  • visible execution cases,
  • and must output one corrected DSL expression.

The experiment trains three fixed-budget adapters:

  • seed_lora: 240 base-family random-trace records.
  • static_bridge_lora: 180 base-family records plus 60 challenge-family static counterexample records.
  • model_loop_lora: 180 base-family records plus 60 challenge-family records selected against seed-adapter wrong programs.

Layout

  • configs/experiment.json: fixed design and hyperparameters.
  • src/: standalone DSL, data, prompt, and model utilities.
  • scripts/: dataset generation, model-candidate mining, training, evaluation, and report entry points.
  • data/: generated JSONL datasets and manifests.
  • reports/: mining JSON, evaluation JSON files, and final report.
  • logs/ and run_logs/: experiment notebook and command output.

Large adapters and checkpoints are intentionally outside this directory:

/workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/

Primary Outputs

  • Final report: reports/qwen35_4b_model_in_loop_counterexamples_report.md.
  • Chronological log: logs/experiment_log.md.
  • Large artifact manifest: large_artifacts_manifest.md.

Main result: static bridge and model-loop bridge both reached 72/72 reranked hidden all-pass on the challenge set, while the seed adapter reached 46/72. Static bridge was cleaner under greedy decoding at 72/72 versus 71/72 for model-loop bridge. All three adapters retained 60/60 on the IID split.

Report

Rendered from reports/qwen35_4b_model_in_loop_counterexamples_report.md

Question

Can counterexamples selected against Qwen-generated wrong DSL programs improve executable program repair beyond static counterexample traces under the same training budget?

Design

  • Base model: Qwen/Qwen3.5-4B.
  • Model output: one executable DSL expression.
  • Training: 4-bit NF4 QLoRA adapters.
  • Training budget: 240 records per trained adapter.
  • Seed adapter: 240 base-family random-trace records.
  • Static bridge adapter: 180 base-family records plus 60 challenge-family static counterexample records.
  • Model-loop bridge adapter: 180 base-family records plus 60 challenge-family records whose traces were selected against seed-adapter wrong programs.
  • Evaluation: parse and execute generated programs on visible and hidden cases.
  • Candidate selection: choose the valid candidate with the most visible-case passes.
  • Large adapter/checkpoint files are stored outside the compact experiment directory.

Dataset

  • Seed train records: 240.
  • Static bridge train records: 240.
  • Model-loop bridge allocation: {'length_contains_code': 40, 'modulo_sum_label': 10, 'tuple_branch_label': 10}.
  • IID eval records: 60.
  • Challenge eval records: 72.
  • Mining pool records: 144.
  • Visible cases per record: 6.
  • Hidden cases per record: 18.

Mining Summary

  • length_contains_code: 36/40 bridge records had sampled model wrong programs; 2 unique wrong programs mined. Top: 105x (if (and (contains text needle) (gt (count_eq text needle) threshold)) "MATCH_LONG" "MISS"); 1x (if (and (contains text needle) (gt (len (first (match-all text needle))) threshold)) "MATCH_LONG" "MISS").
  • modulo_sum_label: 0/10 bridge records had sampled model wrong programs; 0 unique wrong programs mined. Top: none.
  • tuple_branch_label: 10/10 bridge records had sampled model wrong programs; 3 unique wrong programs mined. Top: 14x (if (and (gt (sum item) threshold) (gt (tuple_get item index) 0)) high_label low_label); 6x (if (and (gt (sum item) threshold) (gt (tuple_get item index) threshold)) high_label low_label); 6x (if (and (gt (tuple_get item index) threshold) high_label) low_label).

Main Results

ConditionDataPromptSamplesGreedy HiddenRerank Hiddenmodulo_sum_labellength_contains_codetuple_branch_label
Seed adapterdsl_eval_challenge.jsonltrace362.5% (45/72)63.9% (46/72)100.0% (24/24)25.0% (6/24)66.7% (16/24)
Static bridge adapterdsl_eval_challenge.jsonltrace3100.0% (72/72)100.0% (72/72)100.0% (24/24)100.0% (24/24)100.0% (24/24)
Model-loop bridge adapterdsl_eval_challenge.jsonltrace398.6% (71/72)100.0% (72/72)100.0% (24/24)100.0% (24/24)100.0% (24/24)
Model-loop bridge adapter, no tracedsl_eval_challenge.jsonlno_trace086.1% (62/72)86.1% (62/72)100.0% (24/24)58.3% (14/24)100.0% (24/24)
Model-loop bridge adapter, shuffled tracedsl_eval_challenge.jsonlshuffled_trace036.1% (26/72)36.1% (26/72)45.8% (11/24)41.7% (10/24)20.8% (5/24)
Seed adapter, IIDdsl_eval_iid.jsonltrace0100.0% (60/60)100.0% (60/60)n/an/an/a
Static bridge adapter, IIDdsl_eval_iid.jsonltrace0100.0% (60/60)100.0% (60/60)n/an/an/a
Model-loop bridge adapter, IIDdsl_eval_iid.jsonltrace0100.0% (60/60)100.0% (60/60)n/an/an/a

Readout

  • Challenge reranked hidden all-pass: seed 63.9% (46/72), static bridge 100.0% (72/72), model-loop bridge 100.0% (72/72).
  • length_contains_code reranked hidden all-pass: seed 25.0% (6/24), static bridge 100.0% (24/24), model-loop bridge 100.0% (24/24).
  • tuple_branch_label reranked hidden all-pass: seed 66.7% (16/24), static bridge 100.0% (24/24), model-loop bridge 100.0% (24/24).
  • modulo_sum_label reranked hidden all-pass: seed 100.0% (24/24), static bridge 100.0% (24/24), model-loop bridge 100.0% (24/24).
  • Model-loop trace ablations: aligned trace 98.6% (71/72), no trace 86.1% (62/72), shuffled trace 36.1% (26/72).

Failure Signatures

  • Seed length_contains_code greedy programs: 18/24 (if (and (contains text needle) (gt (count_eq text needle) threshold)) "MATCH_LONG" "MISS"); 6/24 (if (and (contains text needle) (gt (len text) threshold)) "MATCH_LONG" "MISS").
  • Static bridge length_contains_code greedy programs: 24/24 (if (and (contains text needle) (gt (len text) threshold)) "MATCH_LONG" "MISS").
  • Model-loop bridge length_contains_code greedy programs: 24/24 (if (and (contains text needle) (gt (len text) threshold)) "MATCH_LONG" "MISS").

Interpretation

  • Static bridge and model-loop bridge both solved the challenge set under reranking, while static bridge was cleaner under greedy decoding.
  • Model-loop mining was still useful diagnostically: it exposed the seed adapter's stable wrong hypotheses, especially the count_eq substitute for string length.
  • Trace ablations show the symbolic trace is semantically active. Removing it mainly hurts length_contains_code; shuffling it collapses all challenge families.
  • For this task shape, the strongest training recipe is not yet the extra active-mining loop. It is targeted bridge coverage with an execution-based verifier.
  • The next higher-leverage experiment should make bridge selection adaptive only after expanding the held-out challenge space enough that static bridge records no longer saturate it.

Per-Condition Details

seed_lora_challenge

  • Adapter: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/seed_lora.
  • Data: data/eval/dsl_eval_challenge.jsonl.
  • Prompt mode: trace.
  • Samples: 3.
  • Greedy hidden all-pass: 62.5% (45/72).
  • Rerank hidden all-pass: 63.9% (46/72).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
length_contains_code25.0% (6/24)25.0% (6/24)29.2% (7/24)29.2% (7/24)
modulo_sum_label100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)
tuple_branch_label62.5% (15/24)66.7% (16/24)66.7% (16/24)70.8% (17/24)

static_bridge_lora_challenge

  • Adapter: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/static_bridge_lora.
  • Data: data/eval/dsl_eval_challenge.jsonl.
  • Prompt mode: trace.
  • Samples: 3.
  • Greedy hidden all-pass: 100.0% (72/72).
  • Rerank hidden all-pass: 100.0% (72/72).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
length_contains_code100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)
modulo_sum_label100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)
tuple_branch_label100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)

model_loop_lora_challenge

  • Adapter: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/model_loop_lora.
  • Data: data/eval/dsl_eval_challenge.jsonl.
  • Prompt mode: trace.
  • Samples: 3.
  • Greedy hidden all-pass: 98.6% (71/72).
  • Rerank hidden all-pass: 100.0% (72/72).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
length_contains_code100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)
modulo_sum_label100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)
tuple_branch_label95.8% (23/24)100.0% (24/24)95.8% (23/24)100.0% (24/24)

model_loop_lora_no_trace_challenge

  • Adapter: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/model_loop_lora.
  • Data: data/eval/dsl_eval_challenge.jsonl.
  • Prompt mode: no_trace.
  • Samples: 0.
  • Greedy hidden all-pass: 86.1% (62/72).
  • Rerank hidden all-pass: 86.1% (62/72).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
length_contains_code58.3% (14/24)58.3% (14/24)58.3% (14/24)58.3% (14/24)
modulo_sum_label100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)
tuple_branch_label100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)

model_loop_lora_shuffled_trace_challenge

  • Adapter: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/model_loop_lora.
  • Data: data/eval/dsl_eval_challenge.jsonl.
  • Prompt mode: shuffled_trace.
  • Samples: 0.
  • Greedy hidden all-pass: 36.1% (26/72).
  • Rerank hidden all-pass: 36.1% (26/72).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
length_contains_code41.7% (10/24)41.7% (10/24)41.7% (10/24)41.7% (10/24)
modulo_sum_label45.8% (11/24)45.8% (11/24)45.8% (11/24)45.8% (11/24)
tuple_branch_label20.8% (5/24)20.8% (5/24)20.8% (5/24)20.8% (5/24)

seed_lora_iid

  • Adapter: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/seed_lora.
  • Data: data/eval/dsl_eval_iid.jsonl.
  • Prompt mode: trace.
  • Samples: 0.
  • Greedy hidden all-pass: 100.0% (60/60).
  • Rerank hidden all-pass: 100.0% (60/60).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
contains_and_count_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
contains_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
contains_count_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
length_and_mod_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
length_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
length_mod_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
mod_scalar_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
scalar_branch_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sorted_first_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_add_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_and_scalar_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_threshold_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
tuple_get_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
tuple_sum_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)

static_bridge_lora_iid

  • Adapter: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/static_bridge_lora.
  • Data: data/eval/dsl_eval_iid.jsonl.
  • Prompt mode: trace.
  • Samples: 0.
  • Greedy hidden all-pass: 100.0% (60/60).
  • Rerank hidden all-pass: 100.0% (60/60).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
contains_and_count_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
contains_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
contains_count_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
length_and_mod_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
length_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
length_mod_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
mod_scalar_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
scalar_branch_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sorted_first_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_add_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_and_scalar_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_threshold_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
tuple_get_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
tuple_sum_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)

model_loop_lora_iid

  • Adapter: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/model_loop_lora.
  • Data: data/eval/dsl_eval_iid.jsonl.
  • Prompt mode: trace.
  • Samples: 0.
  • Greedy hidden all-pass: 100.0% (60/60).
  • Rerank hidden all-pass: 100.0% (60/60).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
contains_and_count_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
contains_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
contains_count_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
length_and_mod_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
length_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
length_mod_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
mod_scalar_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
scalar_branch_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sorted_first_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_add_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_and_scalar_code100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
sum_threshold_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
tuple_get_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)
tuple_sum_label100.0% (4/4)100.0% (4/4)100.0% (4/4)100.0% (4/4)

Artifact Layout

Experiment log 12

Show the running log (12 entries, 2026-06-22)

2026-06-22

  • Created standalone experiment directory.
  • Selected Qwen/Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
  • Defined the central question: whether counterexamples selected against model-generated wrong DSL programs improve executable repair beyond static counterexamples under the same training budget.

  • Planned fixed-budget adapter conditions:

    • seed adapter: 240 base-family random-trace records,
    • static bridge adapter: 180 base-family records plus 60 challenge-family static counterexample records,
    • model-loop bridge adapter: 180 base-family records plus 60 challenge-family records selected against wrong programs mined from the seed adapter.

  • Bridge allocation:

    • 40 length_contains_code records,
    • 10 modulo_sum_label records,
    • 10 tuple_branch_label records.
  • Large model artifacts will be stored under /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/.

Next step: build and validate the datasets, then train the seed adapter.

Dataset Build

  • Ran python scripts/build_dataset.py.
  • Wrote seed train split: 240 records.
  • Wrote static bridge train split: 240 records.
  • Wrote base anchor split for model-loop training: 180 records.
  • Wrote static challenge bridge records: 60 records.
  • Wrote model mining pool: 144 records, 48 per challenge family.
  • Wrote IID eval split: 60 records.
  • Wrote challenge eval split: 72 records, 24 per challenge family.
  • Each ordinary record has 6 visible cases and 18 hidden cases.
  • Each mining-pool record has an additional 96-case pool for identifying model-generated wrong programs.
  • Dataset manifest: data/dataset_manifest.json.

Next step: train the seed adapter on the 240-record base-family split.

Seed Adapter Training

  • Trained seed_lora on data/seed/dsl_train.jsonl.
  • Training records: 240.
  • Eval records used during training: first 24 IID records from data/eval/dsl_eval_iid.jsonl.
  • Epochs: 2.
  • LoRA rank/alpha/dropout: 32/64/0.05.
  • Final IID eval loss: 0.0000395.
  • Training runtime: about 839 seconds.
  • Adapter output: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/seed_lora.

Next step: evaluate the seed adapter on the challenge split and mine model-generated wrong programs from the mining pool.

Seed Challenge Evaluation

  • Evaluated seed_lora on data/eval/dsl_eval_challenge.jsonl.
  • Prompt mode: aligned trace.
  • Samples: 3.
  • Overall greedy hidden all-pass: 45/72.
  • Overall reranked hidden all-pass: 46/72.

  • Family results, reranked hidden all-pass:

    • modulo_sum_label: 24/24.
    • length_contains_code: 6/24.
    • tuple_branch_label: 16/24.

Next step: mine valid wrong programs from seed_lora on the separate mining pool and build the model-loop training split.

Model-Candidate Mining

  • Ran scripts/mine_model_counterexamples.py with seed_lora.
  • Mining pool records: 144.
  • Candidates per record: 1 greedy + 2 sampled.
  • Wrote mining report: reports/mining/seed_model_mining.json.
  • Wrote model-loop bridge records: data/model_loop/model_mined_bridge_records.jsonl.
  • Wrote model-loop train split: data/model_loop/dsl_train.jsonl.

  • Model-loop train split:

    • 240 records total,
    • 180 base-family records,
    • 60 bridge records.

  • Mined bridge record composition:

    • 50 records with model-generated wrong programs,
    • 10 fallback static records for the solved modulo family.

  • Mining summary:

    • length_contains_code: 36/40 requested bridge records had sampled model wrong programs; top wrong program occurred 105 times: (if (and (contains text needle) (gt (count_eq text needle) threshold)) "MATCH_LONG" "MISS").
    • tuple_branch_label: 10/10 requested bridge records had sampled model wrong programs.
    • modulo_sum_label: 0/10 requested bridge records had sampled model wrong programs, so these use fallback static selectors.

Next step: train the static bridge and model-loop bridge adapters under the same 240-record budget.

Static Bridge Adapter Training

  • Trained static_bridge_lora on data/static_bridge/dsl_train.jsonl.
  • Training records: 240.

  • Composition:

    • 180 base-family records,
    • 60 challenge-family bridge records with static counterexample-selected traces.
  • Epochs: 2.
  • Final IID eval loss: 0.0000992.
  • Training runtime: about 864 seconds.
  • Adapter output: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/static_bridge_lora.

Next step: train the model-loop bridge adapter with the same training budget and hyperparameters.

Model-Loop Bridge Adapter Training

  • Trained model_loop_lora on data/model_loop/dsl_train.jsonl.
  • Training records: 240.

  • Composition:

    • 180 base-family records,
    • 50 challenge-family bridge records selected against model-generated wrong programs,
    • 10 challenge-family fallback static records for the solved modulo family.
  • Epochs: 2.
  • Final IID eval loss: 0.00003141.
  • Training runtime: about 865 seconds.
  • Adapter output: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/models/model_loop_lora.

Next step: run final challenge evaluations and prompt ablations.

Final Challenge Evaluations

  • Evaluated all adapters on data/eval/dsl_eval_challenge.jsonl.

  • Evaluator behavior:

    • prompt mode: aligned trace,
    • samples: 3,
    • candidate selection: choose the valid candidate with the most visible-case passes,
    • success metric: generated DSL program must pass every hidden case for the record.

  • seed_lora:

    • greedy hidden all-pass: 45/72,
    • reranked hidden all-pass: 46/72,
    • length_contains_code: 6/24 reranked hidden all-pass,
    • modulo_sum_label: 24/24 reranked hidden all-pass,
    • tuple_branch_label: 16/24 reranked hidden all-pass.

  • static_bridge_lora:

    • greedy hidden all-pass: 72/72,
    • reranked hidden all-pass: 72/72,
    • all three challenge families: 24/24 reranked hidden all-pass.

  • model_loop_lora:

    • greedy hidden all-pass: 71/72,
    • reranked hidden all-pass: 72/72,
    • length_contains_code: 24/24 reranked hidden all-pass,
    • modulo_sum_label: 24/24 reranked hidden all-pass,
    • tuple_branch_label: 24/24 reranked hidden all-pass.

  • Raw result files:

    • reports/eval/seed_lora_challenge.json,
    • reports/eval/static_bridge_lora_challenge.json,
    • reports/eval/model_loop_lora_challenge.json.

Prompt Controls

  • Evaluated model_loop_lora on the same challenge split with greedy decoding only.

  • No-trace control:

    • prompt mode: no_trace,
    • greedy hidden all-pass: 62/72,
    • length_contains_code: 14/24,
    • modulo_sum_label: 24/24,
    • tuple_branch_label: 24/24.

  • Shuffled-trace control:

    • prompt mode: shuffled_trace,
    • greedy hidden all-pass: 26/72,
    • length_contains_code: 10/24,
    • modulo_sum_label: 11/24,
    • tuple_branch_label: 5/24.

  • Interpretation:

    • aligned traces are semantically active,
    • removing traces mainly damages the length-vs-count distinction,
    • shuffling traces damages every challenge family.

  • Raw result files:

    • reports/eval/model_loop_lora_no_trace_challenge.json,
    • reports/eval/model_loop_lora_shuffled_trace_challenge.json.

IID Retention Checks

  • Evaluated all adapters on data/eval/dsl_eval_iid.jsonl.
  • Prompt mode: aligned trace.
  • Samples: 0.

  • Results:

    • seed_lora: 60/60 greedy hidden all-pass,
    • static_bridge_lora: 60/60 greedy hidden all-pass,
    • model_loop_lora: 60/60 greedy hidden all-pass.

  • Raw result files:

    • reports/eval/seed_lora_iid.json,
    • reports/eval/static_bridge_lora_iid.json,
    • reports/eval/model_loop_lora_iid.json.

Report

  • Generated final report with python scripts/make_report.py.
  • Report path: reports/qwen35_4b_model_in_loop_counterexamples_report.md.

  • Main readout:

    • the seed adapter failed mainly on length_contains_code,
    • static bridge solved the challenge suite at 72/72 greedy and reranked hidden all-pass,
    • model-loop bridge solved the challenge suite at 72/72 after reranking and 71/72 greedy,
    • IID retention remained 60/60 for every adapter.

  • Practical conclusion:

    • the best recipe demonstrated here is targeted bridge coverage plus execution-based verification,
    • model-in-loop mining is valuable as a diagnostic and selector, but it did not beat the simpler static bridge on this challenge suite,
    • the next higher-leverage run should expand the held-out challenge space until static bridge records no longer saturate it, then use active mining to allocate records to unsolved wrong-hypothesis clusters.

Artifact Split

  • Compact experiment directory: /workspace/experiments/qwen35_4b_model_in_loop_counterexamples/.
  • Compact directory size after report generation: about 6.2 MB.
  • Large artifact directory: /workspace/large_artifacts/qwen35_4b_model_in_loop_counterexamples/.
  • Large directory size after training: about 1.4 GB.
  • Large files are LoRA adapter outputs and checkpoint snapshots.
  • Large artifact manifest: large_artifacts_manifest.md.

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗