Research log Small Model Experimentation
GitHub

Qwen 3.5 4B Executable Program Posttraining

Finished2026-06-22imported · line ZBenchmark GeneralizationStructured Execution and CompilersGitHub ↗
Show the worked steps and runnable fixes transfer

The one idea you need

Instead of eyeballing a broken formula and guessing a patch, the model rewrites it as a small runnable program, which the computer then test-drives on real example cases to see if it truly works — like a mechanic rebuilding an engine and road-testing it, not just trusting the manual.

The question

Can a small model learn to fix broken little programs by rewriting them, so the fixes still work on kinds of problems it never saw during training?

What we found

Yes, but with a catch. Shown worked-through reasoning in the prompt, the model fixed unseen problem types about three-quarters of the time, versus one-in-three when the prompt showed no steps. Strip out or scramble those steps and most of the gain vanishes. One two-condition problem type stayed stubborn, rising from never solved to roughly one in three only after adding extra practice examples.

Why it matters

When you post-train a small model to emit runnable fixes, keep matching worked-through steps in every prompt — missing or scrambled steps erase most of the gain. And cover combined two-condition patterns in training, or they quietly fail.

Fixes that worked on unseen problems33% → 75%with no worked steps versus with matching worked steps
The stubborn two-condition problem0 of 24 → 7 of 24solved after adding that pattern to training practice
Two of three unseen problem types24 of 24, and 23 of 24near-perfect transfer to families never trained on
Worked steps scrambled on purposedrops to 27 of 72shuffling the steps wrecks accuracy, proving they carry real signal
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Question
    2. Design
    3. Dataset
    4. Iteration Readout
    5. Results
    6. Artifact Layout
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 3

Success on unseen problems across four training-and-prompt setups

How to read

Each cluster is one setup; bar height is the share of unseen problems fully solved. Within a cluster, one bar is a single best guess, the other picks the best of a few tries. Taller is better.

0%20%40%60%80%initial trace adapter (trace, 12 samples)initial trace adapter (trace, 12 …65.3%66.7%conjunction-support adapter (trace, 3 samples)conjunction-support adapter (trac…72.2%75%conjunction-support (no trace, greedy)conjunction-support (no trace, gr…33.3%33.3%conjunction-support (shuffled trace, greedy)conjunction-support (shuffled tra…37.5%37.5%

Takeaway → The two setups that include matching worked steps tower over the two without — about three-quarters of unseen problems solved versus one-in-three — and best-of-few barely helps.

Data table
Conditiongreedyvisible-test rerank
initial trace adapter (trace, 12 samples)65.3%66.7%
conjunction-support adapter (trace, 3 samples)72.2%75%
conjunction-support (no trace, greedy)33.3%33.3%
conjunction-support (shuffled trace, greedy)37.5%37.5%

Numbers from report table (reports/qwen35_4b_executable_program_posttraining_report.md; backed by reports/eval/*.json)

Technical framing

Held-out hidden-case success by adapter and prompt condition — Executable-program posttraining transfers to held-out families (75%), but only when the prompt carries aligned visible traces.

Success on each unseen problem type, before and after extra practice

How to read

Each cluster is one unseen problem type; bar height is the share fully solved when picking the best of a few tries. The two bars compare the first model with one given extra practice on two-condition problems. Taller is better.

0%25%50%75%100%modulo_sum_labelmodulo_sum_label100%100%length_contains_codelength_contains_code0%29.2%tuple_branch_labeltuple_branch_label100%95.8%

Takeaway → The stubborn two-condition type climbs from never solved to about a third after extra practice, while the other two types stay near-perfect.

Data table
Held-out familyinitial trace adapterconjunction-support adapter
modulo_sum_label100%100%
length_contains_code0%29.2%
tuple_branch_label100%95.8%

Numbers from report table (reports/qwen35_4b_executable_program_posttraining_report.md; backed by reports/eval/trace_lora_trace_holdout.json and trace_and_bridge_lora_trace_holdout_samples3.json)

Technical framing

Per-family held-out success: initial vs conjunction-support adapter — The 0/24 failure on length+contains was a specific mechanism error; adding training-only conjunction families lifted it to 7/24 without hurting the others.

What removing or scrambling the worked steps does to each problem type

How to read

Each cluster is one unseen problem type; bar height is the share fully solved. The three bars are a prompt with matching worked steps, no steps, or scrambled steps. Taller is better.

0%25%50%75%100%modulo_sum_labelmodulo_sum_label100%100%70.8%length_contains_codelength_contains_code20.8%0%20.8%tuple_branch_labeltuple_branch_label95.8%0%20.8%

Takeaway → Removing the steps drops the branching type from near-perfect to zero, and even scrambled steps cap it at a fifth — the steps do real work.

Data table
Held-out familyaligned traceno traceshuffled trace
modulo_sum_label100%100%70.8%
length_contains_code20.8%0%20.8%
tuple_branch_label95.8%0%20.8%

Numbers from report table (reports/qwen35_4b_executable_program_posttraining_report.md; backed by reports/eval/trace_and_bridge_lora_no_trace_holdout_greedy.json and trace_and_bridge_lora_shuffled_trace_holdout_greedy.json)

Technical framing

Trace ablation: per-family hidden success for the conjunction-support adapter — Aligned visible traces do real work: removing or shuffling them collapses tuple_branch_label from 95.8% to at most 20.8%.

In the author’s words from the Overview

This standalone experiment tests whether a small posttraining change can move Qwen 3.5 4B from direct text repair into executable program repair. The model receives a wrong DSL program plus visible failing cases and emits a corrected DSL expression. The evaluator executes generated programs on visible cases, optionally reranks sampled candidates by visible pass count, and then scores hidden cases. configs/experiment.json: fixed model, data, training, and evaluation settings. src/: standalone DSL, prompting, data, and model utilities. scripts/: dataset generation, training, evaluation, and reporting entry points. data/: generated JSONL train/eval records. reports/: metrics, per-record generations, and final writeup. … Read the full result →

Overview

This standalone experiment tests whether a small posttraining change can move Qwen 3.5 4B from direct text repair into executable program repair.

The model receives a wrong DSL program plus visible failing cases and emits a corrected DSL expression. The evaluator executes generated programs on visible cases, optionally reranks sampled candidates by visible pass count, and then scores hidden cases.

Layout

  • configs/experiment.json: fixed model, data, training, and evaluation settings.
  • src/: standalone DSL, prompting, data, and model utilities.
  • scripts/: dataset generation, training, evaluation, and reporting entry points.
  • data/: generated JSONL train/eval records.
  • reports/: metrics, per-record generations, and final writeup.
  • logs/ and run_logs/: experiment notes and command output.

Large generated artifacts are intentionally outside this directory:

The compact experiment directory can be downloaded without adapter weights or checkpoints.

Intended Run

python scripts/build_dataset.py
python scripts/train_dsl_lora.py --mode trace --output-dir /workspace/large_artifacts/qwen35_4b_executable_program_posttraining/models/dsl_trace_lora
python scripts/eval_dsl.py --adapter /workspace/large_artifacts/qwen35_4b_executable_program_posttraining/models/dsl_trace_lora --prompt-mode trace --split holdout --output reports/eval_trace_holdout.json
python scripts/make_report.py

Report

Rendered from reports/qwen35_4b_executable_program_posttraining_report.md

Question

Can a Qwen 3.5 4B adapter trained to emit executable DSL repair programs produce programs that generalize to held-out composition families, and does visible-test reranking improve hidden-case success?

Design

  • Base model: Qwen/Qwen3.5-4B.
  • Model output: one executable DSL expression.
  • Evaluator: parses and executes generated programs on visible and hidden cases.
  • Reranking: samples candidate programs and selects the valid candidate with the most visible-case passes.
  • Main held-out families: modulo_sum_label, length_contains_code, and tuple_branch_label.
  • Adapter weights and checkpoints are stored outside the compact directory under /workspace/large_artifacts/qwen35_4b_executable_program_posttraining/models/.

Dataset

  • Train records: 240.
  • IID eval records: 60.
  • Holdout eval records: 72.
  • Visible cases per record: 6.
  • Hidden cases per record: 18.

Iteration Readout

The first trace-trained executable-program adapter transferred cleanly on two held-out families but failed the length+contains family completely. Inspection showed that failed generations repeatedly substituted count_eq text needle for the needed len text predicate inside a conjunction. A second adapter was trained from scratch with three training-only conjunction families added under the same 240-record budget.

Key held-out results:

ConditionPromptSamplesGreedy HiddenRerank Hiddenmodulo_sum_labellength_contains_codetuple_branch_label
Initial trace adaptertrace1265.3% (47/72)66.7% (48/72)100.0% (24/24)0.0% (0/24)100.0% (24/24)
Conjunction-support trace adaptertrace372.2% (52/72)75.0% (54/72)100.0% (24/24)29.2% (7/24)95.8% (23/24)
Conjunction-support adapterno_trace033.3% (24/72)33.3% (24/72)100.0% (24/24)0.0% (0/24)0.0% (0/24)
Conjunction-support adaptershuffled_trace037.5% (27/72)37.5% (27/72)70.8% (17/24)20.8% (5/24)20.8% (5/24)

Readout:

  • Executable DSL posttraining produced a large held-out signal on modulo_sum_label and tuple_branch_label.
  • The initial failure on length_contains_code was not random formatting noise; it was a specific mechanism error.
  • Adding non-held-out conjunction training families moved length_contains_code from 0/24 to 7/24 under visible reranking, while preserving 24/24 on modulo and 23/24 on tuple.
  • Aligned visible traces mattered: the conjunction-support adapter scored 54/72 with aligned trace plus 3 samples, 24/72 with no trace greedy, and 27/72 with shuffled trace greedy.
  • A full 12-sample evaluation of the second adapter was started but stopped after two records because generations were taking over 90 seconds per record. The reported second-adapter rerank condition uses 3 samples and a 64-token cap.

Results

trace_and_bridge_lora_no_trace_holdout_greedy

  • Adapter: /workspace/large_artifacts/qwen35_4b_executable_program_posttraining/models/dsl_trace_and_bridge_lora.
  • Prompt mode: no_trace.
  • Records: 72.
  • Greedy hidden all-pass: 33.3% (24/72).
  • Visible-rerank hidden all-pass: 33.3% (24/72).
  • Greedy visible all-pass: 34.7% (25/72).
  • Rerank visible all-pass: 34.7% (25/72).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
length_contains_code0.0% (0/24)0.0% (0/24)4.2% (1/24)4.2% (1/24)
modulo_sum_label100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)
tuple_branch_label0.0% (0/24)0.0% (0/24)0.0% (0/24)0.0% (0/24)

trace_and_bridge_lora_shuffled_trace_holdout_greedy

  • Adapter: /workspace/large_artifacts/qwen35_4b_executable_program_posttraining/models/dsl_trace_and_bridge_lora.
  • Prompt mode: shuffled_trace.
  • Records: 72.
  • Greedy hidden all-pass: 37.5% (27/72).
  • Visible-rerank hidden all-pass: 37.5% (27/72).
  • Greedy visible all-pass: 37.5% (27/72).
  • Rerank visible all-pass: 37.5% (27/72).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
length_contains_code20.8% (5/24)20.8% (5/24)20.8% (5/24)20.8% (5/24)
modulo_sum_label70.8% (17/24)70.8% (17/24)70.8% (17/24)70.8% (17/24)
tuple_branch_label20.8% (5/24)20.8% (5/24)20.8% (5/24)20.8% (5/24)

trace_and_bridge_lora_trace_holdout_samples3

  • Adapter: /workspace/large_artifacts/qwen35_4b_executable_program_posttraining/models/dsl_trace_and_bridge_lora.
  • Prompt mode: trace.
  • Records: 72.
  • Greedy hidden all-pass: 72.2% (52/72).
  • Visible-rerank hidden all-pass: 75.0% (54/72).
  • Greedy visible all-pass: 72.2% (52/72).
  • Rerank visible all-pass: 75.0% (54/72).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
length_contains_code20.8% (5/24)29.2% (7/24)20.8% (5/24)29.2% (7/24)
modulo_sum_label100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)
tuple_branch_label95.8% (23/24)95.8% (23/24)95.8% (23/24)95.8% (23/24)

trace_lora_trace_holdout

  • Adapter: /workspace/large_artifacts/qwen35_4b_executable_program_posttraining/models/dsl_trace_lora.
  • Prompt mode: trace.
  • Records: 72.
  • Greedy hidden all-pass: 65.3% (47/72).
  • Visible-rerank hidden all-pass: 66.7% (48/72).
  • Greedy visible all-pass: 66.7% (48/72).
  • Rerank visible all-pass: 68.1% (49/72).
FamilyGreedy HiddenRerank HiddenGreedy VisibleRerank Visible
length_contains_code0.0% (0/24)0.0% (0/24)4.2% (1/24)4.2% (1/24)
modulo_sum_label100.0% (24/24)100.0% (24/24)100.0% (24/24)100.0% (24/24)
tuple_branch_label95.8% (23/24)100.0% (24/24)95.8% (23/24)100.0% (24/24)

Artifact Layout

Experiment log 1

Show the running log (1 entry, 2026-06-22)

2026-06-22

  • Created standalone experiment directory.
  • Selected official base model Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
  • Confirmed the model loads through AutoModelForImageTextToText and can generate from a text-only prompt.
  • Designed the experiment around executable DSL repair with visible-test reranking.
  • Initial trace adapter completed. Full held-out trace evaluation showed strong modulo and tuple transfer but complete length+contains hidden failure.
  • Inspected failed length+contains generations. The model repeatedly emitted (count_eq text needle) forms instead of the required conjunction, indicating missing conjunction support in the supervised DSL distribution.
  • Added training-only conjunction families while keeping the 240-record training budget fixed for the next iteration.

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗