Qwen3.5-4B Code ABI Compiler Heldout Primitive Pilot

Finished2026-06-27imported · line ZStructured Execution and Compilers GitHub ↗

The one idea you need

Think of a fixed box of prebuilt code blocks, tuned so you can snap together solutions to a set of practice coding problems. Before teaching a small model to write programs from that box, first check: do the blocks even fit brand-new problems?

The question

Before training a small model to solve coding tasks by snapping together a fixed kit of prebuilt building blocks, can that kit actually express brand-new problems?

What we found

No. The block kit could build a working solution for 84% of the problems it was tuned against, but only 14% of brand-new ones — a 70-point collapse. Three fresh random batches of new problems all landed near 18%, so it wasn't bad luck. The kit was overfit to its own examples, so model training was skipped entirely.

Why it matters

Before spending compute to train a model on a reusable code library, first check how well it covers unseen problems. Build the blocks from a source independent of your test problems, or you will measure memorization dressed up as reuse.

Problems the kit can build: familiar vs brand-new84% → 14%share of coding tasks the fixed block kit could assemble a solution for

Coverage across random new-problem batchesabout 18%three independent reshuffles, all between 17.5% and 19.4%

Answers that looked right but were actually wrong9% → 70%share passing quick checks yet failing hidden ones, familiar vs new

Solved new problems needing more than one step6 of 22most covered new problems were single-step; multi-step composition barely appeared

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 3

Toolbox solves less and misleads more on new problems

How to read

Bars group three problem sets: familiar practice, brand-new, and the training pool. First bar per group is the share of problems solved (taller is better); second bar is the share of answers that passed quick checks but were actually wrong (taller is worse).

Takeaway → Solving cliff-drops from a tall familiar bar to a stub on new problems, while the wrong-but-convincing bar leaps up — the kit both covers less and misleads more.

Data table

MBPP split	oracle coverage	false-pass rate among visible-pass tasks
calibration	83.8%	8.8%
held-out	13.8%	69.9%
train	22.2%	57.4%

Numbers from experiments/qwen35_4b_code_abi_compiler_heldout_primitive_pilot/reports/gate1_summary.json

Technical framing

Frozen-ABI oracle coverage collapses on held-out tasks (Gate 1 failed) — Coverage drops from 83.8% (calibration) to 13.8% (held-out) while false passes soar, so compiler training was not run.

Every kind of coding task collapses the same way

How to read

Bars group five task categories (dictionaries, lists, numbers, text patterns, other), each with a familiar-problem bar and a brand-new-problem bar showing the share of problems solved; taller is better. Compare the two bars within each category.

Takeaway → In every category the new-problem bar shrinks to a fraction of the familiar one — even the dictionary category that solved 100% of familiar problems — so no task type escapes.

Data table

task category	calibration	held-out
dict/counter	100%	14.3%
list/tuple	89.5%	17.6%
numeric	79.2%	6.7%
string/regex	95.5%	13.8%
other	62.5%	18.5%

Numbers from experiments/qwen35_4b_code_abi_compiler_heldout_primitive_pilot/reports/gate1_summary.json

Technical framing

Coverage collapse holds across every task category — No category is spared: even dict/counter tasks fall from 100% covered on calibration to 14% on held-out tasks.

Fresh random batches confirm the poor result is real

How to read

Three bars, one per randomly drawn batch of brand-new problems; bar height is the share the toolbox could solve, where taller is better. A flat, low set of bars means the weak result was not just an unlucky draw.

Takeaway → All three random batches land in a narrow low band near 18%, confirming the collapse was typical rather than a fluke of one unlucky sample.

Data table

random 160-task held-out sample	frozen-ABI oracle coverage
seed 11	19.4%
seed 17	18.1%
seed 23	17.5%

Numbers from experiments/qwen35_4b_code_abi_compiler_heldout_primitive_pilot/reports/gate1_split_sweep.json

Technical framing

Random held-out resamples confirm the fixed slice was not unlucky — Three random held-out subsets stay at 17.5-19.4% coverage (mean 18.3%), close to the fixed slice's 13.8%.

In the author’s words from the Overview

Standalone experiment for a frozen code-primitive ABI compiler pilot. The experiment freezes a reusable ABI, measures held-out oracle coverage before training, then trains/evaluates a Qwen3.5-4B QLoRA compiler to emit ABI programs that are executed by a deterministic interpreter.

Overview

Standalone experiment for a frozen code-primitive ABI compiler pilot.

The experiment freezes a reusable ABI, measures held-out oracle coverage before training, then trains/evaluates a Qwen3.5-4B QLoRA compiler to emit ABI programs that are executed by a deterministic interpreter. Checkpoints are stored outside this directory under /workspace/large_artifacts/qwen35_4b_code_abi_compiler_heldout_primitive_pilot/.

Report

Rendered from reports/final_report.md

Purpose

This standalone experiment tests whether a frozen code-primitive ABI remains reusable on held-out MBPP tasks before training a Qwen3.5-4B compiler to emit ABI programs.

The package uses a fixed ABI implementation and does not add kernels after seeing the held-out tasks. The first gate is oracle coverage: if the ABI cannot express held-out tasks, compiler training would not test reusable compilation.

Gate 1 Result

Gate 1 failed. Frozen-ABI oracle coverage dropped from 134/160 (83.8%) on the calibration slice to 22/160 (13.8%) on the held-out slice, a drop of 70.0%.

The train split also has low coverage: 83/374 (22.2%), leaving only 66 compiler-training targets after the deterministic validation split. That is not enough to make a QLoRA compiler result meaningful.

Gate 1 coverage

split	n	oracle-covered	oracle coverage	first-visible correct	visible-any tasks	task false-pass rate	candidate hidden-wrong rate
calibration	160	134	83.8%	95	147	8.8%	79.2%
heldout	160	22	13.8%	8	73	69.9%	94.7%
train	374	83	22.2%	45	195	57.4%	88.0%

Split-Sweep Check

To check whether the fixed held-out slice was unlucky, the experiment sampled three random 160-task subsets from the test suffix excluded from the calibration slice. Coverage remained low: mean 18.3%, range 17.5%-19.4%.

seed	n	oracle-covered	oracle coverage	first-visible correct	task false-pass rate
11	160	31	19.4%	15	55.1%
17	160	29	18.1%	15	61.3%
23	160	28	17.5%	13	61.6%

Held-out seed sweep

Slice Diagnostics

slice	calibration coverage	heldout coverage
algorithmic_control	0/0 (0.0%)	0/1 (0.0%)
dict_counter	9/9 (100.0%)	1/7 (14.3%)
list_tuple	51/57 (89.5%)	9/51 (17.6%)
numeric	38/48 (79.2%)	3/45 (6.7%)
other	15/24 (62.5%)	5/27 (18.5%)
string_regex	21/22 (95.5%)	4/29 (13.8%)

Coverage by slice

Depth Diagnostics

Covered held-out tasks are mostly depth-1 programs: {'1': 16, '2': 3, '3': 3}. There are too few held-out depth-2/3 targets to support a composition-training claim.

Depth counts

Decision

The compiler-training arm was intentionally not run. The precondition for interpreting it did not hold: the frozen ABI was not broadly reusable on held-out tasks. Training Qwen to emit this ABI would mostly measure a small, contaminated target set rather than reusable code compilation.

The next productive step is library curation under a strict protocol: define primitives from a source independent of the evaluation tasks, freeze them, then repeat this held-out coverage gate. Adding kernels after inspecting held-out misses would invalidate the purpose of the gate.

Files

src/abi_oracle.py: frozen ABI oracle implementation.
scripts/build_targets.py: calibration, held-out, and train target builder.
scripts/gate1_split_sweep.py: random held-out split sweep.
reports/gate1_summary.json: primary Gate 1 results.
reports/gate1_split_sweep.json: multi-split confirmation.
reports/figures/: generated charts.

Experiment log 1

Show the running log (1 entry, 2026-06-27)

2026-06-27

Created standalone experiment package.
Copied the frozen ABI implementation into this package so it can be rerun independently.
Primary gate: held-out frozen-ABI oracle coverage before any compiler training.
Planned metrics: held-out coverage drop, depth-1 selection vs depth-2/3 composition, parse validity, execution accuracy, confusion patterns, and direct Python sampling on the same held-out records.
Built calibration, held-out, and train target records using the frozen ABI.
Gate 1 primary result:
- calibration coverage: 134/160 (83.8%)
- held-out coverage: 22/160 (13.8%)
- coverage drop: 70.0 percentage points
- train coverage: 83/374 (22.2%)
- compiler train targets after validation split: 66
Ran a three-seed held-out split sweep over the test suffix excluded from calibration:
- seed 11: 31/160 (19.4%)
- seed 17: 29/160 (18.1%)
- seed 23: 28/160 (17.5%)
- mean coverage: 18.3%
Decision: compiler training was gated off. Frozen ABI reuse collapsed on held-out tasks, so a QLoRA compiler pilot would not measure reusable code compilation.
Generated reports/final_report.md and figures under reports/figures/.