Qwen3.5-4B Independent Code ABI Coverage Gate
The one idea you need
Picture a toolbox of generic Python operations locked shut before you see any problem. A new task counts as solved only if one tool, or a short two-step chain, exactly reproduces the required answer, like a preset universal remote where you hope one button already matches.
The question
If you assemble a fixed toolbox of generic coding operations before seeing any problems, how many brand-new coding tasks can it solve exactly, out of the box?
What we found
No, hardly any. The locked toolbox solved only about 14% of brand-new tasks, roughly 1 in 7, versus 37% on the familiar tasks it was shaped around. Reshuffling which tasks are unseen barely moves it, around 18%. And nearly every win was a single tool matching by luck, not a real composed solution, far too thin to build a trainable system on top of.
Why it matters
Before training a small model to translate problems into a fixed operation library, first measure how much of the target tasks that library can even express on fresh problems. About 15% coverage, mostly trivial, leaves nothing broad to learn.
On this page
Results at a glance 4
How to read
Bars show the share of tasks the frozen toolbox solves, for the familiar tuning set, brand-new tasks, and a broader task pool; taller is better.
Takeaway → The brand-new bar falls to well under half the familiar bar, showing the toolbox does not carry over to unseen problems.
Data table
| MBPP split (160 / 160 / 374 tasks) | oracle coverage |
|---|---|
| calibration | 37.5% |
| held-out | 14.4% |
| train | 19.5% |
Numbers from reports/coverage_gate_summary.json (calibration.overall, heldout.overall, train.overall)
Technical framing
Frozen ABI oracle coverage collapses on held-out tasks — The frozen ABI covers 37.5% of calibration tasks but only 14.4% held-out, failing the gate for a compiler-training run.
How to read
Each bar is one random sample of brand-new tasks; height is the share the frozen toolbox solves; taller is better.
Takeaway → All three bars sit near the same low level, around a fifth, so the weak coverage is consistent, not an unlucky draw.
Data table
| Random held-out sample (160 tasks each) | oracle coverage |
|---|---|
| seed 11 | 18.8% |
| seed 17 | 18.1% |
| seed 23 | 18.1% |
Numbers from reports/coverage_gate_summary.json (sweep.coverage_values, sweep.results[].seed)
Technical framing
Held-out coverage is stable across three random-sample seeds — Three-seed sweep averages 18.3% (range 18.1-18.8%), confirming low held-out coverage is not a sampling fluke.
How to read
Bars split brand-new tasks into categories like text, list, and number tasks; height is the share the toolbox solves; taller is better.
Takeaway → No category clears about a fifth and number tasks nearly bottom out, so the gap spans every domain, not one weak spot.
Data table
| Held-out task slice | oracle coverage |
|---|---|
| string_regex | 20.7% |
| other | 18.5% |
| list_tuple | 17.6% |
| dict_counter | 14.3% |
| numeric | 4.4% |
| algorithmic_control | 0% |
Numbers from reports/coverage_gate_summary.json (heldout.by_slice)
Technical framing
Held-out coverage is low in every task slice — No slice exceeds 21% held-out coverage; numeric tasks are worst at 4.4%, so the gap is not confined to one domain.
How to read
For each task group, one shade shows how often tasks with a quick-check match had no true solution, another how often individual matches fail hidden tests; lower is better.
Takeaway → On brand-new tasks over nine in ten quick-check passes still fail hidden tests, so the visible check gives false confidence.
Data table
| MBPP split | task false-pass rate | candidate hidden-wrong rate |
|---|---|---|
| calibration | 35.5% | 82.6% |
| held-out | 59.6% | 91.9% |
| train | 54.1% | 84.3% |
Technical framing
Visible-test filtering is unreliable: false-pass pressure by split — On held-out tasks 59.6% of visible passes have no full winner and 91.9% of visible-consistent candidates fail hidden tests.
In the author’s words from the Report · “Headline Result”
Held-out coverage is low. The frozen independent ABI covers 23/160 held-out tasks (14.4%). The three-seed held-out sweep averages 18.3%, range 18.1%-18.8%. Calibration coverage is higher at 37.5%, but the held-out slice and random sweep are the primary readout.
Overview
Standalone no-training coverage gate for an independently specified code ABI.
The experiment freezes a general-purpose Python/stdlib-style primitive inventory before evaluation, then measures oracle coverage on MBPP calibration and held-out slices. It does not train a model and does not add kernels after inspecting held-out misses.
Final report will be written to reports/final_report.md.
Report
Rendered from reports/final_report.md
Purpose
This standalone no-training experiment tests whether an independently specified Python/stdlib-style ABI covers held-out MBPP tasks at a useful rate. The ABI was frozen before evaluation. No kernels were added after looking at held-out misses.
Frozen ABI
The inventory contains generic Python operations: argument routing, list/tuple transforms, dictionary/counter utilities, regex/string transforms, predicates and label adapters, bounded arithmetic/combinatorics/bit utilities, and small generic compositions. At max arity 4 the inventory enumerates 903 candidates: {'dict_counter': 14, 'generic': 8, 'list_tuple': 189, 'numeric': 296, 'predicate': 32, 'predicate_label': 264, 'string_regex': 100}.
This is an oracle coverage gate, not a learned compiler and not a deployable solver. Coverage means at least one ABI candidate passes all available tests for a task.
Headline Result
Held-out coverage is low. The frozen independent ABI covers 23/160 held-out tasks (14.4%). The three-seed held-out sweep averages 18.3%, range 18.1%-18.8%.
Calibration coverage is higher at 37.5%, but the held-out slice and random sweep are the primary readout.
| split | n | oracle-covered | oracle coverage | first-visible correct | visible-any tasks | task false-pass rate | candidate hidden-wrong rate | mean candidates |
|---|---|---|---|---|---|---|---|---|
| calibration | 160 | 60 | 37.5% | 35 | 93 | 35.5% | 82.6% | 219.8 |
| heldout | 160 | 23 | 14.4% | 7 | 57 | 59.6% | 91.9% | 247.1 |
| train | 374 | 73 | 19.5% | 36 | 159 | 54.1% | 84.3% | 241.6 |
Held-Out Sweep
| seed | n | oracle-covered | oracle coverage | first-visible correct | task false-pass rate |
|---|---|---|---|---|---|
| 11 | 160 | 30 | 18.8% | 10 | 44.4% |
| 17 | 160 | 29 | 18.1% | 13 | 47.3% |
| 23 | 160 | 29 | 18.1% | 9 | 50.8% |
Slice Diagnostics
| slice | n | oracle-covered | oracle coverage | first-visible correct | task false-pass rate |
|---|---|---|---|---|---|
| algorithmic_control | 1 | 0 | 0.0% | 0 | 100.0% |
| dict_counter | 7 | 1 | 14.3% | 1 | 50.0% |
| list_tuple | 51 | 9 | 17.6% | 3 | 50.0% |
| numeric | 45 | 2 | 4.4% | 0 | 88.2% |
| other | 27 | 5 | 18.5% | 1 | 50.0% |
| string_regex | 29 | 6 | 20.7% | 2 | 33.3% |
Depth Diagnostics
Held-out covered tasks by winning depth: {'1': 20, '2': 3}. Most covered tasks are depth-1 single-primitive hits; held-out composition coverage is very small.
False-Pass Pressure
Visible-test filtering is not reliable. On held-out tasks, 57 tasks have at least one visible-consistent candidate, but the task-level visible-pass/no-full-winner rate is 59.6%, and the candidate-level hidden-wrong rate among visible-consistent candidates is 91.9%.
Covered Held-Out Examples
| task | slice | depth | winning program | task |
|---|---|---|---|---|
| 175 | string_regex | 1 | {"arg": 0, "op": "all_distinct_arg0"} | Write a function to verify validity of a string of parentheses. |
| 190 | numeric | 1 | {"args": [2, 0], "op": "floordiv_2_0"} | Write a python function to count the number of integral co-ordinates that lie inside a squ |
| 191 | other | 1 | {"arg": 0, "op": "all_distinct_arg0"} | Write a function to check whether the given month name contains 30 days or not. |
| 192 | string_regex | 1 | {"arg": 0, "op": "has_duplicate_arg0"} | Write a python function to check whether a string has atleast one letter and one number. |
| 201 | list_tuple | 1 | {"arg": 0, "op": "has_duplicate_arg0"} | Write a python function to check whether the elements in a list are same or not. |
| 203 | other | 1 | {"args": [1, 0], "op": "floordiv_1_0"} | Write a python function to find the hamming distance between given two integers. |
| 204 | string_regex | 1 | {"args": [0, 1], "op": "count_0_1"} | Write a python function to count the occurrence of a given character in a string. |
| 211 | other | 1 | {"arg": 0, "op": "bit_count_arg0"} | Write a python function to count numbers whose oth and nth bits are set. |
| 216 | list_tuple | 2 | {"args": [0, 1], "op": "contains_subsequence_0_1"} | Write a function to check if a nested list is a subset of another nested list. |
| 223 | list_tuple | 1 | {"arg": 1, "op": "is_odd_arg1"} | Write a function to check for majority element in the given sorted array. |
Some covered tasks are likely test-suite coincidences rather than semantic equivalence. That makes the low held-out coverage an optimistic upper bound, not a pessimistic one.
Decision
This gate does not support a code-ABI compiler-training run. A frozen independent ABI covers only about 14-18% of held-out MBPP tasks, mostly as depth-1 single-primitive matches. Training a compiler on this ABI would have too little held-out expressivity to test broad reusable code compilation.
The useful next step is not to add task-specific kernels. A better test would use a real independently curated task domain whose tasks are known to be deterministic transformations, then repeat this same held-out coverage gate.
Files
scripts/run_independent_abi_gate.py: frozen ABI evaluator and gate runner.reports/coverage_gate_summary.json: all summary metrics.data/calibration_records.jsonl,data/heldout_records.jsonl,data/train_records.jsonl: per-task records.data/sweep_records.jsonl: per-task sweep records.reports/figures/: generated charts.
Experiment log 1
Show the running log (1 entry, 2026-06-27)
2026-06-27
- Created standalone experiment package.
- Defined the frozen ABI as a general-purpose Python/stdlib-style primitive inventory before running coverage.
- Primary gate: held-out oracle coverage on unseen MBPP test records using only this frozen ABI.
- No training is planned unless held-out coverage is meaningfully high.
- Implemented
scripts/run_independent_abi_gate.pywith the frozen ABI and evaluator. - Ran calibration, held-out, train, and three-seed held-out sweep.
Main results:
- calibration coverage: 60/160 (37.5%)
- held-out coverage: 23/160 (14.4%)
- train coverage: 73/374 (19.5%)
- held-out sweep: 30/160, 29/160, 29/160; mean 18.3%
- held-out covered depth counts: 20 depth-1, 3 depth-2, 0 depth-3
- held-out task-level visible-pass/no-full-winner rate: 59.6%
- held-out candidate-level hidden-wrong rate among visible-consistent candidates: 91.9%
- Decision: no compiler training. The frozen independent ABI does not cover held-out MBPP tasks at a useful rate.
- Generated report and charts at
reports/final_report.mdandreports/figures/.
Figures 5
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Run steps are documented inside the experiment folder (README and scripts).