Qwen3.5-4B Code ABI Oracle Coverage Ladder
The one idea you need
Picture a bin of certified, pre-tested parts. Instead of writing code from scratch, you snap together at most three of these building-block functions to solve a task. This experiment asks how many real programming problems a few dozen such parts can cover.
The question
Can a fixed library of small, already-tested code functions be combined to correctly solve most everyday programming problems, without writing any new code from scratch?
What we found
Yes, mostly. Snapping together at most three verified functions produced a passing solution for 84% of 160 small programming tasks, up from just 13% with a bare-bones library. But the library is a foundation, not a solver: when you accept the first program that passes one public test, only 59% are actually correct, because most passing candidates still fail hidden tests.
Why it matters
When generating code with small models, never trust a single public test: most programs that pass one still fail on hidden cases. Restrict generation to verified building blocks, and run a full test suite before accepting any candidate.
On this page
Results at a glance 4
How to read
Left to right the toolbox grows from tiny to full. For each, one bar is the share of 160 tasks it can build at all, the other the share where the first program passing one test is actually correct. Taller is better.
Takeaway → Both bars rise as the toolbox grows, but the 'actually correct' bar stays far below the 'can be built' bar: being expressible does not mean being auto-solved.
Data table
| ABI rung | oracle coverage (any candidate passes all tests) | first visible-consistent candidate correct |
|---|---|---|
| core | 12.5% | 10.6% |
| expanded | 57.5% | 41.9% |
| final | 83.8% | 59.4% |
Numbers from README table; reports/coverage_summary_initial.json, coverage_summary_expanded.json, coverage_summary_final.json
Technical framing
ABI oracle coverage climbs the rung ladder, first-visible selection lags — The final reusable ABI can express 134/160 tasks, but naive first-visible selection only gets 95/160 — a coverage gate, not a solver.
How to read
Each group is a task category. One bar is the share of that category's tasks the full toolbox can build, the other the share where the first passing program is actually correct. Taller is better.
Takeaway → Every category clears roughly four-fifths coverage except the mixed 'other' bucket, where both building and picking the right program drop sharply.
Data table
| task slice | oracle coverage | first visible candidate correct |
|---|---|---|
| dict_counter | 100% | 66.7% |
| list_tuple | 89.5% | 77.2% |
| numeric | 79.2% | 50% |
| other | 62.5% | 25% |
| string_regex | 95.5% | 68.2% |
Numbers from report table (reports/final_report.md); reports/coverage_summary_final.json
Technical framing
Final-rung coverage by task slice — Coverage stays above 79% in every classified slice except the catch-all 'other' bucket, where both coverage and selection drop.
How to read
Left to right the toolbox grows from tiny to full. Each bar is the share of tasks where a program passed the one shown test yet was still wrong. Here lower is better.
Takeaway → The fooled-by-one-test bar falls from 58% to 9% as the toolbox grows, but real checking is still needed since most individual candidates stay wrong.
Data table
| ABI rung | false visible pass (among tasks with a visible-consistent candidate) |
|---|---|
| core | 58.3% |
| expanded | 22% |
| final | 8.8% |
Numbers from report table (reports/final_report.md); reports/coverage_summary_*.json
Technical framing
Task-level false-visible-pass rate falls as the ABI grows — Richer ABIs cut single-visible-test false passes from 58% to 9%, but per-candidate hidden-wrong pressure stays near 80% — verification is still required.
How to read
Each bar is how many of the 134 solved tasks were built from a chain of that many functions: one, two, or three. Taller means more tasks at that complexity.
Takeaway → The single-block bar towers over the rest: 94 of 134 tasks need one piece, so genuine multi-block assembly covers only 40 tasks.
Data table
| winning ABI program depth | covered tasks (final rung) |
|---|---|
| depth 1 | 94 |
| depth 2 | 31 |
| depth 3 | 9 |
Numbers from report table (reports/final_report.md, 'Winning Program Depths')
Technical framing
Most covered tasks need only a depth-1 ABI program — 94 of 134 covered tasks are solved by a single primitive; genuine composition (depth 2-3) accounts for 40 tasks.
In the author’s words from the Overview
Final report: reports/final_report.md reports/figures/ Main result on the first 160 MBPP test records: final reusable ABI oracle coverage reached 134/160 (83.75%) under the available tests. First visible-consistent candidate accuracy was 95/160 (59.4%), so this is a compiler/substrate coverage gate, not a deployable solver by itself.
Overview
Standalone no-training experiment for a code-primitive ABI oracle coverage gate.
The experiment measures whether a small verified ABI of deterministic code primitives can express decomposable MBPP-style tasks. It does not train or save model checkpoints. It builds task slices, enumerates ABI candidate programs, verifies them by execution, and reports coverage, false-visible-pass rates, candidate counts, and task-slice diagnostics.
Final report:
reports/final_report.mdreports/figures/
Main result on the first 160 MBPP test records: final reusable ABI oracle coverage reached 134/160 (83.75%) under the available tests. First visible-consistent candidate accuracy was 95/160 (59.4%), so this is a compiler/substrate coverage gate, not a deployable solver by itself.
Report
Rendered from reports/final_report.md
Purpose
This standalone no-training experiment asks whether a finite verified code-primitive ABI can express a meaningful slice of MBPP-style Python tasks before spending any compute on a compiler model.
The run uses the first 160 records from the MBPP test split. Each task gets one visible test for visible-consistency accounting, while oracle coverage is measured against every available test in the task record. The experiment does not train Qwen, does not save checkpoints, and does not use reference source code as a template.
ABI Rungs
core: small generic string/list/dict/numeric kernels.expanded: broad reusable kernels for row-sum sorting, counters, simple geometry, bit operations, list transforms, and regex/string transforms.final: expanded plus reusable algorithmic utility kernels such as sequence recurrences, tuple/list conversions, range sums, run-length encoding, divisor/bit counts, and small dynamic reducers.
Headline Results
| rung | oracle coverage | first visible candidate correct | visible-any tasks | task-level false-visible-pass | candidate-level hidden-wrong among visible | mean candidates/task |
|---|---|---|---|---|---|---|
| core | 20/160 (12.5%) | 17/160 (10.6%) | 48 | 28 (58.3%) | 110 (81.5%) | 66.9 |
| expanded | 92/160 (57.5%) | 67/160 (41.9%) | 118 | 26 (22.0%) | 425 (78.0%) | 164.2 |
| final | 134/160 (83.8%) | 95/160 (59.4%) | 147 | 13 (8.8%) | 628 (79.2%) | 224.0 |
Final Rung by Slice
| slice | oracle coverage | first visible candidate correct | task-level false-visible-pass | mean candidates/task |
|---|---|---|---|---|
| dict_counter | 9/9 (100.0%) | 6/9 (66.7%) | 0 (0.0%) | 188.8 |
| list_tuple | 51/57 (89.5%) | 44/57 (77.2%) | 4 (7.3%) | 234.9 |
| numeric | 38/48 (79.2%) | 24/48 (50.0%) | 5 (11.6%) | 201.8 |
| other | 15/24 (62.5%) | 6/24 (25.0%) | 3 (16.7%) | 313.5 |
| string_regex | 21/22 (95.5%) | 15/22 (68.2%) | 1 (4.5%) | 161.1 |
Winning Program Depths
Covered task count by winning ABI depth: 1: 94, 2: 31, 3: 9.
Gate Read
The oracle coverage gate clears for the intended decomposable-code direction: the final reusable ABI covers 134/160 tasks (83.75%) under the available test suites, with coverage above 79% in every automatically classified slice except other.
The result does not make the ABI a deployable solver by itself. First-visible selection is only 95/160 (59.4%), and candidate-level hidden-wrong pressure among visible-consistent candidates remains high at 628 wrong visible-consistent candidates out of 793 total visible-consistent candidates. This means compiler training should use constrained decoding plus verification and should not rely on one public test or first visible-consistent selection.
The strongest caution is that this is test-suite oracle coverage, not a proof of semantic equivalence. Some candidates can satisfy all available tests while being semantically too broad, too narrow, or coincidentally correct. The package keeps these candidates visible in data/abi_coverage_records_final.jsonl so those cases can be audited.
Remaining Uncovered Tasks
Uncovered task IDs after the final rung:
20, 25, 31, 36, 47, 55, 56, 60, 77, 111, 122, 123, 124, 125, 129, 134, 136, 137, 147, 148, 150, 153, 158, 159, 160, 164
The residual is a mix of specialized number theory, bespoke formulas, tasks with ambiguous or thin test suites, and algorithmic problems outside the current ABI. The next step should not be to keep adding one-off kernels indefinitely. The useful next gate is a compiler-training pilot on this frozen final ABI, paired with stronger generated counterexample tests for covered tasks and a held-out slice that excludes primitives added after inspection.
Files
configs/experiment.json: run configuration.scripts/run_coverage_ladder.py: ABI oracle evaluator.scripts/make_report.py: report and figure generator.data/abi_coverage_records_initial.jsonl: core rung records.data/abi_coverage_records_expanded.jsonl: expanded rung records.data/abi_coverage_records_final.jsonl: final rung records.reports/coverage_summary_initial.json: core summary.reports/coverage_summary_expanded.json: expanded summary.reports/coverage_summary_final.json: final summary.reports/figures/: generated charts.
Experiment log 1
Show the running log (1 entry, 2026-06-27)
2026-06-27
- Created standalone experiment package.
- Copied generic safe Python execution and JSONL utilities into this package.
- Planned no-training code ABI oracle coverage ladder over MBPP test tasks.
- Primary gate: measure whether a verified primitive ABI can express decomposable real-code task slices before any compiler training.
- Implemented
scripts/run_coverage_ladder.py. Ran core ABI rung on 160 MBPP test records:
- oracle coverage: 20/160 (12.5%)
- first visible-consistent candidate correct: 17/160 (10.6%)
- mean candidates/task: 66.9
- Inspected misses and added broad reusable primitives for row-sum sorting, counters, string/list transforms, simple formulas, bit predicates, and regex-style tasks.
Ran expanded ABI rung:
- oracle coverage: 92/160 (57.5%)
- first visible-consistent candidate correct: 67/160 (41.9%)
- mean candidates/task: 164.2
- Inspected remaining misses and added a final broad utility layer: tuple/list conversions, sequence recurrences, run-length encoding, range sums, divisor/bit counts, pairwise reducers, and small geometry.
- One final-rung run hung because some enumeration-style primitives were tried outside their intended domain. Killed that run and added explicit numeric/domain bounds before rerunning.
Ran final ABI rung:
- oracle coverage: 134/160 (83.75%)
- first visible-consistent candidate correct: 95/160 (59.4%)
- task-level visible-pass/no-full-winner rate among visible-any tasks: 13/147 (8.8%)
- candidate-level hidden-wrong rate among visible-consistent candidates: 628/793 (79.2%)
- mean candidates/task: 224.0
- Generated charts and standalone report at
reports/final_report.md. - Gate read: code-ABI oracle coverage is high enough to justify a compiler-training pilot on a frozen ABI, but selection/verification remains unresolved and the result is test-suite coverage rather than proof of semantic equivalence.
Figures 5
Data files 3
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/coverage_summary_expanded.json4.6 kBreports/coverage_summary_final.json4.6 kBreports/coverage_summary_initial.json4.5 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).