Research log Small Model Experimentation
GitHub

Qwen Real Task ABI Coverage Gate

Finished2026-06-27imported · line YAlgorithmic Memory and RetrievalEvidence-Conditioned SelectionStructured Execution and CompilersGitHub ↗
A fixed toolkit solves the routine

The one idea you need

Picture a fixed box of LEGO bricks. The question is whether everyday builds can be snapped together from bricks already in the box, combining at most a couple, or whether some designs demand a specialty brick the box simply never contained.

The question

If you build one fixed set of small data-cleanup operations, can everyday office jobs be assembled from those parts alone, without adding anything new?

What we found

It depends, and the split is sharp. A frozen kit of reusable office operations, with no training at all, assembled 84% of realistic tasks from stored parts alone, far above the 21% a bare kit managed, and it fully solved even brand-new task types it had never seen built. But every task deliberately designed to need a missing part failed: zero solved.

Why it matters

Before investing in training around a reusable operation library, measure first what share of your real tasks can be built from existing parts. Routine cleanup covers well, but budget a path to add the missing operations.

Everyday tasks the full kit could build84%32 of 38 realistic tasks, no training
The same tasks with only a bare kit21%most unsolvable without richer parts
Deliberately unusual tasks solved0 of 5custom fiscal-year, discount, latest-row logic all failed
Common office cleanups solved100%10 of 10 routine name, date, and price tasks
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Abstract
    2. Method
    3. Run Configuration
    4. Primary Results
    5. Interpretation
    6. Limitations
    7. Artifacts
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 3

Coverage climbs as the kit gets richer and combines more parts

How to read

Bars show the share of 38 everyday tasks built from prebuilt parts, grouped left to right by how rich the kit is (bare, mid, full), with two shades for allowing one versus two combining steps. Taller is better.

0%25%50%75%100%core ABIcore ABI10.5%21.1%office ABIoffice ABI60.5%71.1%office+table ABIoffice+table ABI73.7%84.2%

Takeaway → Bars rise steadily rightward, from a stub for the bare kit to the tallest for the full kit, so a richer parts library plus more combining solves far more tasks.

Data table
frozen ABI variantdepth 1depth 2
core ABI10.5%21.1%
office ABI60.5%71.1%
office+table ABI73.7%84.2%

Numbers from analysis/overall_summary.csv

Technical framing

Held-out task coverage by ABI variant and search depth — The full office+table ABI at depth 2 covers 84% of 38 real-style tasks; the minimal core ABI covers only 21%.

Common tasks fully solved, deliberately unusual tasks all fail

How to read

Each bar is one group of tasks; height is the share solved from prebuilt parts, so taller is better. The final bar holds tasks deliberately built to need parts the kit lacks.

0%25%50%75%100%calibrationcalibration100%held-out compositionheld-out composition100%held-out new familyheld-out new family100%held-out same familyheld-out same family93.3%stress out-of-ABIstress out-of-ABI0%

Takeaway → The first four bars sit at or near full height while the final unusual-task bar drops to zero, so the kit handles familiar work but nothing beyond its parts.

Data table
task splitheld-out coverage
calibration100%
held-out composition100%
held-out new family100%
held-out same family93.3%
stress out-of-ABI0%

Numbers from report table (reports/qwen_real_task_abi_coverage_gate_report.md, Primary Split Summary)

Technical framing

Coverage by task split under the primary ABI (office+table, depth 2) — Held-out compositions and new families are fully covered, but every deliberately out-of-ABI stress task fails.

Most tasks solved; the few failures split two ways

How to read

Three bars show what happened to all 38 tasks with the full kit: solved on fresh examples, matched only the shown examples, or never built at all. The tall solved bar is best.

0%25%50%75%100%held-out coveredheld-out covered84.2%train-match onlytrain-match only5.3%no train matchno train match10.5%

Takeaway → One tall bar dominates near 84%, with two small failure bars, so a few tasks matched only the shown examples and a few could not be built at all.

Data table
outcomeshare of 38 tasks
held-out covered84.2%
train-match only5.3%
no train match10.5%

Numbers from analysis/overall_summary.csv

Technical framing

Outcome breakdown under the primary gate (office+table, depth 2) — Failures split into overfitting to visible examples (5%) and tasks the ABI/search cannot even fit (11%).

In the author’s words from the Report · “Abstract”

This standalone experiment tests whether a frozen office-data ABI covers real-style deterministic tasks that were not generated from that ABI. It uses oracle enumeration, not model training: the result is a decomposability gate for whether a large compiler corpus is worth building.

Overview

Standalone experiment testing whether a fixed office-data ABI covers real-style deterministic tasks that were not generated from the ABI.

Question

The compiler recipe only matters for real capability if realistic tasks decompose into a reusable operation library. This experiment freezes a general-purpose office ABI first, then evaluates oracle coverage on hand-curated deterministic tasks from contact cleanup, dates, money, URLs/files, product codes, addresses, and small tables.

Method

  • Define ABI primitives before task definitions in src/qwen_real_task_abi_coverage_gate.py.
  • Write task references as ordinary Python functions over examples, not as ABI programs.
  • Use an enumerative oracle synthesizer to search ABI expressions from visible fields and constants.
  • Score candidates on train examples and held-out examples for each task.
  • Report coverage by ABI tier, split, family, search depth, and failure mode.

Primary Metrics

  • heldout_covered: a candidate matches both train and held-out examples.
  • train_match_only: a candidate fits train examples but fails held-out examples.
  • no_train_match: the ABI/search could not even fit the visible examples.
  • Coverage by split and task family under the strongest fixed ABI.

Artifacts

Report

Rendered from reports/qwen_real_task_abi_coverage_gate_report.md

Abstract

This standalone experiment tests whether a frozen office-data ABI covers real-style deterministic tasks that were not generated from that ABI. It uses oracle enumeration, not model training: the result is a decomposability gate for whether a large compiler corpus is worth building.

Method

  • The ABI is a fixed library of scalar string/date/money/contact/file operations plus small-table aggregation templates.
  • Task references are ordinary Python functions over hand-curated examples; task outputs are not produced by stored ABI programs.
  • The oracle enumerates expressions from visible fields/constants, fits train examples, then tests held-out examples for each task.
  • ABI variants are core, office, and office_table; the primary gate is office_table at the largest search depth.

Run Configuration

  • Suite: main.
  • Tasks: 38 total, 6 table tasks under the primary slice.
  • Primary ABI/depth: office_table, depth 2.
  • Large artifacts directory: /workspace/large_artifacts/qwen_real_task_abi_coverage_gate.

Primary Results

  • Overall held-out coverage: 84.2% (32/38 tasks).
  • Calibration coverage: 100.0% (10/10).
  • Non-calibration coverage: 78.6% (22/28).
  • Held-out composition coverage: 100.0% (8/8).
  • Stress/out-of-ABI coverage: 0.0% (0/5).
  • Table-task coverage: 83.3% (5/6).
  • Train-match-only rate: 5.3%.
  • No-train-match rate: 10.5%.

Overall By ABI

variantmax_depthtasksheldout_coveredtrain_match_ratetrain_match_onlyno_train_match
core13810.5%15.8%5.3%84.2%
core23821.1%26.3%5.3%73.7%
office13860.5%65.8%5.3%34.2%
office23871.1%76.3%5.3%23.7%
office_table13873.7%78.9%5.3%21.1%
office_table23884.2%89.5%5.3%10.5%

Primary Split Summary

splittasksheldout_coveredtrain_match_ratetrain_match_onlyno_train_match
calibration10100.0%100.0%0.0%0.0%
heldout_composition8100.0%100.0%0.0%0.0%
heldout_new_family6100.0%100.0%0.0%0.0%
heldout_same_family993.3%100.0%6.7%0.0%
stress_out_of_abi50.0%20.0%20.0%80.0%

Covered Program Examples

task_idfamilysplitprogram_depthprogram
address_cityaddressheldout_new_family1.00city_before_state(FIELD(address))
address_stateaddressheldout_new_family1.00us_state(FIELD(address))
address_zip5addressheldout_new_family1.00zip5(FIELD(address))
contact_email_companycontactheldout_same_family1.00email_company(FIELD(email))
contact_email_domaincontactcalibration1.00email_domain(FIELD(email))
contact_name_initialscontactcalibration1.00initials(FIELD(name))
contact_phone_e164contactcalibration1.00phone_e164_us(FIELD(phone))
contact_phone_last4contactheldout_same_family1.00phone_last4(FIELD(phone))
date_days_betweendateheldout_composition1.00date_diff_days(FIELD(end),FIELD(start))
date_month_namedateheldout_same_family1.00date_month_name(FIELD(date))
date_quarter_labeldateheldout_same_family1.00date_quarter(FIELD(date))
date_to_isodatecalibration1.00date_iso(FIELD(date))
money_centsmoneycalibration1.00money_cents(FIELD(price))
money_line_total_centsmoneyheldout_composition2.00mul(FIELD(qty),alnum(FIELD(price)))
money_percent_decimalmoneycalibration1.00percent_float(FIELD(discount))
code_after_colonproductheldout_same_family1.00digits(FIELD(code))
sku_prefixproductcalibration1.00sku_prefix(FIELD(sku))
sku_suffixproductheldout_same_family1.00sku_suffix(FIELD(sku))

Uncovered Examples

task_idfamilysplitfailure_reasontrain_matchprogram
contact_normalize_namecontactheldout_same_familytrain_match_onlyTruetitle(FIELD(name))
date_fiscal_year_julydatestress_out_of_abino_train_matchFalse
money_discounted_totalmoneystress_out_of_abino_train_matchFalse
sku_middle_segmentproductstress_out_of_abitrain_match_onlyTruedigits(FIELD(sku))
table_latest_paid_amounttablestress_out_of_abino_train_matchFalse
phrase_slugtextstress_out_of_abino_train_matchFalse

coverage_by_depth.png

coverage_by_split.png

coverage_by_family.png

failure_modes.png

program_depth_distribution.png

Interpretation

The fixed ABI covers a meaningful fraction of real-style deterministic tasks, but coverage is far from universal. The gap between calibration and stress splits is the main signal: common office transforms decompose well, while tasks that require bespoke fiscal logic, middle-token extraction, discount formulas, or latest-row selection expose missing primitives or missing control patterns. The positive read is that held-out composition and held-out new-family tasks are covered in this catalog, including table filters and aggregations. The negative read is equally important: every deliberately out-of-ABI stress task fails, so the ABI cannot be treated as an open-ended intelligence multiplier without a retrieval/extension path for missing primitives. A train-match-only result is treated as a warning rather than success: it means the ABI/search can fit visible examples but does not identify the actual task robustly on held-out examples. This is a gate, not a compiler result. If the target domain is restricted to covered families, the ABI direction has a real base to build on. If the target domain includes the uncovered stress patterns, the ABI must be expanded or paired with retrieval/tooling before model training is worth scaling. The next decisive step is a less hand-curated corpus: either production-like task logs or a public deterministic transformation benchmark, with the ABI frozen before evaluating that set.

Limitations

The tasks are hand-curated real-style examples, not production logs or a public benchmark. That is stronger than factory-generated ABI compositions but still not a definitive real-world coverage estimate. The oracle is bounded by the implemented search space, so some misses may be search misses rather than true ABI misses. The table ABI is intentionally small and does not include sorting or window operations.

Artifacts

  • Details: analysis/details.csv
  • Summary: analysis/summary.csv and analysis/overall_summary.csv
  • Task catalog: analysis/task_catalog.json
  • Large artifacts directory: /workspace/large_artifacts/qwen_real_task_abi_coverage_gate

Experiment log 6

Show the running log (6 entries)

Setup

  • Created fresh experiment directory: /workspace/experiments/qwen_real_task_abi_coverage_gate
  • Large artifacts directory: /workspace/large_artifacts/qwen_real_task_abi_coverage_gate
  • Core question: whether a frozen office ABI covers real-style deterministic tasks that are not generated from that ABI.
  • Report format: standalone Markdown and HTML with plots.

Iteration Notes

  • Initial smoke exposed a leakage risk in date tasks: an ISO-date task could be solved by copying an unrelated interval end field. Fixed by separating single-date rows from interval rows before pilot/main.
  • Initial pilot attempted depth-3 brute-force expressions and was interrupted after it proved too slow for a coverage gate. Fixed by bounding the gate to depth-1/2 programs, which covers direct primitives and short office compositions without turning the experiment into a synthesizer benchmark.

Run smoke_v1

  • Started: 2026-06-27 03:15:25 UTC
  • Variants: core,office_table
  • Depths: 1,2
  • Tasks: 12
  • Completed in 19.3s.
  • Primary coverage: 83.3% (10/12 tasks).
  • Train-match-only: 8.3%; no-train-match: 8.3%.

Run pilot_v1

  • Started: 2026-06-27 03:21:32 UTC
  • Variants: core,office,office_table
  • Depths: 1,2
  • Tasks: 30
  • Completed in 25.8s.
  • Primary coverage: 86.7% (26/30 tasks).
  • Train-match-only: 6.7%; no-train-match: 6.7%.

Run main_v1

  • Started: 2026-06-27 03:23:19 UTC
  • Variants: core,office,office_table
  • Depths: 1,2
  • Tasks: 38
  • Completed in 27.0s.
  • Primary coverage: 84.2% (32/38 tasks).
  • Train-match-only: 5.3%; no-train-match: 10.5%.

Final Read

  • The frozen office_table ABI covered 84.2% of the full hand-curated task catalog at depth 2.
  • Non-calibration coverage was 78.6% (22/28), held-out composition coverage was 100.0% (8/8), and held-out new-family coverage was 100.0% (6/6).
  • Table coverage was 83.3% (5/6): count, sum, filtered count, and filtered sum were covered; latest-paid-row selection was not.
  • Every deliberately out-of-ABI stress task failed (0/5): fiscal-year logic, discounted total, middle SKU segment, phrase slugification, and latest paid amount.
  • Read: this is a guarded positive for a fixed office ABI on common deterministic transformations, not proof of open-ended coverage. The next step should use a less hand-curated corpus or freeze the ABI before evaluating a public/production-like task set.

Figures 5

coverage by depth
coverage by depth · analysis/figures/
coverage by family
coverage by family · analysis/figures/
coverage by split
coverage by split · analysis/figures/
failure modes
failure modes · analysis/figures/
program depth distribution
program depth distribution · analysis/figures/

Data files 8

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

The run commands are documented inside the experiment folder (see the README).

Browse the experiment folder on GitHub ↗