Qwen Public PROSE ABI Gate

Finished2026-06-27imported · line YBenchmark Generalization Structured Execution and Compilers GitHub ↗

The one idea you need

Picture a fixed toolbox of preset text-reshaping tools: pull the domain from an email, take the first word, round a number. Freeze it, then hand it an outside pile of formatting jobs, keeping a tool only if it also works on rows it never saw.

The question

If you hand-build a fixed set of text-reshaping recipes, will they handle an independent, outside collection of everyday spreadsheet-cleanup tasks?

What we found

No. The frozen toolkit fully solved only 19% of the 309 outside tasks. In 77% of them no recipe fit even the worked examples, so the toolkit lacked that operation entirely; under 4% overfit. Yet a small four-billion-parameter model, given the same examples, answered a fresh case correctly 73% of the time, and two-thirds right even on tasks the toolkit missed.

Why it matters

Building text normalization from a fixed rule library covers only a fraction of real formats, especially dates and numbers. Prefer letting a small model infer the transformation from examples, or add retrieval to pick domain-specific rules.

Prebuilt toolkit's task coverage19% (59 of 309)solved and verified on rows it never saw

Misses where no recipe fit at all77%the operation was simply missing

Small model answering from examples73% correct4-billion-parameter model, one fresh case per task

Bigger recipe search, tiny payoff13% → 19%500x more recipes tried, still under a fifth solved

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 7

Even the richest recipe set solves under a fifth of tasks

How to read

Three toolbox sizes along the bottom, smallest to largest. Each bar splits its tasks into solved-and-verified-on-unseen-rows, a recipe that fit examples only by luck, and no recipe fitting at all. A taller solved slice is better.

Takeaway → The solved slice never clears about a fifth in any toolbox while the 'no recipe fit' band dominates, meaning the tasks are out of reach, not merely overfit.

Data table

frozen ABI tier	held-out coverage	train-only (coincidence) failures	no train match
core	12.6%	2.6%	84.8%
office	17.8%	3.2%	79%
concat	19.1%	3.6%	77.3%

Numbers from analysis/overall_summary.csv

Technical framing

Frozen transformation ABI covers under 20% of public PROSE tasks — Even the richest frozen tier validates only 19% of tasks; most misses are missing primitives, not held-out coincidences.

The small model answers most tasks the recipes cannot

How to read

Bars show how often a small model, given a few examples, produced the exact right answer on a fresh case: for all sampled tasks, only tasks the toolbox missed, and only tasks it covered. Taller is better.

Takeaway → The model still nails about two-thirds of the tasks the toolbox misses, showing the shortfall is the fixed recipes, not the transformations being impossible.

Data table

task slice (60-task sample)	Qwen direct exact match (1 held-out query)
all sampled tasks	73.3%
ABI not covered	65.9%
ABI covered	93.8%

Numbers from analysis/qwen_direct_summary.csv

Technical framing

Frozen Qwen answers most tasks the ABI cannot cover — The model infers 66% of transformations the frozen ABI misses, so the bottleneck is the ABI's primitives, not the model.

Simple edits work; date rounding and ranges do not

How to read

Each bar is a type of text operation; height is the share of those tasks the best toolbox solved. Left bars are simple surface edits, right bars are date and number rounding and ranges. Taller is better.

Takeaway → Coverage falls from about two-in-five for plain substring edits to zero for date rounding and range tasks (number rounding barely clears one-in-twenty), exactly where the toolbox runs out.

Data table

task feature	held-out coverage (concat tier)
Substring	40.5%
Casing	33.3%
Concatenation	19.2%
Multicolumn	18.8%
DateTime	7.7%
Numeric	7.3%
Conditional	5.4%
NumericRounding	4.8%
DateTimeRounding	0%
DateTimeRange	0%

Numbers from analysis/feature_summary.csv

Technical framing

Where the ABI fails: coverage by task feature (best tier) — Surface substring/casing tasks are reachable; date-time rounding and range arithmetic are completely outside the frozen ABI.

Structured formats work; dates and numbers barely do

How to read

Each bar is a task family such as dates, phone numbers, or emails; height is the share of that family the best toolbox solved. Taller is better.

Takeaway → Tidy families like shipping codes and emails reach half or more, while the two biggest families, dates and numbers, sit near the bottom and drag the total down.

Data table

Task family	Coverage (concat tier)
DateTime	8.5%
Number	7.1%
Name	39.3%
Phone	37.5%
ShippingCode	80%
City	11.1%
UserAgent	42.9%
Email	50%
Address	0%
BillingCode	0%

Numbers from experiments/qwen_public_prose_abi_gate/analysis/family_summary.csv

Technical framing

ABI coverage by task family (best tier), largest families — Semi-structured families like ShippingCode and Email are coverable; the two largest families, DateTime and Number, are nearly uncovered.

Failures are missing recipes, not lucky-but-wrong ones

How to read

For each toolbox size, the bar splits all tasks into solved, a recipe matching the examples only by coincidence, and no recipe fitting at all. The thin coincidence slice is the point.

Takeaway → The coincidence slice stays near 3% while 'no recipe fit' fills roughly four-fifths, so failures are genuinely missing tools, not overfit guesses.

Data table

ABI tier	Covered (held-out validated)	Train-only fit (coincidence)	No train match
core	12.6%	2.6%	84.8%
office	17.8%	3.2%	79%
concat	19.1%	3.6%	77.3%

Numbers from experiments/qwen_public_prose_abi_gate/analysis/overall_summary.csv

Technical framing

Why the ABI fails: no train fit dominates, coincidental fits are rare — Across all tiers roughly 8 in 10 failures are tasks where no candidate even fits the train rows, not overfit programs failing held-out.

The model stumbles most on date tasks

How to read

Each bar is a task family in the sampled set; height is how often the small model gave the exact right answer from examples. Taller is better.

Takeaway → Names and phone numbers hit every case while dates fall to half, the same family where the fixed recipes are weakest.

Data table

Slice	Qwen3-4B direct answer
Overall	73.3%
DateTime	50%
Name	100%
Number	78.6%
Phone	100%

Numbers from experiments/qwen_public_prose_abi_gate/analysis/qwen_direct_summary.csv

Technical framing

Frozen Qwen one-query exact match by task family (60-task sample) — Qwen's misses concentrate in DateTime tasks (50% exact) — the same family where the frozen ABI is weakest.

Trying vastly more recipes barely helps

How to read

Each bar is a toolbox size; height is how many candidate recipes it sifts through per task. A taller bar means far more work, not better results.

Takeaway → The largest toolbox searches over 500 times more recipes than the smallest yet lifts tasks solved only from 13% to 19%, so search size is not the bottleneck.

Data table

ABI tier	Median candidates
core	109
office	1057
concat	56.83k

Numbers from experiments/qwen_public_prose_abi_gate/analysis/overall_summary.csv

Technical framing

Cost of each ABI tier: median candidate programs searched per task — Growing the search space ~500x from core to concat lifts coverage only 12.6% to 19.1% — enumeration size is not the bottleneck.

In the author’s words from the Report · “Abstract”

This standalone experiment evaluates a frozen deterministic transformation ABI on the public Microsoft PROSE Transformation.Text benchmark. A program is selected from train examples and counted only if it also matches held-out examples from the same task.

Overview

Standalone experiment testing whether a frozen deterministic transformation ABI covers public Microsoft PROSE Transformation.Text tasks under within-task held-out validation.

Question

Does the office/transformation ABI cover an independent public benchmark under held-out validation?

Method

Fetch public benchmark data into /workspace/large_artifacts/qwen_public_prose_abi_gate/prose-benchmarks.
Freeze a generic transformation-template ABI before evaluating the benchmark.
For each task with enough examples, split examples into train and held-out rows.
Select a program only if it fits train examples; count it as covered only if it also matches held-out examples.
Report train-only fits separately as semantic/coincidence failures.

Artifacts

Source: src/qwen_public_prose_abi_gate.py
Reports: reports/
Metrics and figures: analysis/
Public benchmark checkout: /workspace/large_artifacts/qwen_public_prose_abi_gate/prose-benchmarks

Report

Rendered from reports/qwen_public_prose_abi_gate_report.md

Abstract

This standalone experiment evaluates a frozen deterministic transformation ABI on the public Microsoft PROSE Transformation.Text benchmark. A program is selected from train examples and counted only if it also matches held-out examples from the same task.

Method

Dataset: Microsoft PROSE public benchmark suite, Transformation.Text.
Tasks with too few examples are excluded so every scored task has held-out examples.
Split: first 4 examples, capped by task size, are train examples; up to 50 following examples are held out.
Frozen ABI tiers: core surface extraction/casing, office adds regex/date/number/domain primitives, concat adds two-part concatenation over ABI expressions.
Train-only fits are counted as failures because they do not validate task semantics on held-out rows.

Run Configuration

Suite: main.
Scored tasks: 309.
Public benchmark checkout: /workspace/large_artifacts/qwen_public_prose_abi_gate/prose-benchmarks.

Primary Results

Best frozen tier coverage: 19.1% (59/309 tasks).
Train-only/coincidence failures: 3.6% (11 tasks).
No-train-match failures: 77.3% (239 tasks).
Synthetic-metadata coverage: 18.3% (55/301 tasks).
Non-synthetic-metadata coverage: 50.0% (4/8 tasks).

Frozen Qwen Direct-Answer Sample

Model: Qwen/Qwen3-4B.
Sample: 60 tasks, seed 20260627.
Prompt: first 4 examples, then one held-out query.
Exact match on that one held-out query: 73.3% (44/60 tasks).
This is a diagnostic baseline, not the same metric as ABI coverage: it scores one held-out query, while ABI coverage requires one program to match all held-out rows.

slice	tasks	exact
overall	60	0.73
abi_covered=False	44	0.66
abi_covered=True	16	0.94
family=DateTime	22	0.50
family=Name	4	1.00
family=Number	14	0.79
family=Phone	6	1.00

Qwen Sample Misses

task_id	family	features	abi_covered	exact	target	prediction
BillingCode.000007	BillingCode	Concatenation	False	False	11529]	[11529]
DateTime.000006	DateTime	Concatenation,DateTimeRange,DateTimeRounding,DateTime,Multicolumn	False	False	Fri 8:00-9:00	Thu 8:00-9:00
DateTime.000014	DateTime	DateTime	False	False	Friday #1 February 2013	Saturday #1 February 2013
DateTime.000046	DateTime	DateTime	False	False	1952 7 2	7/2/1952 2 2
DateTime.000056	DateTime	DateTimeRange,DateTimeRounding,DateTime	False	False	11AM-1PM	Noon-2PM
DateTime.000065	DateTime	DateTimeRange,DateTimeRounding,DateTime	False	False	12:30PM-12:44PM	12:00PM-12:45PM
DateTime.000066	DateTime	DateTimeRange,DateTimeRounding,DateTime	False	False	12:40PM-12:49PM	12:40PM-12:41PM
DateTime.000067	DateTime	DateTimeRange,DateTimeRounding,DateTime	False	False	12:00PM-1:20PM	11:30AM-12:50PM
DateTime.000088	DateTime	DateTimeRounding,DateTime	False	False	4:00PM	8:00AM
DateTime.000093	DateTime	DateTime	False	False	1990-03-07	1990-11-07
DateTime.000106	DateTime	DateTime	False	False	Mon	Wed
DateTime.000115	DateTime	DateTimeRange,DateTimeRounding,DateTime	False	False	40-60	0-20
Noise.000001	Noise	Concatenation,Multicolumn	True	False	%@^zxi*d^gx%i ry!d!!egrumyr%$$#(k	col0=%@^zxi*d^gx%i ry! col1=d!!egrumyr%$$#(k
Number.000015	Number	Numeric,NumericRounding	False	False	-22600	-22500
Number.000031	Number	Numeric,NumericRounding	False	False	43.5	43.0
Number.000094	Number	Concatenation,Numeric	False	False	₹56,343	₹5,6343

Overall By Tier

tier	tasks	coverage	train_match_rate	train_only_rate	no_train_match_rate	median_candidates
concat	309	19.1%	22.7%	3.6%	77.3%	56833.00
core	309	12.6%	15.2%	2.6%	84.8%	109.00
office	309	17.8%	21.0%	3.2%	79.0%	1057.00

Feature Coverage

feature	tasks	coverage	train_only_rate
DateTimeRange	36	0.0%	0.0%
DateTimeRounding	44	0.0%	0.0%
NumericRange	7	0.0%	0.0%
NumericRounding	42	4.8%	2.4%
Conditional	37	5.4%	10.8%
Numeric	82	7.3%	2.4%
DateTime	117	7.7%	4.3%
Multicolumn	16	18.8%	0.0%
Concatenation	52	19.2%	0.0%
Casing	9	33.3%	0.0%
Substring	111	40.5%	3.6%

Lowest-Coverage Families

family	tasks	coverage	train_only_rate
Address	6	0.0%	16.7%
Author	1	0.0%	0.0%
BillingCode	6	0.0%	0.0%
Column	2	0.0%	0.0%
FilePath	1	0.0%	100.0%
Gender	3	0.0%	0.0%
Log	4	0.0%	0.0%
Meteorite	1	0.0%	0.0%
Product	2	0.0%	50.0%
Url	1	0.0%	0.0%
Number	84	7.1%	2.4%
DateTime	106	8.5%	4.7%
City	9	11.1%	0.0%
Phone	16	37.5%	0.0%
Name	28	39.3%	3.6%
UserAgent	7	42.9%	0.0%
Email	6	50.0%	0.0%
EmergencyCall	2	50.0%	0.0%
Language	2	50.0%	0.0%
Rating	2	50.0%	0.0%
Currency	3	66.7%	0.0%
ShippingCode	10	80.0%	0.0%
Abbreviation	1	100.0%	0.0%
Airline	1	100.0%	0.0%
Noise	1	100.0%	0.0%
Song	1	100.0%	0.0%
State	1	100.0%	0.0%
Team	1	100.0%	0.0%
ZipCode	1	100.0%	0.0%

Covered Program Examples

task_id	family	features	num_examples	program_depth	program
Abbreviation.000001	Abbreviation	Concatenation,Conditional,Substring	52	1.00	initials(COL0)
Airline.000002	Airline	Substring	5	1.00	field,,0
City.000012	City	Substring	22	1.00	field,,1
Currency.000003	Currency	Numeric,Substring	20	1.00	number_2dp(COL0)
Currency.000005	Currency	Numeric,Substring	100	1.00	number_2dp(COL0)
DateTime.000003	DateTime	DateTime	10	1.00	time_hour(COL0)
DateTime.000004	DateTime	Concatenation,DateTime,Multicolumn	20	1.00	concat' '
DateTime.000013	DateTime	Conditional,DateTime	230	0.00	COL0
DateTime.000037	DateTime	DateTime	5	1.00	title(COL0)
DateTime.000092	DateTime	DateTime	32	1.00	time_hour(COL0)
DateTime.000102	DateTime	DateTime	20	1.00	time_hour(COL0)
DateTime.000103	DateTime	DateTime	20	1.00	field:,1
DateTime.000104	DateTime	DateTime	20	1.00	alpha(COL0)
DateTime.000105	DateTime	DateTime	20	1.00	field:,2
Email.000010	Email	Substring	10	1.00	email_domain(COL0)
Email.000011	Email	Substring	10	1.00	field.,0
Email.000013	Email	Substring	100	1.00	email_domain(COL0)
EmergencyCall.000003	EmergencyCall	Casing,Substring	12	2.00	title(field;,1)
Language.000002	Language	Multicolumn,Substring	51	1.00	file_stem(COL1)
Name.000008	Name	Concatenation,Substring	20	1.00	first_last_initials(COL0)
Name.000014	Name	Substring	10	1.00	first_word(COL0)
Name.000025	Name	Casing,Concatenation,Substring	10	2.00	lower(initials_sp(COL0))

Train-Only Failures

task_id	family	features	num_examples	program
Address.000012	Address	Conditional,Substring	10	number_int(COL0)
DateTime.000012	DateTime	Conditional,DateTime	10	fieldspace,-1
DateTime.000096	DateTime	DateTime	20	field-,4
DateTime.000097	DateTime	DateTime	20	field-,2
DateTime.000099	DateTime	DateTime	20	field-,0
DateTime.000101	DateTime	DateTime	20	field-,0
FilePath.000001	FilePath	Conditional,Substring	17	file_stem(COL0)
Name.000037	Name	Conditional,Substring	20	field,,1
Number.000010	Number	Numeric,NumericRounding	40	COL0
Number.000082	Number	Numeric	20	number_int(COL0)
Product.000003	Product	Substring	5	concat' ',last_word(COL0))

No-Train-Match Failures

task_id	family	features	num_examples
Address.000002	Address	Substring	5
Address.000003	Address	Substring	5
Address.000009	Address	Concatenation,Multicolumn	10
Address.000013	Address	Conditional,Substring	10
Address.000014	Address	Substring	29
Author.000001	Author	Conditional,DateTime	39
BillingCode.000001	BillingCode	Concatenation	11
BillingCode.000002	BillingCode	Concatenation,Substring	11
BillingCode.000003	BillingCode	Concatenation	5
BillingCode.000004	BillingCode	Concatenation	5
BillingCode.000005	BillingCode	Concatenation,Substring	5
BillingCode.000007	BillingCode	Concatenation	5
City.000004	City	Conditional	20
City.000005	City	Conditional	14
City.000006	City	Conditional	5
City.000007	City	Conditional	5
City.000008	City	Conditional	5
City.000009	City	Conditional,Substring	6
City.000010	City	Conditional,Numeric	6
City.000011	City	Conditional	7
Column.000001	Column	Concatenation,Conditional,Substring	10
Column.000002	Column	Casing,Concatenation,Conditional,Substring	10
Currency.000004	Currency	Numeric,Substring	20
DateTime.000005	DateTime	Conditional,DateTime	27
DateTime.000006	DateTime	Concatenation,DateTimeRange,DateTimeRounding,DateTime,Multicolumn	1000
DateTime.000007	DateTime	DateTime	25
DateTime.000008	DateTime	DateTimeRange,DateTimeRounding,DateTime	20
DateTime.000009	DateTime	DateTime	100

Interpretation

This public benchmark is a hard gate because task definitions come from an independent repository rather than this experiment's generator. It directly tests whether a frozen transformation ABI covers independent string-transformation tasks under held-out validation. The result is negative for this frozen ABI: the best tier covers only 19.1%, and most misses are no-train-match failures rather than held-out coincidence failures. The ABI is therefore too narrow for the public Transformation.Text suite without substantial primitive expansion or a retrieval step that selects a domain-specific ABI. The frozen-Qwen direct-answer diagnostic points in a different direction: the model often infers these public transformations from examples even when the frozen ABI has no matching program. That suggests the immediate bottleneck is not that the transformations are impossible for the model; it is that the fixed ABI does not expose the right operations for this benchmark. The train-only column is load-bearing. Those tasks are exactly the cases where a plausible expression fits examples but fails held-out rows, so they are excluded from coverage.

Limitations

This run covers Transformation.Text, not Split.Text or Extraction.Text. The ABI is a fixed template library rather than a complete PROSE-style DSL, so misses can reflect missing primitives or bounded search. The benchmark itself contains mostly synthetic data according to its metadata, though it is independent public data rather than generated by this experiment.

Artifacts

Details: analysis/details.csv
Overall summary: analysis/overall_summary.csv
Family summary: analysis/family_summary.csv
Feature summary: analysis/feature_summary.csv
Frozen Qwen direct-answer sample: analysis/qwen_direct_sample.csv
Public benchmark checkout: /workspace/large_artifacts/qwen_public_prose_abi_gate/prose-benchmarks

Experiment log 5

Show the running log (5 entries)

Setup

Created fresh experiment directory: /workspace/experiments/qwen_public_prose_abi_gate
Large artifacts directory: /workspace/large_artifacts/qwen_public_prose_abi_gate
Downloaded Microsoft PROSE public benchmark suite into the large artifacts directory.
Core question: whether a frozen transformation ABI covers a less-curated public benchmark under held-out validation.
Report format: standalone Markdown and HTML with plots.

Run `smoke_v1`

Started: 2026-06-27 03:43:06 UTC
Tasks: 40
Train examples per task: 3; held-out cap: 12
Completed in 45.4s.
Primary coverage: 20.0% (8/40 tasks).
Train-only: 7.5%; no-train-match: 72.5%.

Run `pilot_v1`

Started: 2026-06-27 03:44:21 UTC
Tasks: 120
Train examples per task: 4; held-out cap: 20
Completed in 179.9s.
Primary coverage: 8.3% (10/120 tasks).
Train-only: 5.0%; no-train-match: 86.7%.

Run `main_v1`

Started: 2026-06-27 03:47:49 UTC
Tasks: 309
Train examples per task: 4; held-out cap: 50
Completed in 511.5s.
Primary coverage: 19.1% (59/309 tasks).
Train-only: 3.6%; no-train-match: 77.3%.

Frozen Qwen Direct-Answer Sample

Model: Qwen/Qwen3-4B
Sample: 60 public tasks, seed 20260627; first 4 examples shown, one held-out query scored.
Exact match: 73.3% (44/60).
ABI-covered sample exact: 93.8% (15/16); ABI-missed sample exact: 65.9% (29/44).
This baseline scores one held-out query per task, so it is diagnostic rather than directly comparable to full-task ABI coverage.

Figures 6

Data files 13

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

analysis/family_summary.csv2.6 kB
analysis/feature_summary.csv1.6 kB
analysis/overall_summary.csv380 B
analysis/qwen_direct_summary.csv211 B
runs/main_v1/family_summary.csv2.6 kB
runs/main_v1/feature_summary.csv1.6 kB
runs/main_v1/overall_summary.csv380 B
runs/pilot_v1/family_summary.csv944 B
runs/pilot_v1/feature_summary.csv1.1 kB
runs/pilot_v1/overall_summary.csv248 B
runs/smoke_v1/family_summary.csv939 B
runs/smoke_v1/feature_summary.csv1.1 kB
runs/smoke_v1/overall_summary.csv206 B

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗

Qwen Public PROSE ABI Gate

Results at a glance 7

Overview

Question

Method

Artifacts

Report

Abstract

Method

Run Configuration

Primary Results

Frozen Qwen Direct-Answer Sample

Qwen Sample Misses

Overall By Tier

Feature Coverage

Lowest-Coverage Families

Covered Program Examples

Train-Only Failures

No-Train-Match Failures

Interpretation

Limitations

Artifacts

Experiment log 5

Setup

Run smoke_v1

Run pilot_v1

Run main_v1

Frozen Qwen Direct-Answer Sample

Figures 6

Data files 13

Reproduce

Related

Related experiments

Run `smoke_v1`

Run `pilot_v1`

Run `main_v1`