Real Transform ABI Gate with Counterexamples

Finished2026-06-27imported · line ZAlgorithmic Memory and Retrieval Benchmark Generalization Evidence-Conditioned Selection Reliability and Safety+1 moreGitHub ↗

The one idea you need

Picture a fixed toolbox of ready-made data-cleaning functions. For each job you try every tool against a few input-output examples and keep whichever fits. The catch: with so few examples a wrong tool can fit by luck, so trick examples are added to unmask those coincidences.

The question

If you match ready-made transformation functions against a handful of input-output examples, do you get the truly correct function, or just one that happens to fit?

What we found

It depends on how messy the data is. For clean, spreadsheet-style pipeline jobs the toolbox covered every one (100%) and held firm even against deliberately tricky examples. For irregular date, ID, and text cleanup, coverage slipped from 95% to 85% once trick examples exposed functions that only looked right. And a wrong function fit the few shown examples about a third of the time.

Why it matters

When auto-picking transformation functions from examples, never trust a match on a few cases: generate adversarial trick examples to catch lucky fits. Expect clean pipelines to work today; messy formatting needs richer functions or a human step.

Clean pipeline jobs solved100%all 20, still 100% after trick examples

Messy jobs after trick examples95% → 85%date/ID/text; trick examples unmasked 2 lucky matches

Wrong function that still fit the examplesabout 1 in 3passed the few shown examples but was actually wrong

Solved jobs needing two chained steps5 of 37nearly every win was a single operation, not deep chaining

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 4

Jobs solved per group, before and after trick examples

How to read

Grouped bars show the share of tasks solved for clean pipelines, messy date/ID work, and overall. Each group pairs coverage from the few shown examples (lighter) against coverage after trick examples are added (darker). Taller is better.

Takeaway → Clean-pipeline bars stay at full height after trick examples while the messy group drops, so the fixed toolbox holds up only on tidy work.

Data table

Domain	raw coverage	filtered coverage
CSV/ETL clean pipeline	100%	100%
Date/ID irregular	95%	85%
Overall	97.5%	92.5%

Numbers from reports/summary.json (by_domain, overall)

Technical framing

Frozen ABI coverage before and after adversarial counterexamples — Clean pipeline transforms stay fully covered after counterexample filtering; the irregular date/ID domain loses coverage on edge cases.

Planted decoy jobs caught by trick examples

How to read

Two bars for the same two deliberately deceptive tasks: coverage using the few shown examples versus coverage after trick examples are added. Here a drop toward zero is the intended, better outcome.

Takeaway → Coverage falls from full to nothing, confirming trick examples removed every planted match that had fit the shown examples for the wrong reason.

Data table

Condition	raw coverage	filtered coverage
Known-coincidence smoke tasks (n=2)	100%	0%

Numbers from reports/summary.json (smoke.overall)

Technical framing

Smoke test: counterexamples remove known-wrong raw winners — Both deliberately coincidence-prone tasks passed raw thin tests but were fully removed by adversarial counterexamples, so raw coverage alone is unsafe.

How many solved jobs used one tool versus two

How to read

Bars count solved tasks that needed a single tool versus two tools chained together. Lighter bars are before trick examples, darker after; height just shows how common each pattern is, not which is better.

Takeaway → The single-tool bar towers over the two-tool bar, so the toolbox mostly proves it can pick the right one tool, not chain several.

Data table

Program depth	raw covered	filtered covered
depth 1 (single primitive)	34	32
depth 2 (composed)	5	5

Numbers from reports/summary.json (raw_depth_counts, filtered_depth_counts)

Technical framing

Depth of winning programs among covered tasks — Coverage is dominated by depth-1 single-primitive wins, so the gate validates operation selection more than deep program synthesis.

How often a passing function was actually wrong

How to read

One bar per group (clean pipelines, messy work, overall) showing how often a function that passed the few shown examples still failed hidden checks. Lower is better.

Takeaway → All three bars sit near a third, showing thin visible examples routinely let wrong functions slip through and why the trick examples are needed.

Data table

Domain	visible-pass candidates wrong on hidden tests
CSV/ETL clean pipeline	37.5%
Date/ID irregular	33.3%
Overall	35.6%

Numbers from reports/summary.json (visible_hidden_wrong_rate)

Technical framing

Visible false-pass pressure by domain — Roughly a third of candidates that pass visible examples fail hidden tests, showing thin visible tests routinely admit wrong programs.

In the author’s words from the Report · “Summary”

Overall filtered coverage was 37/40 (92.5%). The result supports the narrow claim that a generic ABI is useful for pipeline-shaped transformations, but it does not establish broad coverage of irregular transformation work. The coincidence smoke test worked as intended: both known-wrong raw winners were removed by adversarial examples, so raw coverage alone is not a safe headline metric.

Overview

Standalone no-training experiment for deterministic transformation ABI coverage with counterexample filtering.

The package defines two independently curated transformation domains:

clean CSV/ETL-style row transformations
irregular date/ID/string normalization transformations

It freezes a generic ABI before evaluation, measures raw coverage on visible plus standard hidden tests, then applies adversarial counterexample tests generated from the task reference semantics. No model training or checkpoints are produced.

Reproduce

python scripts/run_gate.py --data-dir data --reports-dir reports
python scripts/make_report.py

The main writeup is reports/report.md; figures are in reports/figures/.

Report

Rendered from reports/report.md

Summary

This no-training gate tested whether a frozen, generic transformation ABI covers two held-out-style deterministic transformation domains, and whether additional counterexamples remove thin-test coincidences.

Main result: the clean pipeline domain stayed fully covered after counterexample filtering, while the irregular date/ID/string domain lost coverage on edge cases. Overall filtered coverage was 37/40 (92.5%). The result supports the narrow claim that a generic ABI is useful for pipeline-shaped transformations, but it does not establish broad coverage of irregular transformation work.

The coincidence smoke test worked as intended: both known-wrong raw winners were removed by adversarial examples, so raw coverage alone is not a safe headline metric.

Charts

Coverage by domain

Known-coincidence smoke test

Winning program depth

False-pass pressure

Method

The ABI was frozen before evaluating the expanded 40-task suite.
Coverage was measured twice: raw coverage on visible plus hidden examples, then filtered coverage after extra adversarial examples.
The suite is curated and self-contained. It is not a public benchmark and should be treated as a gate for whether a larger benchmark build is worth doing.
Counterexamples can refute a candidate program when expected behavior is available. They do not certify correctness in reference-free deployment.

Domain Results

Domain	n	Raw coverage	Filtered coverage	Raw removed	Visible false-pass rate
CSV/ETL clean pipeline	20	20/20 (100.0%)	20/20 (100.0%)	0	37.5%
Date/ID irregular	20	19/20 (95.0%)	17/20 (85.0%)	2	33.3%

Program Depth

Raw covered depth counts: {'1': 34, '2': 5}
Filtered covered depth counts: {'1': 32, '2': 5}

Most covered tasks were depth-1 single-primitive programs. Depth-2 coverage appeared mainly in aggregation and parsing transforms. This is useful but limits the composition claim: the gate mainly validates reusable operation selection in clean transformations, not deep program synthesis.

Counterexample Smoke

Smoke task	Raw	Filtered	Raw winning program
smoke_parentheses	yes	no	{"op": "all_distinct_chars"}
smoke_month_30	yes	no	{"op": "all_distinct_chars"}

The two smoke tasks were designed so a broad generic predicate can pass thin raw examples for the wrong reason. Counterexamples removed all raw smoke winners.

Filtered Misses

Task	Domain	Raw	Filtered	Raw winning program	Reason exposed by filter
id_03	date_id_irregular	yes	no	{"op": "normalize_phone"}	Raw winner failed adversarial counterexample
id_11	date_id_irregular	yes	no	{"op": "normalize_date_iso"}	Raw winner failed adversarial counterexample
id_18	date_id_irregular	no	no	-	No ABI candidate passed raw examples

The filtered misses are informative: they are irregular edge cases rather than broad pipeline failures. Examples include phone extensions, full month names, and hyphenated title casing. These are exactly the cases where a fixed generic ABI needs either richer primitives, task-specific logic, or a human/stronger-model expansion step.

Interpretation

The gate gives a positive result for clean CSV/ETL-style transformations: filtered coverage was 100% on the curated clean domain. It gives a narrower result for irregular date/ID/string transformations: raw coverage was high, but counterexamples removed two tasks and one task had no raw ABI solution.

The practical takeaway is to split future work by domain shape:

Clean row/column/filter/sort/group/normalize pipelines are a plausible target for a compiler-to-ABI system.
Irregular extraction and formatting tasks need a stronger counterexample suite and a broader ABI before training a compiler would be justified.
Any future model-training pilot should report depth-1 operation selection separately from depth-2+ composition, because this gate is dominated by single-primitive wins.

Experiment log 1

Show the running log (1 entry, 2026-06-27)

2026-06-27

Created standalone experiment package.
Defined a frozen generic transformation ABI and two curated deterministic transformation domains.
Planned verifier smoke test on coincidence-style false coverage before reading domain coverage.
Planned headline metrics: raw coverage, counterexample-filtered coverage, depth split, false-pass rate, and coverage by domain.
Smoke test was tightened before the first accepted run so broad generic predicates can pass thin raw examples and then be removed by adversarial counterexamples.
Expanded the evaluation suite to 20 CSV/ETL tasks and 20 date/ID/string tasks while keeping the ABI fixed.
Final run: raw coverage 39/40 (97.5%), counterexample-filtered coverage 37/40 (92.5%).
CSV/ETL filtered coverage: 20/20 (100.0%). Date/ID/string filtered coverage: 17/20 (85.0%).
Smoke test: raw coverage 2/2, filtered coverage 0/2, confirming the counterexample filter catches known coincidence-style false coverage.