Qwen3.5-4B Foofah External Transform Gate
The one idea you need
Imagine cleaning up messy spreadsheets using only a fixed drawer of moves: split a column, flip rows into columns, fold repeated headers. If the one move a task needs is not in the drawer, no cleverness rebuilds the right answer. This checks whether the frozen drawer can reach it at all.
The question
If you lock in a small, fixed set of spreadsheet-reshaping moves, can you actually rebuild the correct clean table for most real cleanup tasks?
What we found
No. Across 250 real spreadsheet-cleanup tasks, the frozen set of moves could rebuild the correct table for only 18% of them. And the weakness is not picking the right sequence: simply grabbing the first move-sequence that fits already solves 17%, almost touching that 18% ceiling. The moves themselves are missing, and showing more worked examples does not budge it.
Why it matters
Before training a model to write or choose reshaping programs, first check the correct program is even buildable from your moves. If it is not for most tasks, no training rescues you; widen the set of moves and re-test first.
On this page
Results at a glance 3
How to read
Three bars score the same 250 tasks three ways: fits the shown before-and-after example, actually works on a fresh unseen table, and just takes the first fitting move-sequence. Bar height is the share solved; taller is better.
Takeaway → All three bars sit near one-fifth, and taking the first fitting sequence nearly matches the best-case bar, so the toolbox itself, not the choosing, is the limit.
Data table
| metric | coverage |
|---|---|
| fits example pair (raw) | 22.8% |
| passes held-out table | 18% |
| first-visible selection | 17.2% |
Numbers from experiments/qwen35_4b_foofah_external_transform_gate/reports/summary.json (overall)
Technical framing
Frozen transformation ABI on external Foofah benchmark (250 cases) — The frozen ABI solves only 18% of held-out cases; selection (17.2%) nearly matches the oracle, so expressivity, not selection, is the bottleneck.
How to read
Each line tracks the share of tasks solved as the number of shown before-and-after example pairs rises from one to five. Higher is better. The fresh-table line stays flat and the first-fit line barely moves; the fits-the-example line only falls to meet them.
Takeaway → The fresh-table line is dead flat across one to five examples, meaning failures come from missing moves in the toolbox, not from too few examples.
Data table
| Number of example pairs | raw example coverage | held-out coverage | first-visible selection |
|---|---|---|---|
| 1 | 40% | 18% | 16% |
| 2 | 20% | 18% | 16% |
| 3 | 18% | 18% | 18% |
| 4 | 18% | 18% | 18% |
| 5 | 18% | 18% | 18% |
Numbers from experiments/qwen35_4b_foofah_external_transform_gate/reports/summary.json (by_num_samples)
Technical framing
Coverage vs number of example pairs shown — Held-out coverage is flat from 1 to 5 examples: failures come from missing primitives, not ambiguity; extra examples only prune coincidence fits.
How to read
Two bars split the 50 task types into fully solved versus completely unsolved on fresh tables. Bar height is the count of task types; the left bar is the good outcome.
Takeaway → The unsolved bar towers over the solved one, 41 versus 9, showing the toolbox handles a narrow slice and misses most task types entirely.
Data table
| family outcome | families (of 50) |
|---|---|
| fully covered (5/5 held-out) | 9 |
| zero held-out coverage | 41 |
Numbers from experiments/qwen35_4b_foofah_external_transform_gate/reports/summary.json (by_family) and report table
Technical framing
Benchmark families: all-or-nothing coverage — Coverage is all-or-nothing by family: 9 fold/unpivot/extract families are fully solved, the other 41 of 50 get zero held-out coverage.
In the author’s words from the Report · “Summary”
Main result: the frozen ABI covered 45/250 held-out cases (18.0%). Raw example coverage was 57/250 (22.8%). First-visible selection solved 43/250 (17.2%), nearly all held-out-covered cases, so selection is not the bottleneck. The gate therefore fails at ABI expressivity on this external benchmark. Model training or constrained scoring would be hard to interpret because the correct program is absent for 82% of cases.
Overview
Standalone external transformation gate using the Foofah benchmark format.
The experiment freezes a compact table-transformation ABI, loads benchmark cases from an external source, and measures:
- raw oracle coverage on
InputTable->OutputTable - held-out coverage on
TestingTable->TestAnswer - coverage by benchmark family and sample count
- constrained selection accuracy for simple baselines
No model training is performed unless oracle coverage is high enough to make compiler selection meaningful.
Source benchmark clone:
/workspace/large_artifacts/external_sources/foofah_benchmarks
Reproduce
python scripts/run_gate.py
python scripts/make_report.pyReport
Rendered from reports/report.md
Summary
This standalone gate evaluated a frozen compact table-transformation ABI on the Foofah benchmark format. Each case provides an example pair (InputTable -> OutputTable) and a separate held-out check (TestingTable -> TestAnswer).
Main result: the frozen ABI covered 45/250 held-out cases (18.0%). Raw example coverage was 57/250 (22.8%). First-visible selection solved 43/250 (17.2%), nearly all held-out-covered cases, so selection is not the bottleneck.
The gate therefore fails at ABI expressivity on this external benchmark. Model training or constrained scoring would be hard to interpret because the correct program is absent for 82% of cases.
Source
- Repository:
https://github.com/markjin1990/foofah_benchmarks - Local clone:
/workspace/large_artifacts/external_sources/foofah_benchmarks - Commit used:
87c0f407e0881622acb02fb20893ca2506713a9a - Files evaluated: 250
Charts
Results
| Metric | Value |
|---|---|
| Raw example coverage | 57/250 (22.8%) |
| Held-out coverage | 45/250 (18.0%) |
| First-visible held-out accuracy | 43/250 (17.2%) |
| Mean candidate count | 5.6 |
| Winner depth counts | {'2': 45} |
| Winner family counts | {'compose': 45} |
Coverage by Example Count
| Num examples | n | Raw coverage | Held-out coverage | First-visible accuracy |
|---|---|---|---|---|
| 1 | 50 | 20/50 (40.0%) | 9/50 (18.0%) | 8/50 (16.0%) |
| 2 | 50 | 10/50 (20.0%) | 9/50 (18.0%) | 8/50 (16.0%) |
| 3 | 50 | 9/50 (18.0%) | 9/50 (18.0%) | 9/50 (18.0%) |
| 4 | 50 | 9/50 (18.0%) | 9/50 (18.0%) | 9/50 (18.0%) |
| 5 | 50 | 9/50 (18.0%) | 9/50 (18.0%) | 9/50 (18.0%) |
Held-out coverage stays essentially flat as examples increase from 1 to 5, which points to missing primitives rather than ambiguity from too few examples.
Covered Families
| Family | Held-out covered | Held-out coverage | Mean candidates |
|---|---|---|---|
| craigslist_data_wrangler | 5/5 | 100.0% | 9.2 |
| potters_wheel_fold | 5/5 | 100.0% | 10.6 |
| potters_wheel_fold_2 | 5/5 | 100.0% | 9.6 |
| proactive_wrangling_fold | 5/5 | 100.0% | 20.8 |
| synthetic_13 | 5/5 | 100.0% | 8.0 |
| synthetic_33 | 5/5 | 100.0% | 49.6 |
| synthetic_4 | 5/5 | 100.0% | 11.4 |
| synthetic_45 | 5/5 | 100.0% | 9.4 |
| synthetic_51 | 5/5 | 100.0% | 9.4 |
Families with zero held-out coverage: 41/50.
Raw-Only False Coverage Examples
These cases had a candidate that fit the example pair but failed the held-out table. They are the external benchmark analogue of a counterexample filter removing coincidence fits.
| File | Family | Samples | Candidate count | First program |
|---|---|---|---|---|
| exp0_15_1.txt | synthetic_15 | 1 | 12 | {"steps": [{"cols": [0, 5], "op": "project_0_5"}, {"cols": [0, 1], "op": "project_0_1"}]} |
| exp0_15_2.txt | synthetic_15 | 2 | 33 | {"steps": [{"header": true, "id_cols": 1, "op": "unpivot_h_skip_id1_s2", "skip_empty": true, "start": 2}, {"cols": [0, 1, 2], "op": "project_first_3"}]} |
| exp0_24_1.txt | synthetic_24 | 1 | 9 | {"steps": [{"op": "transpose"}, {"delim": "-", "op": "split_-"}]} |
| exp0_26_1.txt | synthetic_26 | 1 | 11 | {"steps": [{"op": "transpose"}, {"delim": " ", "op": "split_ "}]} |
| exp0_27_1.txt | synthetic_27 | 1 | 240 | {"steps": [{"cols": [0, 1], "op": "project_0_1"}, {"cols": [0, 1], "op": "project_0_1"}]} |
| exp0_28_1.txt | synthetic_28 | 1 | 9 | {"steps": [{"op": "transpose"}, {"delim": ",", "op": "split_,"}]} |
| exp0_34_1.txt | synthetic_34 | 1 | 90 | {"steps": [{"delim": ",", "op": "split_,"}, {"delim": ",", "op": "split_,"}]} |
| exp0_40_1.txt | synthetic_40 | 1 | 132 | {"steps": [{"cols": [0, 1, 2, 3], "op": "project_first_4"}, {"cols": [0, 1, 2, 3], "op": "project_first_4"}]} |
Interpretation
This is a negative but useful breadth test. A compact frozen ABI handles a narrow slice of fold/unpivot and regex-extraction cases, but it does not cover most external Foofah transformations. Because first-visible selection nearly matches oracle held-out coverage, model-side selection is not the next bottleneck here.
The next useful experiment is not compiler training on this ABI. It is strict library expansion under a source-independent protocol: add generic table primitives from documentation or a training-only split, freeze them, then rerun this exact held-out gate. Adding task-specific primitives after inspecting the failed files would invalidate the gate.
Experiment log 1
Show the running log (1 entry, 2026-06-27)
2026-06-27
- Created standalone external transformation gate package.
- Selected Foofah benchmark format because each case provides input/output examples and separate held-out test tables.
- Frozen ABI scope: projection, header dropping, unpivot/melt, key:value folding, and regex extraction.
- Planned gate: run oracle coverage first; only run model selection if held-out oracle coverage is large enough to interpret.
- Smoke run on 20 files completed: 5/20 held-out coverage.
- Full run on 250 files completed: raw coverage 57/250 (22.8%), held-out coverage 45/250 (18.0%).
- First-visible selection solved 43/250 (17.2%), nearly all held-out-covered cases, so model-side selection was not run.
- Generated final report and charts under
reports/.
Figures 3
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/summary.json15 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).