Qwen3.5-4B Foofah Ephemeral Program Induction
The one idea you need
The job is reshaping a messy table into a clean one. Instead of doing it directly, the model writes a small reshaping script, tests it on the one visible example, then runs it on hidden rows — like a spreadsheet macro that passes one row but breaks on the rest.
The question
If a small model writes a little reshaping program and tests it on the one example it can see, does that beat just asking it for the answer directly?
What we found
No. Asking directly reshaped 55% of tables correctly; the write-and-test-a-program route managed just 15%. Even a magic chooser that always picked the right route each time would reach only 59% — four points above asking directly. And the self-check leaks: about 1 in 5 programs that passed the one visible example were still wrong on the hidden data.
Why it matters
For small-model table reshaping, direct generation beats having the model write and self-check code. A single passing example is a leaky correctness gate — about a fifth of programs that clear it are still wrong — so never ship on that signal alone.
On this page
Results at a glance 4
How to read
Each bar is one method; bar height is the share of 250 tables reshaped exactly right, so taller is better. Asking directly, and directly with a script backup, sit near 55%; a perfect chooser reaches 59%; the script-only bar is far shorter at 15%.
Takeaway → The script-only bar is under a third the height of asking directly, and even a perfect chooser barely clears the direct bar — the script route adds almost nothing.
Data table
| Strategy | exact held-out rate |
|---|---|
| Direct JSON generation | 55.2% |
| Visible-verified program | 15.2% |
| Direct + program fallback | 55.6% |
| Oracle union (direct OR program) | 59.2% |
Numbers from reports/summary.json
Technical framing
Held-out exact-match rate on Foofah by strategy (n=250) — Ephemeral program induction is far weaker than direct output; even a perfect selector would add only 4pp over direct.
How to read
Bars count the 250 tables by who solved them: both methods, direct-answering only, script only, or neither. Comparing the direct-only bar against the script-only bar shows which approach uniquely rescues more; taller is more.
Takeaway → The direct-only bar towers over script-only, 110 versus 10 — asking directly uniquely solves eleven times as many tables as the script route does.
Data table
| Outcome bucket | cases |
|---|---|
| Both correct | 28 |
| Direct only | 110 |
| Program only | 10 |
| Neither | 102 |
Numbers from report table (reports/report.md), matching reports/summary.json counts
Technical framing
Where direct output and induced programs succeed (250 cases) — Programs recover only 10 cases direct generation misses, while direct alone solves 110 that programs miss.
How to read
Bars follow the script route left to right: code written, code that runs on the visible example, code that passes it, and code actually right on the hidden data. Each stage drops, showing where scripts fail; taller is better.
Takeaway → The count collapses from 250 written to 47 passing the visible example, and 9 of those 47 still fail on hidden data — passing one example proves little.
Data table
| Pipeline stage | cases |
|---|---|
| Code found | 250 |
| Runs on visible example | 202 |
| Passes visible example | 47 |
| Correct on held-out | 38 |
Numbers from reports/summary.json
Technical framing
Program-induction pipeline attrition (250 cases) — The bottleneck is passing the visible example (47/250), and 9 of those 47 visible-pass programs are still wrong on held-out data.
How to read
Grouped bars per test show asking directly, the script route, and a perfect chooser. The first two tests use easy hand-picked cases; the last two use a harder mixed sample, where higher is better.
Takeaway → On the easy cases direct answering and the perfect chooser hit 100% while the script route already trailed at 63% and 88%; on the harder sample the script route collapses to 12% — the easy cases had flattered it.
Data table
| Prompt smoke | direct | program | oracle union |
|---|---|---|---|
| induce prefix8 (n=8) | 100% | 62.5% | 100% |
| context prefix8 (n=8) | 100% | 87.5% | 100% |
| context spread25 (n=25) | 44% | 12% | 48% |
| context_v2 spread25 (n=25) | 44% | 12% | 44% |
Numbers from report table (reports/report.md, Iteration section)
Technical framing
Prompt-iteration smokes before the full run — Easy prefix cases flattered the program arm; on the harder spread it collapsed to 12%, and 'context' kept the only direct-failure recovery.
In the author’s words from the Report · “Result”
Program induction found code for 250/250 cases and executed on the visible example for 202/250. It passed the visible example on 47/250, but 9 of those visible-pass programs were hidden-wrong.
Overview
Standalone experiment on Foofah table transformations:
- Direct Qwen3.5-4B table generation.
- Ephemeral Python program induction: Qwen writes a
transform(table)function from examples. - Visible-example verification of generated programs.
- Held-out execution on Foofah
TestingTable. - Agreement analysis between direct output and executable program output.
The experiment asks whether "write a bespoke executable transformer, verify it, then execute it" adds value over direct output generation on an external table-transformation benchmark.
Benchmark source:
/workspace/large_artifacts/external_sources/foofah_benchmarks
Reproduce
python scripts/build_cases.py
python scripts/eval_qwen.py --limit 8 --program-prompt induce --max-direct-tokens 384 --max-code-tokens 512
python scripts/eval_qwen.py --program-prompt context --max-direct-tokens 768 --max-code-tokens 768 --progress-every 10
python scripts/make_report.py --records reports/eval_records_context.jsonlAll scoring is exact table equality against held-out TestAnswer after converting cells to strings.
Report
Rendered from reports/report.md
Question
Can Qwen3.5-4B improve external table transformations by writing a bespoke executable transform(table) function, verifying it on the visible example, and executing it on the held-out input?
The benchmark is Foofah (https://github.com/markjin1990/foofah_benchmarks), scored by exact equality to held-out TestAnswer tables.
Result
| arm | exact held-out | rate |
|---|---|---|
| Direct JSON generation | 138/250 | 55.2% |
| Visible-verified generated program | 38/250 | 15.2% |
| Direct with program fallback on direct parse failure | 139/250 | 55.6% |
| Oracle union: direct OR program | 148/250 | 59.2% |
Program induction found code for 250/250 cases and executed on the visible example for 202/250. It passed the visible example on 47/250, but 9 of those visible-pass programs were hidden-wrong.
Overlap
| bucket | count |
|---|---|
| Both direct and program correct | 28 |
| Direct only | 110 |
| Program only | 10 |
| Neither | 102 |
Agreement between direct output and program execution occurred on 34/250 cases, with precision 28/34 (82.4%).
Program-only recoveries: exp0_13_2.txt, exp0_13_4.txt, exp0_13_5.txt, exp0_45_2.txt, exp0_51_5.txt, exp0_8_2.txt, exp0_potters_wheel_merge_split_2.txt, exp0_potters_wheel_merge_split_3.txt, exp0_potters_wheel_merge_split_4.txt, exp0_potters_wheel_merge_split_5.txt.
Agreement-hidden-wrong cases: exp0_22_2.txt, exp0_22_3.txt, exp0_22_4.txt, exp0_27_1.txt, exp0_34_1.txt, exp0_40_1.txt.
Iteration
Before the full run, four prompt smokes were run:
| prompt smoke | n | direct | program | oracle union | agreement precision |
|---|---|---|---|---|---|
| induce prefix8 | 8 | 100.0% | 62.5% | 100.0% | 100.0% |
| context prefix8 | 8 | 100.0% | 87.5% | 100.0% | 100.0% |
| context spread25 | 25 | 44.0% | 12.0% | 48.0% | 33.3% |
| context_v2 spread25 | 25 | 44.0% | 12.0% | 44.0% | 75.0% |
The full run used context: it had lower agreement precision than context_v2 on the hard spread, but it preserved the only direct-failure recovery in that spread and therefore had higher coverage headroom.
Read
The generated-program route tests a tool-use idea: the model emits a bespoke executable artifact, the artifact is checked on the visible example, and the checked artifact is executed on the held-out input. The result should be read through coverage and selection separately.
The executable-program arm is real but weak on this benchmark. It creates some correct programs outside direct generation (program_only=10), but visible-example verification is not enough to make it deployable: false-pass among visible-pass programs is 9/47 (19.1%).
The decisive number is the oracle union. If it is meaningfully above direct generation, there is headroom for a better selector or verifier over direct-vs-program outputs. If it is close to direct generation, ephemeral program induction is not adding much capability on Foofah.
Caveats
- This is greedy single-sample direct generation and greedy single-sample code generation.
- Generated code is sandboxed and limited to safe builtins plus
re,math,Counter, anddefaultdict. - The program is verified only on the visible example before held-out execution; visible-pass hidden-wrong is expected and measured.
- Exact table matching is strict after converting cells to strings.
Experiment log 1
Show the running log (1 entry, 2026-06-27)
2026-06-27
- Created standalone ephemeral-program-induction experiment package.
- Objective: compare direct Qwen table output with generated executable Python transformers on the same Foofah held-out
TestAnswerrows. Planned iteration:
- Build Foofah case JSONL.
- Smoke test strict induction and context-aware program prompts.
- Select the better prompt for the full 250-case run.
- Report direct, program, agreement, fallback, and oracle-union metrics with figures.
- Built 250 Foofah cases across 50 families.
- Safe executor smoke passed, including safe-import stripping for generated programs.
Prefix smoke, 8 cases:
- strict induction: direct 8/8, program hidden 5/8, visible-pass 5/8, agreement precision 5/5.
- context prompt: direct 8/8, program hidden 7/8, visible-pass 7/8, agreement precision 7/7.
Hard spread smoke, 25 cases with stride 10:
- context prompt: direct 11/25, program hidden 3/25, oracle union 12/25, visible-pass 7/25, agreement precision 2/6.
- context_v2 prompt: direct 11/25, program hidden 3/25, oracle union 11/25, visible-pass 6/25, agreement precision 3/4.
- Selected
contextprompt for full run because it preserved the only direct-failure recovery in the hard spread.context_v2is retained as an iteration record: more conservative agreement, less coverage. Full 250-case
contextrun complete:- Direct JSON generation: 138/250 exact (55.2%), parse 236/250 (94.4%).
- Visible-verified generated program: 38/250 exact (15.2%).
- Direct with program fallback on direct parse failure: 139/250 exact (55.6%).
- Oracle union, direct OR program: 148/250 (59.2%).
- Program-only recoveries: 10 cases.
- Visible-pass hidden-wrong: 9/47 (19.1%).
- Direct/program agreement: 34 cases, 28 correct (82.4% precision).
- Generated final report and figures under
reports/.
Figures 5
Data files 7
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/eval_summary_context.json35 kBreports/eval_summary_context_limit8.json5.0 kBreports/eval_summary_context_stride10_limit25.json18 kBreports/eval_summary_context_v2_stride10_limit25.json18 kBreports/eval_summary_induce_limit8.json4.9 kBreports/family_summary.json34 kBreports/summary.json45 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).