Qwen3.5-4B Foofah Program Strategy Portfolio

Finished2026-06-28imported · line ZBenchmark Generalization Structured Execution and Compilers GitHub ↗

The one idea you need

To reshape a messy table, the model writes several small programs and runs each against the one worked example it can check, like trying a ring of keys against a test lock. Programs that pass become candidates; a commit rule picks which to trust.

The question

When a small model reshapes messy tables, does having it write and test several small programs beat just asking it for the finished table directly?

What we found

Yes, but modestly, and which passing program you trust matters more than the programs. Asking directly for the finished table got 42% right. The rule the team locked in, commit only when two programs agree, reached just 48%. A simpler rule they also wrote down beforehand, trust the first program that passes the check, hit 56%, nearly the 58% best-possible ceiling. Consensus quietly discarded correct lone hits.

Why it matters

When candidate programs self-check against a worked example, commit the first that passes rather than requiring two to agree, since one correct program is usually right. Gains cluster in a few table shapes, so route effort, do not run every strategy everywhere.

Asking directly vs trusting the first passing program42% → 56%exact table matches on 50 reserved test cases

The cautious rule they actually shipped48%committed only when two programs agreed

Best possible from these candidates58%if the right program were always picked

Extra compute the five-strategy portfolio spentabout 12x73,911 → 890,030 forward tokens

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 3

Accuracy: asking directly versus each rule for picking a program

How to read

Height is exact-match accuracy on 50 reserved test tables; taller is better. Left bar asks the model directly; the two middle bars require two or three programs to agree; the next trusts the first program that passes; the rightmost is the best-possible ceiling.

Takeaway → The first-passing bar (56%) nearly touches the 58% ceiling while the agree-rules stall near 48%, so the picking rule, not the programs, capped the gain.

Data table

condition	exact-match accuracy
direct JSON	42%
consensus_2 (dev-selected)	48%
consensus_3	48%
first visible program	56%
direct-or-program oracle	58%

Numbers from reports/test_summary.json

Technical framing

Held-out Foofah accuracy: direct JSON vs frozen strategy portfolio — The dev-selected consensus_2 policy gains modestly (42%→48%); the predeclared first-visible selector nearly reaches the 58% oracle.

What each picking rule commits, rescues, and breaks

How to read

For three picking rules, grouped bars count cases out of 50: programs committed, wrong direct answers rescued (higher is better), and correct direct answers broken (lower is better).

Takeaway → Trusting the first passing program commits 23 and rescues 8 wrong answers at just 1 break; the agree-rules commit fewer and rescue only 3 to 4, safer but too timid.

Data table

condition	program commits	direct-miss recoveries	direct-correct losses
consensus_2 (dev-selected)	14	4	1
consensus_3	11	3	0
first visible program	23	8	1

Numbers from reports/test_summary.json

Technical framing

Program-commit precision vs recoveries by selector (held-out) — First-visible commits more programs and recovers 8 direct misses at only 1 loss; consensus rules are safer but too conservative.

Compute cost: asking directly versus running five program strategies

How to read

Two bars show total forward tokens across the 50-case test run; lower is cheaper. Left is asking the model directly; right runs all five program strategies with example-checked repairs on every case.

Takeaway → The portfolio burns about twelve times more tokens (890,030 versus 73,911) for a modest accuracy gain, real added coverage but not yet an efficient policy.

Data table

condition	forward tokens
direct JSON	73.91k
5-strategy portfolio	89k

Numbers from report table (reports/report.md); reports/test_summary.json (total_forward_tokens)

Technical framing

Token cost of the strategy portfolio vs direct generation (held-out run) — The portfolio spends about 12x more tokens than direct generation, so the accuracy gain is not yet an efficient policy.

In the author’s words from the Overview

Standalone experiment for searching a small portfolio of executable program-generation strategies for Foofah table transformations. Strategy selection is performed on train/dev task families, then the frozen portfolio is evaluated on held-out families.

Overview

Standalone experiment for searching a small portfolio of executable program-generation strategies for Foofah table transformations.

The experiment uses only the local Foofah benchmark files under /workspace/large_artifacts/external_sources/foofah_benchmarks, with cases materialized in data/cases.jsonl. Strategy selection is performed on train/dev task families, then the frozen portfolio is evaluated on held-out families.

Report

Rendered from reports/report.md

Question

Can a small, searched portfolio of executable program-generation strategies improve Foofah table-transformation accuracy over direct JSON generation on held-out task families?

The experiment searches strategy prompts on train/dev families, freezes the selected portfolio, and evaluates it once on held-out families. Hidden answers are used for measurement and for train/dev strategy selection only, never as inputs to generation.

Selected Portfolio

Selected variants: verified_structural, cell_parser, row_column_rule, header_aware, split_fold_unpivot
Selected policy: consensus_2
Selection rule: greedy ordered variants on train; choose prefix and selector maximizing dev exact accuracy with loss/precision/token tie-breaks

Main Held-Out Result

arm	exact	rate	program commits	program precision	direct-miss recoveries	direct-correct losses	forward tokens
Direct JSON	21/50	42.0%	0	-	0	0	73911
Selected portfolio	24/50	48.0%	14	78.6%	4	1	890030
Direct OR selected-program oracle	29/50	58.0%	-	-	-	-	-

Held-Out Selector Tradeoff

selector	exact	rate	program commits	program precision	recoveries	losses
`direct`	21/50	42.0%	0	-	0	0
`first_visible_program`	28/50	56.0%	23	78.3%	8	1
`consensus_2`	24/50	48.0%	14	78.6%	4	1
`consensus_3`	24/50	48.0%	11	90.9%	3	0

Figures

Read

The primary dev-selected policy is a real but modest positive: consensus_2 improves held-out accuracy from 21/50 to 24/50, with 4 direct-miss recoveries and 1 direct-correct loss. It is safer than committing every visible-passing program, but it leaves a large share of candidate coverage unused.

The strongest predeclared held-out selector is first_visible_program: 28/50, with 8 direct-miss recoveries and 1 direct-correct loss. This nearly reaches the direct-or-program oracle of 29/50. It was not chosen by the dev selector, so it should be read as an important selector-mismatch finding rather than as the primary frozen-policy result.

The gain is highly family-concentrated. potters_wheel_merge_split moves from 0/5 direct to 4/5 with first-visible programs, and synthetic_8 moves from 0/5 direct to 4/5. Other families, especially crime_data_wrangler, potters_wheel_unfold, and potters_wheel_unfold2, mostly consume repair budget without producing deployable correct programs.

The cost is substantial: direct generation used 73,911 forward tokens on test, while the five-strategy portfolio used 890,030. This is not yet an efficient policy. The experiment establishes that strategy portfolios can create real held-out executable coverage, but the next iteration should be an adaptive budget/router that spends program attempts only on families or cases likely to benefit.

The most actionable result is therefore not "use all five strategies everywhere." It is: executable program strategies create complementary candidates, visible-pass first fallback captures most of the candidate oracle on some structural families, and the dev-selected consensus rule is too conservative for singleton-correct strategy hits.

Family Readout

family	direct	first-visible	consensus_2	oracle
agriculture	4/5	3/5	3/5	4/5
crime_data_wrangler	0/5	0/5	0/5	0/5
potters_wheel_merge_split	0/5	4/5	3/5	4/5
potters_wheel_unfold	2/5	2/5	2/5	2/5
potters_wheel_unfold2	0/5	0/5	0/5	0/5
proactive_wrangling_fold	5/5	5/5	5/5	5/5
synthetic_12	5/5	5/5	5/5	5/5
synthetic_25	4/5	4/5	4/5	4/5
synthetic_48	1/5	1/5	1/5	1/5
synthetic_8	0/5	4/5	1/5	4/5

Iteration Notes

Eight strategy prompts were smoke-tested first. The full eight-strategy sweep was too slow, and three variants were repair-heavy or non-incremental in the smoke.
The train/dev pilot used five pruned strategies on one case per train/dev family. Train showed direct 9/30, consensus_2 10/30, oracle 12/30. Dev showed direct 4/10, consensus_2 5/10, oracle 5/10.
The frozen held-out run evaluated all 50 cases from the 10 test families with the dev-selected five-strategy portfolio.
The held-out split revealed a mismatch: dev favored consensus_2, but singleton correct visible-program hits on synthetic_8 made first-visible much better on test.

Caveats

Family-heldout evaluation is stricter than a random case split but still uses one external benchmark.
Program candidates are verified on the visible example only; hidden answers are used solely for evaluation and train/dev strategy selection.
The selected strategy portfolio is greedy and small; it is not a global optimum over all possible prompts.
The dev split is only 10 cases, so selector choice is noisy; this run directly showed that dev selected a conservative policy that underfit singleton-correct held-out families.
The portfolio is expensive because every test case receives five program attempts and visible-feedback repairs. Accuracy gains should not be read without the token-cost column.
Exact table matching normalizes all cells to strings and requires exact row/column equality.

Experiment log 1

Show the running log (1 entry, 2026-06-28)

2026-06-28

Created standalone experiment package for Foofah program-strategy portfolio search.
Protocol: deterministic family split; search candidate executable-program prompt variants on train/dev families; freeze selected portfolio; evaluate on held-out families.
Primary comparison: direct JSON versus frozen selected program portfolio, with direct-miss recoveries, direct-correct losses, visible-program oracle, commit precision, and forward-token accounting.
Smoke run started with eight prompt variants. It was stopped after two cases because the full eight-variant sweep was too slow for the intended staged protocol. The smoke also showed shape_first, aggregation_grouping, and transpose_restructure were repair-heavy or non-incremental on the first two cases, so the pilot candidate set was pruned to five variants: verified_structural, row_column_rule, split_fold_unpivot, header_aware, and cell_parser.
Fixed a selector tie-sort bug found by the tiny smoke output and added --max-cases-per-family for family-balanced pilots.
Train pilot completed on one case per train family: direct 9/30, consensus_2 10/30, first-visible 8/30, direct-or-program oracle 12/30.
Dev pilot completed on one case per dev family: direct 4/10, consensus_2 5/10, direct-or-program oracle 5/10. The frozen selected portfolio uses all five pruned variants with consensus_2.
Held-out test completed on all 50 cases from 10 test families: direct 21/50, selected consensus_2 24/50, first-visible 28/50, direct-or-program oracle 29/50.
Generated final report and figures under reports/.