Qwen3.5-4B Foofah Strategy Discovery Live
The one idea you need
A small model studies a few already-solved table-reshaping puzzles and writes itself two named recipes: one pivots columns into rows, the other merges repeated rows. It freezes them, then follows a recipe to write fresh code for brand-new tables it has never seen.
The question
If a small model invents its own reusable recipes for reshaping messy data tables, does it then solve more new tables than just asking it directly?
What we found
Barely, and nothing genuinely new. The self-written recipes lift exactly-solved tables from 42% to 46%, edging out plain prompting. But a ready-made library of solved programs already handled 56 to 58%, and every table the recipes cracked, that library already covered. Net new tables unlocked: exactly zero. Discovery just re-derived a slice of what the model could already reach.
Why it matters
Before trusting a model to invent reusable strategies, check what a curated library of solved programs already covers — here self-discovery only re-derived existing wins. If you do commit a discovered recipe, gate it on a table-shape match, which made every commit correct.
On this page
Results at a glance 3
How to read
Each bar is one approach; height is the share of 50 tables solved exactly, so taller is better. The left four bars are asking directly and three invented-recipe selection rules; the right two are the ready-made program library.
Takeaway → The invented-recipe bars land just above asking-directly at 42-46%, but the two library bars jump to 56-58% — clever self-made recipes still trail the existing toolbox.
Data table
| condition | exact-match accuracy |
|---|---|
| Direct JSON | 42% |
| Discovered first-visible | 44% |
| Discovered shape-triggered | 46% |
| Discovered oracle union | 46% |
| Baseline first-visible | 56% |
| Baseline oracle union | 58% |
Numbers from reports/final_summary.json (also README/report table)
Technical framing
Held-out exact-match accuracy: discovered strategies vs baselines (50 Foofah tasks) — Discovered strategies beat direct JSON by up to 4pp but add zero tasks beyond the included baseline pool (56-58%).
How to read
Three task families along the bottom; within each, grouped bars compare asking directly, the invented recipes, and the ready-made library. Bar height is the share of that family's five tables solved exactly; taller is better.
Takeaway → On merge-and-split tables the recipes recover cases direct generation misses, yet the library solves those plus a whole family the recipes never crack.
Data table
| Task family | direct JSON | discovered first-visible | baseline first-visible |
|---|---|---|---|
| agriculture | 80% | 80% | 60% |
| potters_wheel_merge_split | 0% | 40% | 80% |
| synthetic_8 | 0% | 0% | 80% |
Numbers from reports/final_summary.json summary.by_family (also report table)
Technical framing
Where policies diverge: per-family accuracy on the three differentiating families — Discovered strategies recover merge_split tasks that direct generation misses, but the baseline pool already solves those plus synthetic_8.
How to read
The two invented recipes along the bottom; paired bars show the share of 50 tables where the recipe passes the shown example versus matches the hidden answer. Taller is better.
Takeaway → Each recipe solves only about one-in-eight to one-in-five tables alone — narrow, family-specific reach, not a general-purpose reshaping skill.
Data table
| condition | visible-example pass | hidden exact match |
|---|---|---|
| pivot_long_to_wide (strat_01) | 12% | 12% |
| flatten_row_groups (strat_02) | 20% | 18% |
Numbers from reports/final_summary.json summary.variant_summary (also report table)
Technical framing
Quality of the two discovered strategy cards across all 50 held-out tasks — Each discovered strategy solves only 6-10 of 50 tasks on its own; their value is narrow, family-specific coverage.
In the author’s words from the Report · “Summary”
This experiment uses Qwen3.5-4B to propose reusable program-generation strategy prompts from calibration examples, freezes those strategies, and evaluates fresh executable-program generations on held-out Foofah-style table transformations.
Overview
This standalone package tests strategy discovery for Foofah-style table transformations.
The experiment uses Qwen3.5-4B locally in two roles:
- Propose reusable strategy prompts from calibration examples.
- Generate fresh executable
transform(table)programs on held-out tasks under the frozen discovered strategy prompts.
Run a smoke test:
python scripts/run_strategy_discovery.py \
--root /workspace/experiments/qwen35_4b_foofah_strategy_discovery_live \
--limit-test 6 \
--max-discovered 2 \
--max-repairs 0 \
--overwriteRun the full held-out evaluation:
python scripts/run_strategy_discovery.py \
--root /workspace/experiments/qwen35_4b_foofah_strategy_discovery_live \
--max-discovered 2 \
--max-repairs 1 \
--resumeOutputs are written under reports/.
Report
Rendered from reports/report.md
Summary
This experiment uses Qwen3.5-4B to propose reusable program-generation strategy prompts from calibration examples, freezes those strategies, and evaluates fresh executable-program generations on held-out Foofah-style table transformations.
Discovered Strategy Cards
strat_01
Name: pivot_long_to_wide
Identify the first column as the identifier and subsequent columns as value columns. Iterate through each value column, extracting rows where the identifier matches, and construct new rows where the identifier is paired with the column header and its corresponding value, effectively transposing the data structure.
strat_02
Name: flatten_row_groups
Detect when multiple rows share identical values in the first column and the remaining columns contain data that should be aggregated horizontally. Group these rows by the first column's value, then concatenate the non-identifier columns from each group into a single row, reducing the row count while expanding the column count.
Held-Out Result
| Policy | Exact | Accuracy | Tokens | Recoveries | Losses | Commit precision |
|---|---|---|---|---|---|---|
| Direct JSON | 21/50 | 42.0% | included | 0 | 0 | n/a |
| Discovered first-visible | 22/50 | 44.0% | 435,131 | 2 | 1 | 69.2% |
| Discovered shape-triggered | 23/50 | 46.0% | 435,131 | 2 | 0 | 100.0% |
| Discovered oracle union | 23/50 | 46.0% | 435,131 | n/a | n/a | n/a |
| Included baseline first-visible | 28/50 | 56.0% | included | n/a | n/a | n/a |
| Included baseline oracle union | 29/50 | 58.0% | included | n/a | n/a | n/a |
New visible-correct tasks over the included baseline oracle: 0.
Strategy Quality
| Strategy | Visible pass | Hidden exact | Tokens |
|---|---|---|---|
strat_01 | 6/50 | 6/50 | 172,558 |
strat_02 | 10/50 | 9/50 | 188,662 |
Family Breakdown
| Family | n | Direct | Discovered | Shape-triggered | Baseline first-visible | Discovered oracle |
|---|---|---|---|---|---|---|
agriculture | 5 | 4/5 | 4/5 | 4/5 | 3/5 | 4/5 |
crime_data_wrangler | 5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
potters_wheel_merge_split | 5 | 0/5 | 2/5 | 2/5 | 4/5 | 2/5 |
potters_wheel_unfold | 5 | 2/5 | 2/5 | 2/5 | 2/5 | 2/5 |
potters_wheel_unfold2 | 5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
proactive_wrangling_fold | 5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
synthetic_12 | 5 | 5/5 | 4/5 | 5/5 | 5/5 | 5/5 |
synthetic_25 | 5 | 4/5 | 4/5 | 4/5 | 4/5 | 4/5 |
synthetic_48 | 5 | 1/5 | 1/5 | 1/5 | 1/5 | 1/5 |
synthetic_8 | 5 | 0/5 | 0/5 | 0/5 | 4/5 | 0/5 |
Figures
Interpretation
The decisive question is whether discovered strategies add held-out recoveries that are not already available to the included baseline pool. The first-visible row measures deployable commitment if any discovered program passes the public example; the shape-triggered row avoids committing discovered programs outside a simple public column-contraction trigger; the oracle row measures coverage if selection were perfect.
Limitations
- Evaluation used
max_discovered=2,max_repairs=1, andlimit_test=all. - Direct and included baseline metrics are read from local records packaged with this experiment; discovered strategy programs are freshly generated in this run.
- This is still a small held-out benchmark and should be repeated across additional family splits before any strategy card is treated as robust.
Experiment log 1
Show the running log (1 entry, 2026-06-28)
2026-06-28
- Created standalone live strategy-discovery package.
- Copied calibration records, held-out baseline records, task cases, and sandbox utilities into this package.
- Planned a smoke run before full held-out evaluation so parsing/execution failures are caught before spending full generation budget.
- Ran a 6-task smoke test, found the first discovered strategy card mentioned pandas, and tightened the discovery prompt/sanitizer to require plain Python list transforms.
- Ran the full 50-task held-out evaluation with
max_discovered=2andmax_repairs=1. - Final held-out result: direct JSON 21/50, discovered first-visible 22/50, discovered shape-triggered 23/50, discovered oracle union 23/50, included baseline first-visible 28/50, included baseline oracle 29/50.
- Primary gate result: discovered strategies added 0 visible-correct tasks beyond the included baseline oracle. The strategy-discovery arm is therefore a clean negative on held-out coverage expansion.
Figures 3
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Run steps are documented inside the experiment folder (README and scripts).