Qwen3.5-4B Foofah Program Repair Agent
The one idea you need
The task: reshape a messy table into a target layout, given one solved example. The model can name the answer directly, or write a reshaping program, test it against that example, and patch bugs until it matches — like fixing a spreadsheet macro on a sample before the real run.
The question
If a small model writes and debugs a table-reshaping program against a worked example before running it, does that beat just guessing the final answer directly?
What we found
No. Guessing the answer directly won outright, solving 55% of unseen tables versus only 25% for the debugged program. But the program is a useful complement, not a replacement: it rescued 18 tables the direct guess botched, so running both and keeping whichever works lifts success to 62%. And the repair rounds earned their keep, raising program accuracy from 16% to 25% while breaking nothing.
Why it matters
Don't replace direct output with a self-debugging code path; run both and keep whichever succeeds. And never treat "passed the worked example" as proof — one in five example-passing programs was silently wrong on the real table.
On this page
Results at a glance 4
How to read
Each bar is one strategy; bar height is the share of unseen tables reshaped exactly right. The first three bars are single strategies, the last two combine direct guessing with the program path; taller is better.
Takeaway → The program-only bars sit well below direct guessing, yet the combined bar is the tallest, showing the two approaches solve different tables.
Data table
| arm | exact held-out accuracy |
|---|---|
| direct JSON | 55.2% |
| initial program | 16% |
| final repaired program | 24.8% |
| direct + program fallback | 56.8% |
| oracle union (direct OR program) | 62.4% |
Numbers from reports/eval_summary.json (matches report table)
Technical framing
Held-out exact-match accuracy by arm (Foofah, n=250) — Repair lifts program accuracy from 16% to 24.8%, and the program path adds +18 cases over direct alone (oracle union 62.4%).
How to read
The horizontal axis is the repair round; lines count tables. One line is tables still being repaired, one is tables now matching the worked example, one is tables that also match the real table. Bigger gains per round are better.
Takeaway → Correct programs arrive fastest early — 40 in the first round, then 16, then 5, then 1 — so repair helps most up front and fades fast.
Data table
| repair round | attempted | visible-example pass | visible pass and hidden-correct |
|---|---|---|---|
| 0 | 250 | 50 | 40 |
| 1 | 200 | 20 | 16 |
| 2 | 180 | 5 | 5 |
| 3 | 175 | 3 | 1 |
Numbers from reports/report.md (Repair By Round table) and reports/eval_summary.json round_stats
Technical framing
Repair progress by round (cases out of 250) — Each repair round converts more failed programs, with diminishing returns: 40 correct at round 0, then +16, +5, +1.
How to read
Bars count tables among programs that passed the worked example: the total that passed, how many were truly right on the real table, and how many were secretly wrong. A smaller wrong bar is better.
Takeaway → Of 78 programs that passed the example, 16 were still wrong on the real table, so passing the example is no guarantee.
Data table
| outcome | cases (of 250) |
|---|---|
| final visible-pass programs | 78 |
| hidden-correct | 62 |
| hidden-wrong (false pass) | 16 |
Numbers from reports/eval_summary.json (final_visible_pass, final_hidden_exact, visible_false_pass)
Technical framing
Verification risk: visible-example pass vs held-out truth — About one fifth (20.5%) of programs that pass the visible example are still wrong on the held-out table, so naive commit-on-pass is unsafe.
How to read
Grouped bars compare two instructions across three measures on a 25-table trial; one color is the plain instruction, the other explicitly forbids copying the example's output. For the two correctness measures higher is better; for the misleading-pass measure lower is better.
Takeaway → The stricter instruction doubled program correctness and cut misleading passes from about seven in ten to four in ten, so telling the model not to memorize helped.
Data table
| metric | standard repair prompt | strict anti-hardcoding prompt |
|---|---|---|
| final program correct | 8% | 16% |
| oracle union | 44% | 52% |
| visible false-pass rate | 71.4% | 42.9% |
Numbers from reports/report.md (Iteration table)
Technical framing
Prompt iteration on the 25-case smoke spread: standard vs strict repair prompt — Explicitly warning the model against hardcoding visible outputs doubled final program correctness and cut the false-pass rate.
In the author’s words from the Report · “Result”
Repair raised visible-verified program correctness from 40 to 62 cases, adding 22 program-correct cases while losing 0. The final program arm contributed 18 direct-miss recoveries. The oracle union reached 156/250, a +18 case headroom over direct JSON generation.
Overview
Standalone experiment on Foofah table transformations.
The experiment asks whether Qwen3.5-4B becomes more useful when it writes an executable table-transform program, observes visible-example failures, and repairs the program over several rounds before executing it on the held-out input table.
Arms and measurements:
- Direct JSON output for the held-out table.
- Initial generated
transform(table)program. - Repair loop with visible feedback from the example input/output.
- Final visible-verified program, scored on held-out
TestAnswer. - Direct/program oracle union and deployable fallback metrics.
- False-pass rate: visible-example pass but held-out failure.
Benchmark source:
/workspace/large_artifacts/external_sources/foofah_benchmarks
Reproduce
python scripts/build_cases.py
python scripts/eval_repair_agent.py --limit 6 --max-repairs 2 --max-direct-tokens 384 --max-code-tokens 512
python scripts/eval_repair_agent.py --max-repairs 3 --max-direct-tokens 768 --max-code-tokens 768 --progress-every 10
python scripts/make_report.pyAll scoring uses exact equality to Foofah's held-out TestAnswer table after converting all cells to strings.
Report
Rendered from reports/report.md
Question
Can Qwen3.5-4B improve table transformation accuracy by writing an executable transform(table) program, observing visible-example failures, and repairing the program over several rounds before held-out execution?
The benchmark is Foofah (https://github.com/markjin1990/foofah_benchmarks), scored by exact equality to held-out TestAnswer tables.
Result
| arm | exact held-out | rate |
|---|---|---|
| Direct JSON generation | 138/250 | 55.2% |
| Initial visible-verified program | 40/250 | 16.0% |
| Final repaired visible-verified program | 62/250 | 24.8% |
| Direct with program fallback on direct parse failure | 142/250 | 56.8% |
| Oracle union: direct OR final program | 156/250 | 62.4% |
Repair raised visible-verified program correctness from 40 to 62 cases, adding 22 program-correct cases while losing 0.
The final program arm contributed 18 direct-miss recoveries. The oracle union reached 156/250, a +18 case headroom over direct JSON generation.
Verification Risk
Final programs passed the visible example on 78/250 cases. Of those, 16 were hidden-wrong, a false-pass rate of 20.5%.
Direct/program agreement occurred on 55 cases, with 44 correct (80.0% precision).
Repair By Round
| round | attempted | visible pass | visible-pass and hidden-correct |
|---|---|---|---|
| 0 | 250 | 50 | 40 |
| 1 | 200 | 20 | 16 |
| 2 | 180 | 5 | 5 |
| 3 | 175 | 3 | 1 |
Mean code-generation rounds per case: 3.22. Including direct JSON generation, the mean model-generation calls per case were about 4.22.
Iteration
Before the full run, the repair prompt was tested and revised:
| smoke | n | initial program | final program | oracle union | program-only | visible false-pass |
|---|---|---|---|---|---|---|
| prefix6 r2 | 6 | 83.3% | 83.3% | 100.0% | 0 | 0.0% |
| standard spread25 r2 | 25 | 8.0% | 8.0% | 44.0% | 0 | 71.4% |
| strict spread25 r2 | 25 | 8.0% | 16.0% | 52.0% | 2 | 42.9% |
The first hard-spread repair prompt increased visible-pass but added no hidden-correct programs. The stricter repair prompt explicitly warned against visible-output hardcoding and improved the same spread from 2 to 4 final program-correct cases, with 2 program-only recoveries.
Diagnostics
Program-only recoveries: exp0_11_5.txt, exp0_13_3.txt, exp0_13_4.txt, exp0_13_5.txt, exp0_22_1.txt, exp0_45_2.txt, exp0_51_3.txt, exp0_51_5.txt, exp0_5_3.txt, exp0_5_4.txt, exp0_8_1.txt, exp0_8_2.txt, exp0_8_4.txt, exp0_8_5.txt, exp0_potters_wheel_divide_3.txt, exp0_potters_wheel_merge_split_3.txt, exp0_potters_wheel_merge_split_4.txt, exp0_potters_wheel_merge_split_5.txt.
Repair-added program-correct files: exp0_11_2.txt, exp0_11_3.txt, exp0_11_4.txt, exp0_11_5.txt, exp0_22_1.txt, exp0_26_3.txt, exp0_27_3.txt, exp0_27_5.txt, exp0_33_5.txt, exp0_40_2.txt, exp0_40_3.txt, exp0_40_4.txt, exp0_40_5.txt, exp0_47_3.txt, exp0_47_4.txt, exp0_5_4.txt, exp0_5_5.txt, exp0_8_1.txt, exp0_8_2.txt, exp0_8_4.txt, exp0_potters_wheel_divide_3.txt, exp0_proactive_wrangling_fold_4.txt.
Visible-pass hidden-wrong files: exp0_13_1.txt, exp0_22_2.txt, exp0_22_3.txt, exp0_22_4.txt, exp0_24_1.txt, exp0_24_2.txt, exp0_26_1.txt, exp0_27_1.txt, exp0_29_4.txt, exp0_33_1.txt, exp0_34_1.txt, exp0_40_1.txt, exp0_48_3.txt, exp0_5_1.txt, exp0_potters_wheel_merge_split_1.txt, exp0_potters_wheel_unfold2_2.txt.
Agreement-hidden-wrong files: exp0_22_2.txt, exp0_22_3.txt, exp0_22_4.txt, exp0_24_3.txt, exp0_24_5.txt, exp0_26_1.txt, exp0_27_1.txt, exp0_34_1.txt, exp0_40_1.txt, exp0_48_3.txt, exp0_potters_wheel_unfold2_3.txt.
Read
The repair loop produced a real coverage gain over one-shot program induction: execution feedback converted additional failed programs into held-out-correct programs. It is not a standalone replacement for direct generation, but it is a complementary tool path.
The deployability issue remains selection. Visible-example verification is useful but incomplete; about one fifth of visible-passing final programs were hidden-wrong. The clean positive signal is the oracle union and the direct-miss recoveries, not naive commit-on-visible-pass.
Caveats
- This is greedy single-sample direct generation and greedy repair generation.
- The loop stops at the first visible-passing program, matching deployment where hidden answers are unavailable.
- Generated code is sandboxed and limited to safe builtins plus
re,math,Counter, anddefaultdict. - Exact table matching is strict after converting cells to strings.
Experiment log 1
Show the running log (1 entry, 2026-06-27)
2026-06-27
- Created standalone Foofah program-repair-agent experiment.
- Objective: test whether execution feedback on the visible example lets Qwen3.5-4B repair generated
transform(table)programs and improve held-out table transformation accuracy. Planned process:
- Build Foofah case JSONL.
- Implement restricted Python execution and visible-output diff feedback.
- Smoke test initial generation plus repair loop.
- Iterate prompt or harness if smoke reveals mechanical failures.
- Run full 250-case evaluation with three repair rounds.
- Generate report, charts, and validation evidence.
- Built 250 Foofah cases across 50 families.
- Safe executor smoke passed, including stripping safe import lines before AST validation.
Prefix smoke, 6 cases, max 2 repairs:
- direct 6/6, initial program 5/6, final repaired program 5/6.
- only one case exercised repair; repair did not fix it.
Hard spread smoke, 25 cases with stride 10, max 2 repairs, first repair prompt:
- direct 11/25, initial program 2/25, final program 2/25, oracle union 11/25.
- repair increased visible-pass from 5/25 to 7/25 but added no hidden-correct programs.
- false-pass among final visible-pass programs was 5/7.
- Revised repair prompt to forbid visible-output hardcoding and emphasize generalization to the new input shape.
Hard spread smoke, same 25 cases, strict repair prompt:
- direct 11/25, initial program 2/25, final program 4/25, oracle union 13/25.
- program-only recoveries 2/25.
- false-pass among final visible-pass programs dropped to 3/7.
- Selected strict repair prompt for full run.
Full 250-case run, max 3 repairs:
- direct JSON exact 138/250 (55.2%).
- initial visible-verified program exact 40/250 (16.0%).
- final repaired visible-verified program exact 62/250 (24.8%).
- repair added 22 program-correct cases and lost 0 initially-correct program cases.
- direct with program fallback on direct parse failure reached 142/250 (56.8%).
- oracle union of direct OR final program reached 156/250 (62.4%), with 18 program-only direct-miss recoveries.
- final visible-pass programs: 78/250; hidden-wrong among those: 16/78 (20.5%).
- direct/program agreement: 55 cases; agreement correct: 44/55 (80.0%).
Generated final report and figures:
reports/report.mdreports/summary.jsonreports/family_summary.jsonreports/figures/overall_accuracy.pngreports/figures/round_progress.pngreports/figures/verification_diagnostics.pngreports/figures/by_num_samples.pngreports/figures/prompt_iteration.pngreports/figures/repair_advantage_families.png
Validation:
data/cases.jsonlandreports/eval_records.jsonlboth contain 250 records.python -m py_compile src/*.py scripts/*.pypassed.- required report artifacts are non-empty.
- standalone-content grep found no references to broader experiment context.
Figures 6
Data files 6
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/eval_summary.json60 kBreports/eval_summary_limit6.json8.9 kBreports/eval_summary_stride10_limit25.json30 kBreports/eval_summary_stride10_strict_limit25.json30 kBreports/family_summary.json41 kBreports/summary.json55 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).