Qwen3.5-4B Foofah Selective Program Fallback
The one idea you need
Reshaping a messy table has two routes: hand-write the finished answer, or write a tiny program that does the reshaping. Run that program on the worked examples you can already see; if it reproduces them exactly, it has earned the right to handle the hidden test.
The question
When a small model can either write a table's answer directly or write a program to produce it, which should you trust, and do extra sanity checks help decide?
What we found
Trust the program the moment it reproduces the visible worked examples. Doing that lifted exact-match accuracy from 55% to 62% across 250 table tasks, rescuing 18 answers the direct route got wrong while losing none it got right. The fancier idea of stress-testing the program against fresh made-up inputs did not help and slightly lowered accuracy.
Why it matters
When a generated program reproduces the examples you can see, deploy it as a fallback and stop there. Skip synthetic double-checks: the model's own answers on invented inputs repeat the same mistakes, so they veto good programs more than bad ones.
On this page
Results at a glance 4
held-out exact match · selector policy →
Data table
| selector policy | accuracy |
|---|---|
| direct JSON only | 55.2% |
| program if parse fails | 56.8% |
| program if probe support >= 0.50 | 56% |
| visible program, probe veto < 0.67 | 58.4% |
| program whenever visible passes | 62.4% |
Numbers from reports/report_metrics.json (summary.policies)
hidden-correct cases (of 26) · answer source →
Data table
| answer source | hidden-correct |
|---|---|
| direct JSON answer | 0 |
| visible-passing program | 18 |
Numbers from reports/report_metrics.json (summary.visible_disagree_direct_correct, visible_disagree_program_correct)
cases · selector policy →
Data table
| selector policy | program commits | correct commits |
|---|---|---|
| program if parse fails | 5 | 4 |
| program if probe support >= 0.50 | 5 | 2 |
| program on visible disagreement | 26 | 18 |
| visible program, probe veto < 0.67 | 62 | 52 |
| program whenever visible passes | 78 | 62 |
Numbers from reports/report_metrics.json (summary.policies program_commits / program_commit_correct)
mean probe agreement · program hidden-correctness →
Data table
| program hidden-correctness | mean probe support |
|---|---|
| hidden-correct programs | 25% |
| hidden-wrong programs | 33.3% |
In the author’s words from the Overview
This standalone experiment tests whether a generated Python table-transform program should be used as a fallback to direct JSON generation on Foofah table-transformation tasks. The experiment starts from a self-contained candidate set with one direct JSON answer and one repaired executable transform(table) candidate per task. It then evaluates selector policies that decide whether to keep the direct answer or commit the visible-verified program. Some policies use extra counterexample-style probe inputs and independent direct model answers on those probes. Primary question: > Can counterexample-stressed agreement safely convert visible-verified program candidates into deployed accuracy beyond direct JSON generation? … Read the full result →
Overview
This standalone experiment tests whether a generated Python table-transform program should be used as a fallback to direct JSON generation on Foofah table-transformation tasks.
The experiment starts from a self-contained candidate set with one direct JSON answer and one repaired executable transform(table) candidate per task. It then evaluates selector policies that decide whether to keep the direct answer or commit the visible-verified program. Some policies use extra counterexample-style probe inputs and independent direct model answers on those probes.
Primary question:
Can counterexample-stressed agreement safely convert visible-verified program candidates into deployed accuracy beyond direct JSON generation?
The report is generated by scripts/make_report.py.
Report
Rendered from reports/report.md
Question
Can Qwen3.5-4B improve deployed Foofah table-transformation accuracy by using a visible-verified executable transform(table) program as a fallback to direct JSON generation, and do counterexample-style probe inputs make that fallback decision safer?
The benchmark is Foofah (https://github.com/markjin1990/foofah_benchmarks). Each task has visible input-output examples and a held-out test input. Hidden answers are used only for evaluation.
Candidate Pool
The package contains 250 task records. Each record has:
- one direct JSON answer for the held-out input,
- one executable program candidate after visible-example repair,
- visible-example execution status for the program,
- held-out exact-match labels for evaluation.
Main Result
| selector | exact held-out | rate | program commits | program precision | direct-miss recoveries | direct-correct losses |
|---|---|---|---|---|---|---|
| Direct JSON | 138/250 | 55.2% | 0 | - | 0 | 0 |
| Program only if direct parse fails | 142/250 | 56.8% | 5 | 80.0% | 4 | 0 |
| Program on disagreement if probe support >= 0.50 | 140/250 | 56.0% | 5 | 40.0% | 2 | 0 |
| Visible program, veto disagreement if probe support < 0.67 | 146/250 | 58.4% | 62 | 83.9% | 8 | 0 |
| Program on visible disagreement only | 156/250 | 62.4% | 26 | 69.2% | 18 | 0 |
| Program whenever visible example passes | 156/250 | 62.4% | 78 | 79.5% | 18 | 0 |
The strongest deployed policy was also the simplest: commit the program whenever it passes the visible example. It reached 156/250 (62.4%), improving direct JSON by +18 cases with 0 direct-correct losses.
The key diagnostic is the visible-disagreement slice. There were 26 cases where the program passed the visible example but disagreed with the direct answer. In that slice, direct JSON was hidden-correct on 0 cases, while the program was hidden-correct on 18 cases. That made visible-program fallback strongly complementary to direct generation in this candidate pool.
Counterexample Probes
For each visible-disagreement case, the evaluator generated up to three deterministic probe input tables and asked Qwen3.5-4B for direct JSON outputs on those probes. The candidate program was also executed on the same probes. Probe support is the fraction of comparable probes where direct output and program output agreed.
The probe mechanism did not improve selection:
- Probe support >= 0.50 recovered 2 direct misses and reached 140/250 (56.0%).
- Visible-program fallback with a probe-support veto recovered 8 direct misses and reached 146/250 (58.4%).
- Mean probe agreement on the decision slice was 28.3%.
Probe support was not a reliable correctness signal. Among visible-disagreement cases with comparable probes, mean support was 25.0% for hidden-correct programs and 33.3% for hidden-wrong programs.
Iteration
The experiment used three stages:
- A no-model selector diagnostic over all 250 cases established that visible-program fallback reached 156/250 and parse-failure fallback reached 142/250.
- A small model-probe smoke on 8 visible-disagreement cases showed probe thresholds rejecting most useful program wins.
- A full model-probe pass on all 26 visible-disagreement cases confirmed that probe support was weaker than the simple visible-pass rule.
Read
The useful result is not that counterexample-stressed direct agreement solved selection. It did not. The useful result is that visible-example execution alone was a strong fallback gate for this candidate pool: every direct-correct case survived, and the visible-passing programs recovered 18 direct misses.
The counterexample probes failed for a concrete reason: the independent direct channel often agreed with the same wrong extrapolation, while rejecting many correct programs. Generated probes added cost and reduced accuracy under thresholded policies.
Caveats
- The candidate pool is fixed and included in
data/candidate_records.jsonl. - Program fallback is evaluated only for programs that pass the visible example.
- Probe answers are greedy Qwen3.5-4B direct JSON generations on synthetic probe inputs, not ground truth.
- Hidden answers are used only for evaluation and policy comparison.
- Exact table matching is strict after normalizing cells to strings.
Experiment log 1
Show the running log (1 entry, 2026-06-27)
2026-06-27
- Created standalone selective-program-fallback experiment.
- Objective: test whether visible-verified executable program candidates can be safely used as fallback answers for Foofah table transformations, and whether counterexample-style probe agreement improves the fallback decision.
- Copied self-contained Foofah cases and candidate records into
data/. Planned process:
- Build selector/probe evaluator.
- Run no-model diagnostic to establish deterministic selector baselines.
- Run smoke with model-scored probe inputs.
- Iterate selector thresholds if smoke reveals useful signal.
- Run full visible-program probe evaluation.
- Generate report, figures, and validation evidence.
- Built
scripts/eval_selective_fallback.pywith policies for direct-only, parse-failure fallback, visible-program fallback, visible-disagreement fallback, and counterexample-probe support thresholds. No-model full diagnostic:
- direct JSON: 138/250 (55.2%).
- program only if direct parse fails: 142/250 (56.8%).
- program whenever visible example passes: 156/250 (62.4%).
- visible-program fallback recovered 18 direct misses and lost 0 direct-correct cases.
- visible-disagreement slice: 26 cases; direct correct 0, program correct 18.
Model-probe smoke on 8 visible-disagreement cases:
- visible-program fallback: 5/8.
- probe-support threshold policies: 0/8 or 1/8.
- probe support was higher on some hidden-wrong programs than on hidden-correct programs, so broader probing was restricted to the decision slice.
Deterministic program-probe full pass:
- all 78 visible-passing programs executed on generated probe inputs.
- crashes, constant outputs, and output-variety features did not separate correct programs from hidden-wrong programs.
Full model-probe pass on all 26 visible-disagreement cases:
- visible-program fallback: 18/26.
- probe support >= 0.50: 2/26.
- visible-program fallback with probe veto: 8/26.
- mean probe support was 25.0% for hidden-correct programs and 33.3% for hidden-wrong programs among comparable cases.
Final merged all-250 policy comparison:
- direct JSON: 138/250 (55.2%).
- parse-failure fallback: 142/250 (56.8%).
- probe support >= 0.50: 140/250 (56.0%).
- visible-program fallback with probe veto: 146/250 (58.4%).
- program on visible disagreement only: 156/250 (62.4%).
- program whenever visible example passes: 156/250 (62.4%).
Generated report and figures:
reports/report.mdreports/report_metrics.jsonreports/final_summary.jsonreports/final_records.jsonlreports/figures/policy_accuracy.pngreports/figures/program_commit_tradeoff.pngreports/figures/probe_support_histogram.pngreports/figures/family_gains.png
Figures 4
Data files 8
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/final_family_summary.json141 kBreports/final_summary.json2.7 kBreports/model_probe_visible_disagree_summary.json2.8 kBreports/program_probe_summary.json2.7 kBreports/report_metrics.json9.3 kBreports/selective_family_summary.json141 kBreports/selective_summary.json2.7 kBreports/smoke_visible_disagree_summary.json2.6 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).