Qwen3.5-4B Foofah Program Ensemble Consensus

Finished2026-06-27imported · line ZBenchmark Generalization Evidence-Conditioned Selection Interpretability and Diagnostics Reliability and Safety+1 moreGitHub ↗

The one idea you need

Think of fixing a messy spreadsheet: the model writes a few small transform-the-table programs, then checks each against one worked example whose answer is known. The question is whether making several programs agree beats simply running the first one that passes the example.

The question

When a small model writes several little programs to reshape a table, does making them agree beat just running the first one that passes a checkable example?

What we found

No. The simplest rule won: run the first program that passes a single worked example. It solved 52% of tables versus 44% when the model just answered directly, rescuing 23 tables it had otherwise botched while breaking only 4. Requiring several programs to agree was more careful but far too timid, recovering fewer mistakes and topping out at 49%.

Why it matters

For structured tasks where you can write code and check it against a known example, have the small model generate programs and deploy the first one that passes. Don't gate on ensemble agreement — it trades away most of the wins.

Answering directly44%tables solved out of 250

First working program52%tables solved out of 250 — the best rule

Requiring agreement49%the more cautious voting rule fell short

Best any rule could reach54%ceiling if selection were perfect

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 4

Held-out exact-match accuracy by selection policy (250 Foofah tasks) The simple first-visible-passing selector beats consensus and reaches 52%, capturing most of the 54% oracle union over direct JSON's 44.4%.

exact-match accuracy · selection policy →

Data table

selection policy	accuracy
direct JSON	44.4%
consensus >= 3	45.6%
consensus >= 2	48.8%
first visible-passing program	52%
oracle union	54%

Numbers from reports/final_summary.json (oracle union = oracle_union/n)

Recoveries vs losses relative to direct JSON, per selector Consensus never loses a direct-correct case but recovers few misses; first-visible recovers 23 misses at the cost of only 4 losses.

tasks changed vs direct JSON · selection policy →

Data table

selection policy	direct-miss recoveries	direct-correct losses
consensus >= 3	3	0
consensus >= 2	11	0
first visible-passing program	23	4

Numbers from reports/final_summary.json

Program-commit precision by selector Stricter consensus makes each committed program more likely correct, but the precision gain is too small to offset its far lower coverage.

commit precision · selection policy →

Data table

selection policy	precision
consensus >= 3	84.2%
consensus >= 2	81.1%
first visible-passing program	73.3%

Numbers from reports/final_summary.json

Visible-example passes per prompt variant: initial vs repair-added One visible-feedback repair round nearly doubles structural_python's passes but adds little to the other variants.

tasks passing visible example · prompt variant →

Data table

prompt variant	passed initially	added by repair
row_column_rule	52	12
structural_python	29	28
verified_structural	53	6

Numbers from reports/variant_summary.json

In the author’s words from the Overview

This standalone experiment tests whether multiple independently prompted executable table-transform programs can produce safer fallback answers than a single visible-verified program. Each Foofah task is evaluated with: one direct JSON answer, multiple program candidates from different prompts, one visible-feedback repair attempt per program candidate, visible-example filtering, held-out output clustering for consensus policies. Primary question: > Does agreement among independently prompted visible-verified programs improve deployed exact-match accuracy over direct JSON generation and over a single visible-verified program fallback? The final report is generated by scripts/make_report.py.

Overview

This standalone experiment tests whether multiple independently prompted executable table-transform programs can produce safer fallback answers than a single visible-verified program.

Each Foofah task is evaluated with:

one direct JSON answer,
multiple program candidates from different prompts,
one visible-feedback repair attempt per program candidate,
visible-example filtering,
held-out output clustering for consensus policies.

Primary question:

Does agreement among independently prompted visible-verified programs improve deployed exact-match accuracy over direct JSON generation and over a single visible-verified program fallback?

The final report is generated by scripts/make_report.py.

Report

Rendered from reports/report.md

Question

Can Qwen3.5-4B improve Foofah table-transformation accuracy by generating several independently prompted executable transform(table) programs, verifying them on the visible example, and selecting by output consensus on the held-out input?

The benchmark is Foofah (https://github.com/markjin1990/foofah_benchmarks). Hidden answers are used only for evaluation.

Setup

Each of 250 tasks was evaluated with one direct JSON answer and three program variants:

verified_structural
structural_python
row_column_rule

Each program variant received one visible-feedback repair attempt if the initial program failed the visible example. A program candidate was eligible for selection only if it passed the visible example and executed on the held-out input.

Main Result

selector	exact held-out	rate	program commits	program precision	direct-miss recoveries	direct-correct losses
Direct JSON	111/250	44.4%	0	-	0	0
Direct/program agreement only	111/250	44.4%	0	-	0	0
Program consensus >= 3	114/250	45.6%	19	84.2%	3	0
Program consensus >= 2	122/250	48.8%	53	81.1%	11	0
First visible-passing program	130/250	52.0%	101	73.3%	23	4

The best deployed policy was first visible-passing program: 130/250 (52.0%), versus direct JSON at 111/250 (44.4%). It recovered 23 direct misses but lost 4 direct-correct cases.

The oracle union of direct JSON or any visible-correct program reached 135/250 (54.0%). That leaves 5 cases of selector headroom after the best deployed policy.

Consensus

Consensus was safer but too conservative:

Consensus >= 2 committed 53 times with 81.1% precision, recovering 11 direct misses.
Consensus >= 3 committed 19 times with 84.2% precision, recovering 3 direct misses.

There were 101 tasks with at least one visible-passing program, 57 with at least two visible-passing programs, and 53 with an output cluster of size at least two. The ensemble had 24 direct-miss tasks where at least one visible program was hidden-correct.

Variant Diagnostics

variant	visible pass	visible precision	initial visible pass	repair-added visible
row_column_rule	64/250 (25.6%)	79.7%	52	12
structural_python	57/250 (22.8%)	73.7%	29	28
verified_structural	59/250 (23.6%)	71.2%	53	6

Iteration

The first six-case smoke exposed direct-output parsing fragility, so JSON extraction was updated to accept the first valid array prefix and the direct prompt was tightened to forbid prose or markdown.

After that fix, a six-case smoke reached direct 5/6 and first-visible program 5/6. A harder stride-10 smoke with the first prompt set showed no oracle gain and poor visible-program precision, so the weak minimal-code variant was replaced with verified_structural.

The full run used the revised three-variant ensemble with one repair round per variant.

Read

The ensemble generated real additional candidate coverage. Direct JSON solved 111 tasks; direct plus any visible-correct program could solve 135. The simple first-visible selector captured most of that gain, reaching 130.

The specific hypothesis that independent program consensus would be the best selector did not hold. Consensus improved precision over first-visible fallback but under-recovered too many direct misses. On this benchmark, visible-example pass plus fixed prompt order was a better deployed selector than requiring agreement.

Caveats

The full run uses one greedy direct answer and one greedy generation per program variant, with one greedy repair attempt per variant.
The ensemble has three prompt variants; larger or sampled ensembles may change the coverage/precision tradeoff.
Program execution is sandboxed and limited to safe builtins plus re, math, Counter, and defaultdict.
Exact table matching is strict after converting cells to strings.

Experiment log 1

Show the running log (1 entry, 2026-06-27)

2026-06-27

Created standalone Foofah program-ensemble-consensus experiment.
Objective: test whether multiple independently prompted executable programs, filtered by the visible example and selected by output consensus, outperform direct JSON generation and single-program fallback.
Copied 250 Foofah cases into data/cases.jsonl.
Planned process:
- Build evaluator with direct JSON generation, diverse program generation, one visible-feedback repair per program, safe execution, and output clustering.
- Run a small smoke and inspect whether prompt variants produce non-identical visible-passing candidates.
- Iterate prompt set if the ensemble collapses to duplicates or fails to produce visible-passing programs.
- Run full 250-case evaluation.
- Generate report, figures, and validation evidence.
Built scripts/eval_program_ensemble.py with three program variants, one visible-feedback repair round, safe execution, and selector policies.
First six-case smoke:
- direct parser accepted only 1/6 due to valid JSON followed by stray markdown or an extra bracket.
- program candidates produced visible-passing correct programs on 5/6 cases.
- consensus-2 committed on 3/6 and was correct on all committed cases.
- Updated direct prompt to require first/last bracket and updated JSON extraction to accept the first valid array prefix before rerunning the smoke.
Second six-case smoke after parser/prompt fix:
- direct parse 6/6, direct exact 5/6.
- visible-program oracle 5/6.
- first visible-program fallback, consensus-2, and direct all reached 5/6.
- The easy prefix did not test direct-miss recovery, so selected a stride-10 25-case smoke next.
First stride-10 25-case smoke, three variants, one repair:
- direct 9/25.
- visible-program oracle 3/25 and oracle union 9/25, so programs added no direct-miss coverage.
- first visible-program fallback 8/25 with one direct-correct loss.
- consensus-2 committed 7 times but only 2/7 were hidden-correct.
- The minimal_python variant produced visible-passing hidden-wrong candidates and no correct visible programs.
- Replaced minimal_python with verified_structural and selected a second stride smoke with two repairs.
Second stride-10 smoke, revised variants with two repairs:
- stopped after 12/25 records because direct and program oracle were still tied at 6/12 and runtime was high.
- overlap inspection showed some formerly program-recoverable cases were now solved by the stricter direct prompt/parser, so the full run remains informative but should use resume support.
- Added --resume support to the evaluator before launching the full run.
Full 250-case run, revised three-variant ensemble, one repair per variant:
- direct JSON: 111/250 (44.4%), parse 234/250.
- first visible-passing program fallback: 130/250 (52.0%).
- consensus >= 2: 122/250 (48.8%).
- consensus >= 3: 114/250 (45.6%).
- oracle union of direct JSON or any visible-correct program: 135/250 (54.0%).
- tasks with at least one visible-passing program: 101/250.
- first visible-passing program recovered 23 direct misses and lost 4 direct-correct cases.
- consensus >= 2 committed 53 times with 43 correct (81.1% precision), recovering 11 direct misses with 0 direct-correct losses.
- consensus was safer but under-recovered; first visible-passing program was the best deployed selector.
Generated final report and figures:
- reports/report.md
- reports/final_summary.json
- reports/variant_summary.json
- reports/family_summary.json
- reports/prefix_summary.json
- reports/consensus_diagnostics.json
- reports/figures/policy_accuracy.png
- reports/figures/selector_tradeoff.png
- reports/figures/prefix_progress.png
- reports/figures/variant_quality.png
- reports/figures/family_gains.png
Validation:
- data/cases.jsonl and reports/full_ensemble_records.jsonl both contain 250 records.
- python -m py_compile src/*.py scripts/*.py passed.
- required report artifacts are non-empty.
- human-authored standalone-content grep found no broader-context references.
- no evaluator process remained after the run.