Qwen3.5-4B Foofah Direct vs ABI
The one idea you need
Imagine fixing a messy spreadsheet two ways: a converter that can only chain moves from a fixed menu of table operations, or a person shown one before-and-after example who simply retypes the corrected table. The fixed menu cannot express most oddball reshapes.
The question
To reshape messy tables, is it better to build a converter out of fixed operations, or to just ask a small model to rewrite the table itself?
What we found
Just ask the model. Directly generating the reshaped table got 55% of 250 table tasks exactly right, versus only 18% for the fixed-operation converter. The model even nailed 103 reshapes the converter could not even express, while the converter uniquely solved just 10. Direct generation is far from perfect, with 45% still failing, but the converter is a narrow backup, not the main route.
Why it matters
For small-model table wrangling, do not build a fixed-operation converter first. A plain "here is a before-and-after example, now rewrite this table" prompt, with a generous output length, wins on most real reshapes; keep any converter only as a small backup.
On this page
Results at a glance 2
How to read
Height shows the fraction of 250 table tasks each method got exactly right. Leftmost bar is asking the model directly; the two middle bars are the fixed-operation converter (best-case coverage and realistic selection); the right bar combines model plus converter. Taller is better.
Takeaway → The direct-ask bar towers over both converter bars, beating the specialized converter by roughly three to one; adding the converter on top barely helps.
Data table
| arm | exact held-out accuracy |
|---|---|
| direct Qwen (greedy) | 55.2% |
| ABI oracle coverage | 18% |
| ABI first-visible | 17.2% |
| direct OR ABI fallback | 58.8% |
Numbers from experiments/qwen35_4b_foofah_direct_vs_abi/reports/comparison_summary.json
Technical framing
Foofah held-out exact accuracy: direct generation vs frozen ABI — Directly asking the 4B model to emit the transformed table (55.2%) beats the ABI/compiler route (18%); ABI adds only a small fallback slice.
How to read
Bars split the 250 tasks into four outcome groups: solved by both methods, by asking directly only, by the converter only, or by neither. Taller bars for the direct-only and both groups favor the model.
Takeaway → The 'directly only' bar of 103 dwarfs the 'converter only' bar of 10, so the model handles about ten times more cases the converter cannot than the reverse.
Data table
| outcome bucket | cases |
|---|---|
| both correct | 35 |
| direct only | 103 |
| ABI only | 10 |
| neither | 102 |
Numbers from experiments/qwen35_4b_foofah_direct_vs_abi/reports/comparison_summary.json
Technical framing
Who solves which cases: overlap of direct generation and ABI (250 cases) — Direct generation uniquely solves 103 cases; the ABI's unique contribution is just 10, making it a fallback rather than the main route.
In the author’s words from the Report · “Result”
Direct Qwen is the stronger arm on this external structural-transform benchmark (table on the experiment page). Direct parse rate was 236/250 (94.4%) with a 768-token generation cap.
Overview
Standalone external benchmark comparison:
- Frozen ABI oracle coverage on Foofah table transformations.
- Direct Qwen3.5-4B generation of the held-out
TestAnswertable from the example pair andTestingTable.
The experiment asks whether the ABI/compiler route adds value over simply asking the base model to perform the transformation directly on external structural table tasks.
Source benchmark clone:
/workspace/large_artifacts/external_sources/foofah_benchmarks
Reproduce
python scripts/build_cases.py
python scripts/eval_direct_qwen.py --limit 3 --max-new-tokens 220
python scripts/eval_direct_qwen.py --max-new-tokens 768 --progress-every 10
python scripts/make_report.pyThe full direct-generation arm uses greedy decoding with enable_thinking=False and a 768-token output cap. Outputs are scored by exact equality to Foofah's held-out TestAnswer table after string normalization.
Report
Rendered from reports/report.md
Question
Does the frozen Foofah table-transform ABI add value over directly asking Qwen3.5-4B to transform the held-out table?
This package compares exact held-out TestAnswer accuracy on the same 250 Foofah cases from https://github.com/markjin1990/foofah_benchmarks.
Result
Direct Qwen is the stronger arm on this external structural-transform benchmark:
| arm | exact held-out | rate |
|---|---|---|
| Direct Qwen greedy JSON generation | 138/250 | 55.2% |
| Frozen ABI oracle coverage | 45/250 | 18.0% |
| Frozen ABI first-visible selection | 43/250 | 17.2% |
| Direct Qwen OR ABI first-visible fallback | 147/250 | 58.8% |
Direct parse rate was 236/250 (94.4%) with a 768-token generation cap.
Overlap
| bucket | count |
|---|---|
| Direct and ABI both correct/covered | 35 |
| Direct only | 103 |
| ABI only | 10 |
| Neither | 102 |
Direct accuracy on ABI-covered cases: 35/45 (77.8%).
Direct accuracy on ABI-uncovered cases: 103/205 (50.2%).
The practical fallback union (direct exact OR ABI first-visible) reaches 147/250 (58.8%), a +9 case lift over direct generation alone.
Read
The remaining compiler niche did not appear as the dominant path on Foofah under this test. The ABI's structural table-transform coverage was only 18.0%, and direct Qwen solved many cases outside the ABI's expressivity (direct_only=103). The frozen ABI still has a small complementary slice (abi_only=10; first-visible adds 9 deployable cases), but it is a fallback, not the main route.
The important interpretation is not that direct generation is perfect. It is not: exact accuracy is 55.2%, parse failures remain 14, and long-output cases are penalized by the 768-token cap. The point is narrower and decisive for this gate: on an independent Foofah benchmark, the ABI/compiler route does not beat simply asking the base model to emit the transformed table.
Diagnostics
By NumSamples, direct-vs-ABI accuracy is in reports/comparison_summary.json and reports/figures/by_num_samples.png.
Example direct-only files: exp0_10_1.txt, exp0_10_2.txt, exp0_10_3.txt, exp0_10_4.txt, exp0_10_5.txt, exp0_11_1.txt, exp0_11_2.txt, exp0_11_3.txt, exp0_11_4.txt, exp0_12_1.txt, exp0_12_2.txt, exp0_12_3.txt, exp0_12_4.txt, exp0_12_5.txt, exp0_17_3.txt, exp0_17_4.txt, exp0_17_5.txt, exp0_19_1.txt, exp0_19_2.txt, exp0_19_3.txt.
Example ABI-only files: exp0_13_1.txt, exp0_13_2.txt, exp0_13_3.txt, exp0_13_4.txt, exp0_13_5.txt, exp0_33_1.txt, exp0_45_2.txt, exp0_51_3.txt, exp0_51_4.txt, exp0_51_5.txt.
Caveats
- This is greedy direct generation with a 768-token cap, not a best-possible direct-generation system.
- The ABI baseline is frozen from the prior Foofah gate and imported into
data/abi_*for a standalone comparison. - Exact table matching is strict string-table equality after normalizing cells to strings.
Experiment log 1
Show the running log (1 entry, 2026-06-27)
2026-06-27
- Created standalone direct-vs-ABI comparison package.
- Scope: exact held-out
TestAnswertable on Foofah cases. - Planned arms: frozen ABI held-out coverage, first-visible ABI selection, direct Qwen generation.
- Imported the frozen Foofah ABI gate outputs into
data/abi_*for a standalone apples-to-apples comparison. - Smoke-tested direct Qwen with no-thinking chat template: 3/3 parse, 3/3 exact on the first three cases.
- Launched full direct Qwen greedy JSON generation with
max_new_tokens=768. - Full run complete: direct Qwen 138/250 exact (55.2%), parse 236/250 (94.4%).
- Frozen ABI oracle coverage 45/250 (18.0%), first-visible 43/250 (17.2%).
- Direct-or-ABI-first-visible fallback 147/250 (58.8%), a +9 case lift over direct alone.
- Interpretation: the Foofah structural-transform ABI is complementary but not the main route; direct generation dominates the frozen ABI gate on this external benchmark.
Figures 3
Data files 3
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/comparison_summary.json17 kBreports/direct_qwen_summary.json7.7 kBreports/direct_qwen_summary_limit3.json672 B
Reproduce
Run steps are documented inside the experiment folder (README and scripts).