Qwen3.5-4B Transform ABI Compiler Pilot
The one idea you need
Think of cleaning a messy spreadsheet or pulling an ID out of ragged text. Instead of asking the model to type the code, the system pre-writes every valid short recipe, shows a few input-to-output examples, and the model just ranks which recipe fits. A trusted program then runs the winner exactly.
The question
If you hand a small model a menu of ready-made data-transformation recipes instead of asking it to write the code itself, can it reliably pick the right one?
What we found
Yes. After a light round of tuning, the model chose a recipe that produced the correct output on all 48 test tasks, matching a perfect answer key and beating the untuned model's 92%. It recovered the harder multi-step chains the untuned model flubbed, 19 of 21 up to 21 of 21. Honest caveat: simply matching recipes against the shown examples already scored about 90%, so the menu was fairly easy.
Why it matters
For structured tasks, let the model score valid pre-built options instead of generating code from scratch: this erases syntax errors, and a tiny tune can push the choice to perfect. But seed the menu with hard look-alike decoys, or high accuracy is an illusion.
On this page
Results at a glance 3
How to read
Each bar is a way of choosing the recipe; bar height is the share of 48 tasks whose output came out correct, taller is better. The rightmost bar is a perfect answer key; the two leftmost are simple rules that pick any recipe matching the shown examples.
Takeaway → The tuned-model bar rises to the same full height as the perfect answer key, while the untuned model's bar stops short, missing four tasks.
Data table
| arm | filtered execution accuracy |
|---|---|
| first visible | 93.8% |
| random visible (mean) | 90% |
| frozen Qwen scorer | 91.7% |
| QLoRA scorer | 100% |
| oracle | 100% |
Numbers from reports/report.md table; reports/eval/{first_visible,random_visible,base,lora,oracle}_summary.json
Technical framing
Filtered execution accuracy: QLoRA scorer closes the oracle gap — A small QLoRA on the constrained scorer reaches 48/48, matching the oracle; frozen Qwen misses 4 tasks and non-model baselines are already ~90%.
How to read
Each method shows two bars: one for single-step tasks and one for tasks needing several chained steps; height is the share solved correctly, taller is better. Only the tuned model reaches full height on both.
Takeaway → Only the tuned model tops out on both bars; the untuned model's multi-step bar dips below its single-step one, showing chaining was the weak spot.
Data table
| arm | depth-1 (operation selection) | depth-2+ (composition) |
|---|---|---|
| first visible | 88.9% | 100% |
| random visible (mean) | 89.6% | 90.5% |
| frozen Qwen scorer | 92.6% | 90.5% |
| QLoRA scorer | 100% | 100% |
Numbers from reports/report.md table; reports/eval/{first_visible,base,lora}_summary.json
Technical framing
Accuracy by composition depth — The QLoRA scorer recovers both depth-2+ composition tasks frozen Qwen missed (19/21 to 21/21) while also perfecting depth-1 selection.
How to read
Each bar shows how often a method chose the exact intended recipe, not just any recipe that happens to produce the right output; height is the share, taller is better.
Takeaway → The tuned model's bar towers to full height while every other bar sits near or below half, meaning the others often pass with look-alike recipes.
Data table
| arm | exact program match |
|---|---|
| first visible | 43.8% |
| random visible (mean) | 45.4% |
| frozen Qwen scorer | 52.1% |
| QLoRA scorer | 100% |
Numbers from reports/report.md table; reports/eval/{first_visible,base,lora}_summary.json
Technical framing
Exact target-program match rate — Baselines often pass via visible-consistent alternatives; only the QLoRA scorer selects the exact target ABI program every time.
In the author’s words from the Report · “Summary”
The QLoRA scorer reached 48/48 filtered execution accuracy (100.0%), compared with 44/48 (91.7%) for frozen Qwen and 48/48 (100.0%) for the oracle. On the composition slice, QLoRA reached 21/21 (100.0%) versus frozen Qwen's 19/21 (90.5%). This is a positive compiler-learnability result inside the generated transformation distribution. It should not be read as a broad production result: the tasks are generated from the same frozen ABI grammar, and most coverage remains shallow. The next gate is a less-curated task source with the ABI frozen before task inspection.
Overview
Standalone compiler pilot for deterministic transformation tasks.
The experiment tests whether Qwen3.5-4B can choose executable ABI programs for a frozen transformation library. It uses constrained candidate scoring rather than free-form JSON decoding: candidate ABI programs are enumerated, the model scores each candidate under the task prompt, and the selected program is executed by a deterministic interpreter.
The headline metrics separate:
- depth-1 operation selection
- depth-2/3 composition
- raw example accuracy
- counterexample-filtered accuracy
Large model adapters are written under /workspace/large_artifacts/qwen35_4b_transform_abi_compiler_pilot/.
Reproduce
python scripts/build_data.py
python scripts/eval_constrained.py --arm base
python scripts/train_lora.py
python scripts/eval_constrained.py --arm lora --adapter /workspace/large_artifacts/qwen35_4b_transform_abi_compiler_pilot/lora
python scripts/make_report.pyReport
Rendered from reports/report.md
Summary
This pilot tested a constrained compiler surface for deterministic transformation tasks. Candidate ABI programs were enumerated, Qwen scored each candidate under the task prompt, and the selected program was executed by a deterministic interpreter. This removes JSON syntax validity as a confound and isolates operation/composition choice.
The QLoRA scorer reached 48/48 filtered execution accuracy (100.0%), compared with 44/48 (91.7%) for frozen Qwen and 48/48 (100.0%) for the oracle. On the composition slice, QLoRA reached 21/21 (100.0%) versus frozen Qwen's 19/21 (90.5%).
This is a positive compiler-learnability result inside the generated transformation distribution. It should not be read as a broad production result: the tasks are generated from the same frozen ABI grammar, and most coverage remains shallow. The next gate is a less-curated task source with the ABI frozen before task inspection.
Charts
Dataset
- Train records: 180
- Validation records: 40
- Eval records: 48
- Eval depth counts:
{'1': 27, '2': 15, '3': 6} - Eval domain counts:
{'csv_etl': 30, 'date_id_irregular': 18} - Mean eval candidate count: 29.5
Results
| Arm | Overall filtered | Depth-1 filtered | Depth-2+ filtered | Exact program |
|---|---|---|---|---|
| First visible | 93.8% | 88.9% | 100.0% | 43.8% |
| Random visible mean | 90.0% | 89.6% | 90.5% | 45.4% |
| Frozen Qwen scorer | 91.7% | 92.6% | 90.5% | 52.1% |
| QLoRA scorer | 100.0% | 100.0% | 100.0% | 100.0% |
| Oracle | 100.0% | 100.0% | 100.0% | 100.0% |
Random visible baseline seed range: 87.5%-93.8%, mean 90.0%.
Base Misses Recovered By QLoRA
| Task | Domain | Depth | Base selected | Target |
|---|---|---|---|---|
| eval_0001_score_select | csv_etl | 2 | {"steps":[{"cols":["name","score"],"op":"select_cols"},{"col":"id","numeric":false,"op":"sort_by","reverse":false},{"col":"id","numeric":false,"op":"sort_by","reverse":true}]} | {"steps":[{"cmp":"ge","col":"score","op":"filter_num_cmp","threshold":50},{"cols":["name","score"],"op":"select_cols"}]} |
| eval_0018_ticket | date_id_irregular | 1 | {"steps":[{"op":"extract_regex","pattern":"sku_like"},{"op":"extract_regex","pattern":"sku_like"},{"op":"extract_regex","pattern":"sku_like"}]} | {"steps":[{"op":"extract_regex","pattern":"ticket"}]} |
| eval_0020_score_select | csv_etl | 2 | {"steps":[{"cols":["name","score"],"op":"select_cols"},{"col":"amount","numeric":false,"op":"sort_by","reverse":false},{"col":"amount","numeric":false,"op":"sort_by","reverse":true}]} | {"steps":[{"cmp":"ge","col":"score","op":"filter_num_cmp","threshold":50},{"cols":["name","score"],"op":"select_cols"}]} |
| eval_0037_ticket | date_id_irregular | 1 | {"steps":[{"op":"extract_regex","pattern":"sku_like"},{"op":"extract_regex","pattern":"sku_like"},{"op":"extract_regex","pattern":"sku_like"}]} | {"steps":[{"op":"extract_regex","pattern":"ticket"}]} |
Training
- Trainable LoRA parameters: 10.6M.
- Training steps: 80.
- Final train loss reported by Trainer: 0.1261.
- Training runtime: 546.2s.
The adapter learned the compact compiler language strongly. Because the constrained scorer evaluates candidate programs directly, the gain is not from better parseability; it is from moving target ABI programs above plausible visible-consistent alternatives.
Interpretation
The useful signal is the depth-2+ result: the adapter recovered the two composition tasks frozen Qwen missed and closed the oracle gap on this generated suite. That supports running a harder compiler pilot on a less-curated transformation benchmark.
The limiting caveat is also clear: the non-model baselines are already strong because candidate enumeration filters by visible examples. First-visible reached 93.8%, and random-visible averaged 90.0%. Future tasks need more adversarial visible-equivalent candidates and a less templated source to distinguish robust compiler skill from an easy candidate set.
Decision
Proceed to the next gate only if the ABI and task set are frozen from an external source before evaluation. The next experiment should keep the constrained scorer, but use a larger less-curated pipeline-transform corpus and report depth-1 operation selection separately from depth-2+ composition.
Experiment log 1
Show the running log (1 entry, 2026-06-27)
2026-06-27
- Created standalone compiler-pilot package.
- Chose constrained candidate scoring to isolate operation/composition choice from JSON syntax failures.
- Planned arms: first-visible candidate baseline, frozen Qwen constrained scorer, QLoRA constrained scorer.
- Planned headline split: depth-1 operation selection vs depth-2/3 composition.
- Built 180 train, 40 validation, and 48 eval records. Eval depth split: 27 depth-1, 15 depth-2, 6 depth-3.
- Oracle constrained candidate coverage: 48/48.
- First-visible baseline: 45/48 filtered execution accuracy.
- Random-visible baseline: 87.5%-93.8% filtered accuracy across five seeds, mean 90.0%.
- Frozen Qwen constrained scorer: 44/48 filtered accuracy, 19/21 on depth-2+.
- QLoRA constrained scorer: 48/48 filtered accuracy, 21/21 on depth-2+.
- Generated final report and charts under
reports/.
Figures 4
Data files 11
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/train_metrics.json295 Breports/eval/base_summary.json1.6 kBreports/eval/first_visible_summary.json1.5 kBreports/eval/lora_summary.json1.4 kBreports/eval/oracle_summary.json1.4 kBreports/eval/random_visible_summary.json1.6 kBreports/eval/random_visible_summary_seed_11.json1.6 kBreports/eval/random_visible_summary_seed_17.json1.6 kBreports/eval/random_visible_summary_seed_23.json1.6 kBreports/eval/random_visible_summary_seed_29.json1.6 kBreports/eval/random_visible_summary_seed_31.json1.6 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).