Qwen Constrained ABI Parser
The one idea you need
Picture a small AI turning a plain-English request into a short recipe of steps a computer then runs. As it writes, a guardrail blocks any half-finished step that would break the recipe's grammar, so every finished recipe comes out at least well-formed.
The question
When a small model turns a plain-English request into a runnable recipe of steps, does forcing each step to follow strict grammar rules make the result correct more often?
What we found
Yes, mostly. On the hardest six-step requests, blocking any invalid step as the model writes lifted correctly-running recipes from 60% to 75%, and it won on all five training runs. It also beat merely re-rolling until valid, which reached only about 63%, and writing no steps at all, which got 0%. The catch: the guardrail also makes some wrong recipes look well-formed.
Why it matters
When your small model writes machine-runnable steps, block anything that breaks your format as it writes, rather than re-rolling failed outputs. The guardrail rescues cases re-rolling misses, and its edge grows when requests are worded unfamiliarly.
On this page
Results at a glance 4
How to read
Each bar is a different way to turn the request into steps; bar height is the share of runs whose steps produced the right final answer. Taller is better. The rightmost bar is the perfect-input ceiling.
Takeaway → The guardrail bar clears free writing and cheap re-rolling; writing no steps scores nothing, and only the perfect-input ceiling stands higher.
Data table
| arm | execution accuracy |
|---|---|
| direct answer | 0% |
| free greedy | 60% |
| resample-to-valid | 62.5% |
| parse-then-emit (constrained) | 60% |
| constrained (FSM) | 75% |
| oracle parse (ceiling) | 100% |
Numbers from report table (reports/qwen_constrained_abi_parser_report.md); matches analysis/summary_by_arm.csv
Technical framing
Depth-6 execution accuracy: constrained decoding beats free decoding and cheap resampling — At held-out depth 6, finite-state constrained decoding reaches 75% execution vs 60% free — resampling to validity only gets 62.5%.
How to read
Left to right is how many steps the recipe needs, so further right is harder; height is the share of recipes running to the right answer, higher is better. Each line is one way of writing the steps.
Takeaway → The guardrail line sits on top at every length, staying above free writing and cheap re-rolling as recipes grow longer.
Data table
| procedure depth | constrained (FSM) | free greedy | resample-to-valid | parse-then-emit (constrained) |
|---|---|---|---|---|
| 2 | 92.5% | 75% | 75% | 77.5% |
| 3 | 82.5% | 72.5% | 72.5% | 75% |
| 4 | 80% | 60% | 67.5% | 55% |
| 6 | 75% | 60% | 62.5% | 60% |
Numbers from report table (reports/qwen_constrained_abi_parser_report.md); matches analysis/summary_by_arm.csv
Technical framing
Execution accuracy vs procedure depth (held-out compositions) — The constrained decoder leads free decoding by 10-20 points at every depth; resampling tracks free decoding closely.
How to read
Left to right is recipe length; height is the share running to the right answer, higher is better. Lines show each writing method when the request uses unfamiliar phrasing.
Takeaway → The guardrail line stays high near 75% while free writing drops to about 48% and the parse-first method collapses far below it.
Data table
| procedure depth | constrained (FSM) | free greedy | parse-then-emit (constrained) |
|---|---|---|---|
| 2 | 82.5% | 55% | 37.5% |
| 4 | 77.5% | 47.5% | 22.5% |
| 6 | 75% | 47.5% | 27.5% |
Numbers from report table (reports/qwen_constrained_abi_parser_report.md); matches analysis/summary_by_arm.csv
Technical framing
Wording-shifted prompts: constrained decoding is robust, free decoding degrades — Under template shift the constrained decoder holds 75-83% while free decoding falls to ~48% and the parse stage collapses.
How to read
Three measures: how often the recipe is well-formed, how often it runs to the right answer, and how often well-formed recipes are actually right. Two bars each, free writing versus guardrail. Higher is better.
Takeaway → The guardrail makes every recipe well-formed and lifts right-answers to 75%, but that last measure also dips to 75%, so some forced-valid recipes are still wrong.
Data table
| metric | free greedy | constrained (FSM) |
|---|---|---|
| valid-program rate | 60% | 100% |
| execution accuracy | 60% | 75% |
| correct given valid | 100% | 75% |
Numbers from report table (reports/qwen_constrained_abi_parser_report.md); matches analysis/summary_by_arm.csv
Technical framing
Depth 6: validity vs execution vs correct-given-valid — The grammar forces 100% validity and lifts execution to 75%, but some forced-valid programs are wrong (correct-given-valid drops to 75%).
In the author’s words from the Report · “Abstract”
This standalone experiment tests whether a finite-state stack-ABI decoder and a canonical parse stage make a small model a more reliable compiler from natural language into executable procedures. The headline metric is external execution accuracy, not valid-program rate.
Overview
This standalone experiment tests whether a finite-state stack-ABI decoder and a canonical parse stage improve a local 4B model's reliability as a compiler from natural language into executable procedures.
The headline metric is external execution accuracy. Valid-program rate is tracked, but validity alone is not a success criterion.
Large adapter checkpoints are stored outside this experiment directory:
/workspace/large_artifacts/qwen_constrained_abi_parserReport
Rendered from reports/qwen_constrained_abi_parser_report.md
Abstract
This standalone experiment tests whether a finite-state stack-ABI decoder and a canonical parse stage make a small model a more reliable compiler from natural language into executable procedures. The headline metric is external execution accuracy, not valid-program rate.
Method
Training examples contain one primitive operation. Evaluation uses held-out procedure depths 2, 3, 4, and 6, plus wording-shifted prompts at depths 2, 4, and 6. The task families are string, number, table, date, list, and path transformations.
Three targets are trained: direct final answers, raw stack programs, and parse-plus-program outputs. The raw stack adapter is evaluated with free greedy decoding, finite-state constrained decoding, and a resample-to-valid baseline. The parse adapter is evaluated both by executing its free program section and by deterministically emitting a stack program from its parse block. Oracle parse and gold ABI sanity arms bound the decoder and interpreter.
A valid-rate increase alone is pre-registered as insufficient. A useful constraint must improve execution accuracy while preserving correct-given-valid accuracy; otherwise the grammar merely forces wrong programs to become well formed.
Run Configuration
- Primary suite:
main. - Seeds:
101,202,303,404,505. - Evaluation rows:
320metric rows,2560scored examples across arms. - QLoRA update steps per adapter:
24. - Large adapters are stored outside the experiment tree.
Primary Results
- Depth-6 standard execution: free raw stack 60.0%; constrained raw stack 75.0%; constraint delta 15.0%.
- Depth-6 valid-rate/correct-given-valid: free valid 60.0%, cgv 100.0%; constrained valid 100.0%, cgv 75.0%.
- Depth-6 constrained raw stack beats free decoding on
5/5seeds; mean per-seed delta 15.0%. - Depth-6 divergence: constrained-only correct 15.0%; free-only correct 0.0%.
- Depth-6 resample-to-valid execution: 62.5%; delta versus free 2.5%; mean attempts 1.40.
- Depth-6 parse-then-emit execution: 60.0%; parse exactness 57.5%.
- Depth-6 direct-answer baseline: 0.0%.
- Template-shift depth-6 constrained execution: 75.0%; drop from standard constrained depth-6 0.0%.
- Template-shift depth-6 free execution: 47.5%; drop from standard free depth-6 12.5%.
- Template-shift depth-6 constraint delta over free: 27.5%.
- Oracle parse and gold ABI depth-6 sanity: oracle parse 100.0%; gold ABI 100.0%.
| arm | split | depth | runs | n_total | exec_accuracy_mean | exec_accuracy_std | valid_exec_rate_mean | correct_given_valid_mean | parse_exact_rate_mean | divergence_rate_mean | constrained_only_rate_mean | free_only_rate_mean | mean_attempts_mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| answer_only | eval_comp_d2 | 2 | 5 | 40 | 0.0% | 0.0% | 0.0% | n/a | 0.0% | n/a | n/a | n/a | 1.00 |
| gold_abi_constrained | eval_comp_d2 | 2 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | n/a | n/a | n/a | 0.00 |
| oracle_parse_constrained | eval_comp_d2 | 2 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| parse_then_emit_constrained | eval_comp_d2 | 2 | 5 | 40 | 77.5% | 13.7% | 82.5% | 93.8% | 77.5% | 7.5% | 2.5% | 2.5% | 1.00 |
| parse_then_emit_free | eval_comp_d2 | 2 | 5 | 40 | 77.5% | 10.5% | 80.0% | 96.7% | 77.5% | n/a | n/a | n/a | 1.00 |
| program_stack_constrained | eval_comp_d2 | 2 | 5 | 40 | 92.5% | 11.2% | 100.0% | 92.5% | 0.0% | 25.0% | 17.5% | 0.0% | 1.00 |
| program_stack_free | eval_comp_d2 | 2 | 5 | 40 | 75.0% | 12.5% | 75.0% | 100.0% | 0.0% | n/a | n/a | n/a | 1.00 |
| program_stack_resample_valid | eval_comp_d2 | 2 | 5 | 40 | 75.0% | 12.5% | 75.0% | 100.0% | 0.0% | 0.0% | 0.0% | 0.0% | 1.25 |
| answer_only | eval_comp_d3 | 3 | 5 | 40 | 0.0% | 0.0% | 0.0% | n/a | 0.0% | n/a | n/a | n/a | 1.00 |
| gold_abi_constrained | eval_comp_d3 | 3 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | n/a | n/a | n/a | 0.00 |
| oracle_parse_constrained | eval_comp_d3 | 3 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| parse_then_emit_constrained | eval_comp_d3 | 3 | 5 | 40 | 75.0% | 12.5% | 80.0% | 94.2% | 75.0% | 2.5% | 0.0% | 0.0% | 1.00 |
| parse_then_emit_free | eval_comp_d3 | 3 | 5 | 40 | 75.0% | 12.5% | 80.0% | 94.2% | 75.0% | n/a | n/a | n/a | 1.00 |
| program_stack_constrained | eval_comp_d3 | 3 | 5 | 40 | 82.5% | 6.8% | 100.0% | 82.5% | 0.0% | 27.5% | 10.0% | 0.0% | 1.00 |
| program_stack_free | eval_comp_d3 | 3 | 5 | 40 | 72.5% | 16.3% | 77.5% | 93.1% | 0.0% | n/a | n/a | n/a | 1.00 |
| program_stack_resample_valid | eval_comp_d3 | 3 | 5 | 40 | 72.5% | 16.3% | 77.5% | 93.1% | 0.0% | 0.0% | 0.0% | 0.0% | 1.23 |
| answer_only | eval_comp_d4 | 4 | 5 | 40 | 0.0% | 0.0% | 0.0% | n/a | 0.0% | n/a | n/a | n/a | 1.00 |
| gold_abi_constrained | eval_comp_d4 | 4 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | n/a | n/a | n/a | 0.00 |
| oracle_parse_constrained | eval_comp_d4 | 4 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| parse_then_emit_constrained | eval_comp_d4 | 4 | 5 | 40 | 55.0% | 6.8% | 62.5% | 88.7% | 55.0% | 0.0% | 0.0% | 0.0% | 1.00 |
| parse_then_emit_free | eval_comp_d4 | 4 | 5 | 40 | 55.0% | 6.8% | 62.5% | 88.7% | 55.0% | n/a | n/a | n/a | 1.00 |
| program_stack_constrained | eval_comp_d4 | 4 | 5 | 40 | 80.0% | 6.8% | 100.0% | 80.0% | 0.0% | 40.0% | 20.0% | 0.0% | 1.00 |
| program_stack_free | eval_comp_d4 | 4 | 5 | 40 | 60.0% | 13.7% | 60.0% | 100.0% | 0.0% | n/a | n/a | n/a | 1.00 |
| program_stack_resample_valid | eval_comp_d4 | 4 | 5 | 40 | 67.5% | 11.2% | 67.5% | 100.0% | 0.0% | 10.0% | 7.5% | 0.0% | 1.40 |
| answer_only | eval_comp_d6 | 6 | 5 | 40 | 0.0% | 0.0% | 0.0% | n/a | 0.0% | n/a | n/a | n/a | 1.00 |
| gold_abi_constrained | eval_comp_d6 | 6 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | n/a | n/a | n/a | 0.00 |
| oracle_parse_constrained | eval_comp_d6 | 6 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| parse_then_emit_constrained | eval_comp_d6 | 6 | 5 | 40 | 60.0% | 10.5% | 60.0% | 100.0% | 57.5% | 27.5% | 17.5% | 0.0% | 1.00 |
| parse_then_emit_free | eval_comp_d6 | 6 | 5 | 40 | 42.5% | 6.8% | 45.0% | 96.0% | 57.5% | n/a | n/a | n/a | 1.00 |
| program_stack_constrained | eval_comp_d6 | 6 | 5 | 40 | 75.0% | 8.8% | 100.0% | 75.0% | 0.0% | 40.0% | 15.0% | 0.0% | 1.00 |
| program_stack_free | eval_comp_d6 | 6 | 5 | 40 | 60.0% | 10.5% | 60.0% | 100.0% | 0.0% | n/a | n/a | n/a | 1.00 |
| program_stack_resample_valid | eval_comp_d6 | 6 | 5 | 40 | 62.5% | 12.5% | 62.5% | 100.0% | 0.0% | 7.5% | 2.5% | 0.0% | 1.40 |
| answer_only | eval_indist_d1 | 1 | 5 | 40 | 0.0% | 0.0% | 0.0% | n/a | 0.0% | n/a | n/a | n/a | 1.00 |
| gold_abi_constrained | eval_indist_d1 | 1 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | n/a | n/a | n/a | 0.00 |
| oracle_parse_constrained | eval_indist_d1 | 1 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| parse_then_emit_constrained | eval_indist_d1 | 1 | 5 | 40 | 95.0% | 6.8% | 97.5% | 97.5% | 95.0% | 7.5% | 2.5% | 2.5% | 1.00 |
| parse_then_emit_free | eval_indist_d1 | 1 | 5 | 40 | 95.0% | 6.8% | 97.5% | 97.5% | 95.0% | n/a | n/a | n/a | 1.00 |
| program_stack_constrained | eval_indist_d1 | 1 | 5 | 40 | 97.5% | 5.6% | 100.0% | 97.5% | 0.0% | 2.5% | 2.5% | 0.0% | 1.00 |
| program_stack_free | eval_indist_d1 | 1 | 5 | 40 | 95.0% | 11.2% | 97.5% | 97.1% | 0.0% | n/a | n/a | n/a | 1.00 |
| program_stack_resample_valid | eval_indist_d1 | 1 | 5 | 40 | 92.5% | 11.2% | 95.0% | 97.1% | 0.0% | 2.5% | 0.0% | 2.5% | 1.05 |
| answer_only | eval_template_d2 | 2 | 5 | 40 | 0.0% | 0.0% | 0.0% | n/a | 0.0% | n/a | n/a | n/a | 1.00 |
| gold_abi_constrained | eval_template_d2 | 2 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | n/a | n/a | n/a | 0.00 |
| oracle_parse_constrained | eval_template_d2 | 2 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| parse_then_emit_constrained | eval_template_d2 | 2 | 5 | 40 | 37.5% | 0.0% | 42.5% | 92.0% | 32.5% | 7.5% | 0.0% | 0.0% | 1.00 |
| parse_then_emit_free | eval_template_d2 | 2 | 5 | 40 | 37.5% | 0.0% | 37.5% | 100.0% | 32.5% | n/a | n/a | n/a | 1.00 |
| program_stack_constrained | eval_template_d2 | 2 | 5 | 40 | 82.5% | 14.3% | 100.0% | 82.5% | 0.0% | 52.5% | 30.0% | 2.5% | 1.00 |
| program_stack_free | eval_template_d2 | 2 | 5 | 40 | 55.0% | 6.8% | 57.5% | 96.0% | 0.0% | n/a | n/a | n/a | 1.00 |
| program_stack_resample_valid | eval_template_d2 | 2 | 5 | 40 | 57.5% | 14.3% | 62.5% | 91.0% | 0.0% | 12.5% | 5.0% | 2.5% | 1.43 |
| answer_only | eval_template_d4 | 4 | 5 | 40 | 0.0% | 0.0% | 0.0% | n/a | 0.0% | n/a | n/a | n/a | 1.00 |
| gold_abi_constrained | eval_template_d4 | 4 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | n/a | n/a | n/a | 0.00 |
| oracle_parse_constrained | eval_template_d4 | 4 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| parse_then_emit_constrained | eval_template_d4 | 4 | 5 | 40 | 22.5% | 10.5% | 25.0% | 93.3% | 22.5% | 5.0% | 0.0% | 0.0% | 1.00 |
| parse_then_emit_free | eval_template_d4 | 4 | 5 | 40 | 22.5% | 10.5% | 22.5% | 100.0% | 22.5% | n/a | n/a | n/a | 1.00 |
| program_stack_constrained | eval_template_d4 | 4 | 5 | 40 | 77.5% | 10.5% | 100.0% | 77.5% | 0.0% | 57.5% | 35.0% | 5.0% | 1.00 |
| program_stack_free | eval_template_d4 | 4 | 5 | 40 | 47.5% | 13.7% | 47.5% | 100.0% | 0.0% | n/a | n/a | n/a | 1.00 |
| program_stack_resample_valid | eval_template_d4 | 4 | 5 | 40 | 47.5% | 13.7% | 50.0% | 95.0% | 0.0% | 2.5% | 0.0% | 0.0% | 1.50 |
| answer_only | eval_template_d6 | 6 | 5 | 40 | 0.0% | 0.0% | 0.0% | n/a | 0.0% | n/a | n/a | n/a | 1.00 |
| gold_abi_constrained | eval_template_d6 | 6 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | n/a | n/a | n/a | 0.00 |
| oracle_parse_constrained | eval_template_d6 | 6 | 5 | 40 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | n/a | n/a | n/a | 0.00 |
| parse_then_emit_constrained | eval_template_d6 | 6 | 5 | 40 | 27.5% | 18.5% | 27.5% | 100.0% | 25.0% | 22.5% | 5.0% | 0.0% | 1.00 |
| parse_then_emit_free | eval_template_d6 | 6 | 5 | 40 | 22.5% | 13.7% | 25.0% | 91.7% | 25.0% | n/a | n/a | n/a | 1.00 |
| program_stack_constrained | eval_template_d6 | 6 | 5 | 40 | 75.0% | 12.5% | 100.0% | 75.0% | 0.0% | 50.0% | 30.0% | 2.5% | 1.00 |
| program_stack_free | eval_template_d6 | 6 | 5 | 40 | 47.5% | 13.7% | 55.0% | 86.7% | 0.0% | n/a | n/a | n/a | 1.00 |
| program_stack_resample_valid | eval_template_d6 | 6 | 5 | 40 | 47.5% | 13.7% | 52.5% | 90.0% | 0.0% | 10.0% | 0.0% | 0.0% | 1.48 |
Interpretation
The experiment separates two possible bottlenecks. If constrained decoding increases validity and execution together, malformed syntax was suppressing an otherwise useful compiler. If validity rises while execution stays flat or correct-given-valid falls, malformed syntax was mainly a symptom of unresolved semantic uncertainty. The divergence diagnostics show whether the grammar rescues examples free decoding missed or overrides examples free decoding already had right. At depth 6, constrained decoding changes execution by 15.0% relative to free raw-stack decoding. The valid-rate change is 40.0%, and the correct-given-valid change is -25.0%. The resample-to-valid baseline is the cheap alternative. At depth 6 it is -12.5% versus the constrained decoder, so this comparison determines whether a full grammar adds value beyond simply rejecting invalid samples. The parse stage helps standard depth-6 execution relative to its own free program section: 60.0% versus 42.5%. But the parse stage does not solve wording shift in this form: template depth-6 parse-then-emit is 27.5%, far below constrained raw stack at 75.0%. For constrained raw-stack decoding on composition splits, generated procedures break down as: correct_exact 76.2%, wrong_op_order_or_choice 16.9%, correct_semantic_variant 6.2%, wrong_constant_or_argument 0.6%.
Limitations
This experiment tests robustness of compilation over a fixed known primitive library. It does not test invention of operations outside the ABI. The finite-state grammar is tied to the synthetic task schema and uses task-visible constants and type information.
Artifacts
- Metrics:
analysis/summary_by_arm.csvandanalysis/all_metrics.csv - Details:
analysis/all_details.csv - Training logs:
analysis/all_train_logs.csv - Checkpoints:
/workspace/large_artifacts/qwen_constrained_abi_parser/checkpoints
Experiment log 4
Show the running log (4 entries)
Setup
- Created: 2026-06-26 UTC
- Fresh experiment directory:
experiments/qwen_constrained_abi_parser - Large artifact directory:
large_artifacts/qwen_constrained_abi_parser - Base model:
Qwen/Qwen3-4B - Primary question: can a grammar-constrained stack ABI decoder and a canonical parse stage improve executable procedure accuracy, not merely valid-program rate?
- Primary readouts: execution accuracy by composition depth, template-shift execution, validity versus execution, correct-given-valid, parse exactness, decoder divergence, and failure taxonomy.
Run smoke_v1
- Started: 2026-06-26 15:43:38 UTC
- Suite:
smoke - Model:
Qwen/Qwen3-4B - Seeds:
101 - Training targets:
answer_only,program_stack,parse_emit - Training examples per seed:
18 - Eval examples per split:
3 - Steps:
1 - Resample attempts:
2
Completed smoke_v1 in 704.5s.
- Metric rows: 64
- Detail rows: 192
- Training log rows: 3
Run pilot_v1
- Started: 2026-06-26 15:56:19 UTC
- Suite:
pilot - Model:
Qwen/Qwen3-4B - Seeds:
101 - Training targets:
answer_only,program_stack,parse_emit - Training examples per seed:
120 - Eval examples per split:
6 - Steps:
12 - Resample attempts:
2
Completed pilot_v1 in 714.1s.
- Metric rows: 64
- Detail rows: 384
- Training log rows: 15
Run main_v1
- Started: 2026-06-26 16:08:48 UTC
- Suite:
main - Model:
Qwen/Qwen3-4B - Seeds:
101,202,303,404,505 - Training targets:
answer_only,program_stack,parse_emit - Training examples per seed:
180 - Eval examples per split:
8 - Steps:
24 - Resample attempts:
2
Completed main_v1 in 4058.7s.
- Metric rows: 320
- Detail rows: 2560
- Training log rows: 75
Figures 7
Data files 5
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv90 kBanalysis/summary_by_arm.csv58 kBruns/main_v1/metrics.csv65 kBruns/pilot_v1/metrics.csv18 kBruns/smoke_v1/metrics.csv14 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).