Qwen Batched Transduction Consistency
The one idea you need
Picture reformatting one column of a spreadsheet. You can hand the model each cell alone, or show it every cell at once, hoping it locks onto a single rule and applies it uniformly. Shared context should breed consistency, but one slip now spoils the whole column.
The question
If a model reformats a batch of similar entries all together instead of one by one, does it get the whole set right more often?
What we found
No. Answering every entry in one combined pass got the whole task right 45 percent of the time, versus 50 percent when each entry was handled alone, a five-point loss. Bundling never rescued a task the solo approach had failed, and it spoiled two tasks that had been fully solved. Asking the model to reason first or format strictly did not close the gap.
Why it matters
To cut calls by packing similar rows into one generation, know the tax: one wrong element fails the whole batch, and shared context can corrupt rows the model would have nailed alone. Keep independent rows independent.
On this page
Results at a glance 3
How to read
Six methods run along the bottom, from one-at-a-time up through ever-larger bundles to all-at-once with order shuffled or a rule hint; taller is better. Dark bars show how often the whole task was fully correct, light bars how often a single entry was correct.
Takeaway → The leftmost one-at-a-time bar is tallest on whole-task correctness, so no bundling method beats handling entries separately, though bigger bundles beat smaller ones.
Data table
| prompting method | full-task exact | row exact |
|---|---|---|
| row-by-row | 50% | 72.1% |
| batch of 2 | 40% | 70.2% |
| batch of 4 | 40% | 71.7% |
| batch all | 45% | 72.5% |
| batch all, shuffled order | 42.5% | 65.8% |
| batch all + rule hint | 45% | 69% |
Numbers from experiments/qwen_batched_transduction_consistency/analysis/iteration_comparison.csv
Technical framing
Batched transduction vs row-by-row prompting (40 text-transformation tasks) — No batched arm beats row-by-row on strict full-task exact; larger batches help but the best (45%) still trails row-by-row (50%).
How to read
Four methods along the bottom, taller is better: handling entries one at a time versus three ways of bundling them with tougher instructions. Dark bars are whole-task correctness; light bars are single-entry correctness.
Takeaway → Every bundled bar sits at or below the one-at-a-time bar on whole-task correctness, so tighter instructions never close the gap.
Data table
| prompting method (v2 iteration) | full-task exact | row exact |
|---|---|---|
| row-by-row | 50% | 72.1% |
| batch all | 45% | 72.5% |
| batch all + verify hint | 45% | 70% |
| batch all + structured JSON input | 42.5% | 67.5% |
Numbers from experiments/qwen_batched_transduction_consistency/analysis/summary.csv
Technical framing
Stricter batch prompts do not recover the gap — A verify hint ties plain batch-all at 45% full-task exact and structured JSON input falls to 42.5%; prompt iteration cannot close the gap.
How to read
Three bars count, out of forty tasks, how many bundling improved, left unchanged, or worsened versus one at a time. A tall wins bar would be good; a tall loses bar is bad.
Takeaway → The wins bar is empty, thirty-eight tasks tie, and two tasks lose, so bundling only ever hurt or made no difference.
Data table
| task outcome | tasks (of 40) |
|---|---|
| batch-all wins | 0 |
| ties | 38 |
| batch-all loses | 2 |
Numbers from report table (reports/qwen_batched_transduction_consistency_report.md, Batch-All Task Flips)
Technical framing
Per-task outcome: batch-all vs row-by-row on full-task exact — Batch-all never flips a failed task into a full-task success and loses two tasks that row-by-row solved: a clean negative result.
In the author’s words from the Report · “Abstract”
This standalone experiment tests whether answering multiple query rows in one shared generation context improves task-level consistency on public text-transformation tasks. The strict primary metric is full-task exact: all held-out rows for a task must be answered exactly.
Overview
Standalone experiment testing whether batched transduction improves task-level consistency on public text-transformation tasks.
Question
Given a few input-output examples and multiple query rows, does answering all queries in one shared generation context improve full-task consistency compared with answering each row independently?
Main Arms
row_by_row: one prompt per held-out row.batch_2: held-out rows answered in batches of two.batch_4: held-out rows answered in batches of four.batch_all: all held-out rows for the task answered in one JSON list.batch_all_shuffled: all held-out rows answered in one JSON list, but query order is deterministically shuffled and then unshuffled for scoring.batch_all_rule_hint: all held-out rows answered together with an instruction to infer one rule internally before emitting the JSON list.
Layout
src/qwen_batched_transduction_consistency.py: runner and report generator.runs/: raw per-run outputs.analysis/: consolidated CSVs and figures.reports/: Markdown and HTML reports.
Report
Rendered from reports/qwen_batched_transduction_consistency_report.md
Abstract
This standalone experiment tests whether answering multiple query rows in one shared generation context improves task-level consistency on public text-transformation tasks. The strict primary metric is full-task exact: all held-out rows for a task must be answered exactly.
Method
- Dataset: public
Transformation.Texttasks. - Split: first
4examples are train examples; up to6held-out examples are scored. - Row-by-row baseline: one prompt per held-out row.
- Batched transduction: one JSON-array output per query batch, with batch sizes 2, 4, and all held-out rows.
- Shuffled-order control: query rows are shuffled inside the batch and then unshuffled for scoring.
- Rule-hint arm: one-batch output with an instruction to use one consistent internal rule.
- Parse failures are counted directly; malformed or wrong-length JSON arrays receive empty predictions for missing rows.
Run Configuration
- Suite:
main_v2_prompt_iteration. - Qwen model:
Qwen/Qwen3-4B. - Tasks:
40. - Held-out cap:
6. - Sample seed:
20260627.
Primary Results
Prompt Iteration Result
| method | tasks | full_task_exact | row_exact | parse_ok_rate |
|---|---|---|---|---|
| batch_all | 40 | 45.0% | 72.5% | 100.0% |
| batch_all_structured | 40 | 42.5% | 67.5% | 100.0% |
| batch_all_verify_hint | 40 | 45.0% | 70.0% | 100.0% |
| row_by_row | 40 | 50.0% | 72.1% | 100.0% |
Iteration Comparison
The first main run swept batch size and basic controls. The second main run kept the same 40-task sample and tested stricter batch prompts.
| iteration | method | tasks | full_task_exact | row_exact | parse_ok_rate |
|---|---|---|---|---|---|
| v1_batch_sweep | row_by_row | 40 | 50.0% | 72.1% | 100.0% |
| v1_batch_sweep | batch_2 | 40 | 40.0% | 70.2% | 99.2% |
| v1_batch_sweep | batch_4 | 40 | 40.0% | 71.7% | 99.2% |
| v1_batch_sweep | batch_all | 40 | 45.0% | 72.5% | 100.0% |
| v1_batch_sweep | batch_all_shuffled | 40 | 42.5% | 65.8% | 100.0% |
| v1_batch_sweep | batch_all_rule_hint | 40 | 45.0% | 69.0% | 100.0% |
| v2_prompt_iteration | row_by_row | 40 | 50.0% | 72.1% | 100.0% |
| v2_prompt_iteration | batch_all | 40 | 45.0% | 72.5% | 100.0% |
| v2_prompt_iteration | batch_all_verify_hint | 40 | 45.0% | 70.0% | 100.0% |
| v2_prompt_iteration | batch_all_structured | 40 | 42.5% | 67.5% | 100.0% |
Batch-All Task Flips
- Batch-all wins over row-by-row on
0tasks. - Batch-all loses to row-by-row on
2tasks. - Batch-all ties row-by-row on
38tasks.
Family Summary
| family | method | tasks | full_task_exact | row_exact |
|---|---|---|---|---|
| Address | batch_all | 2 | 0.0% | 50.0% |
| Address | batch_all_structured | 2 | 0.0% | 50.0% |
| Address | batch_all_verify_hint | 2 | 0.0% | 50.0% |
| Address | row_by_row | 2 | 0.0% | 50.0% |
| BillingCode | batch_all | 1 | 0.0% | 0.0% |
| BillingCode | batch_all_structured | 1 | 0.0% | 0.0% |
| BillingCode | batch_all_verify_hint | 1 | 0.0% | 0.0% |
| BillingCode | row_by_row | 1 | 0.0% | 33.3% |
| City | batch_all | 2 | 50.0% | 87.5% |
| City | batch_all_structured | 2 | 100.0% | 100.0% |
| City | batch_all_verify_hint | 2 | 100.0% | 100.0% |
| City | row_by_row | 2 | 50.0% | 87.5% |
| Column | batch_all | 1 | 100.0% | 100.0% |
| Column | batch_all_structured | 1 | 100.0% | 100.0% |
| Column | batch_all_verify_hint | 1 | 100.0% | 100.0% |
| Column | row_by_row | 1 | 100.0% | 100.0% |
| DateTime | batch_all | 16 | 50.0% | 75.0% |
| DateTime | batch_all_structured | 16 | 37.5% | 69.8% |
| DateTime | batch_all_verify_hint | 16 | 43.8% | 72.9% |
| DateTime | row_by_row | 16 | 56.2% | 70.8% |
| FilePath | batch_all | 1 | 100.0% | 100.0% |
| FilePath | batch_all_structured | 1 | 100.0% | 100.0% |
| FilePath | batch_all_verify_hint | 1 | 100.0% | 100.0% |
| FilePath | row_by_row | 1 | 100.0% | 100.0% |
| Gender | batch_all | 1 | 0.0% | 66.7% |
| Gender | batch_all_structured | 1 | 0.0% | 66.7% |
| Gender | batch_all_verify_hint | 1 | 0.0% | 66.7% |
| Gender | row_by_row | 1 | 0.0% | 66.7% |
| Language | batch_all | 1 | 100.0% | 100.0% |
| Language | batch_all_structured | 1 | 100.0% | 100.0% |
| Language | batch_all_verify_hint | 1 | 100.0% | 100.0% |
| Language | row_by_row | 1 | 100.0% | 100.0% |
| Name | batch_all | 1 | 100.0% | 100.0% |
| Name | batch_all_structured | 1 | 100.0% | 100.0% |
| Name | batch_all_verify_hint | 1 | 100.0% | 100.0% |
| Name | row_by_row | 1 | 100.0% | 100.0% |
| Number | batch_all | 10 | 20.0% | 62.5% |
| Number | batch_all_structured | 10 | 20.0% | 48.3% |
| Number | batch_all_verify_hint | 10 | 20.0% | 53.3% |
| Number | row_by_row | 10 | 30.0% | 64.2% |
| Phone | batch_all | 2 | 100.0% | 100.0% |
| Phone | batch_all_structured | 2 | 100.0% | 100.0% |
| Phone | batch_all_verify_hint | 2 | 100.0% | 100.0% |
| Phone | row_by_row | 2 | 100.0% | 100.0% |
| ShippingCode | batch_all | 1 | 0.0% | 33.3% |
| ShippingCode | batch_all_structured | 1 | 0.0% | 33.3% |
| ShippingCode | batch_all_verify_hint | 1 | 0.0% | 33.3% |
| ShippingCode | row_by_row | 1 | 0.0% | 33.3% |
| UserAgent | batch_all | 1 | 100.0% | 100.0% |
| UserAgent | batch_all_structured | 1 | 100.0% | 100.0% |
| UserAgent | batch_all_verify_hint | 1 | 100.0% | 100.0% |
| UserAgent | row_by_row | 1 | 100.0% | 100.0% |
Task Details
| task_id | family | method | heldout_rows | full_task_exact | row_exact | parse_ok_rate |
|---|---|---|---|---|---|---|
| Address.000002 | Address | batch_all | 3 | False | 33.3% | 100.0% |
| Address.000002 | Address | row_by_row | 3 | False | 33.3% | 100.0% |
| Address.000013 | Address | batch_all | 6 | False | 66.7% | 100.0% |
| Address.000013 | Address | row_by_row | 6 | False | 66.7% | 100.0% |
| BillingCode.000007 | BillingCode | batch_all | 3 | False | 0.0% | 100.0% |
| BillingCode.000007 | BillingCode | row_by_row | 3 | False | 33.3% | 100.0% |
| City.000010 | City | batch_all | 3 | True | 100.0% | 100.0% |
| City.000010 | City | row_by_row | 3 | True | 100.0% | 100.0% |
| City.000011 | City | batch_all | 4 | False | 75.0% | 100.0% |
| City.000011 | City | row_by_row | 4 | False | 75.0% | 100.0% |
| Column.000001 | Column | batch_all | 6 | True | 100.0% | 100.0% |
| Column.000001 | Column | row_by_row | 6 | True | 100.0% | 100.0% |
| DateTime.000004 | DateTime | batch_all | 6 | True | 100.0% | 100.0% |
| DateTime.000004 | DateTime | row_by_row | 6 | True | 100.0% | 100.0% |
| DateTime.000007 | DateTime | batch_all | 6 | True | 100.0% | 100.0% |
| DateTime.000007 | DateTime | row_by_row | 6 | True | 100.0% | 100.0% |
| DateTime.000017 | DateTime | batch_all | 6 | True | 100.0% | 100.0% |
| DateTime.000017 | DateTime | row_by_row | 6 | True | 100.0% | 100.0% |
| DateTime.000025 | DateTime | batch_all | 6 | True | 100.0% | 100.0% |
| DateTime.000025 | DateTime | row_by_row | 6 | True | 100.0% | 100.0% |
| DateTime.000027 | DateTime | batch_all | 6 | False | 66.7% | 100.0% |
| DateTime.000027 | DateTime | row_by_row | 6 | False | 33.3% | 100.0% |
| DateTime.000034 | DateTime | batch_all | 6 | True | 100.0% | 100.0% |
| DateTime.000034 | DateTime | row_by_row | 6 | True | 100.0% | 100.0% |
| DateTime.000051 | DateTime | batch_all | 3 | False | 33.3% | 100.0% |
| DateTime.000051 | DateTime | row_by_row | 3 | False | 33.3% | 100.0% |
| DateTime.000076 | DateTime | batch_all | 6 | False | 66.7% | 100.0% |
| DateTime.000076 | DateTime | row_by_row | 6 | False | 66.7% | 100.0% |
| DateTime.000081 | DateTime | batch_all | 6 | False | 50.0% | 100.0% |
| DateTime.000081 | DateTime | row_by_row | 6 | False | 50.0% | 100.0% |
| DateTime.000094 | DateTime | batch_all | 4 | True | 100.0% | 100.0% |
| DateTime.000094 | DateTime | row_by_row | 4 | True | 100.0% | 100.0% |
| DateTime.000104 | DateTime | batch_all | 6 | True | 100.0% | 100.0% |
| DateTime.000104 | DateTime | row_by_row | 6 | True | 100.0% | 100.0% |
| DateTime.000108 | DateTime | batch_all | 6 | True | 100.0% | 100.0% |
| DateTime.000108 | DateTime | row_by_row | 6 | True | 100.0% | 100.0% |
| DateTime.000111 | DateTime | batch_all | 6 | False | 83.3% | 100.0% |
| DateTime.000111 | DateTime | row_by_row | 6 | True | 100.0% | 100.0% |
| DateTime.000114 | DateTime | batch_all | 6 | False | 50.0% | 100.0% |
| DateTime.000114 | DateTime | row_by_row | 6 | False | 0.0% | 100.0% |
| DateTime.000115 | DateTime | batch_all | 6 | False | 0.0% | 100.0% |
| DateTime.000115 | DateTime | row_by_row | 6 | False | 0.0% | 100.0% |
| DateTime.000116 | DateTime | batch_all | 6 | False | 50.0% | 100.0% |
| DateTime.000116 | DateTime | row_by_row | 6 | False | 50.0% | 100.0% |
| FilePath.000001 | FilePath | batch_all | 6 | True | 100.0% | 100.0% |
| FilePath.000001 | FilePath | row_by_row | 6 | True | 100.0% | 100.0% |
| Gender.000001 | Gender | batch_all | 3 | False | 66.7% | 100.0% |
| Gender.000001 | Gender | row_by_row | 3 | False | 66.7% | 100.0% |
| Language.000002 | Language | batch_all | 6 | True | 100.0% | 100.0% |
| Language.000002 | Language | row_by_row | 6 | True | 100.0% | 100.0% |
| Name.000028 | Name | batch_all | 6 | True | 100.0% | 100.0% |
| Name.000028 | Name | row_by_row | 6 | True | 100.0% | 100.0% |
| Number.000008 | Number | batch_all | 6 | False | 16.7% | 100.0% |
| Number.000008 | Number | row_by_row | 6 | False | 33.3% | 100.0% |
| Number.000015 | Number | batch_all | 6 | False | 50.0% | 100.0% |
| Number.000015 | Number | row_by_row | 6 | False | 33.3% | 100.0% |
| Number.000016 | Number | batch_all | 6 | False | 50.0% | 100.0% |
| Number.000016 | Number | row_by_row | 6 | False | 83.3% | 100.0% |
| Number.000022 | Number | batch_all | 6 | False | 33.3% | 100.0% |
| Number.000022 | Number | row_by_row | 6 | True | 100.0% | 100.0% |
| Number.000028 | Number | batch_all | 3 | True | 100.0% | 100.0% |
| Number.000028 | Number | row_by_row | 3 | True | 100.0% | 100.0% |
| Number.000029 | Number | batch_all | 3 | False | 66.7% | 100.0% |
| Number.000029 | Number | row_by_row | 3 | False | 66.7% | 100.0% |
| Number.000043 | Number | batch_all | 6 | True | 100.0% | 100.0% |
| Number.000043 | Number | row_by_row | 6 | True | 100.0% | 100.0% |
| Number.000049 | Number | batch_all | 4 | False | 75.0% | 100.0% |
| Number.000049 | Number | row_by_row | 4 | False | 25.0% | 100.0% |
| Number.000075 | Number | batch_all | 6 | False | 66.7% | 100.0% |
| Number.000075 | Number | row_by_row | 6 | False | 66.7% | 100.0% |
| Number.000077 | Number | batch_all | 3 | False | 66.7% | 100.0% |
| Number.000077 | Number | row_by_row | 3 | False | 33.3% | 100.0% |
| Phone.000008 | Phone | batch_all | 6 | True | 100.0% | 100.0% |
| Phone.000008 | Phone | row_by_row | 6 | True | 100.0% | 100.0% |
| Phone.000011 | Phone | batch_all | 3 | True | 100.0% | 100.0% |
| Phone.000011 | Phone | row_by_row | 3 | True | 100.0% | 100.0% |
| ShippingCode.000008 | ShippingCode | batch_all | 3 | False | 33.3% | 100.0% |
| ShippingCode.000008 | ShippingCode | row_by_row | 3 | False | 33.3% | 100.0% |
| UserAgent.000003 | UserAgent | batch_all | 6 | True | 100.0% | 100.0% |
| UserAgent.000003 | UserAgent | row_by_row | 6 | True | 100.0% | 100.0% |
Figures
Interpretation
Row-by-row inference reaches 72.1% row exact and 50.0% full-task exact. Batch-all reaches 72.5% row exact and 45.0% full-task exact, a full-task delta of -5.0% and a row-exact delta of 0.4%. The result is a clean negative for batched transduction as a general consistency fix in this setup. Batch-all slightly improves average row exact, but it never flips a row-by-row-failed task into a full-task success on the 40-task sample, and it loses two tasks that row-by-row solved. The batch-size sweep shows larger batches are better than smaller batches, but the best batched arm still trails row-by-row on full-task exact. The stricter prompt iteration does not recover the gap: verify-hint batch-all ties plain batch-all at 45.0%, while structured JSON input falls to 42.5%. The shuffled-order control is lower than normal batch-all, so ordering and shared context carry some signal. The signal is not enough to overcome the new batch failure mode: one wrong element in the JSON array spoils full-task exact, and batch prompting sometimes changes correct row-by-row outputs into wrong batched outputs.
Limitations
This run uses deterministic decoding and a capped task sample. Batched JSON output is stricter than ordinary text output, so parse cleanliness is reported separately. Full-task exact is intentionally harsh; a method can have high row exact while failing full-task exact because one row is wrong.
Artifacts
- Task-level details:
analysis/task_details.csv - Row-level details:
analysis/row_details.csv - Summary:
analysis/summary.csv - Iteration comparison:
analysis/iteration_comparison.csv - Figures:
analysis/figures/ - Benchmark mirror:
/workspace/large_artifacts/qwen_batched_transduction_consistency/prose-benchmarks
Experiment log 4
Show the running log (4 entries)
Setup
- Created a fresh standalone experiment directory.
- Copied the public transformation benchmark mirror into
/workspace/large_artifacts/qwen_batched_transduction_consistency/prose-benchmarks. - Primary metric: full-task exact on held-out rows. A task counts only if every held-out row is exact.
- Secondary metrics: row exact, parse success, and parsed item count.
Run smoke_v1
- Time UTC:
2026-06-27T06:16:05.586794+00:00 - Elapsed seconds:
12.4 - Config:
{"batch_max_new_tokens": 160, "heldout_cap": 3, "methods": "row_by_row,batch_all", "min_heldout": 3, "qwen_task_limit": 4, "row_max_new_tokens": 48, "sample_seed": 20260627, "suite": "smoke_v1", "task_limit": 0, "train_n": 4} batch_all: row exact 91.7%; full-task exact 75.0%; parse ok 100.0%row_by_row: row exact 83.3%; full-task exact 75.0%; parse ok 100.0%
Run main_v1
- Time UTC:
2026-06-27T06:25:27.762033+00:00 - Elapsed seconds:
530.3 - Config:
{"batch_max_new_tokens": 320, "heldout_cap": 6, "methods": "", "min_heldout": 3, "qwen_task_limit": 40, "row_max_new_tokens": 64, "sample_seed": 20260627, "suite": "main_v1", "task_limit": 0, "train_n": 4} batch_2: row exact 70.2%; full-task exact 40.0%; parse ok 99.2%batch_4: row exact 71.7%; full-task exact 40.0%; parse ok 99.2%batch_all: row exact 72.5%; full-task exact 45.0%; parse ok 100.0%batch_all_rule_hint: row exact 69.0%; full-task exact 45.0%; parse ok 100.0%batch_all_shuffled: row exact 65.8%; full-task exact 42.5%; parse ok 100.0%row_by_row: row exact 72.1%; full-task exact 50.0%; parse ok 100.0%
Run main_v2_prompt_iteration
- Time UTC:
2026-06-27T06:32:43.777202+00:00 - Elapsed seconds:
357.6 - Config:
{"batch_max_new_tokens": 360, "heldout_cap": 6, "methods": "row_by_row,batch_all,batch_all_verify_hint,batch_all_structured", "min_heldout": 3, "qwen_task_limit": 40, "row_max_new_tokens": 64, "sample_seed": 20260627, "suite": "main_v2_prompt_iteration", "task_limit": 0, "train_n": 4} batch_all: row exact 72.5%; full-task exact 45.0%; parse ok 100.0%batch_all_structured: row exact 67.5%; full-task exact 42.5%; parse ok 100.0%batch_all_verify_hint: row exact 70.0%; full-task exact 45.0%; parse ok 100.0%row_by_row: row exact 72.1%; full-task exact 50.0%; parse ok 100.0%
Figures 5
Data files 4
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/summary.csv200 Bruns/main_v1/summary.csv330 Bruns/main_v2_prompt_iteration/summary.csv200 Bruns/smoke_v1/summary.csv134 B
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.