Pairwise Table Judge

Finished2026-06-27imported · line YActive Evidence Acquisition Benchmark Generalization Evidence-Conditioned Selection Interpretability and Diagnostics+1 moreGitHub ↗

The one idea you need

Picture asking a small model to referee two filled-in spreadsheets and crown the more correct one. On the genuinely hard match-ups it stops reading the cells and behaves like a rigged coin that always lands on the top-listed contestant.

The question

When a small model produces several versions of a reformatted data table, can it reliably compare them side by side and pick the one that is actually more correct?

What we found

No. Letting the model run a knockout comparison among its own candidate tables did not help: fully-correct tables slipped from 50% (just trusting its first attempt) to 47.5%. On the hard cases where a correct table existed, it picked the right one 100% of the time when that table was shown first and 0% when shown second. It judged position, not content.

Why it matters

If you let a small model choose among its own outputs, never trust a raw side-by-side vote. Randomize candidate order and require the pick to survive a swap; otherwise you are selecting by slot, not by quality.

Fully correct tables: first attempt vs judged pick50% → 47.5%the comparison step slightly hurt instead of helping

Which table it picked on hard cases, by position100% vs 0%chose the correct table when shown first, never when shown second

Ceiling if it always kept the best candidate62.5%the best table already sitting in the pool, versus 47.5% achieved

Misleadingly rosy overall pick rate90%inflated by easy tasks where both tables were already identical

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 7

How often each method produced a fully correct table

How to read

Grouped bars for four methods: the model's first table, its judged pick between candidates, the best table already in its pool, and a version fed the correct rows. Each shows whole-table and single-row correctness; taller is better.

Takeaway → The judged-pick bars sit just below the first-try bars, so the comparison step slightly lowered accuracy, while the pool's best towers above both.

Data table

Method	full-task exact	row exact
direct greedy	50%	72.3%
pairwise tournament	47.5%	71.9%
table oracle	62.5%	82.5%
row repair (oracle diagnostic)	80%	91.3%

Numbers from experiments/qwen_pairwise_table_judge/analysis/summary.csv

Technical framing

Full-task exact accuracy by selection method — The deployable pairwise tournament loses 2.5 points vs direct greedy; the 12.5-point oracle gap is unrealized selection headroom.

The model just picks whichever table is shown first

How to read

Two bars for the hard tables where the first try was wrong: left, the correct table is placed second; right, placed first. Height is how often it picked the correct table; higher is better.

Takeaway → The rate jumps from none to all when the correct table moves to first position, meaning the model followed order, not content.

Data table

Candidate order (normal mode, tasks where greedy is wrong)	judge picks the correct table
correct table is candidate B (base)	0%
correct table is candidate A (swapped)	100%

Numbers from experiments/qwen_pairwise_table_judge/analysis/diagnostic_headroom_summary.csv

Technical framing

On headroom tasks the judge just picks candidate A — 0% vs 100% flip with candidate order: pure position bias, not semantic table judging, on the tasks that actually matter.

Overall pick rate looks fine but is misleading

How to read

Four bars, one per prompt style, showing how often the model picked the correct table across all comparisons; higher looks better.

Takeaway → All bars sit near 80-90%, but that height comes from easy tables where both choices were identical, hiding the failure on hard ones.

Data table

Judge prompt mode	judge picks hidden-correct table
normal	90%
no examples	80%
shuffled examples	80%
row-shuffled candidate	80%

Numbers from experiments/qwen_pairwise_table_judge/analysis/diagnostic_summary.csv

Technical framing

Aggregate direct-vs-correct diagnostic looks fine, misleadingly — 80-90% aggregate pick rates are inflated by saturated tasks where both candidates are identical; the headroom subset reveals the failure.

Per-row accuracy versus whole-table accuracy by method

How to read

For each of the four methods, one bar for how many individual rows were right and one for how many whole tables were fully right; higher is better.

Takeaway → Every method's whole-table bar drops well below its per-row bar, showing one wrong row is enough to sink an otherwise-good table.

Data table

condition	Row exact	Full-task exact
Row repair (diagnostic)	91.3%	80%
Table oracle	82.5%	62.5%
Direct greedy	72.3%	50%
Pairwise tournament	71.9%	47.5%

Numbers from experiments/qwen_pairwise_table_judge/reports/qwen_pairwise_table_judge_report.md (Main Result table)

Technical framing

Row-level vs full-table exact accuracy by method — Every method loses 11-24 points going from per-row accuracy to whole-table exactness; one bad row sinks the table.

On fixable tasks a better table existed but went unpicked

How to read

Ten hard tasks along the bottom; for each, a bar for the first-try row accuracy and a bar for the best-available table's row accuracy; higher is better.

Takeaway → The best-available bar towers over the first-try bar on all ten, several reaching a perfect score, yet the comparison step never chose them.

Data table

Tasks where the oracle-selected table beats direct greedy	Direct greedy	Table oracle
DateTime.000027	50%	83.3%
DateTime.000076	66.7%	100%
DateTime.000081	50%	66.7%
DateTime.000114	16.7%	66.7%
Number.000008	33.3%	50%
Number.000015	33.3%	100%
Number.000016	83.3%	100%
Number.000049	0%	75%
Number.000075	66.7%	100%
Number.000077	33.3%	100%

Numbers from experiments/qwen_pairwise_table_judge/reports/qwen_pairwise_table_judge_report.md (Selected Tables table)

Technical framing

Where the missed headroom lives: per-task row accuracy, greedy vs oracle pick — All 10 recoverable tasks are DateTime or Number; a better pick exists in the pool (six reach 100%) but the judge never selects it.

A small trial hid the result the full run revealed

How to read

Four methods, each with a bar from a small six-task trial and a bar from the full forty-task run, measuring how often a fully correct table appeared; higher is better.

Takeaway → In the small trial the comparison step tied the first try, but the full run shows it falling below the first try.

Data table

condition	Pilot, 6 tasks	Main, 40 tasks
Table oracle	100%	62.5%
Direct greedy	66.7%	50%
Pairwise tournament	66.7%	47.5%
Row repair (diagnostic)	16.7%	80%

Numbers from experiments/qwen_pairwise_table_judge/experiment_log.md (Pilot and Main Run result lists)

Technical framing

Pilot (6 tasks) vs main run (40 tasks): full-task exact by method — The saturated 6-task pilot hid the story: at 40 tasks the tournament falls below greedy while row repair jumps from 16.7% to 80%.

The single table the comparison step changed got worse

How to read

One task the step altered; a bar for the first-try table's row accuracy and a bar for the swapped-in table's row accuracy; higher is better.

Takeaway → The swapped table's bar is lower than the original, turning a perfect table into an imperfect one; the only swap it made hurt.

Data table

condition	Direct greedy pick	Tournament pick
DateTime.000111 (1 of 40 tasks changed)	100%	83.3%

Numbers from experiments/qwen_pairwise_table_judge/reports/qwen_pairwise_table_judge_report.md (Tournament Changes table)

Technical framing

The only table the tournament changed got worse — Across 40 tasks the tournament swapped exactly one table, turning a perfect 100% task into 83.3% and losing full-task exactness.

In the author’s words from the Overview · “Summary”

Can a model choose the more task-consistent full output table when shown examples, query rows, and two candidate tables?

Overview

This top-level README was generated during repository normalization because the imported experiment did not include one.

Source track: track-y
Primary report: reports/qwen_pairwise_table_judge_report.md
Metadata: metadata.yaml

How To Read

Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.

Summary

Can a model choose the more task-consistent full output table when shown examples, query rows, and two candidate tables?

Report

Rendered from reports/qwen_pairwise_table_judge_report.md

Question

Can a model choose the more task-consistent full output table when shown examples, query rows, and two candidate tables?

This experiment evaluates pairwise table judging on public text-transformation tasks. The primary deployable method is a tournament over a non-label shortlist of candidate tables. A separate diagnostic compares direct greedy tables against hidden-correct tables when the hidden-correct table is present.

Setup

Benchmark root: /workspace/large_artifacts/qwen_pairwise_table_judge/prose-benchmarks
Run: main_qwen_pairwise_40
Tasks: 40
Candidate table rows: 1370
Pairwise judgment rows: 336
Shortlist size: 6
Train rows per task: 4
Held-out cap per task: 6

Main Result

method	tasks	row_exact	full_task_exact	table_oracle_rate	median_candidate_tables
row_repair_diagnostic	25	91.3%	80.0%	100.0%	9.00
table_oracle	40	82.5%	62.5%	62.5%	9.50
direct_row_greedy	40	72.3%	50.0%	62.5%	9.50
pairwise_tournament	40	71.9%	47.5%	62.5%	9.50

Interpretation

The deployable pairwise tournament changes full-task exact by -2.5 points relative to direct greedy. The table oracle is 12.5 points above direct greedy, so any gap between tournament and oracle is selection headroom. In the all-oracle-task direct-vs-hidden-correct diagnostic, the normal judge picks the hidden-correct table 90.0% of the time, but that aggregate includes saturated tasks where direct and oracle are identical.

On the actual headroom subset, where direct greedy is wrong and a hidden-correct table exists, the normal judge picks the hidden-correct table 0.0% when direct is candidate A and 100.0% when the hidden-correct table is candidate A. This is the critical diagnostic: a large base/swapped gap indicates position bias rather than semantic table judging.

The row-repair diagnostic reaches 80.0% full-task exact. It is not deployable because it uses hidden oracle row alternatives; it measures whether the judge can accept correct row-level replacements when they are explicitly supplied.

Charts

Full-task exact by method

Row versus task accuracy

Diagnostic pick oracle

Headroom diagnostic pick oracle

Tournament changes

Family heatmap

Diagnostic Summary

mode	pair_kind	comparisons	picked_oracle	direct_full_exact	oracle_full_exact
no_examples	direct_vs_correct	25	80.0%	80.0%	100.0%
normal	direct_vs_correct	50	90.0%	80.0%	100.0%
row_shuffled_candidate	direct_vs_correct	25	80.0%	80.0%	100.0%
shuffled_examples	direct_vs_correct	25	80.0%	80.0%	100.0%

Headroom-Only Diagnostic

mode	order_tag	comparisons	unique_tasks	picked_oracle	picked_candidate_a
no_examples	base	5	5	0.0%	100.0%
normal	base	5	5	0.0%	100.0%
normal	swapped	5	5	100.0%	100.0%
row_shuffled_candidate	base	5	5	0.0%	100.0%
shuffled_examples	base	5	5	0.0%	100.0%

Selected Tables

task_id	family	method	source	row_exact	full_task_exact	candidate_tables	table_candidate_oracle
Address.000002	Address	direct_row_greedy	row_greedy	33.3%	False	9	False
Address.000013	Address	direct_row_greedy	row_greedy	66.7%	False	17	False
BillingCode.000007	BillingCode	direct_row_greedy	row_greedy	33.3%	False	16	False
City.000010	City	direct_row_greedy	row_greedy	100.0%	True	9	True
City.000011	City	direct_row_greedy	row_greedy	75.0%	False	12	False
Column.000001	Column	direct_row_greedy	row_greedy	100.0%	True	9	True
DateTime.000004	DateTime	direct_row_greedy	row_greedy	100.0%	True	9	True
DateTime.000007	DateTime	direct_row_greedy	row_greedy	100.0%	True	9	True
DateTime.000017	DateTime	direct_row_greedy	row_greedy	100.0%	True	44	True
DateTime.000025	DateTime	direct_row_greedy	row_greedy	100.0%	True	16	True
DateTime.000027	DateTime	direct_row_greedy	row_greedy	50.0%	False	88	False
DateTime.000034	DateTime	direct_row_greedy	row_greedy	100.0%	True	9	True
DateTime.000051	DateTime	direct_row_greedy	row_greedy	33.3%	False	12	False
DateTime.000076	DateTime	direct_row_greedy	row_greedy	66.7%	False	14	True
DateTime.000081	DateTime	direct_row_greedy	row_greedy	50.0%	False	14	False
DateTime.000094	DateTime	direct_row_greedy	row_greedy	100.0%	True	9	True
DateTime.000104	DateTime	direct_row_greedy	row_greedy	100.0%	True	9	True
DateTime.000108	DateTime	direct_row_greedy	row_greedy	100.0%	True	9	True
DateTime.000111	DateTime	direct_row_greedy	row_greedy	100.0%	True	10	True
DateTime.000114	DateTime	direct_row_greedy	row_greedy	16.7%	False	512	False
DateTime.000115	DateTime	direct_row_greedy	row_greedy	0.0%	False	9	False
DateTime.000116	DateTime	direct_row_greedy	row_greedy	50.0%	False	9	False
FilePath.000001	FilePath	direct_row_greedy	row_greedy	100.0%	True	9	True
Gender.000001	Gender	direct_row_greedy	row_greedy	66.7%	False	11	False
Language.000002	Language	direct_row_greedy	row_greedy	100.0%	True	9	True
Name.000028	Name	direct_row_greedy	row_greedy	100.0%	True	9	True
Number.000008	Number	direct_row_greedy	row_greedy	33.3%	False	16	False
Number.000015	Number	direct_row_greedy	row_greedy	33.3%	False	224	True
Number.000016	Number	direct_row_greedy	row_greedy	83.3%	False	40	True
Number.000022	Number	direct_row_greedy	row_greedy	100.0%	True	32	True
Number.000028	Number	direct_row_greedy	row_greedy	100.0%	True	9	True
Number.000029	Number	direct_row_greedy	row_greedy	66.7%	False	14	False
Number.000043	Number	direct_row_greedy	row_greedy	100.0%	True	9	True
Number.000049	Number	direct_row_greedy	row_greedy	0.0%	False	56	False
Number.000075	Number	direct_row_greedy	row_greedy	66.7%	False	16	True
Number.000077	Number	direct_row_greedy	row_greedy	33.3%	False	26	True
Phone.000008	Phone	direct_row_greedy	row_greedy	100.0%	True	9	True
Phone.000011	Phone	direct_row_greedy	row_greedy	100.0%	True	9	True
ShippingCode.000008	ShippingCode	direct_row_greedy	row_greedy	33.3%	False	9	False
UserAgent.000003	UserAgent	direct_row_greedy	row_greedy	100.0%	True	9	True
Address.000002	Address	pairwise_tournament	row_greedy	33.3%	False	9	False
Address.000013	Address	pairwise_tournament	row_greedy	66.7%	False	17	False
BillingCode.000007	BillingCode	pairwise_tournament	row_greedy	33.3%	False	16	False
City.000010	City	pairwise_tournament	row_greedy	100.0%	True	9	True
City.000011	City	pairwise_tournament	row_greedy	75.0%	False	12	False
Column.000001	Column	pairwise_tournament	row_greedy	100.0%	True	9	True
DateTime.000004	DateTime	pairwise_tournament	row_greedy	100.0%	True	9	True
DateTime.000007	DateTime	pairwise_tournament	row_greedy	100.0%	True	9	True
DateTime.000017	DateTime	pairwise_tournament	row_greedy	100.0%	True	44	True
DateTime.000025	DateTime	pairwise_tournament	row_greedy	100.0%	True	16	True
DateTime.000027	DateTime	pairwise_tournament	row_greedy	50.0%	False	88	False
DateTime.000034	DateTime	pairwise_tournament	row_greedy	100.0%	True	9	True
DateTime.000051	DateTime	pairwise_tournament	row_greedy	33.3%	False	12	False
DateTime.000076	DateTime	pairwise_tournament	row_greedy	66.7%	False	14	True
DateTime.000081	DateTime	pairwise_tournament	row_greedy	50.0%	False	14	False
DateTime.000094	DateTime	pairwise_tournament	row_greedy	100.0%	True	9	True
DateTime.000104	DateTime	pairwise_tournament	row_greedy	100.0%	True	9	True
DateTime.000108	DateTime	pairwise_tournament	row_greedy	100.0%	True	9	True
DateTime.000111	DateTime	pairwise_tournament	batch_plain	83.3%	False	10	True
DateTime.000114	DateTime	pairwise_tournament	row_greedy	16.7%	False	512	False
DateTime.000115	DateTime	pairwise_tournament	row_greedy	0.0%	False	9	False
DateTime.000116	DateTime	pairwise_tournament	row_greedy	50.0%	False	9	False
FilePath.000001	FilePath	pairwise_tournament	row_greedy	100.0%	True	9	True
Gender.000001	Gender	pairwise_tournament	row_greedy	66.7%	False	11	False
Language.000002	Language	pairwise_tournament	row_greedy	100.0%	True	9	True
Name.000028	Name	pairwise_tournament	row_greedy	100.0%	True	9	True
Number.000008	Number	pairwise_tournament	row_greedy	33.3%	False	16	False
Number.000015	Number	pairwise_tournament	row_greedy	33.3%	False	224	True
Number.000016	Number	pairwise_tournament	row_greedy	83.3%	False	40	True
Number.000022	Number	pairwise_tournament	row_greedy	100.0%	True	32	True
Number.000028	Number	pairwise_tournament	row_greedy	100.0%	True	9	True
Number.000029	Number	pairwise_tournament	row_greedy	66.7%	False	14	False
Number.000043	Number	pairwise_tournament	row_greedy	100.0%	True	9	True
Number.000049	Number	pairwise_tournament	row_greedy	0.0%	False	56	False
Number.000075	Number	pairwise_tournament	row_greedy	66.7%	False	16	True
Number.000077	Number	pairwise_tournament	row_greedy	33.3%	False	26	True
Phone.000008	Phone	pairwise_tournament	row_greedy	100.0%	True	9	True
Phone.000011	Phone	pairwise_tournament	row_greedy	100.0%	True	9	True
ShippingCode.000008	ShippingCode	pairwise_tournament	row_greedy	33.3%	False	9	False
UserAgent.000003	UserAgent	pairwise_tournament	row_greedy	100.0%	True	9	True
City.000010	City	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
Column.000001	Column	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
DateTime.000004	DateTime	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
DateTime.000007	DateTime	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
DateTime.000017	DateTime	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	44	True
DateTime.000025	DateTime	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	16	True
DateTime.000034	DateTime	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
DateTime.000076	DateTime	row_repair_diagnostic	row_repair_diagnostic	66.7%	False	14	True
DateTime.000094	DateTime	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
DateTime.000104	DateTime	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
DateTime.000108	DateTime	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
DateTime.000111	DateTime	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	10	True
FilePath.000001	FilePath	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
Language.000002	Language	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
Name.000028	Name	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
Number.000015	Number	row_repair_diagnostic	row_repair_diagnostic	33.3%	False	224	True
Number.000016	Number	row_repair_diagnostic	row_repair_diagnostic	83.3%	False	40	True
Number.000022	Number	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	32	True
Number.000028	Number	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
Number.000043	Number	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
Number.000075	Number	row_repair_diagnostic	row_repair_diagnostic	66.7%	False	16	True
Number.000077	Number	row_repair_diagnostic	row_repair_diagnostic	33.3%	False	26	True
Phone.000008	Phone	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
Phone.000011	Phone	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
UserAgent.000003	UserAgent	row_repair_diagnostic	row_repair_diagnostic	100.0%	True	9	True
Address.000002	Address	table_oracle	row_greedy	33.3%	False	9	False
Address.000013	Address	table_oracle	row_greedy	66.7%	False	17	False
BillingCode.000007	BillingCode	table_oracle	row_greedy	33.3%	False	16	False
City.000010	City	table_oracle	row_greedy	100.0%	True	9	True
City.000011	City	table_oracle	row_greedy	75.0%	False	12	False
Column.000001	Column	table_oracle	row_greedy	100.0%	True	9	True
DateTime.000004	DateTime	table_oracle	row_greedy	100.0%	True	9	True
DateTime.000007	DateTime	table_oracle	row_greedy	100.0%	True	9	True
DateTime.000017	DateTime	table_oracle	row_greedy	100.0%	True	44	True
DateTime.000025	DateTime	table_oracle	row_greedy	100.0%	True	16	True
DateTime.000027	DateTime	table_oracle	row_combo	83.3%	False	88	False
DateTime.000034	DateTime	table_oracle	row_greedy	100.0%	True	9	True
DateTime.000051	DateTime	table_oracle	row_greedy	33.3%	False	12	False
DateTime.000076	DateTime	table_oracle	row_combo	100.0%	True	14	True
DateTime.000081	DateTime	table_oracle	batch_plain	66.7%	False	14	False
DateTime.000094	DateTime	table_oracle	row_greedy	100.0%	True	9	True
DateTime.000104	DateTime	table_oracle	row_greedy	100.0%	True	9	True
DateTime.000108	DateTime	table_oracle	row_greedy	100.0%	True	9	True
DateTime.000111	DateTime	table_oracle	row_greedy	100.0%	True	10	True
DateTime.000114	DateTime	table_oracle	row_combo	66.7%	False	512	False
DateTime.000115	DateTime	table_oracle	row_greedy	0.0%	False	9	False
DateTime.000116	DateTime	table_oracle	row_greedy	50.0%	False	9	False
FilePath.000001	FilePath	table_oracle	row_greedy	100.0%	True	9	True
Gender.000001	Gender	table_oracle	row_greedy	66.7%	False	11	False
Language.000002	Language	table_oracle	row_greedy	100.0%	True	9	True
Name.000028	Name	table_oracle	row_greedy	100.0%	True	9	True
Number.000008	Number	table_oracle	row_sample0	50.0%	False	16	False
Number.000015	Number	table_oracle	row_combo	100.0%	True	224	True
Number.000016	Number	table_oracle	row_combo	100.0%	True	40	True
Number.000022	Number	table_oracle	row_greedy	100.0%	True	32	True
Number.000028	Number	table_oracle	row_greedy	100.0%	True	9	True
Number.000029	Number	table_oracle	row_greedy	66.7%	False	14	False
Number.000043	Number	table_oracle	row_greedy	100.0%	True	9	True
Number.000049	Number	table_oracle	row_combo	75.0%	False	56	False
Number.000075	Number	table_oracle	batch_plain	100.0%	True	16	True
Number.000077	Number	table_oracle	row_combo	100.0%	True	26	True
Phone.000008	Phone	table_oracle	row_greedy	100.0%	True	9	True
Phone.000011	Phone	table_oracle	row_greedy	100.0%	True	9	True
ShippingCode.000008	ShippingCode	table_oracle	row_greedy	33.3%	False	9	False
UserAgent.000003	UserAgent	table_oracle	row_greedy	100.0%	True	9	True

Tournament Changes

task_id	family	direct_source	tournament_source	direct_row_exact	tournament_row_exact	direct_full_exact	tournament_full_exact	delta_row_exact	delta_full_exact
DateTime.000111	DateTime	row_greedy	batch_plain	100.0%	83.3%	True	False	-16.7%	-100.0%

Files

runs/main_qwen_pairwise_40/table_candidates.csv
runs/main_qwen_pairwise_40/oracle_summary.csv
runs/main_qwen_pairwise_40/pairwise_judgments.csv
runs/main_qwen_pairwise_40/judge_details.csv
runs/main_qwen_pairwise_40/selected_tables.csv
runs/main_qwen_pairwise_40/diagnostic_direct_vs_correct.csv
analysis/summary.csv
analysis/diagnostic_summary.csv
analysis/diagnostic_headroom_summary.csv
analysis/tournament_changes.csv
analysis/selected_tables.csv
analysis/judge_details.csv

Experiment log 8

Show the running log (8 entries, 2026-06-27)

Objective

Test whether a language model can select the more task-consistent full output table when shown concrete alternatives for the same transformation task.

The primary method is pairwise table judging. The experiment includes:

A deployable tournament over a non-label shortlist of candidate tables.
A diagnostic direct-vs-hidden-correct comparison where the correct table is present but not labeled as such.
Controls that remove or corrupt the task context.

The experiment is standalone: all run-local inputs, candidate pools, judgments, analyses, charts, and reports are stored under this directory.

Initial Plan

Create a fresh experiment directory.
Copy a fixed candidate-table pool into the run directory.
Load the public text-transformation tasks and render the train examples/query rows.
Ask Qwen to judge candidate table pairs.
Compare deployable tournament, diagnostic direct-vs-correct judging, no-example control, shuffled-example control, and row-shuffled-candidate control.
Generate CSVs, charts, Markdown report, and HTML report.

Run Notes

2026-06-27 08:06 UTC - Scaffold

Created a fresh experiment directory at /workspace/experiments/qwen_pairwise_table_judge.
Created a separate large-artifact root at /workspace/large_artifacts/qwen_pairwise_table_judge.
Symlinked the public PROSE benchmark data under the large-artifact root instead of duplicating it in the experiment directory.
Implemented src/qwen_pairwise_table_judge.py.
Verified syntax with python -m py_compile.

2026-06-27 08:07 UTC - No-Qwen Smoke

Command:

python /workspace/experiments/qwen_pairwise_table_judge/src/qwen_pairwise_table_judge.py \
  --run_name smoke_no_qwen \
  --task_limit 6 \
  --heldout_cap 4 \
  --shortlist 4 \
  --no_qwen

Purpose: validate filesystem layout, candidate-pool copying, benchmark loading, CSV output, chart generation, Markdown report generation, and HTML report generation without spending model calls.

Fixes made during smoke:

Replaced a pandas Series boolean fallback with an explicit None check.
Fixed a mode_summary.mode attribute collision by indexing mode_summary["mode"].

Smoke metrics are not interpreted because --no_qwen defaults uncached judgments to candidate A.

2026-06-27 08:08 UTC - Real-Qwen Pilot

Command:

python /workspace/experiments/qwen_pairwise_table_judge/src/qwen_pairwise_table_judge.py \
  --run_name pilot_qwen_6 \
  --task_limit 6 \
  --heldout_cap 4 \
  --shortlist 4

Result:

52 real model judgments.
table_oracle: 100.0% full-task exact.
direct_row_greedy: 66.7% full-task exact.
pairwise_tournament: 66.7% full-task exact.
row_repair_diagnostic: 16.7% full-task exact.

Pilot diagnosis:

Qwen emitted parseable A/B choices.
The small task sample was mostly saturated, so it was only used to validate mechanics and prompt parsing.

2026-06-27 08:09 UTC - Main Run

Command:

python /workspace/experiments/qwen_pairwise_table_judge/src/qwen_pairwise_table_judge.py \
  --run_name main_qwen_pairwise_40 \
  --task_limit 40 \
  --heldout_cap 6 \
  --shortlist 6

Result:

40 public text-transformation tasks.
336 unique cached model judgments.
No blank choices in the cached judgment records.
table_oracle: 62.5% full-task exact.
direct_row_greedy: 50.0% full-task exact.
pairwise_tournament: 47.5% full-task exact.
row_repair_diagnostic: 80.0% full-task exact on the 25 tasks with an exact candidate.

Main diagnosis:

The deployable pairwise tournament is a negative result: it is 2.5 points worse than direct greedy and does not capture the 12.5-point oracle headroom.
The all-oracle-task diagnostic is misleading if read alone: normal direct-vs-hidden-correct judging picks the hidden-correct table 90.0% of the time, but most of those comparisons are saturated cases where direct and oracle are the same candidate.
On the five true headroom tasks, Qwen picks candidate A regardless of semantics:
- direct as A, hidden-correct as B: 0.0% picked hidden-correct.
- hidden-correct as A, direct as B: 100.0% picked hidden-correct.
- no examples, shuffled examples, and row-shuffled candidate controls with direct as A: 0.0% picked hidden-correct.
The useful signal is row-local and diagnostic only: when hidden correct row replacements are explicitly supplied, row repair reaches 80.0% full-task exact. That is not deployable because it uses oracle alternatives.

2026-06-27 08:11 UTC - Report Hardening

Added analysis/diagnostic_headroom_summary.csv.
Added analysis/tournament_changes.csv.
Added analysis/figures/diagnostic_headroom_pick_oracle.png.
Updated the Markdown and HTML report to make the headroom-only position-bias result explicit.
Verified the report is standalone and contains no references to earlier experiments.

Final artifacts:

src/qwen_pairwise_table_judge.py
runs/main_qwen_pairwise_40/pairwise_judgments.csv
runs/main_qwen_pairwise_40/judge_details.csv
runs/main_qwen_pairwise_40/selected_tables.csv
runs/main_qwen_pairwise_40/diagnostic_direct_vs_correct.csv
analysis/summary.csv
analysis/diagnostic_summary.csv
analysis/diagnostic_headroom_summary.csv
analysis/tournament_changes.csv
analysis/figures/*.png
reports/qwen_pairwise_table_judge_report.md
reports/qwen_pairwise_table_judge_report.html

Figures 6

diagnostic headroom pick oracle · analysis/figures/

diagnostic pick oracle · analysis/figures/

Data files 10

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗

Pairwise Table Judge

Results at a glance 7

Overview

How To Read

Summary

Report

Question

Setup

Main Result

Interpretation

Charts

Diagnostic Summary

Headroom-Only Diagnostic

Selected Tables

Tournament Changes

Files

Experiment log 8

Objective

Initial Plan

Run Notes

2026-06-27 08:06 UTC - Scaffold

2026-06-27 08:07 UTC - No-Qwen Smoke

2026-06-27 08:08 UTC - Real-Qwen Pilot

2026-06-27 08:09 UTC - Main Run

2026-06-27 08:11 UTC - Report Hardening

Figures 6

Data files 10

Reproduce

Related

Related experiments