Qwen Disagreement-Probe Program Induction

Finished2026-06-27imported · line YBenchmark Generalization Structured Execution and Compilers GitHub ↗

The one idea you need

A small model writes several short programs to reformat messy text — dates, prices, names. Because the programs disagree on some inputs, you feed it those tricky cases and ask it to judge which output is right, hoping its verdicts reveal the single best program.

The question

When a small model writes several programs to reformat text, does quizzing it on the inputs where those programs disagree help pick the best one?

What we found

No. The disagreement quiz picked the same programs whether its judge answers were real, randomly assigned, or skipped entirely — all three landed at 64% of tasks fully solved. The only genuine gain came from a plain caution rule: run a generated program only when it clearly fits, otherwise answer each item by hand. That rule lifted fully-solved tasks from 56% to 64%.

Why it matters

When a small model writes code to handle a task, invest in a conservative fallback that runs the code only when it clearly fits — not elaborate schemes to rank rival programs, which here changed no decisions at all.

Tasks fully solved: plain answering vs adding caution56% → 64%share of 25 tasks solved exactly, before and after the cautious program gate

Extra tasks won by the disagreement quiznonereal-quiz, random-quiz, and no-quiz gates all tied at 64%

Tasks where the quiz changed the choice0 of 25quiz-gated and no-quiz decisions were identical on every task

Where the gains landed29% → 57%number-formatting tasks solved exactly; every other category matched plain answering

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 6

How many tasks each method fully solved

How to read

Bars show the share of tasks solved exactly right, one per method — plain by-hand answering, program only, and gates using real, random, no, or scrambled quiz answers; taller is better.

Takeaway → The real-quiz, random-quiz, and no-quiz gates land at exactly the same height, so the disagreement quiz bought nothing — the caution gate, not the quiz, delivered the lift.

Data table

method	full-task exact
direct row-by-row	56%
program only	40%
gate (no probes)	64%
disagreement-probe gate	64%
random-probe gate	64%
shuffled-label gate	56%

Numbers from experiments/qwen_disagreement_probe_program_induction/analysis/summary.csv

Technical framing

Full-task exact match: gated program use ties across probe variants — The simple direct-fallback gate lifts 56% to 64%; disagreement probes add nothing beyond it (probe and random-probe gates also 64%).

Tasks improved versus tasks broken, compared to plain answering

How to read

For each method, one bar counts tasks it improved over plain answering and another counts tasks it broke, out of 25; more improved and fewer broken is better. Raw programs broke six to fourteen tasks; both caution gates broke none.

Takeaway → Running programs unguarded wrecks many tasks; the cautious gate keeps the two wins while breaking nothing — its whole value is restraint.

Data table

method	tasks helped	tasks hurt
disagreement-probe gate	2	0
gate (no probes)	2	0
program only	2	6
monolithic program	2	7
shuffled recursive	1	14

Numbers from experiments/qwen_disagreement_probe_program_induction/analysis/summary.csv

Technical framing

Tasks helped vs hurt relative to direct answering (25 tasks) — Raw generated programs hurt 6-7 tasks; the conservative fallback gates keep the 2 wins while hurting none.

Tasks solved by category: plain answering vs the quiz gate

How to read

Paired bars per text-format category compare plain answering against the disagreement-quiz gate; taller is better. Every category matches except number-formatting, where the gate rises from about 29% to about 57%.

Takeaway → All of the gate's advantage comes from number-formatting tasks; everywhere else it simply equals plain answering.

Data table

task family	direct row-by-row	disagreement-probe gate
BillingCode	100%	100%
City	0%	0%
Currency	100%	100%
DateTime	28.6%	28.6%
Email	100%	100%
FilePath	100%	100%
Name	100%	100%
Number	28.6%	57.1%
Phone	100%	100%
Rating	100%	100%

Numbers from report table (Family Summary), experiments/qwen_disagreement_probe_program_induction/reports/qwen_disagreement_probe_program_induction_report.md

Technical framing

Full-task exact by task family: direct vs probe-gated — All of the gated method's gain comes from Number tasks; every other family matches direct answering exactly.

Item-by-item accuracy by method

How to read

Bars show the share of individual items transformed exactly right, one per method; taller is better. All gated variants tie near 83%, plain answering trails slightly, and the ungated program arms fall to 45% or below.

Takeaway → At the item level the gated methods tie again, while trusting raw programs roughly halves accuracy and scrambled answers collapse it to near nothing.

Data table

Method	Row exact
Recursive gated	82.7%
Probe gated	82.7%
Random-probe gated	82.7%
Direct row	80.7%
Shuffled-probe gated	80.7%
Direct batch	76%
Recursive program	44.7%
Probe program	44.7%
Monolithic program	40.7%
Recursive shuffled	6%

Numbers from experiments/qwen_disagreement_probe_program_induction/analysis/summary.csv

Technical framing

Row-level exact match by method — At the row level all gated variants tie at 82.7%; ungated program arms roughly halve accuracy and shuffled labels collapse to 6%.

How often a fitting program is actually trusted

How to read

Per method, bars show the share of tasks where a generated program passed the visible examples or was actually used — this measures supply of usable programs, not final accuracy. Programs fit on 52% of tasks; the cautious gates trust only 40 to 44%.

Takeaway → Programs often fit the shown examples, yet the gate deliberately trusts fewer of them, trading coverage for safety; scrambled-answer quizzes trust none.

Data table

Method	Train-pass rate
Recursive program	52%
Probe program	52%
Recursive oracle	52%
Monolithic program	48%
Recursive gated	44%
Probe gated	40%
Random-probe gated	40%
Recursive shuffled	20%
Shuffled-probe gated	0%

Numbers from experiments/qwen_disagreement_probe_program_induction/analysis/summary.csv

Technical framing

Share of tasks with a train-passing program used or found, by method — Programs pass visible examples on 52% of tasks, but the conservative gates only trust them on 40-44%; shuffled-label probes trust none.

Item accuracy by category: plain answering vs raw programs

How to read

Paired bars per category compare plain answering against raw unguarded programs at the item level; taller is better. They match on prices, emails, phones, and ratings, but programs collapse on file paths, billing codes, and especially dates and times.

Takeaway → Raw programs handle simple formats fine but fall apart on structured ones, explaining why running them unguarded is risky.

Data table

Task family	Direct row	Recursive program (ungated)
BillingCode	100%	0%
City	66.7%	33.3%
Currency	100%	100%
DateTime	59.5%	9.5%
Email	100%	100%
FilePath	100%	0%
Name	100%	66.7%
Number	76.2%	45.2%
Phone	100%	100%
Rating	100%	100%

Numbers from experiments/qwen_disagreement_probe_program_induction/analysis/family_summary.csv

Technical framing

Row exact by task family: direct answering vs raw generated programs — Ungated programs match direct answering on Currency/Email/Phone/Rating but collapse on BillingCode, FilePath, and especially DateTime.

In the author’s words from the Report · “Interpretation”

Direct row-by-row answering solves 56.0% of tasks under strict full-task exactness. The selected executable program without probe selection solves 40.0%. The non-probe gated method solves 64.0%. The disagreement-probe gated method solves 64.0%. Random-probe gated selection solves 64.0%, and shuffled-label probe selection solves 56.0%. The disagreement-probe gated method helps 2 tasks and hurts 0 tasks relative to direct row-by-row answering. The hidden diagnostic oracle over train-passing generated programs solves 40.0%, and at least one candidate passes visible examples on 52.0% of tasks. The shuffled-label control checks whether probe labels matter. The random-probe control checks whether disagreement ranking matters. … Read the full result →

Overview

Standalone experiment for selecting task-local executable transformation programs with model-labeled disagreement probes.

Large reusable artifacts are stored under:

/workspace/large_artifacts/qwen_disagreement_probe_program_induction

Primary outputs:

src/qwen_disagreement_probe_program_induction.py
experiment_log.md
runs/
analysis/
reports/

Report

Rendered from reports/qwen_disagreement_probe_program_induction_report.md

Question

Can model-labeled disagreement probes select better task-local executable programs than visible examples alone?

The method asks the model to write Python transform(row) candidate functions. Candidate programs are executed on visible examples. Synthetic probe inputs are generated from the visible examples, ranked by candidate disagreement, and labeled by the model. Program selection uses visible examples plus probe labels. Held-out rows are used only for final scoring.

Setup

Run: main_v1
Dataset: public text-transformation tasks.
Tasks: 25
Visible examples per task: 4
Held-out cap per task: 6
Program variants: monolithic,robust
Repair rounds: 1
Disagreement probes per task: 4
Probe label variants: plain,consistency
Probe score threshold for gated use: 0.67
Elapsed seconds: 1556.2

Main Result

method	tasks	row_exact	full_task_exact	train_pass_rate	tasks_helped_vs_direct	tasks_hurt_vs_direct
recursive_gated_direct	25	82.7%	64.0%	44.0%	2	0
probe_gated_direct	25	82.7%	64.0%	40.0%	2	0
random_probe_gated_direct	25	82.7%	64.0%	40.0%	2	0
direct_batch	25	76.0%	60.0%		2	1
direct_row	25	80.7%	56.0%		0	0
shuffled_probe_gated_direct	25	80.7%	56.0%	0.0%	0	0
recursive_program	25	44.7%	40.0%	52.0%	2	6
probe_program	25	44.7%	40.0%	52.0%	2	6
recursive_oracle	25	44.7%	40.0%	52.0%	2	6
monolithic_program	25	40.7%	36.0%	48.0%	2	7
recursive_shuffled	25	6.0%	4.0%	20.0%	1	14

Interpretation

Direct row-by-row answering solves 56.0% of tasks under strict full-task exactness. The selected executable program without probe selection solves 40.0%. The non-probe gated method solves 64.0%. The disagreement-probe gated method solves 64.0%. Random-probe gated selection solves 64.0%, and shuffled-label probe selection solves 56.0%.

The disagreement-probe gated method helps 2 tasks and hurts 0 tasks relative to direct row-by-row answering. The hidden diagnostic oracle over train-passing generated programs solves 40.0%, and at least one candidate passes visible examples on 52.0% of tasks.

The shuffled-label control checks whether probe labels matter. The random-probe control checks whether disagreement ranking matters. A useful result should beat direct answering, the non-probe gate, and both probe controls.

Verdict

Disagreement probes did not add measurable selection power in this run. The probe-gated method tied the non-probe gate at 64.0% and tied the random-probe gate at 64.0%. Probe-gated and non-probe gated decisions differed on 0 of 25 tasks, while probe-gated and random-probe gated outcomes matched on 25 of 25 tasks.

The deployable gain came from the conservative direct-fallback gate: it used generated programs on 11 tasks and improved full-task exactness from 56.0% to 64.0% without hurting any direct-answer successes. The disagreement-probe gate used generated programs on 10 tasks, and the random-probe gate used generated programs on 10 tasks, but neither improved on that simpler gate.

Program Diagnostics

The raw executable-program arm is not deployable by itself: train-passing programs can be too narrow, often literal branches or partial parsers. The gated arms use a generated program only when the selected train-passing candidate does not look like a literal example table and its probe agreement clears the configured threshold; otherwise they fall back to direct row-by-row answering.

The probe score distribution is reported separately because a method that only wins by falling back everywhere is not a useful program selector.

Charts

Method full-task exact

Row scatter

Wins and losses

Family breakdown

Train-passing candidates

Probe agreement scores

Task Details

task_id	family	heldout_rows	direct_full_exact	recursive_full_exact	recursive_gated_full_exact	probe_gated_full_exact	random_probe_gated_full_exact	shuffled_probe_gated_full_exact	recursive_oracle_full_exact	recursive_train_pass_count	probe_gated_used_program	probe_gated_score	probe_gated_helped_vs_direct	probe_gated_hurt_vs_direct
Number.000044	Number	6	False	True	True	True	True	False	True	1	True	0.75	True	False
Number.000093	Number	3	False	True	True	True	True	False	True	1	True	0.75	True	False
BillingCode.000002	BillingCode	6	True	False	True	True	True	True	False	0	False	0	False	False
City.000011	City	3	False	False	False	False	False	False	False	1	False	0	False	False
Currency.000004	Currency	6	True	True	True	True	True	True	True	1	True	1	False	False
DateTime.000012	DateTime	6	False	False	False	False	False	False	False	1	True	1	False	False
DateTime.000035	DateTime	6	True	False	True	True	True	True	False	0	False	0	False	False
DateTime.000077	DateTime	6	False	False	False	False	False	False	False	0	False	0	False	False
DateTime.000083	DateTime	6	False	False	False	False	False	False	False	0	False	0	False	False
DateTime.000088	DateTime	6	False	False	False	False	False	False	False	0	False	0	False	False
DateTime.000094	DateTime	4	True	False	True	True	True	True	False	0	False	0	False	False
DateTime.000098	DateTime	6	False	False	False	False	False	False	False	0	False	0	False	False
Email.000013	Email	6	True	True	True	True	True	True	True	2	True	1	False	False
FilePath.000001	FilePath	6	True	False	True	True	True	True	False	0	False	0	False	False
Name.000013	Name	6	True	True	True	True	True	True	True	2	False	0.5	False	False
Name.000026	Name	6	True	False	True	True	True	True	False	0	False	0	False	False
Name.000028	Name	6	True	True	True	True	True	True	True	2	True	1	False	False
Number.000048	Number	4	True	False	True	True	True	True	False	0	False	0	False	False
Number.000074	Number	6	False	False	False	False	False	False	False	0	False	0	False	False
Number.000075	Number	6	False	False	False	False	False	False	False	0	False	0	False	False
Number.000081	Number	6	False	False	False	False	False	False	False	1	False	0	False	False
Number.000088	Number	6	True	True	True	True	True	True	True	1	True	1	False	False
Phone.000005	Phone	6	True	True	True	True	True	True	True	1	True	1	False	False
Phone.000017	Phone	6	True	True	True	True	True	True	True	1	True	1	False	False
Rating.000001	Rating	6	True	True	True	True	True	True	True	1	True	1	False	False

Family Summary

family	direct_row	monolithic_program	probe_gated_direct	recursive_gated_direct	recursive_oracle	recursive_program
BillingCode	1	0	1	1	0	0
City	0	0	0	0	0	0
Currency	1	1	1	1	1	1
DateTime	0.286	0	0.286	0.286	0	0
Email	1	1	1	1	1	1
FilePath	1	0	1	1	0	0
Name	1	0.667	1	1	0.667	0.667
Number	0.286	0.429	0.571	0.571	0.429	0.429
Phone	1	1	1	1	1	1
Rating	1	0	1	1	1	1

Limitations

Generated code is sandboxed by a conservative AST pass, so some potentially valid programs may be rejected. The benchmark tasks are public text transformations and do not cover arbitrary software engineering problems. Full-task exact is intentionally strict and can be much lower than row accuracy.

Artifacts

Run directory: /workspace/experiments/qwen_disagreement_probe_program_induction/runs/main_v1
Summary: analysis/summary.csv
Task details: analysis/task_summary.csv
Candidate programs: analysis/candidates.csv
Probe labels: analysis/probe_labels.csv
Figures: analysis/figures/

Experiment log 4

Show the running log (4 entries)

Setup

Created a fresh standalone experiment directory.
Large artifacts are kept under /workspace/large_artifacts/qwen_disagreement_probe_program_induction.
Primary metric: strict full-task exact on held-out rows.
Secondary metrics: row exact, generated-program train-pass rate, probe-label usefulness, and candidate-selection flips.

Run `smoke_no_qwen`

Time UTC: 2026-06-27T18:05:55.692908+00:00
Elapsed seconds: 0.8
Config: {"answer_max_new_tokens": 64, "batch_max_new_tokens": 320, "code_max_new_tokens": 520, "heldout_cap": 3, "max_probe_pool": 40, "min_heldout": 3, "no_qwen": true, "probe_count": 3, "probe_label_variants": "plain", "probe_score_min": 0.75, "repair_max_new_tokens": 560, "repair_rounds": 1, "repair_variants": "minimal,broaden,conditional", "run_name": "smoke_no_qwen", "run_shuffled": true, "sample_seed": 20260627, "task_limit": 3, "train_n": 4, "variants": "monolithic,robust"}
Tasks: 3
Direct row-by-row full-task exact: 0.0%
Recursive selected-program full-task exact: 0.0%
Recursive gated-direct full-task exact: 0.0%
Disagreement-probe gated-direct full-task exact: 0.0%
Recursive train-pass rate: 0.0%
Recursive oracle among train-passing candidates: 0.0%

Run `smoke_qwen_5`

Time UTC: 2026-06-27T18:10:38.193113+00:00
Elapsed seconds: 255.2
Config: {"answer_max_new_tokens": 48, "batch_max_new_tokens": 220, "code_max_new_tokens": 420, "heldout_cap": 3, "max_probe_pool": 40, "min_heldout": 3, "no_qwen": false, "probe_count": 3, "probe_label_variants": "plain", "probe_score_min": 0.67, "repair_max_new_tokens": 420, "repair_rounds": 1, "repair_variants": "minimal,broaden,conditional", "run_name": "smoke_qwen_5", "run_shuffled": true, "sample_seed": 20260627, "task_limit": 5, "train_n": 4, "variants": "monolithic,robust"}
Tasks: 5
Direct row-by-row full-task exact: 60.0%
Recursive selected-program full-task exact: 20.0%
Recursive gated-direct full-task exact: 60.0%
Disagreement-probe gated-direct full-task exact: 60.0%
Recursive train-pass rate: 80.0%
Recursive oracle among train-passing candidates: 20.0%

Run `main_v1`

Time UTC: 2026-06-27T18:37:15.412379+00:00
Elapsed seconds: 1556.2
Config: {"answer_max_new_tokens": 56, "batch_max_new_tokens": 300, "code_max_new_tokens": 420, "heldout_cap": 6, "max_probe_pool": 40, "min_heldout": 3, "no_qwen": false, "probe_count": 4, "probe_label_variants": "plain,consistency", "probe_score_min": 0.67, "repair_max_new_tokens": 420, "repair_rounds": 1, "repair_variants": "minimal,broaden,conditional", "run_name": "main_v1", "run_shuffled": true, "sample_seed": 20260627, "task_limit": 25, "train_n": 4, "variants": "monolithic,robust"}
Tasks: 25
Direct row-by-row full-task exact: 56.0%
Recursive selected-program full-task exact: 40.0%
Recursive gated-direct full-task exact: 64.0%
Disagreement-probe gated-direct full-task exact: 64.0%
Recursive train-pass rate: 52.0%
Recursive oracle among train-passing candidates: 40.0%
Interpretation: disagreement probes did not add measurable selection power. Probe-gated, non-probe gated, and random-probe gated selection all reached 64.0%; shuffled probe labels fell back to the direct baseline at 56.0%. Probe-gated and non-probe gated outcomes differed on 0/25 tasks, so the deployable gain came from conservative program/direct gating rather than probe-based discrimination.

Figures 6

method full task exact · analysis/figures/

probe agreement scores · analysis/figures/

row scatter direct vs recursive · analysis/figures/

train passing candidates · analysis/figures/

wins losses vs direct · analysis/figures/

Data files 12

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

analysis/family_summary.csv2.7 kB
analysis/summary.csv662 B
analysis/task_summary.csv11 kB
runs/main_v1/family_summary.csv2.7 kB
runs/main_v1/summary.csv662 B
runs/main_v1/task_summary.csv11 kB
runs/smoke_no_qwen/family_summary.csv649 B
runs/smoke_no_qwen/summary.csv498 B
runs/smoke_no_qwen/task_summary.csv2.7 kB
runs/smoke_qwen_5/family_summary.csv1.1 kB
runs/smoke_qwen_5/summary.csv604 B
runs/smoke_qwen_5/task_summary.csv3.7 kB

Reproduce

The run commands are documented inside the experiment folder (see the README).

Browse the experiment folder on GitHub ↗

Qwen Disagreement-Probe Program Induction

Results at a glance 6

Overview

Report

Question

Setup

Main Result

Interpretation

Verdict

Program Diagnostics

Charts

Task Details

Family Summary

Limitations

Artifacts

Experiment log 4

Setup

Run smoke_no_qwen

Run smoke_qwen_5

Run main_v1

Figures 6

Data files 12

Reproduce

Related

Related experiments

Run `smoke_no_qwen`

Run `smoke_qwen_5`

Run `main_v1`