Research log Small Model Experimentation
GitHub

Qwen Active Example Acquisition

Finished2026-06-27imported · line YActive Evidence AcquisitionBenchmark GeneralizationGitHub ↗
One well-chosen example helps; random extras backfire

The one idea you need

A model reformatting messy data learns a rule from four worked examples and may request more. The smart version asks only for the one example it keeps second-guessing, where its answer flips depending on how the question is phrased, instead of grabbing three at random.

The question

If a small model can request extra worked examples before answering, does letting it pick the one it is most unsure about beat just handing it random ones?

What we found

Barely, and the real lesson is the downside. Letting the model request the single example it was most unsure about lifted fully-solved tasks from 20 to 21 out of 30. But three random extras dropped it to 19, and feeding it examples with scrambled answers sank it to 18. Choosing well matters far more than adding more.

Why it matters

When steering a small model with examples, spend effort finding the one case it is most unsure about and verify its answer is correct. Padding prompts with random extras can quietly lower accuracy, especially on date- and number-formatting tasks.

Tasks fully solved, before then after20 → 21 of 30four given examples vs adding one carefully chosen example
Three random extra examples19 of 30fewer solved than adding nothing
Examples with scrambled answers18 of 30wrong answers hurt more than adding no example
Best any tested pick could reach22 of 30ceiling if the ideal extra example were always chosen
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Question
    2. Setup
    3. Main Result
    4. Interpretation
    5. Charts
    6. Task-Level Active Versus Random
    7. Acquisition Diagnostics
    8. Family Breakdown
    9. Files
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 7

How each way of picking extra examples affects tasks solved

How to read

Each bar is the share of unseen test tasks fully solved, one bar per strategy for choosing extra examples; taller is better. Leftmost adds only the four given examples; rightmost is the best pick that was tried.

0%20%40%60%80%base (4 examples)base (4 examples)66.7%random +3random +363.3%active +3 (shuffled labels)active +3 (shuffled labels)60%diverse +3diverse +366.7%active +1active +170%active +3active +370%oracle over tested picksoracle over tested picks73.3%

Takeaway → The two actively-chosen bars rise above the no-extras baseline while random and scrambled-answer bars fall below it, so how you pick beats how many you add.

Data table
acquisition policyfull-task exact
base (4 examples)66.7%
random +363.3%
active +3 (shuffled labels)60%
diverse +366.7%
active +170%
active +370%
oracle over tested picks73.3%

Numbers from report table (reports/qwen_active_example_acquisition_report.md, Main Result)

Technical framing

Held-out task accuracy by example-acquisition policy — Actively chosen extra examples lift task accuracy from 66.7% to 70.0%; random or label-shuffled extras hurt, and the oracle ceiling is 73.3%.

Accuracy as more extra examples are added, by strategy

How to read

Each line is a strategy for choosing extra examples; the horizontal axis is how many were added (none, one, or three), the vertical axis is the share of tasks fully solved; up is better.

62.5%65%67.5%70%72.5%0123activeorder (pool order)diverserandom

Takeaway → Only the active-choice line stays above its no-extras starting point at both amounts, while the random and diversity lines dip below, so extras help only when chosen well.

Data table
extra examples acquiredactiveorder (pool order)diverserandom
066.7%66.7%66.7%66.7%
170%70%63.3%66.7%
370%66.7%66.7%63.3%

Numbers from report table (reports/qwen_active_example_acquisition_report.md, Main Result)

Technical framing

Accuracy vs number of acquired examples, by selection strategy — Only active selection stays above the 4-example baseline at both budgets; random and diverse acquisition can make things worse.

Date-and-time tasks: the largest hard group

How to read

Each bar is the share of the 13 date-and-time tasks fully solved, one bar per method, from no extras through one and three actively chosen, then random and scrambled-answer extras; taller is better.

0%20%40%60%basebase46.2%active +1active +153.8%active +3active +346.2%random +3random +338.5%shuffled-label +3shuffled-label +338.5%

Takeaway → One actively chosen example lifts this hard group the most, while random or scrambled-answer extras push it down to the lowest bars.

Data table
methodfull-task exact (13 DateTime tasks)
base46.2%
active +153.8%
active +346.2%
random +338.5%
shuffled-label +338.5%

Numbers from report table (reports/qwen_active_example_acquisition_report.md, Family Breakdown)

Technical framing

DateTime tasks: the largest hard family under example acquisition — On DateTime, the largest hard family (13 tasks), one actively chosen example lifts accuracy 46.2% to 53.8%, while random extras drop it to 38.5%.

Getting every row right versus getting single rows right

How to read

Two bars per strategy: the lighter bar is the share of individual test rows answered correctly, the darker bar the share of tasks where every row is right; the darker bar is stricter, and higher is better.

0%25%50%75%100%basebase81.1%66.7%random1random181.1%66.7%random3random381.1%63.3%diverse1diverse180%63.3%diverse3diverse382.2%66.7%order3order383.3%66.7%active1active184.4%70%active3active384.4%70%shuffled labelsshuffled labels76.7%60%oracleoracle87.8%73.3%

Takeaway → Whole-task bars always trail single-row bars, meaning one wrong row sinks a task, and active choosing narrows that gap the most.

Data table
Acquisition policyRow exactFull-task exact
base81.1%66.7%
random181.1%66.7%
random381.1%63.3%
diverse180%63.3%
diverse382.2%66.7%
order383.3%66.7%
active184.4%70%
active384.4%70%
shuffled labels76.7%60%
oracle87.8%73.3%

Numbers from experiments/qwen_active_example_acquisition/reports/qwen_active_example_acquisition_report.md (Main Result table)

Technical framing

Row-level vs full-task strict accuracy by acquisition policy — Full-task exact runs 11-18 points below row exact for every policy; active selection narrows the gap most.

Which task types still have room to improve

How to read

Two bars per task type: accuracy with the four given examples versus the best pick of extras that was tried; each bar is the share of tasks fully solved, higher is better.

0%25%50%75%100%CityCity100%100%CurrencyCurrency100%100%DateTimeDateTime46.2%53.8%LanguageLanguage100%100%MeteoriteMeteorite0%0%NumberNumber66.7%83.3%PhonePhone100%100%StateState100%100%UserAgentUserAgent100%100%

Takeaway → Six of nine task types are already fully solved, so extras can only move the date-time and number groups, and one type stays stuck at zero.

Data table
Task familyBase (4 examples)Oracle acquisition
City100%100%
Currency100%100%
DateTime46.2%53.8%
Language100%100%
Meteorite0%0%
Number66.7%83.3%
Phone100%100%
State100%100%
UserAgent100%100%

Numbers from experiments/qwen_active_example_acquisition/reports/qwen_active_example_acquisition_report.md (Family Breakdown table)

Technical framing

Where the headroom is: base vs oracle full-task accuracy by family — Six of nine families are already saturated at 100%; extra examples can only move DateTime and Number, and Meteorite stays at 0%.

Number-formatting tasks: smart picks help, scrambled answers hurt

How to read

Each bar is the share of the six number-formatting tasks fully solved, one bar per method, from no extras through random, ordered, diverse, and actively chosen, then scrambled-answer and best-pick extras; taller is better.

0%25%50%75%100%basebase66.7%random3random366.7%order3order366.7%diverse3diverse383.3%active3active383.3%shuffled labelsshuffled labels50%oracleoracle83.3%

Takeaway → Active and diverse picks reach the best-achieved level, while examples with scrambled answers drag accuracy below the no-extras start.

Data table
Acquisition policyNumber tasks (n=6)
base66.7%
random366.7%
order366.7%
diverse383.3%
active383.3%
shuffled labels50%
oracle83.3%

Numbers from experiments/qwen_active_example_acquisition/reports/qwen_active_example_acquisition_report.md (Family Breakdown table)

Technical framing

Number family: three extra examples help, but only if labels are real — On Number tasks, active3 and diverse3 hit the oracle ceiling (83.3%) while shuffled-label examples drag accuracy below base.

Do different self-check styles agree on which example to add?

How to read

Three bars per hard task, one for each style of self-check the model uses to gauge its own uncertainty; each bar is the share of that task's rows solved, higher is better.

0%25%50%75%100%DateTime.000022DateTime.00002275%75%50%DateTime.000026DateTime.00002675%75%100%DateTime.000029DateTime.00002925%0%25%Meteorite.000001Meteorite.0000010%0%25%Number.000017Number.00001750%50%25%Number.000082Number.00008275%75%100%

Takeaway → The three self-check styles mostly agree, but on the hardest tasks one style swings the result by a noticeable chunk in either direction.

Data table
TaskPlain probeFormat probeConsistency probe
DateTime.00002275%75%50%
DateTime.00002675%75%100%
DateTime.00002925%0%25%
Meteorite.0000010%0%25%
Number.00001750%50%25%
Number.00008275%75%100%

Numbers from experiments/qwen_active_example_acquisition/reports/qwen_active_example_acquisition_report.md (Acquisition Diagnostics table)

Technical framing

Probe-variant sensitivity: leave-one-out accuracy on tasks where probes disagree — Probe variants mostly agree, but on hard tasks the consistency probe can swing leave-one-out accuracy by 25 points either way.

In the author’s words from the Report · “Interpretation”

The baseline with four examples solves 66.7% of tasks. Active acquisition solves 70.0% with one extra example and 70.0% with three extra examples. Input-diversity acquisition solves 66.7% with three extra examples. Random three-example acquisition solves 63.3%, order-three solves 66.7%, and the shuffled-label control solves 60.0%. The hidden diagnostic oracle over the tested acquisition policies reaches 73.3%, which measures whether any tested extra-example choice contained a better move. Visible-example portfolio selection reaches 66.7%.

Overview

Standalone experiment testing whether actively selected clarifying examples improve strict held-out text-transformation accuracy.

Report

Rendered from reports/qwen_active_example_acquisition_report.md

Question

Can a small number of actively selected clarifying examples improve strict held-out text-transformation accuracy?

Each task starts with four visible examples. The selector may reveal one or three additional examples from a separate acquisition pool before the model answers three held-out rows. The held-out rows are never used for acquisition.

Setup

  • Run: main_final
  • Dataset: public text-transformation tasks.
  • Tasks: 30
  • Visible examples per task: 4
  • Acquisition pool examples per task: 5
  • Held-out evaluation rows per task: 3
  • Probe variants for active disagreement: plain, format, consistency
  • Generation records: 1890

Main Result

methodtasksmean_budgetrow_exactfull_task_exact
oracle_among_tested_acquisitions301.1387.8%73.3%
active1_plain301.0084.4%70.0%
order1_plain301.0084.4%70.0%
active3_plain303.0084.4%70.0%
random1_plain301.0081.1%66.7%
base_plain300.0081.1%66.7%
loo_portfolio_base300.0081.1%66.7%
diverse3_plain303.0082.2%66.7%
order3_plain303.0083.3%66.7%
diverse1_plain301.0080.0%63.3%
random3_plain303.0081.1%63.3%
active3_shuffled_labels303.0076.7%60.0%

Interpretation

The baseline with four examples solves 66.7% of tasks. Active acquisition solves 70.0% with one extra example and 70.0% with three extra examples. Input-diversity acquisition solves 66.7% with three extra examples. Random three-example acquisition solves 63.3%, order-three solves 66.7%, and the shuffled-label control solves 60.0%.

The hidden diagnostic oracle over the tested acquisition policies reaches 73.3%, which measures whether any tested extra-example choice contained a better move. Visible-example portfolio selection reaches 66.7%.

Charts

Full-task exact by method

Row versus full-task exact

Budget comparison

Wins and losses versus base

Probe entropy distribution

Family heatmap

Task-Level Active Versus Random

task_idfamilyfeaturesbaseactive3_exactrandom3_exactactive_helpedactive_hurtactive3_indicesrandom3_indicesloo_variant
Number.000017NumberNumeric,NumericRoundingFalseTrueFalseTrueFalse[3, 0, 1][1, 2, 4]plain
City.000004CityConditionalTrueTrueTrueFalseFalse[0, 1, 2][2, 3, 4]plain
City.000012CitySubstringTrueTrueTrueFalseFalse[0, 1, 2][2, 3, 4]plain
Currency.000003CurrencyNumeric,SubstringTrueTrueTrueFalseFalse[0, 1, 2][0, 1, 2]plain
Currency.000004CurrencyNumeric,SubstringTrueTrueTrueFalseFalse[0, 1, 2][1, 2, 3]plain
DateTime.000007DateTimeDateTimeTrueTrueTrueFalseFalse[0, 1, 2][1, 2, 4]plain
DateTime.000015DateTimeDateTimeFalseFalseFalseFalseFalse[1, 2, 3][1, 2, 4]plain
DateTime.000022DateTimeDateTimeTrueTrueTrueFalseFalse[0, 1, 2][0, 1, 2]plain
DateTime.000023DateTimeDateTimeFalseFalseFalseFalseFalse[0, 1, 2][0, 2, 3]plain
DateTime.000026DateTimeDateTimeFalseFalseFalseFalseFalse[0, 1, 2][1, 2, 4]consistency
DateTime.000028DateTimeDateTimeFalseFalseFalseFalseFalse[1, 4, 0][0, 1, 3]plain
DateTime.000029DateTimeDateTimeFalseFalseFalseFalseFalse[0, 2, 4][0, 1, 4]plain
DateTime.000034DateTimeDateTime,SubstringTrueTrueTrueFalseFalse[0, 1, 2][0, 2, 4]plain
DateTime.000090DateTimeDateTimeTrueTrueFalseFalseFalse[0, 1, 2][2, 3, 4]plain
DateTime.000092DateTimeDateTimeTrueTrueTrueFalseFalse[0, 1, 2][0, 1, 3]plain
DateTime.000107DateTimeDateTimeTrueTrueTrueFalseFalse[0, 1, 2][0, 2, 3]plain
DateTime.000109DateTimeDateTimeFalseFalseFalseFalseFalse[0, 1, 3][0, 1, 4]plain
DateTime.000115DateTimeDateTimeRange,DateTimeRounding,DateTimeFalseFalseFalseFalseFalse[0, 1, 2][0, 1, 3]plain
Language.000002LanguageMulticolumn,SubstringTrueTrueTrueFalseFalse[0, 1, 2][1, 3, 4]plain
Meteorite.000001MeteoriteCasing,Concatenation,DateTime,Multicolumn,Numeric,SubstringFalseFalseFalseFalseFalse[3, 0, 1][1, 2, 4]consistency
Number.000011NumberNumeric,NumericRoundingTrueTrueTrueFalseFalse[0, 1, 2][1, 2, 3]plain
Number.000082NumberNumericTrueTrueTrueFalseFalse[0, 1, 2][1, 3, 4]consistency
Number.000084NumberNumeric,NumericRange,NumericRoundingFalseFalseFalseFalseFalse[3, 0, 1][2, 3, 4]plain
Number.000087NumberNumeric,NumericRange,NumericRoundingTrueTrueTrueFalseFalse[0, 1, 2][0, 3, 4]plain
Number.000088NumberSubstringTrueTrueTrueFalseFalse[0, 1, 2][0, 1, 2]plain
Phone.000003PhoneSubstringTrueTrueTrueFalseFalse[0, 1, 2][1, 2, 3]plain
Phone.000005PhoneSubstringTrueTrueTrueFalseFalse[0, 1, 2][1, 2, 4]plain
Phone.000006PhoneSubstringTrueTrueTrueFalseFalse[0, 1, 2][1, 2, 4]plain
State.000003StateSubstringTrueTrueTrueFalseFalse[0, 1, 2][0, 2, 4]plain
UserAgent.000006UserAgentSubstringTrueTrueTrueFalseFalse[0, 1, 2][0, 2, 3]plain

Acquisition Diagnostics

task_idfamilyfeaturesactive1active3diverse1diverse3random1random3loo_variantloo_plainloo_formatloo_consistency
City.000004CityConditional[0][0, 1, 2][0][0, 1, 2][2][2, 3, 4]plain75.0%75.0%75.0%
City.000012CitySubstring[0][0, 1, 2][2][2, 0, 4][2][2, 3, 4]plain100.0%100.0%100.0%
Currency.000003CurrencyNumeric,Substring[0][0, 1, 2][0][0, 1, 2][2][0, 1, 2]plain100.0%100.0%100.0%
Currency.000004CurrencyNumeric,Substring[0][0, 1, 2][0][0, 1, 2][2][1, 2, 3]plain100.0%100.0%100.0%
DateTime.000007DateTimeDateTime[0][0, 1, 2][0][0, 1, 2][2][1, 2, 4]plain100.0%100.0%100.0%
DateTime.000015DateTimeDateTime[1][1, 2, 3][4][4, 3, 0][4][1, 2, 4]plain0.0%0.0%0.0%
DateTime.000022DateTimeDateTime[0][0, 1, 2][3][3, 0, 1][2][0, 1, 2]plain75.0%75.0%50.0%
DateTime.000023DateTimeDateTime[0][0, 1, 2][3][3, 0, 1][0][0, 2, 3]plain75.0%75.0%75.0%
DateTime.000026DateTimeDateTime[0][0, 1, 2][3][3, 0, 1][1][1, 2, 4]consistency75.0%75.0%100.0%
DateTime.000028DateTimeDateTime[1][1, 4, 0][3][3, 2, 0][1][0, 1, 3]plain0.0%0.0%0.0%
DateTime.000029DateTimeDateTime[0][0, 2, 4][3][3, 0, 1][1][0, 1, 4]plain25.0%0.0%25.0%
DateTime.000034DateTimeDateTime,Substring[0][0, 1, 2][0][0, 1, 2][0][0, 2, 4]plain100.0%100.0%100.0%
DateTime.000090DateTimeDateTime[0][0, 1, 2][0][0, 1, 2][3][2, 3, 4]plain50.0%50.0%50.0%
DateTime.000092DateTimeDateTime[0][0, 1, 2][0][0, 1, 2][3][0, 1, 3]plain100.0%100.0%100.0%
DateTime.000107DateTimeDateTime[0][0, 1, 2][0][0, 1, 2][0][0, 2, 3]plain100.0%100.0%100.0%
DateTime.000109DateTimeDateTime[0][0, 1, 3][0][0, 1, 2][4][0, 1, 4]plain25.0%25.0%25.0%
DateTime.000115DateTimeDateTimeRange,DateTimeRounding,DateTime[0][0, 1, 2][0][0, 1, 2][1][0, 1, 3]plain100.0%100.0%100.0%
Language.000002LanguageMulticolumn,Substring[0][0, 1, 2][2][2, 4, 3][4][1, 3, 4]plain100.0%100.0%100.0%
Meteorite.000001MeteoriteCasing,Concatenation,DateTime,Multicolumn,Numeric,Substring[3][3, 0, 1][3][3, 1, 4][2][1, 2, 4]consistency0.0%0.0%25.0%
Number.000011NumberNumeric,NumericRounding[0][0, 1, 2][1][1, 0, 2][3][1, 2, 3]plain75.0%75.0%75.0%
Number.000017NumberNumeric,NumericRounding[3][3, 0, 1][1][1, 0, 2][1][1, 2, 4]plain50.0%50.0%25.0%
Number.000082NumberNumeric[0][0, 1, 2][0][0, 1, 2][3][1, 3, 4]consistency75.0%75.0%100.0%
Number.000084NumberNumeric,NumericRange,NumericRounding[3][3, 0, 1][3][3, 0, 1][2][2, 3, 4]plain100.0%100.0%75.0%
Number.000087NumberNumeric,NumericRange,NumericRounding[0][0, 1, 2][0][0, 1, 2][0][0, 3, 4]plain100.0%75.0%75.0%
Number.000088NumberSubstring[0][0, 1, 2][2][2, 0, 4][2][0, 1, 2]plain100.0%100.0%100.0%
Phone.000003PhoneSubstring[0][0, 1, 2][0][0, 1, 2][2][1, 2, 3]plain100.0%100.0%100.0%
Phone.000005PhoneSubstring[0][0, 1, 2][3][3, 0, 1][2][1, 2, 4]plain100.0%75.0%100.0%
Phone.000006PhoneSubstring[0][0, 1, 2][3][3, 2, 4][2][1, 2, 4]plain100.0%100.0%100.0%
State.000003StateSubstring[0][0, 1, 2][2][2, 0, 4][4][0, 2, 4]plain100.0%100.0%100.0%
UserAgent.000006UserAgentSubstring[0][0, 1, 2][0][0, 1, 2][3][0, 2, 3]plain100.0%100.0%100.0%

Family Breakdown

methodfamilytasksrow_exactfull_task_exact
active1_plainCity2100.0%100.0%
active1_plainCurrency2100.0%100.0%
active1_plainDateTime1374.4%53.8%
active1_plainLanguage1100.0%100.0%
active1_plainMeteorite133.3%0.0%
active1_plainNumber688.9%66.7%
active1_plainPhone3100.0%100.0%
active1_plainState1100.0%100.0%
active1_plainUserAgent1100.0%100.0%
active3_plainCity2100.0%100.0%
active3_plainCurrency2100.0%100.0%
active3_plainDateTime1369.2%46.2%
active3_plainLanguage1100.0%100.0%
active3_plainMeteorite166.7%0.0%
active3_plainNumber694.4%83.3%
active3_plainPhone3100.0%100.0%
active3_plainState1100.0%100.0%
active3_plainUserAgent1100.0%100.0%
active3_shuffled_labelsCity2100.0%100.0%
active3_shuffled_labelsCurrency2100.0%100.0%
active3_shuffled_labelsDateTime1366.7%38.5%
active3_shuffled_labelsLanguage1100.0%100.0%
active3_shuffled_labelsMeteorite133.3%0.0%
active3_shuffled_labelsNumber666.7%50.0%
active3_shuffled_labelsPhone3100.0%100.0%
active3_shuffled_labelsState1100.0%100.0%
active3_shuffled_labelsUserAgent1100.0%100.0%
base_plainCity2100.0%100.0%
base_plainCurrency2100.0%100.0%
base_plainDateTime1369.2%46.2%
base_plainLanguage1100.0%100.0%
base_plainMeteorite133.3%0.0%
base_plainNumber683.3%66.7%
base_plainPhone3100.0%100.0%
base_plainState1100.0%100.0%
base_plainUserAgent1100.0%100.0%
diverse1_plainCity2100.0%100.0%
diverse1_plainCurrency2100.0%100.0%
diverse1_plainDateTime1364.1%38.5%
diverse1_plainLanguage1100.0%100.0%
diverse1_plainMeteorite133.3%0.0%
diverse1_plainNumber688.9%66.7%
diverse1_plainPhone3100.0%100.0%
diverse1_plainState1100.0%100.0%
diverse1_plainUserAgent1100.0%100.0%
diverse3_plainCity2100.0%100.0%
diverse3_plainCurrency2100.0%100.0%
diverse3_plainDateTime1366.7%38.5%
diverse3_plainLanguage1100.0%100.0%
diverse3_plainMeteorite133.3%0.0%
diverse3_plainNumber694.4%83.3%
diverse3_plainPhone3100.0%100.0%
diverse3_plainState1100.0%100.0%
diverse3_plainUserAgent1100.0%100.0%
loo_portfolio_baseCity2100.0%100.0%
loo_portfolio_baseCurrency2100.0%100.0%
loo_portfolio_baseDateTime1369.2%46.2%
loo_portfolio_baseLanguage1100.0%100.0%
loo_portfolio_baseMeteorite133.3%0.0%
loo_portfolio_baseNumber683.3%66.7%
loo_portfolio_basePhone3100.0%100.0%
loo_portfolio_baseState1100.0%100.0%
loo_portfolio_baseUserAgent1100.0%100.0%
oracle_among_tested_acquisitionsCity2100.0%100.0%
oracle_among_tested_acquisitionsCurrency2100.0%100.0%
oracle_among_tested_acquisitionsDateTime1376.9%53.8%
oracle_among_tested_acquisitionsLanguage1100.0%100.0%
oracle_among_tested_acquisitionsMeteorite166.7%0.0%
oracle_among_tested_acquisitionsNumber694.4%83.3%
oracle_among_tested_acquisitionsPhone3100.0%100.0%
oracle_among_tested_acquisitionsState1100.0%100.0%
oracle_among_tested_acquisitionsUserAgent1100.0%100.0%
order1_plainCity2100.0%100.0%
order1_plainCurrency2100.0%100.0%
order1_plainDateTime1371.8%53.8%
order1_plainLanguage1100.0%100.0%
order1_plainMeteorite166.7%0.0%
order1_plainNumber688.9%66.7%
order1_plainPhone3100.0%100.0%
order1_plainState1100.0%100.0%
order1_plainUserAgent1100.0%100.0%
order3_plainCity2100.0%100.0%
order3_plainCurrency2100.0%100.0%
order3_plainDateTime1371.8%46.2%
order3_plainLanguage1100.0%100.0%
order3_plainMeteorite133.3%0.0%
order3_plainNumber688.9%66.7%
order3_plainPhone3100.0%100.0%
order3_plainState1100.0%100.0%
order3_plainUserAgent1100.0%100.0%
random1_plainCity2100.0%100.0%
random1_plainCurrency2100.0%100.0%
random1_plainDateTime1366.7%46.2%
random1_plainLanguage1100.0%100.0%
random1_plainMeteorite133.3%0.0%
random1_plainNumber688.9%66.7%
random1_plainPhone3100.0%100.0%
random1_plainState1100.0%100.0%
random1_plainUserAgent1100.0%100.0%
random3_plainCity2100.0%100.0%
random3_plainCurrency2100.0%100.0%
random3_plainDateTime1366.7%38.5%
random3_plainLanguage1100.0%100.0%
random3_plainMeteorite133.3%0.0%
random3_plainNumber688.9%66.7%
random3_plainPhone3100.0%100.0%
random3_plainState1100.0%100.0%
random3_plainUserAgent1100.0%100.0%

Files

  • runs/main_final/generations.csv
  • runs/main_final/method_details.csv
  • runs/main_final/acquisition_details.csv
  • runs/main_final/summary.csv
  • analysis/summary.csv
  • analysis/method_details.csv
  • analysis/acquisition_details.csv
  • analysis/family_summary.csv

Experiment log 6

Show the running log (6 entries, 2026-06-27)

2026-06-27

  • Created standalone experiment directory.
  • Planned active clarifying-example acquisition benchmark with Markdown/HTML report and charts.

Run smoke_v1

  • Tasks: 4
  • Generation records: 168
  • base_plain: 50.0% full-task exact.
  • active1_plain: 75.0% full-task exact.
  • active3_plain: 75.0% full-task exact.
  • random3_plain: 50.0% full-task exact.
  • active3_shuffled_labels: 50.0% full-task exact.

Run main_v1

  • Tasks: 30
  • Generation records: 1620
  • base_plain: 66.7% full-task exact.
  • active1_plain: 70.0% full-task exact.
  • active3_plain: 70.0% full-task exact.
  • random3_plain: 63.3% full-task exact.
  • active3_shuffled_labels: 60.0% full-task exact.

Run main_v2_diverse

  • Tasks: 30
  • Generation records: 1800
  • base_plain: 66.7% full-task exact.
  • active1_plain: 70.0% full-task exact.
  • active3_plain: 70.0% full-task exact.
  • diverse3_plain: 66.7% full-task exact.
  • random3_plain: 63.3% full-task exact.
  • active3_shuffled_labels: 60.0% full-task exact.

Run main_final

  • Tasks: 30
  • Generation records: 1890
  • base_plain: 66.7% full-task exact.
  • active1_plain: 70.0% full-task exact.
  • active3_plain: 70.0% full-task exact.
  • diverse3_plain: 66.7% full-task exact.
  • random3_plain: 63.3% full-task exact.
  • active3_shuffled_labels: 60.0% full-task exact.

Run main_final

  • Tasks: 30
  • Generation records: 1890
  • base_plain: 66.7% full-task exact.
  • active1_plain: 70.0% full-task exact.
  • active3_plain: 70.0% full-task exact.
  • diverse3_plain: 66.7% full-task exact.
  • random3_plain: 63.3% full-task exact.
  • active3_shuffled_labels: 60.0% full-task exact.

Figures 6

budget comparison
budget comparison · analysis/figures/
family heatmap
family heatmap · analysis/figures/
method full task exact
method full task exact · analysis/figures/
probe entropy distribution
probe entropy distribution · analysis/figures/
row vs full task
row vs full task · analysis/figures/
wins losses vs base
wins losses vs base · analysis/figures/

Data files 10

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗