Active Evidence Acquisition

Choose or synthesize the next visible example, probe, test, or trace that collapses uncertainty.

What we have learned

Seed Experiments

Current Read

Active evidence has visible promise but modest lift in current prototypes. The next phase should optimize the evidence channel for downstream selection, not just direct answer improvement.

Scorecard

Program: charter
Current read: active examples and probes are promising, but the acquisition objective must be tied to downstream decision quality.
Best next experiment: compare active probe policies by downstream selector lift under a fixed evidence budget.
Strong anchors: qwen_active_example_acquisition, qwen35_4b_active_counterexample_trace_selection, qwen35_4b_learned_active_trace_policy.
Avoid repeating: optimizing probe informativeness without showing that decisions improve.
Evidence that advances the program: budget-normalized gains from probes that do not rely on expected answers at deployment time.

Charter

Show charter.md

Purpose

Choose or synthesize the next example, probe, trace, query, test, or tool action that most reduces uncertainty for a small model or selector.

Why This Is A Program

The initial active-learning experiments are prototypes for a much broader loop: systems should spend scarce evidence budget where it changes decisions, not where it merely adds context.

Progress Signals

A budgeted policy beats random, order, diversity-only, and oracle-naive baselines.
The selected evidence improves downstream decisions, not just intermediate confidence.
Acquisition policies transfer across families or expose why they do not.
Generated probes help without hidden expected answers.

Boundaries

This program owns evidence gathering. The final commit/abstain decision belongs to Evidence-Conditioned Selection.

Backlog

Show backlog.md

Next Experiments

Compare uncertainty, disagreement, information gain, and learned policies on the same acquisition pool.
Generate expected-output-free tests where candidates can be compared by agreement or invariants.
Train family-aware acquisition policies for date/time, numeric, table, and code tasks.
Measure acquisition value under strict token or tool budgets.
Feed acquired evidence into selector training rather than direct prompting only.

Required Controls

Random acquisition.
Input-diversity acquisition.
Fixed-order acquisition.
Oracle among tested acquisitions, clearly labeled.

Stop Conditions

Do not count an acquisition policy as successful if it helps only because it leaks hidden labels or increases budget without improving selection quality per unit evidence.

Experiments 16

2026-07-13 Semantic-policy headroom tournament
No rule produced a stable training target after a failed test: every one of 72 cases reached a correct patch. But before seeing test output, the model got zero of 54 ambiguous first patches fully right and never opened t
2026-07-13 Counterfactual evidence-acquisition curriculum
The run stopped at its first checkpoint-compatibility test. Across 48 frozen contexts, the starting checkpoint differed from the comparison anchor by 0.110735 centered logits, above the fixed 0.100000 limit. Entropy stay
2026-07-11 Interactive policy curriculum: oracle DAgger to execution-reward RL
No — it backfired badly. Success on the practice tasks fell from about 60% to 35%, and on tasks never touched in training from about 69% to 35%. It kept editing fluently but stopped running its checks: the verify-and-com
2026-07-10 Qwen3.5-4B Verified Macro Invention Long-Context Rerun
No. The model was not short on thinking space, it was stuck repeating itself, and more space only bought longer loops. Handed the blueprint, all 16 tasks finished cleanly. Asked to invent the program from examples alone,
2026-06-28 Qwen Oracle-Distilled Acquisition Policy
No. The trained picker does read real signal: it beats revealing nothing (50 to 57 percent of tasks fully solved) and crushes a version fed scrambled answers (27 percent). But it lost to simply grabbing a varied spread o
2026-06-28 Qwen Learned Active Interrogation
No. Unlocking four extra answer keys did raise fully-solved tasks from 63% to 70%, and scrambling those answers sank it to 47%, so real labels genuinely matter. But the model's deliberate picks scored exactly the same 70
imported 2026-07-12 Sampled Query Filter Executor Experiment
Yes. Graded on just one sampled final answer per problem, the model rebuilt the entire set of still-possible number pairs it was never shown, capturing 94 to 98 percent of it, because holding that full set is the cheapes
imported 2026-07-12 Query Filter Executor Experiment
Yes, and that is the surprise. Trained only to name one final answer, the model taught itself to run each instruction in order and track every still-possible pair of the two hidden numbers. Given enough internal steps, i
imported 2026-07-12 Dense Latent Query Executor Experiment
Partly. Giving the model enough internal thinking steps to walk through the whole program lifts accuracy sharply, and it clearly beats a model that reads everything in one glance. But the memory stays approximate: at its
2026-06-27 Pairwise Table Judge
No. Letting the model run a knockout comparison among its own candidate tables did not help: fully-correct tables slipped from 50% (just trusting its first attempt) to 47.5%. On the hard cases where a correct table exist
2026-06-27 Qwen Active Example Acquisition
Barely, and the real lesson is the downside. Letting the model request the single example it was most unsure about lifted fully-solved tasks from 20 to 21 out of 30. But three random extras dropped it to 19, and feeding
2026-06-27 Qwen Active Crystallizer Public Gate
No. Using the model's votes to choose a rule worked on 25% of tasks — barely above the 22.5% you get from scrambled, meaningless votes, and it never beat the best rule the candidate pool could offer. The model answered i
2026-06-24 Qwen3.5-4B Learned Active Trace Policy
It depends. On the main test set a simple even-splitting rule beat the trained picker after one extra input — 91 percent of programs fully correct versus 87 — and the best-possible choice reached 97 percent. The picker d
2026-06-24 Qwen3.5-4B Adaptive Evidence Budget Policy
Yes. Trained just for this single decision, the model matched the accuracy of running all ten checks — solving about 92 of every 100 tasks — while using only about five checks on average, near a perfect-hindsight stopper
2026-06-24 Qwen3.5-4B Active Counterexample Trace Selection
Yes, but choose them well. Committing on the visible examples alone left about a quarter of picks secretly wrong, even though every one passed all the examples shown. Requesting six new test cases where the surviving pro
2026-06-22 Qwen 3.5 4B Unsaturated Frontier Active Bridge
Spread evenly. Giving each of ten problem types the same six extra correction examples let the model fully fix 98% of hard cases. Piling those same examples onto whichever types it failed most reached only 85%, and starv

Claims

Promising C4 · Active evidence must be coupled to downstream decisions