Future experiment queue

Hypothesis. Explicit abstention labels and coverage reporting will separate useful evidence selection from silent cherry-picking.

Minimal protocol. Build one shared candidate pool with visible-only features, train selector variants, and report precision, recall, coverage, and selected accuracy by family.

Success signal. A visible-only selector beats first-visible, shortest-visible, and random-order baselines at matched coverage.

Failure signal. Accuracy gains disappear when coverage is matched or family-held-out evaluation is applied.

Avoid. Do not use hidden expected answers as features or training labels unless clearly reported as an oracle ceiling.

Next step. Run idea intake against selector and oracle-ceiling anchor experiments.

Expected artifacts.

• candidate_pool_manifest
• selector_metrics
• abstention_slices
• report

Hypothesis. Agreement, invariants, metamorphic checks, and execution consistency can improve selection while preserving deployability.

Minimal protocol. Generate candidate-specific tests with no hidden answer key, compare by invariants and cross-candidate agreement, then evaluate selected outputs only after commit.

Success signal. Selection improves over random acquisition and fixed-order acquisition under the same evidence budget.

Failure signal. The method helps only when hidden answers or task-specific leaks enter the evidence loop.

Avoid. Do not treat post-hoc oracle scoring as deployable evidence.

Next step. Start from active counterexample and model-in-loop counterexample experiments.

Expected artifacts.

• test_generation_manifest
• evidence_budget_table
• hidden_boundary_note
• report

Hypothesis. A standard scorecard will reveal reusable mechanisms and retire variants that depend on non-deployable evidence.

Minimal protocol. Select high-impact oracle-ceiling experiments, annotate evidence visibility, baselines, artifact location, and deployable follow-up.

Success signal. Every reviewed ceiling result has a clear keep, rerun, or retire decision.

Failure signal. Scorecards cannot distinguish mechanism promise from hidden-label dependence.

Avoid. Do not rewrite old results as deployable when they were explicitly oracle-bounded.

Next step. Seed from selector, code ABI, and coverage-gate reports.

Expected artifacts.

• scorecard_table
• claim_updates
• decision_record

Hypothesis. Typed state helps most under compositional shifts, while latent state helps only when supervision and task families align.

Minimal protocol. Run direct text, typed bytecode, latent slot, and execution-feedback variants on one shared suite with length, family, and paraphrase shifts.

Success signal. At least one representation has a reproducible advantage across non-IID splits and matched budget.

Failure signal. All gains collapse under shared splits or shuffled/corrupted state controls.

Avoid. Do not compare representations on different task mixes or budgets.

Next step. Use compiler, latent executor, and structured slot anchors to define the suite.

Expected artifacts.

• shared_suite_manifest
• representation_configs
• shift_metrics
• report

Hypothesis. Some strong results are robust to seed changes but brittle to family and paraphrase shifts.

Minimal protocol. Replicate top compiler-style experiments across a small grid of seeds and shifts with identical reporting.

Success signal. The grid identifies stable effects and brittle effects without requiring a new mechanism.

Failure signal. Replication variance overwhelms claimed mechanism differences.

Avoid. Do not add a new model variant until the shift grid is defined.

Next step. Pick three anchor experiments from the compiler program scorecard.

Expected artifacts.

• replication_manifest
• shift_taxonomy
• metrics_grid
• report

Hypothesis. Memory is most useful when converted into constraints or tests rather than appended as context.

Minimal protocol. Retrieve the same source items, transform them into multiple memory modes, and evaluate on matched held-out families.

Success signal. Constraint or test memory beats prompt memory, corrupted retrieval, and no-memory baselines.

Failure signal. All memory modes underperform direct inference or rely on same-family leakage.

Avoid. Do not treat larger prompts as a free budget increase.

Next step. Use independent retrieval and feature-factorized rule experiments as anchors.

Expected artifacts.

• memory_schema
• retrieval_manifest
• mode_metrics
• report

Hypothesis. Learned policies help only when trained on features that transfer across task families.

Minimal protocol. Build one acquisition pool and run random, diversity, disagreement, information gain, and learned policies with fixed budgets.

Success signal. A policy improves selection quality per unit evidence across at least two families.

Failure signal. The best policy changes unpredictably across families or loses to fixed-order acquisition.

Avoid. Do not compare policies on different candidate or evidence pools.

Next step. Derive pool features from active trace policy and adaptive evidence budget experiments.

Expected artifacts.

• acquisition_pool
• policy_configs
• budget_curve
• report

Hypothesis. A STOP/MORE policy can beat fixed-budget and always-continue heuristics when trained on deployable features.

Minimal protocol. Define tool-state features, train STOP/MORE policies without hidden correctness labels at decision time, and compare against fixed budgets.

Success signal. The learned controller improves selected accuracy per cost under matched evidence constraints.

Failure signal. The controller only wins by using hidden labels or by silently spending more budget.

Avoid. Do not let hidden evaluation correctness enter runtime state.

Next step. Anchor on adaptive tool controller and adaptive evidence budget policy experiments.

Expected artifacts.

• tool_state_schema
• controller_configs
• cost_accuracy_curve
• report

Hypothesis. Method ranking changes when all methods share data, model, budget, controls, and held-out families.

Minimal protocol. Prepare one substrate, train method variants with frozen and shuffled-label controls, and evaluate on IID plus held-out splits.

Success signal. One method shows a robust advantage over frozen, shuffled, and weak-baseline controls.

Failure signal. All gains are explained by data selection, extra tokens, or weak baselines.

Avoid. Do not check adapters or model checkpoints into git.

Next step. Use DPO, DAgger, and distillation experiments to define shared data.

Expected artifacts.

• training_manifest
• artifact_manifest
• method_configs
• heldout_metrics
• report

Hypothesis. A standardized manifest is enough to preserve provenance while keeping large artifacts external.

Minimal protocol. Audit experiments with training outputs, require external artifact pointers, model IDs, data hashes, and reproduction notes.

Success signal. All trained-run folders either have manifests or explicit no-large-artifact notes.

Failure signal. Important trained outputs cannot be located or reproduced without local private files.

Avoid. Do not add model weights, adapters, or checkpoints to git.

Next step. Use artifact manifest index and readiness matrix to prioritize missing manifests.

Expected artifacts.

• artifact_manifest_updates
• audit_table
• quality_gate_update

Hypothesis. A lightweight text and metadata linter can prevent hidden-label confusion without slowing pilots.

Minimal protocol. Add a required report section or metadata field, scan new experiments, and document accepted boundary labels.

Success signal. New experiments fail validation when hidden-label boundary documentation is missing.

Failure signal. The rule creates noisy failures or misses obvious oracle/deployable ambiguity.

Avoid. Do not require heavy formal proof for early pilots.

Next step. Prototype against high-impact selector and oracle-ceiling reports.

Expected artifacts.

• validation_rule
• template_update
• boundary_docs

Hypothesis. Program indexes, claims, readiness, and related-work search reduce duplicate proposals and improve anchor selection.

Minimal protocol. Give agents held-out rough ideas, compare raw rg search against documented discovery workflow, and score anchor quality and time.

Success signal. The documented workflow finds better anchors with lower duplicate rate.

Failure signal. Agents ignore or cannot use the generated knowledge surfaces.

Avoid. Do not evaluate only on ideas already named in file titles.

Next step. Use existing discovery workflow and related-work CLI.

Expected artifacts.

• navigation_tasks
• agent_logs
• duplicate_rate_metrics
• report

Hypothesis. The round-2 model's collapsed truncation cascade raises harvest yield everywhere, opening the three starved families and buying a second menagerie step; compounding will be smaller than round 2's jump because the emission-seam repair is one-time.

Minimal protocol. Fast-profile re-harvest with the round-2 merged model (same seeds namespace, K=2), rebuild SFT with the round-2 recipe (recovery arm + weighted loss), train round-3 adapter from BASE on the union data, then paired quick events on two fresh seeds via the HF backend plus the recovery-arm-only and breadth-vs-matched-dose ablations.

Success signal. Round-3 harvest keepers in glyphgate/loomfix/stallwright at 3x round-1 levels; menagerie quick delta over base grows beyond round 2's +0.22/+0.29 mean on fresh paired seeds.

Failure signal. Yield rises but menagerie plateaus (emission-seam repair was the whole effect) or gym-internal gains stop transferring (locality reappears above the protocol layer).

Avoid. Never evaluate adapters through vLLM runtime LoRA (C49); never mix HF-backend and vLLM-backend scores; keep menagerie seeds fresh per event.

Next step. Run the fast-profile re-harvest with the round-2 merged model once the fla-kernel medium-tier fault on the host is triaged.

Expected artifacts.

• round-3 harvest yield tables
• paired menagerie quick/medium events
• ablation table separating recovery-arm from breadth effects

Hypothesis. Consensus helps when failures are uncorrelated, while learned selectors help when visible evidence is structured but sparse.

Minimal protocol. Create matched candidate pools with independent generation, learned selector scores, and consensus scores; evaluate under family holdouts.

Success signal. The comparison identifies regimes where consensus, selection, or hybrid routing dominates.

Failure signal. Consensus gains vanish after controlling for extra samples or shared prompt artifacts.

Avoid. Do not compare methods with unequal sample budgets.

Next step. Use existing independent retrieval and counterexample selection anchors as prior evidence.

Expected artifacts.

• matched_candidate_pool
• consensus_metrics
• selector_metrics
• routing_table

Hypothesis. State supervision contributes only when state variables map cleanly to task factors.

Minimal protocol. Run matched ablations with final-answer only, state-prefix, program-token, shuffled-state, and corrupted-state supervision.

Success signal. A named supervision source survives shuffled and corrupted controls.

Failure signal. Lift tracks data filtering or token budget rather than the intended representation.

Avoid. Do not infer causality from a single no-control training run.

Next step. Start from structured slot and dense supervision ladder anchors.

Expected artifacts.

• ablation_configs
• control_metrics
• diagnostic_slices
• report

Hypothesis. Failure-case memory can create discriminating tests while reducing direct-answer contamination.

Minimal protocol. Retrieve similar verified failures, extract constraints, generate candidate-specific counterexamples, and evaluate selection impact.

Success signal. Generated counterexamples improve selection under visible-only constraints.

Failure signal. Counterexamples are either redundant with public tests or leak hidden labels.

Avoid. Do not copy expected outputs from retrieved examples into the new task.

Next step. Run related-work discovery for counterexample memory and active acquisition.

Expected artifacts.

• failure_memory_bank
• generated_tests
• selection_metrics
• report

Hypothesis. Examples, invariants, aliases, failure modes, and type signatures are enough for shortlisting and debugging.

Minimal protocol. Define a small card schema, encode one existing operator bank, and test retrieval and shortlisting against a baseline bank.

Success signal. Cards improve diagnosis or top-k operator retrieval without increasing manual ambiguity.

Failure signal. The schema is too heavy or fails to support held-out operator lookup.

Avoid. Do not create a schema that only fits one existing task family.

Next step. Sample operator inventory experiments and extract common fields.

Expected artifacts.

• operator_card_schema
• encoded_bank
• retrieval_metrics
• report

Hypothesis. A learned top-k shortlister will beat random and heuristic search when trained with held-out primitive families.

Minimal protocol. Scale bank size, train top-k shortlisters, compare full-search oracle, random shortlister, and closed-vocabulary baselines.

Success signal. Top-k cost falls while deployable selected accuracy stays above heuristic baselines.

Failure signal. Coverage rises but selected accuracy drops without a reliable selection plan.

Avoid. Do not report coverage without selected accuracy and search cost.

Next step. Use inventory shortlister and scaling-stress experiments as anchors.

Expected artifacts.

• operator_bank_manifest
• shortlister_configs
• cost_accuracy_curve
• report

Hypothesis. Family-aware features improve evidence value, but only when family labels are coarse and deployable.

Minimal protocol. Train policy variants with no family, coarse family, and oracle family features; evaluate on held-out family mixtures.

Success signal. Coarse deployable family features improve budget-normalized selection over no-family controls.

Failure signal. Only oracle family labels help or performance collapses on held-out families.

Avoid. Do not use benchmark split identity as a deployable feature.

Next step. Use current active evidence experiments to define feature families.

Expected artifacts.

• family_feature_manifest
• heldout_metrics
• budget_table
• report

Hypothesis. Robust controllers should learn when to distrust or repeat tool evidence, not just call tools more often.

Minimal protocol. Inject controlled tool noise, latency, and misleading examples into an existing tool loop and compare learned, heuristic, and oracle policies.

Success signal. A controller preserves accuracy-cost tradeoffs under noise better than fixed and always-tool baselines.

Failure signal. The policy overuses noisy tools or transfers poorly across noise regimes.

Avoid. Do not evaluate only on clean tools after training with noise.

Next step. Use process-control experiments with explicit tool states as anchors.

Expected artifacts.

• noise_manifest
• tool_state_logs
• robustness_metrics
• report

Hypothesis. Hard-negative training helps only when negatives represent held-out families and not just surface artifacts.

Minimal protocol. Train with mined hard negatives, shuffled negatives, and random negatives; evaluate coverage, selected accuracy, and false positives on held-out families.

Success signal. Mined negatives reduce false positives without lowering recall or transfer.

Failure signal. The model learns benchmark-specific artifacts and fails held-out negative families.

Avoid. Do not report aggregate accuracy without false-positive slices.

Next step. Start from offline hard-negative and constrained coverage DPO anchors.

Expected artifacts.

• negative_manifest
• training_configs
• false_positive_slices
• report

Hypothesis. A small standard slice set will change follow-up decisions more often than ad hoc diagnostics.

Minimal protocol. Define slices by operator, family, length, parse status, evidence state, and artifact boundary; apply to three anchor reports.

Success signal. The slices reveal at least one concrete branch, stop, or rerun decision per anchor.

Failure signal. Slices are descriptive but do not change any experiment decision.

Avoid. Do not add diagnostics without a decision they can change.

Next step. Apply to one compiler, one selector, and one posttraining anchor.

Expected artifacts.

• diagnostic_template
• slice_tables
• decision_notes
• report

Hypothesis. A scorecard over seeds, data, commands, artifacts, and boundary labels will prioritize curation work.

Minimal protocol. Score top anchor experiments from claims and program scorecards, then add curation tasks for missing pieces.

Success signal. Each high-impact anchor has a reproducibility grade and concrete repair task.

Failure signal. Scorecards duplicate the readiness matrix without adding decision value.

Avoid. Do not block exploratory work on full reproducibility unless it is used as shared evidence.

Next step. Use experiment readiness and claim evidence counts to select anchors.

Expected artifacts.

• scorecard_schema
• anchor_scores
• curation_backlog

Hypothesis. A sample-based audit against catalog, claims, and future queue will catch most near duplicates.

Minimal protocol. Sample proposed ideas, run related-work search, compare against human duplicate judgments, and update intake requirements.

Success signal. The audit catches near duplicates without suppressing genuinely new variants.

Failure signal. Duplicate warnings are too noisy or miss obvious repeated ideas.

Avoid. Do not equate shared topic tags with duplicate mechanisms.

Next step. Generate cases from future queue items and recent scaffolded ideas.

Expected artifacts.

• dedupe_cases
• related_search_metrics
• intake_update

Hypothesis. Structured intermediate programs will transfer better than direct vision-language answers on held-out visual table layouts.

Minimal protocol. Adapt one table-transform task to image-rendered tables, compare direct answer, extracted structure, and executable intermediate variants.

Success signal. Executable intermediates improve held-out layout accuracy without manual table extraction leakage.

Failure signal. The gain depends entirely on an external OCR or parser oracle.

Avoid. Do not hide vision parsing errors by evaluating only on gold extracted tables.

Next step. Run intake using Foofah and table-transform anchors.

Expected artifacts.

• visual_table_dataset
• extraction_boundary_note
• shift_metrics
• report

Hypothesis. Failure-mined curricula transfer better than broad model-generated curricula when held-out primitives are explicit.

Minimal protocol. Generate three curriculum sources for one held-out primitive suite and compare transfer under matched data volume and model budget.

Success signal. One curriculum source improves held-out performance without hurting direct baselines.

Failure signal. Curriculum gains are IID-only or disappear under matched difficulty.

Avoid. Do not judge curricula only on training or IID validation.

Next step. Reuse bridge, recombination, and held-out primitive split patterns.

Expected artifacts.

• curriculum_manifest
• data_generation_config
• transfer_metrics
• report

Hypothesis. Role specialization helps only when roles contribute non-redundant evidence and share a strict cost accounting.

Minimal protocol. Assign roles on one candidate pool, compare single-policy, homogeneous ensemble, role ensemble, and oracle routing at equal cost.

Success signal. Role ensembles improve selected accuracy or coverage at matched budget.

Failure signal. All gains come from extra samples or correlated role outputs.

Avoid. Do not compare a multi-role system to a weaker single-model budget.

Next step. Use consensus and selector anchors to define the shared pool.

Expected artifacts.

• role_prompt_manifest
• budget_accounting
• ensemble_metrics
• report

Hypothesis. Some evidence mechanisms remain useful when converted into early-exit and small-k policies.

Minimal protocol. Rerun one selector/tool-control experiment with strict latency, token, and tool-call budgets plus early-exit policies.

Success signal. A constrained policy retains a meaningful fraction of the unconstrained gain over simple baselines.

Failure signal. The method collapses when evidence budget is capped or action latency is counted.

Avoid. Do not ignore preprocessing or tool-call latency.

Next step. Pick an existing adaptive controller with a clear smoke path.

Expected artifacts.

• latency_budget_manifest
• early_exit_metrics
• cost_accuracy_curve
• report

Hypothesis. A dashboard over programs, claims, readiness, artifacts, and future queue will reduce search time and duplicate proposals.

Minimal protocol. Build a static generated dashboard from existing CSV/JSON indexes and evaluate it with navigation tasks.

Success signal. Users or agents find higher-quality anchors faster than with raw Markdown navigation.

Failure signal. The dashboard duplicates existing docs without changing decisions.

Avoid. Do not require a server or external service for core navigation.

Next step. Start with generated data files already produced by make catalog.

Expected artifacts.

• static_dashboard
• navigation_eval
• report

Hypothesis. A visible-signal STOP/MORE policy over thinking tokens can match or beat a fixed budget at lower mean cost by spending more thinking only on hard inputs.

Minimal protocol. Reuse the thinking_budget_scaling MBPP harness; learn or threshold a per-task budget from visible features; compare deployable accuracy-vs-mean-thinking-tokens against fixed budgets and unbudgeted.

Success signal. Controller Pareto-dominates fixed budgets on deployable accuracy vs mean thinking tokens, beating a length/difficulty-blind baseline.

Failure signal. No improvement over the best fixed budget once mean token cost is matched.

Avoid. Do not let the controller read hidden test outcomes; budget decisions must use visible information only.

Next step. Anchor on qwen35_4b_thinking_budget_scaling and the evidence-budget controller experiments.

Expected artifacts.

• budget_policy
• accuracy_vs_tokens_pareto
• report

Hypothesis. Substituting a different task's thinking (removing token-presence, not just order) will collapse more of the gain than shuffling, isolating the genuine-reasoning share.

Minimal protocol. On the thinking_budget_scaling substrate, add controls: shuffled-thinking (token order), foreign-task thinking (token-presence removed), and matched-compute no-think; compare deployable accuracy at matched budgets.

Success signal. A clear ordering real > foreign > shuffled > no_think that quantifies the reasoning vs compute/scaffold split.

Failure signal. Foreign-task thinking matches real thinking, i.e. the gain is essentially compute + scaffold, not reasoning.

Avoid. Do not infer a content effect from a single budget; sweep and report noise.

Next step. Extend qwen35_4b_thinking_budget_scaling controls with foreign-task thinking.

Expected artifacts.

• control_conditions
• decomposition_table
• report

Hypothesis. A few targeted examples can disambiguate type-colliding operators better than passive shortlist scores.

Minimal protocol. Construct operator pairs with shared signatures, ask an acquisition policy for distinguishing probes, and evaluate downstream operator choice.

Success signal. Targeted probes improve operator selection per evidence token over random probes.

Failure signal. Disambiguation requires hidden outputs or more budget than full search.

Avoid. Do not count probes that reveal the hidden target operator directly.

Next step. Identify collisions from current operator inventory runs.

Expected artifacts.

• collision_bank
• probe_manifest
• selection_metrics
• report

Hypothesis. Preflight probes can detect capacity or tool-pressure failures before full training or large sweeps.

Minimal protocol. Run tiny diagnostic probes on candidate mechanisms, compare their signals with later full-run results, and estimate decision precision.

Success signal. Preflight results correctly predict stop or proceed decisions on multiple mechanisms.

Failure signal. Preflight signals have no relation to full-run outcomes.

Avoid. Do not treat preflight as evidence for final claims without full evaluation.

Next step. Use token-pressure and tool-control experiments as anchors.

Expected artifacts.

• preflight_probe_suite
• prediction_table
• decision_metrics
• report

Hypothesis. Tool-state schemas can represent UI observations and action history with only small extensions.

Minimal protocol. Create a tiny UI-state task suite, map observations into the existing tool-state schema, and compare heuristic versus learned controllers.

Success signal. The controller transfers basic STOP/MORE or repair behavior to UI tasks under a fixed budget.

Failure signal. UI state requires a new representation that invalidates the current process-control schema.

Avoid. Do not start with unconstrained open-web tasks.

Next step. Prototype with static UI states before live browser control.

Expected artifacts.

• ui_task_suite
• tool_state_mapping
• controller_metrics
• report

Hypothesis. Failure-mined cases help when tied to a falsifiable slice and held-out control, not when used as unfiltered augmentation.

Minimal protocol. Mine common failure slices, generate synthetic cases, train or prompt with them, and evaluate on untouched held-out failures.

Success signal. The targeted slice improves without creating new false positives elsewhere.

Failure signal. Generated cases overfit surface patterns or degrade unrelated slices.

Avoid. Do not train on evaluation failures without a clean holdout.

Next step. Start with diagnostic failure-slicing template once available.

Expected artifacts.

• failure_slice_manifest
• generated_case_manifest
• before_after_slices
• report

Hypothesis. A small committee can learn budget-aware escalation better than fixed critique rounds.

Minimal protocol. Run fixed-round, single-controller, and committee-controller loops on identical pools with explicit action costs.

Success signal. Committee control beats fixed rounds and single-controller baselines on accuracy per cost.

Failure signal. Committee discussion increases cost without improving commit quality.

Avoid. Do not evaluate with unlimited debate rounds.

Next step. Start from STOP/MORE controller and consensus experiments.

Expected artifacts.

• committee_logs
• action_cost_table
• controller_metrics
• report

Hypothesis. Constraint and test memories compress better than raw examples.

Minimal protocol. Compare raw-example, summary, constraint, and test-memory compression under fixed context and storage ceilings.

Success signal. Compressed constraint or test memory beats no-memory and raw-summary controls.

Failure signal. Compression removes the signal that made memory useful.

Avoid. Do not count retrieval time as free.

Next step. Use memory mode comparison suite once its schema exists.

Expected artifacts.

• compression_manifest
• memory_mode_metrics
• latency_table
• report

Hypothesis. Branch choices improve when claims, readiness gaps, related experiments, and program backlogs are visible in one workflow.

Minimal protocol. Prototype a static branching view for a small set of anchor experiments and compare branch quality with the current docs workflow.

Success signal. Branch proposals cite more relevant evidence and fewer near duplicates.

Failure signal. The interface adds navigation overhead without improving proposal quality.

Avoid. Do not build an interface before defining the branch-quality rubric.

Next step. Use high-impact claim evidence as the first anchor set.

Expected artifacts.

• branching_view
• proposal_quality_rubric
• navigation_logs
• report