Collective Experimentation Infrastructure
Make the repository itself better at spawning, comparing, validating, and remembering many independent lines.
What we have learned
Seed Experiments And Artifacts
- qwen_compiler_multiseed_reattribution
- qwen35_4b_sampler_portfolio_scheduler
- knowledge/experiment_catalog.md
- scripts/build_knowledgebase.py
Current Read
The initial repo scaffold was an archive. This program upgrades the repo into a system that can spawn many future lines and make them comparable.
Scorecard
- Program: charter
- Current read: the repository now has program structure, validation, CI, and collaboration templates; the next lift is faster research navigation.
- Best next experiment: measure whether a new agent can find prior evidence and propose a non-duplicate experiment faster using scorecards and intake records.
- Strong anchors:
knowledge/research_program_index.md,knowledge/claims/initial_claims.md,docs/quality_gates.md. - Avoid repeating: adding process that slows pilots without improving memory or decision quality.
- Evidence that advances the program: navigation artifacts reduce duplicate proposals and improve citation of prior evidence.
Charter
Show charter.md
Purpose
Make the repository better at creating, comparing, validating, and remembering many independent research lines.
Why This Is A Program
The user's intended outcome is not a polished archive of two tracks. It is a compounding experimentation system. This program owns the system-level machinery that lets many future people and agents expand the frontier without losing lessons.
Progress Signals
- New programs can be created without changing repository conventions.
- Experiment metadata maps into programs, claims, evidence, and follow-up queues.
- Validation catches missing scaffolding and artifact mistakes.
- Agents can find relevant prior work before proposing a new run.
- Negative results remain discoverable.
Boundaries
This program is about the research operating system itself. It does not replace local experiment autonomy.
Backlog
Show backlog.md
Next Infrastructure
- Add lightweight navigation tests for whether new agents find prior evidence faster.
- Turn program scorecards into a generated dashboard when enough hand-curated examples exist.
- Add first-class decision records for major strategic pivots as they occur.
- Add schema checks for hidden-label boundaries and artifact manifests.
- Add a deduplication audit that samples new proposals against prior experiments.
Required Controls
- Generated indexes must be reproducible.
- Hand-curated strategic claims must link to evidence.
- Templates should support new lines, not just new runs.
Stop Conditions
Do not add process that makes fast pilots heavy. Infrastructure should reduce repeated work and improve memory, not create ceremony.
Experiments 4
- 2026-06-26 Episodic ECHO-TTT
No. Letting the model spend a few seconds studying an episode's real observations left its four-choice decisions stuck near blind-guess level, about 25%, no better than doing nothing. The tell: feeding it scrambled or no
- 2026-06-25 → 26 Qwen3.5-4B Sampler Portfolio Scheduler
In theory yes, in practice no. A flawless chooser that knew which style would crack each problem solved 75% of coding tasks at the cheapest run's cost, while just doubling tries from one style reached 67% at twice the co
- 2026-06-24 → 25 Qwen Compiler Multi-Seed Reattribution
No. The best schedule averaged 43% correct on plainly worded problems, yet the identical training swung from total failure to 81% just by changing the random starting number — so no schedule earns credit for the wins. Bo
- 2026-06-21 Rule-Family Diversity Scaling Experiment
Yes. Holding training at 240 examples but spreading them across 12 kinds of problems instead of 3 lifted fixes on completely unseen problem types from 0% to 29%, while fixes on familiar problems stayed high. But it only