Algorithmic Memory and Retrieval

Use libraries of verified algorithms, examples, traces, skills, or failures as reusable memory.

What we have learned

Seed Experiments

Current Read

Algorithmic memory can add candidate coverage, but naive skill-card prompting can be negative. The promising direction is memory as a source of executable candidates, constraints, tests, or evidence.

Scorecard

Program: charter
Current read: memory helps when it supplies verifiable candidates, constraints, or tests; naive context stuffing is weak.
Best next experiment: compare retrieved examples, retrieved algorithms, retrieved tests, and retrieved failure cases on one task family.
Strong anchors: qwen_verified_skill_memory_rag, qwen35_4b_verified_algorithm_retrieval_adaptation, learned_sparse_slot_executor.
Avoid repeating: retrieval demos without negative retrieval controls or verification of the retrieved artifact.
Evidence that advances the program: memory improves transfer while random or mismatched memory fails under the same budget.

Charter

Show charter.md

Purpose

Make past verified algorithms, examples, traces, skills, programs, failures, and repairs reusable by future experiments and models.

Why This Is A Program

Plain retrieval into a prompt is not enough. The repository needs a sustained line on what memories should store, how they should be retrieved, how they should be adapted, and when they mislead.

Progress Signals

Retrieval improves control-clean coverage or selection, not just similarity.
Retrieved memories supply constraints, tests, or candidate programs that survive deployable verification.
Negative memory results are preserved so future work avoids repeating weak RAG setups.
Memory schemas become reusable across substrates.

Boundaries

This line owns reusable memory and adaptation. Operator-bank enumeration is part of Operator And Skill Inventories; final selection is part of Evidence-Conditioned Selection.

Backlog

Show backlog.md

Next Experiments

Compare prompt-memory, constraint-memory, test-memory, and candidate-memory using the same retrieved items.
Build a small memory schema for verified skills, failure cases, invariants, and retrieval diagnostics.
Measure same-family retrieval against random, corrupted, and shuffled controls.
Use memory to generate counterexamples rather than direct answers.
Study when retrieved analogies hurt and how to detect that before committing.

Required Controls

Random retrieval.
Corrupted retrieval.
Shuffled query retrieval.
Direct no-memory baseline.

Stop Conditions

Do not promote a memory mechanism that fails to beat direct inference and controls on strict task metrics.

Experiments 30

2026-07-16 Clean Gym-Mix Dose
The mix failed cleanly and instructively. Splitting the standard 160-lesson budget across three skills — sixty trick-instruction episodes, fifty procedure chains, fifty answer-or-abstain puzzles — taught none of them: on
2026-07-15 Statechain-Only Dose
Three results in one event. First, the state-tracking dose passed its local gate cleanly — the skill installed again and this time forgetting stayed inside the calibrated margin. Second, on the real benchmark the trained
2026-07-07 Qwen3.5-4B: Is In-Context Learning Retrieval or Induction?
Only patterns it already has. Told a scrambled counting order outright, the model applied it almost perfectly — 97 percent right. But shown examples of that same order and asked to work it out, it scored 12 percent, no b
2026-07-04 Qwen3.5-4B Tool-Seeded Banking: does tool-search + banking cross the depth-3 wall?
Partly. A brute-force search found working three-step programs the model never produces itself, and retraining on them lifted three-step success from a hard zero to 5 of 40 fresh, never-seen tasks when it can reason acro
2026-07-04 Qwen3.5-4B Depth-3 Dose-Response: data-limited or representational cap?
Just starved for examples. Feeding a fixed 4-billion-parameter model more search-found solutions lifted its solve rate on fresh three-layer puzzles from 0% to 38% when given sixteen tries — climbing steadily at every dos
2026-07-03 Qwen3.5-4B Wall Climbing: does banking shallow composition unlock deeper coverage?
No. Fine-tuning the model on the two-step solutions it could already produce tripled its two-step success on fresh tasks, from 12% to 36%. But its three-step success stayed at exactly zero, unchanged from before: both mo
2026-07-03 Qwen3.5-4B Coverage vs Selection: anatomy of the generation wall
It never generates it. Whenever a correct program shows up among 32 tries, running each candidate against eight known examples finds it every single time — the model judging its own work, and even a random pick among sur
2026-07-03 Qwen3.5-4B Coverage Banking: does banking shift the proposal distribution?
Yes, but it depends on difficulty. On easy one-step problems it just pulls answers it already knew into its top guess (60% to 80%), gaining no new ground. On harder two-step problems something new happens: on brand-new t
2026-06-30 Qwen3.5-4B Thinking-Budget Scaling
Yes, but with a sweet spot. Turning on the model's built-in reasoning lifted the accuracy of the answer you would actually ship from 76% to 91% on basic Python tasks — a genuine capability gain, not just a wider pool of
2026-06-27 Real Transform ABI Gate with Counterexamples
It depends on how messy the data is. For clean, spreadsheet-style pipeline jobs the toolbox covered every one (100%) and held firm even against deliberately tricky examples. For irregular date, ID, and text cleanup, cove
2026-06-27 Qwen Verified Skill Memory RAG
No. Handing the model a matched, verified example made it slightly worse, not better: every value came out correct on 47.5% of jobs, versus 50% when it worked through each value alone with no example. The lookup itself w
2026-06-27 Qwen Real Task ABI Coverage Gate
It depends, and the split is sharp. A frozen kit of reusable office operations, with no training at all, assembled 84% of realistic tasks from stored parts alone, far above the 21% a bare kit managed, and it fully solved
2026-06-27 Qwen3.5-4B Independent Code ABI Coverage Gate
No, hardly any. The locked toolbox solved only about 14% of brand-new tasks, roughly 1 in 7, versus 37% on the familiar tasks it was shaped around. Reshuffling which tasks are unseen barely moves it, around 18%. And near
2026-06-27 Qwen3.5-4B Code ABI Oracle Coverage Ladder
Yes, mostly. Snapping together at most three verified functions produced a passing solution for 84% of 160 small programming tasks, up from just 13% with a bare-bones library. But the library is a foundation, not a solve
~2026-06-27 Qwen 3.5 4B Balanced Discriminative Bridge
An even mix, clearly. Sixty evenly-spread ordinary examples across ten new program types lifted the model's success on unseen hard problems from 60% (with no examples at all) to 99% fully solved. Hand-picking only the tr
2026-06-26 Qwen3.5-4B Verified Algorithm Retrieval Adaptation
Yes, but modestly. Fetching the most similar already-solved problem and adapting its code cracked 3 of the 8 problems that repeated tries kept missing, lifting overall success from 67% to 79%. Matching by meaning clearly
2026-06-26 Qwen3.5-4B Substrate Coverage Ladder
Only when a building-block was hand-carved for each specific problem. A shared library that reshapes old solutions to fit new ones solved zero of the nine stuck tasks, no better than nothing. Custom-written parts solved
2026-06-26 Qwen3.5-4B Retrieval Adapt Verify Scale
It splits. Pulling a similar solved problem from a 364-entry library and rewriting it recovered 8 of 24 stuck tasks — a third, double the 4 that a random-library control found. But the only safe check, the visible sample
2026-06-26 Qwen3.5-4B Pass@k Coverage RL
No. Retraining the model to spread its tries across different problems actually lowered coverage: it solved about 44% of unseen coding problems within four tries, versus about 69% from simply raising the randomness dial
2026-06-26 Qwen3.5-4B Offline Hard Negative Coverage DPO
It depends — and even the gentle version failed its own bar. A light touch matched the task-solving reach of generating twice as many candidate answers at roughly half the compute, and beat a shuffled-pair placebo. But i
2026-06-26 Qwen3.5-4B Independent Retrieval Consensus
No. Requiring independently reshaped solutions to agree on invented tricky inputs, the system committed to zero of 24 unsolved tasks — the separate versions never converged. Simply taking the first solution that passes t
2026-06-26 Qwen3.5-4B Constrained Coverage DPO
Not quite. The nudged model solved 62% of problems within four tries versus 58% for the untrained model, and kept its first-try accuracy — a real, cheap gain. But simply doubling to eight tries solved 67%, still more. Th
2026-06-25 Qwen3.5-4B Verifier-Guided Self-Improvement Report
No. After a four-billion-parameter model retrained on its own test-passing code, its success on unseen problems barely moved, going from about 65% to about 65% (a hair lower). Retraining made its several attempts look mo
2026-06-25 Qwen3.5-4B Trained vs Frozen Repair MDP Report
Retry from scratch. Given the same compute budget, a few more fresh attempts rescued 5 of 57 stuck problems, while a fixer trained to patch broken code rescued only 2 — and its repairs secretly failed the hidden tests ne
2026-06-25 Qwen3.5-4B Real Sample Verify Commit
It's the writing. When a 4-billion-parameter model writes several code attempts per problem, choosing is not the bottleneck. On easier problems nearly every attempt is already correct, so the best result sits near 97%. O
2026-06-25 Qwen3.5-4B Diversity-Keyed Coverage Gate
Mostly the second. Of 24 Python problems a 4-billion-parameter model missed on four tries, spending more and more varied sampling recovered 15, lifting the share solved from 70% to nearly 89%. Mixing three creativity set
2026-06-24 Qwen3.5-4B Sketch Coverage Shift Probe
Only when the template explicitly names the new operation. On tasks needing a brand-new operation, hand-written templates that named and typed it kept the correct program every single time; auto-generated templates kept
2026-06-21 Sparse Support Memory Executor Experiment
Only when its scratch memory held one slot for every possible starting value. With that, it answered every question correctly through the longest 24-step programs. Cut the memory roughly in half and accuracy fell to abou
2026-06-21 Learned Sparse Slot Executor Experiment
Yes, but only small. With eleven possible values and a ready-made scratchpad, the network kept the fully correct answer in view 95.5% of the time, even on longer chains than it trained on. Widen to thirty-one values and
2026-06-21 Feature-Factorized Rule Diversity
No. All three practice diets fixed only about 1 in 5 brand-new bug combinations, so mixing bought nothing over drilling either kind alone. What actually mattered was showing worked, step-by-step repair reasoning during t

Claims

Negative C3 · Plain prompt memory is not enough