Adaptive Cognitive Kernel
The one idea you need
Picture a calculator that, for each instruction in a program, physically swaps in a fresh set of gears — temporarily rewiring itself — instead of a fixed machine that reads each instruction as an input code. Does the rewiring execute programs any better?
The question
If a small network temporarily rewires itself for each operation in a program, does it run and generalize to longer programs better than an ordinary fixed network?
What we found
No advantage. The self-rewiring is genuinely doing ordered work: scrambling the operation order collapses its step-by-step accuracy from about 12% to 2%, and switching the rewiring off cripples it. But it never beats a plain fixed network. On the longest test programs the ordinary one matched or slightly edged it, and both crashed far below their training-length scores.
Why it matters
Letting a small model rewrite its own weights step by step adds real machinery and cost but buys no extra generalization here — a plain fixed network does as well or better. Test any fancy mechanism against a simple baseline first.
On this page
Results at a glance 7
How to read
Each bar is one method's share of correct final answers on the hardest twelve-step programs; taller is better. Left to right: self-rewiring calculator, its instructions-scrambled version, its rewiring-switched-off version, the ordinary fixed calculator, and a read-it-all-at-once model.
Takeaway → The self-rewiring calculator tops its own crippled versions but sits just below the ordinary fixed calculator, so its extra machinery does not win.
Data table
| arm | final-answer accuracy (length 12) |
|---|---|
| ACK ordered | 7.7% |
| ACK shuffled-code | 5.7% |
| ACK no-delta | 6.5% |
| fixed recurrent (GRU) | 8.3% |
| direct transformer | 5.8% |
Numbers from report table (reports/adaptive_cognitive_kernel_report.md)
Technical framing
Final-answer accuracy on hardest held-out length (12 ops) — ACK's dynamic weight edits beat its own ablations but do not beat an ordinary fixed recurrent controller at length 12.
How to read
Each line tracks one method's share of correct final answers as programs grow from four to twelve steps; higher is better, and everything past six steps is longer than training covered. The self-rewiring and ordinary fixed calculators are the top two lines.
Takeaway → All lines plunge steeply as programs lengthen, with the self-rewiring calculator tracking the fixed one rather than pulling ahead.
Data table
| Program length (ops; trained on 2-6) | ACK ordered | fixed recurrent (GRU) | direct transformer | ACK no-delta | ACK shuffled-code |
|---|---|---|---|---|---|
| 4 | 35.3% | 33.4% | 31.2% | 16.2% | 10.8% |
| 6 | 18.4% | 17.7% | 13.9% | 8.9% | 7.5% |
| 8 | 13.2% | 12.6% | 8.9% | 7.2% | 7.6% |
| 10 | 8.7% | 8.1% | 6.6% | 5.9% | 6.8% |
| 12 | 7.7% | 8.3% | 5.8% | 6.5% | 5.7% |
Numbers from report table (reports/adaptive_cognitive_kernel_report.md)
Technical framing
Final-answer accuracy vs program length — All arms degrade steeply beyond the trained lengths; ACK tracks the fixed controller rather than exceeding it.
How to read
Each bar shows how often a method gets each intermediate step right on twelve-step programs; taller is better. Bars compare the self-rewiring calculator run normally versus with its instructions scrambled, randomized, or switched off, plus the ordinary fixed calculator.
Takeaway → Scrambling or randomizing the instructions crushes the self-rewiring calculator from tall to tiny, proving it does ordered work — yet the plain calculator still tracks steps best.
Data table
| condition | per-step state accuracy (length 12) |
|---|---|
| ACK ordered | 11.6% |
| ACK shuffled-code | 2% |
| ACK random-code | 1.5% |
| ACK no-delta | 4% |
| fixed recurrent (GRU) | 14.4% |
Numbers from report table (reports/adaptive_cognitive_kernel_report.md)
Technical framing
Mechanism check: intermediate-state step accuracy at length 12 — Shuffling or randomizing the conditioning codes collapses ACK's state tracking, so the edits do ordered work - yet a plain GRU still tracks state better.
How to read
Grouped bars show final-answer accuracy on programs with unfamiliar operation combinations, at eight and twelve steps; taller is better. The four methods are the self-rewiring calculator, the ordinary fixed calculator, the read-it-all-at-once model, and the rewiring-switched-off version.
Takeaway → The self-rewiring calculator edges ahead at eight steps but the gap disappears at twelve steps, where every method sinks together.
Data table
| Held-out composition split | ACK dynamic (ordered) | Fixed GRU | Direct transformer | ACK no-delta |
|---|---|---|---|---|
| comp8 | 12.6% | 11.2% | 8.3% | 7.9% |
| comp12 | 7.1% | 7% | 6.6% | 5.7% |
Numbers from experiments/adaptive_cognitive_kernel/analysis/summary_by_arm.csv
Technical framing
Held-out operation compositions: final-answer accuracy — On held-out op compositions ACK edges the fixed GRU at comp8 but the gap vanishes at comp12; no arm escapes the collapse.
How to read
Grouped bars show final-answer accuracy on the longest programs across three separate training runs; taller is better. Each cluster compares the self-rewiring calculator, the ordinary fixed calculator, the rewiring-switched-off version, and the read-it-all-at-once model.
Takeaway → The bounce between training runs is about as big as the gap between the self-rewiring and fixed calculators, so neither reliably wins.
Data table
| Training seed | ACK dynamic (ordered) | Fixed GRU | ACK no-delta | Direct transformer |
|---|---|---|---|---|
| seed 101 | 7.7% | 8.9% | 7.2% | 5.6% |
| seed 202 | 8.9% | 7.2% | 6.2% | 6.2% |
| seed 303 | 6.5% | 8.9% | 6% | 5.6% |
Numbers from experiments/adaptive_cognitive_kernel/analysis/metrics.csv
Technical framing
Seed spread at the hardest length (12 ops) — Seed-to-seed variation at length 12 is as large as the ACK-vs-GRU gap, so neither recurrent arm reliably wins.
How to read
Each line shows accuracy climbing as training proceeds; higher and steeper is better. The self-rewiring calculator and the ordinary fixed calculator are the top pair, while the rewiring-switched-off version is the flat bottom line.
Takeaway → The self-rewiring and fixed calculators rise together almost identically, while turning off the self-rewiring leaves learning nearly flat.
Data table
| Training step | ACK dynamic | Fixed GRU | Direct transformer | ACK no-delta |
|---|---|---|---|---|
| 1 | 0.1% | 0.7% | 0% | 0.1% |
| 500 | 4.2% | 4.2% | 3.8% | 1.8% |
| 1000 | 8.2% | 7.7% | 7.9% | 2.6% |
| 1500 | 11.6% | 12.8% | 9.6% | 2.6% |
| 2000 | 14.7% | 15.1% | 12.1% | 2.5% |
| 2500 | 15.8% | 16.8% | 13.5% | 2.7% |
Numbers from experiments/adaptive_cognitive_kernel/analysis/training_log.csv
Technical framing
Training curves: validation strict-pair accuracy (seed 101) — ACK and the fixed GRU learn at nearly identical rates; disabling the dynamic weight edits stalls learning almost completely.
How to read
Each line tracks how often the self-rewiring calculator gets each step right as programs grow, comparing real instruction order against scrambled and randomized versions; higher is better.
Takeaway → The real-order line stays well above the scrambled and randomized lines at every length, confirming the calculator genuinely relies on instruction order.
Data table
| Program length (ops) | ACK ordered codes | ACK shuffled codes | ACK random codes |
|---|---|---|---|
| 4 | 31.2% | 4.5% | 3.7% |
| 6 | 21.9% | 3.2% | 3% |
| 8 | 17.8% | 2.8% | 2.2% |
| 10 | 13.9% | 2.4% | 1.7% |
| 12 | 11.6% | 2% | 1.5% |
Numbers from experiments/adaptive_cognitive_kernel/analysis/summary_by_arm.csv
Technical framing
Ordered vs shuffled vs random conditioning across program lengths — Shuffling or randomizing the op-conditioning stream collapses state tracking at every length, so ACK's computation is genuinely ordered.
In the author’s words from the Report · “Summary”
Key longest-length metrics: ACK ordered length-12 final-answer accuracy: 7.7%. ACK shuffled-code length-12 final-answer accuracy: 5.7%. ACK no-delta length-12 final-answer accuracy: 6.5%. Fixed recurrent controller length-12 final-answer accuracy: 8.3%. Direct transformer length-12 final-answer accuracy: 5.8%. Strict final-pair accuracy on length 12: ACK ordered: 0.7%. ACK shuffled-code: 0.5%. ACK no-delta: 0.5%. Fixed recurrent controller: 0.7%. Direct transformer: 0.3%. Intermediate-state step accuracy on length 12: ACK ordered: 11.6%. ACK shuffled-code: 2.0%. ACK no-delta: 4.0%. Fixed recurrent controller: 14.4%. Held-out-composition metrics: ACK ordered composition-12 final-answer accuracy: 7.1%. … Read the full result →
Overview
This experiment tests whether a task-conditioned recurrent runtime with temporary adapter-style weight edits can learn compositional execution better than ordinary fixed-weight controllers.
The experiment is standalone. Small artifacts live in this directory. Large checkpoints live under:
/workspace/large_artifacts/adaptive_cognitive_kernel/checkpoints/Layout
src/ runner and report generation code
runs/ run metrics and logs
analysis/ aggregate CSVs and figures
reports/ Markdown and HTML reportsMain Question
Can a prompt-conditioned generator select temporary computation kernels that run recurrently and generalize to longer or held-out operation compositions?
Controls
- Direct transformer over the full prompt.
- Fixed recurrent controller with ordinary op-token inputs.
- ACK runtime with dynamic low-rank weight edits disabled.
- ACK shuffled-code evaluation, which preserves runtime capacity while breaking ordered task-conditioned computation.
Report
Rendered from reports/adaptive_cognitive_kernel_report.md
Summary
This standalone experiment tests whether a task-conditioned recurrent runtime can use temporary adapter-style weight edits as a useful computation substrate. The runtime receives an initial two-register state and a sequence of symbolic operations. It must execute the program and predict both the final register pair and intermediate states.
The result is mixed but not a breakthrough: the dynamic ACK runtime learns an ordered conditioning signal and its intermediate-state accuracy is far above no-delta and shuffled-code controls, but it does not beat the ordinary fixed recurrent controller on the longest split.
Key longest-length metrics:
- ACK ordered length-12 final-answer accuracy: 7.7%.
- ACK shuffled-code length-12 final-answer accuracy: 5.7%.
- ACK no-delta length-12 final-answer accuracy: 6.5%.
- Fixed recurrent controller length-12 final-answer accuracy: 8.3%.
- Direct transformer length-12 final-answer accuracy: 5.8%.
Strict final-pair accuracy on length 12:
- ACK ordered: 0.7%.
- ACK shuffled-code: 0.5%.
- ACK no-delta: 0.5%.
- Fixed recurrent controller: 0.7%.
- Direct transformer: 0.3%.
Intermediate-state step accuracy on length 12:
- ACK ordered: 11.6%.
- ACK shuffled-code: 2.0%.
- ACK no-delta: 4.0%.
- Fixed recurrent controller: 14.4%.
Held-out-composition metrics:
- ACK ordered composition-12 final-answer accuracy: 7.1%.
- Fixed recurrent controller composition-12 final-answer accuracy: 7.0%.
Mechanism Under Test
The ACK arm maps each operation token to coefficients over a bank of learned low-rank transition atoms. At each recurrent step, those coefficients temporarily edit the transition applied to the runtime state. The same learned atom bank is reused across tasks and steps.
The no-delta ACK arm keeps the same recurrent shell and operation-conditioned drive but disables the low-rank weight edits. The shuffled-code control keeps the trained ACK runtime and candidate operation codes but permutes the operation-conditioning sequence at evaluation time. A mechanism-level positive requires ACK ordered to outperform fixed controls and to degrade when codes are shuffled.
Dataset
- Register values: integers modulo
17. - Operation set:
12symbolic register operations. - Training lengths:
2..6. - Held-out lengths:
8, 10, 12. - Held-out adjacent operation pairs:
swap -> add_y_to_x, double_x -> add_x_to_y, branch_x -> mix_y, diff_y_x -> double_y. - Evaluation examples per split:
768. - Online training examples:
True. - Training seeds:
101,202,303.
Aggregate Results
| arm | control | split | final_pair_mean | final_pair_std | final_x_mean | state_step_mean | state_all_mean | code_entropy_mean |
|---|---|---|---|---|---|---|---|---|
| ack_dynamic | ordered | eval_comp12 | 0.4% | 0.2% | 7.1% | 11.1% | 0.0% | 0.332 |
| ack_dynamic | random | eval_comp12 | 0.6% | 0.1% | 6.0% | 1.6% | 0.0% | 0.341 |
| ack_dynamic | shuffle | eval_comp12 | 0.3% | 0.1% | 6.2% | 2.0% | 0.0% | 0.332 |
| ack_no_delta | ordered | eval_comp12 | 0.3% | 0.2% | 5.7% | 4.1% | 0.0% | 0.002 |
| direct_transformer | ordered | eval_comp12 | 0.5% | 0.3% | 6.6% | 10.3% | 0.0% | n/a |
| fixed_gru | ordered | eval_comp12 | 0.4% | 0.2% | 7.0% | 13.7% | 0.0% | n/a |
| ack_dynamic | ordered | eval_comp8 | 1.4% | 0.4% | 12.6% | 16.5% | 0.0% | 0.328 |
| ack_dynamic | random | eval_comp8 | 0.3% | 0.3% | 5.6% | 2.0% | 0.0% | 0.338 |
| ack_dynamic | shuffle | eval_comp8 | 0.6% | 0.3% | 7.8% | 2.7% | 0.0% | 1.134 |
| ack_no_delta | ordered | eval_comp8 | 0.4% | 0.2% | 7.9% | 5.2% | 0.0% | 0.002 |
| direct_transformer | ordered | eval_comp8 | 0.6% | 0.2% | 8.3% | 16.3% | 0.0% | n/a |
| fixed_gru | ordered | eval_comp8 | 0.8% | 0.1% | 11.2% | 20.2% | 0.0% | n/a |
| ack_dynamic | ordered | eval_len10 | 0.9% | 0.2% | 8.7% | 13.9% | 0.0% | 0.345 |
| ack_dynamic | random | eval_len10 | 0.2% | 0.1% | 6.1% | 1.7% | 0.0% | 0.343 |
| ack_dynamic | shuffle | eval_len10 | 0.5% | 0.4% | 6.8% | 2.4% | 0.0% | 0.750 |
| ack_no_delta | ordered | eval_len10 | 0.6% | 0.3% | 5.9% | 4.5% | 0.0% | 0.002 |
| direct_transformer | ordered | eval_len10 | 0.5% | 0.3% | 6.6% | 12.6% | 0.0% | n/a |
| fixed_gru | ordered | eval_len10 | 0.7% | 0.3% | 8.1% | 16.6% | 0.0% | n/a |
| ack_dynamic | ordered | eval_len12 | 0.7% | 0.2% | 7.7% | 11.6% | 0.0% | 0.344 |
| ack_dynamic | random | eval_len12 | 0.4% | 0.1% | 6.0% | 1.5% | 0.0% | 0.338 |
| ack_dynamic | shuffle | eval_len12 | 0.5% | 0.2% | 5.7% | 2.0% | 0.0% | 0.344 |
| ack_no_delta | ordered | eval_len12 | 0.5% | 0.1% | 6.5% | 4.0% | 0.0% | 0.002 |
| direct_transformer | ordered | eval_len12 | 0.3% | 0.1% | 5.8% | 10.5% | 0.0% | n/a |
| fixed_gru | ordered | eval_len12 | 0.7% | 0.3% | 8.3% | 14.4% | 0.0% | n/a |
| ack_dynamic | ordered | eval_len4 | 10.5% | 1.9% | 35.3% | 31.2% | 5.6% | 0.343 |
| ack_dynamic | random | eval_len4 | 0.4% | 0.2% | 7.3% | 3.7% | 0.0% | 0.341 |
| ack_dynamic | shuffle | eval_len4 | 0.9% | 0.3% | 10.8% | 4.5% | 0.0% | 1.934 |
| ack_no_delta | ordered | eval_len4 | 2.3% | 0.3% | 16.2% | 10.2% | 0.1% | 0.002 |
| direct_transformer | ordered | eval_len4 | 6.4% | 0.3% | 31.2% | 35.0% | 3.6% | n/a |
| fixed_gru | ordered | eval_len4 | 9.3% | 0.4% | 33.4% | 39.2% | 4.8% | n/a |
| ack_dynamic | ordered | eval_len6 | 3.5% | 0.9% | 18.4% | 21.9% | 0.3% | 0.350 |
| ack_dynamic | random | eval_len6 | 0.3% | 0.1% | 6.7% | 3.0% | 0.0% | 0.343 |
| ack_dynamic | shuffle | eval_len6 | 0.5% | 0.3% | 7.5% | 3.2% | 0.0% | 1.550 |
| ack_no_delta | ordered | eval_len6 | 0.9% | 0.2% | 8.9% | 7.2% | 0.0% | 0.002 |
| direct_transformer | ordered | eval_len6 | 1.6% | 0.3% | 13.9% | 24.5% | 0.1% | n/a |
| fixed_gru | ordered | eval_len6 | 2.8% | 0.3% | 17.7% | 26.6% | 0.6% | n/a |
| ack_dynamic | ordered | eval_len8 | 2.0% | 0.5% | 13.2% | 17.8% | 0.0% | 0.347 |
| ack_dynamic | random | eval_len8 | 0.3% | 0.1% | 5.9% | 2.2% | 0.0% | 0.343 |
| ack_dynamic | shuffle | eval_len8 | 0.6% | 0.3% | 7.6% | 2.8% | 0.0% | 1.148 |
| ack_no_delta | ordered | eval_len8 | 0.5% | 0.3% | 7.2% | 5.6% | 0.0% | 0.002 |
| direct_transformer | ordered | eval_len8 | 0.7% | 0.3% | 8.9% | 16.7% | 0.0% | n/a |
| fixed_gru | ordered | eval_len8 | 1.5% | 0.2% | 12.6% | 20.9% | 0.0% | n/a |
| ack_dynamic | ordered | val | 15.0% | 2.2% | 36.2% | 30.8% | 10.9% | 0.344 |
| ack_no_delta | ordered | val | 2.9% | 0.5% | 17.1% | 10.6% | 1.0% | 0.002 |
| direct_transformer | ordered | val | 12.9% | 1.8% | 32.6% | 33.9% | 11.2% | n/a |
| fixed_gru | ordered | val | 16.9% | 1.2% | 38.7% | 38.1% | 14.3% | n/a |
Figures
Interpretation
The central question is not whether a recurrent neural network can fit short programs. It is whether task-conditioned temporary weight edits create a reusable computation substrate that generalizes better than ordinary fixed-weight controllers. The shuffled-code evaluation is the load-bearing control: if it matches ordered ACK, then extra runtime capacity is not evidence of ordered computation.
The held-out-length and held-out-composition splits are the primary readouts. Trained-length accuracy alone is insufficient, because a prompt transducer can memorize short input-output mappings without learning a reusable executor.
The evidence separates two claims. First, task-conditioned ACK computation is not inert: randomizing or shuffling the conditioning stream collapses state accuracy, and disabling dynamic deltas substantially weakens the ACK runtime. Second, this is not enough to justify the stronger claim that temporary weight edits are a superior computation substrate. A conventional fixed recurrent controller remains as good or better on the longest held-out splits, and all arms show steep degradation as length grows beyond the training range.
Artifacts
- Run directory:
/workspace/experiments/adaptive_cognitive_kernel/runs/main_ack_v1 - Metrics CSV:
/workspace/experiments/adaptive_cognitive_kernel/runs/main_ack_v1/metrics.csv - Training log:
/workspace/experiments/adaptive_cognitive_kernel/runs/main_ack_v1/training_log.csv - Analysis directory:
/workspace/experiments/adaptive_cognitive_kernel/analysis - Large checkpoints:
/workspace/large_artifacts/adaptive_cognitive_kernel/checkpoints/main_ack_v1
Experiment log 4
Show the running log (4 entries)
Setup
Created a fresh experiment directory and separate large-artifact checkpoint root.
Experiment root:
/workspace/experiments/adaptive_cognitive_kernelLarge artifacts:
/workspace/large_artifacts/adaptive_cognitive_kernel/checkpointsInitial Design
The mechanism under test is a task-conditioned recurrent runtime. A prompt encoder reads an initial two-register state plus a sequence of operations. The ACK arm maps each operation token to coefficients over a learned bank of low-rank transition atoms. At runtime, those coefficients temporarily edit the recurrent transition used for that step.
Primary controls:
- direct sequence transformer;
- fixed recurrent controller;
- ACK ablation without dynamic low-rank deltas;
- shuffled-code evaluation for ACK.
Primary readouts:
- trained-length final pair accuracy;
- held-out longer-length final pair accuracy;
- held-out adjacent-composition final pair accuracy;
- state-step accuracy;
- ACK ordered versus shuffled-code gap.
Iteration Notes
Smoke smoke_ack_v1 validated the code path and artifact generation.
Pilot pilot_ack_v1 showed that 450 steps on a finite training set was too weak to test the mechanism. A longer focused run, pilot_ack_long_v2, drove training loss down but left validation near chance, indicating memorization of a finite random program table rather than learned operation semantics.
Patch: training now defaults to online generation of fresh programs each step. This better matches the benchmark: the target is to learn reusable operation semantics, not memorize a fixed table.
Pilot pilot_ack_online_v3 confirmed that online generation made the task learnable: short-length final-answer and state accuracy rose meaningfully, and ACK ordered conditioning separated from shuffled/random controls.
Pilot pilot_ack_drive_v4 added an operation drive vector to the ACK shell. Dynamic ACK improved clearly over no-delta ACK, but fixed GRU remained competitive or stronger.
Main Run
Run: main_ack_v1
Configuration:
- seeds:
101,202,303 - arms:
ack_dynamic,ack_no_delta,fixed_gru,direct_transformer - train steps per arm/seed:
2500 - online train examples: enabled
- eval examples per split:
768 - model width:
128 - ACK atoms:
24 - ACK atom rank:
16
Result:
- ACK ordered length-12 final-answer accuracy:
7.7% - ACK shuffled-code length-12 final-answer accuracy:
5.7% - ACK no-delta length-12 final-answer accuracy:
6.5% - fixed GRU length-12 final-answer accuracy:
8.3% - direct transformer length-12 final-answer accuracy:
5.8% - ACK ordered length-12 state-step accuracy:
11.6% - ACK shuffled-code length-12 state-step accuracy:
2.0% - ACK no-delta length-12 state-step accuracy:
4.0% - fixed GRU length-12 state-step accuracy:
14.4%
Interpretation: dynamic ACK computation is not inert. Shuffling/randomizing the conditioning stream collapses state accuracy, and disabling dynamic deltas weakens the runtime. However, the dynamic ACK does not beat a conventional fixed recurrent controller on the longest held-out split, and all arms degrade steeply with length. The mechanism is therefore mixed but not a breakthrough under this test.
Figures 4
Data files 14
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/metrics.csv17 kBanalysis/summary_by_arm.csv6.6 kBruns/main_ack_v1/metrics.csv18 kBruns/main_ack_v1/run_summary.json201 Bruns/pilot_ack_drive_v4/metrics.csv4.3 kBruns/pilot_ack_drive_v4/run_summary.json181 Bruns/pilot_ack_long_v2/metrics.csv9.3 kBruns/pilot_ack_long_v2/run_summary.json176 Bruns/pilot_ack_online_v3/metrics.csv3.4 kBruns/pilot_ack_online_v3/run_summary.json170 Bruns/pilot_ack_v1/metrics.csv5.8 kBruns/pilot_ack_v1/run_summary.json195 Bruns/smoke_ack_v1/metrics.csv4.2 kBruns/smoke_ack_v1/run_summary.json195 B
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.