Research log Small Model Experimentation
GitHub

Targeted Bridge Allocation

Finished2026-06-22imported · line ZBenchmark GeneralizationGitHub ↗
Worked-out reasoning beats clever example targeting

The one idea you need

A small model must fix code fusing two skills it only ever saw apart — say, sorting a list and reading a tuple, never both at once. Your worked examples are a fixed pool: spread them across every new fusion, or pile them onto the hardest?

The question

With a fixed budget of worked examples for code patterns a model has never seen combined, should you spread them evenly or pile them onto the hardest patterns?

What we found

Barely. Piling examples on the hardest never-seen combinations repaired 33% of test cases versus 28% for spreading them evenly — a three-case edge on sixty tests, and no better than the same budget spent on easy combinations. The real lever was not allocation: strip or scramble the step-by-step solution attached to each example and success crashed from 33% to under 10%.

Why it matters

When you cannot add more training data, don't over-engineer which cases to target — the gain is marginal. Spend your effort attaching a correct step-by-step solution to every example; that pairing, not the allocation, is what carries the transfer.

Concentrating on hard cases vs spreading evenly28% → 33%never-seen combinations repaired: 17 of 60 vs 20 of 60
Hard cases vs same budget on easy cases33% vs 32%20 of 60 vs 19 of 60 — essentially tied
When the step-by-step solution is removed33% → 8%success collapses without correct reasoning attached
Untrained starting model0%0 of 60 never-seen combinations fixed before training
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Question
    2. Design
    3. Allocation Plans
    4. Overall Results
    5. Allocation Comparison
    6. Light Single-Family Probes
    7. Prompt Ablations
    8. Recombination Holdout By Family
    9. Hard-Target Holdout By Factor
    10. Readout
    11. Figures
    12. Artifact Layout
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 8

Where you spend a fixed set of worked examples changes repair

How to read

Each bar is the share of never-seen skill-combinations a trained model repairs; bars run left to right from the untrained model, through spreading examples evenly, to concentrating them on the hardest combinations. Taller is better.

0%10%20%30%40%frozenfrozen0%uniform 2/familyuniform 2/family28.3%uniform 4/familyuniform 4/family25%hard-target 8/8/8hard-target 8/8/833.3%hard-target seen-preservinghard-target seen-preserving25%easy-target controleasy-target control31.7%

Takeaway → The concentrated hard-case bar is tallest at 33%, just five points above even spreading, yet every trained bar towers over the untrained model's 0%.

Data table
Conditiontrace-conditioned LoRA
frozen0%
uniform 2/family28.3%
uniform 4/family25%
hard-target 8/8/833.3%
hard-target seen-preserving25%
easy-target control31.7%

Numbers from reports/final_results.csv (repair_at_1, recombination_holdout)

Technical framing

Recombination-holdout repair by bridge allocation strategy (fixed 240-record budget) — Targeting hard families wins narrowly (33.3% vs 28.3% uniform), but all trace-trained allocations land in a 25-33% band far above frozen 0%.

Training without a correct solution guts every kind of test

How to read

Three test types along the bottom — familiar problems, reworded versions, and never-seen combinations; colored bars are training examples that include a correct step-by-step solution, none, or a scrambled one. Taller is better.

0%20%40%60%80%seen-combination IIDseen-combination IID77.8%27.8%30.6%format shiftformat shift69.4%19.4%25%recombination holdoutrecombination holdout33.3%8.3%10%

Takeaway → The correct-solution bars dwarf the rest everywhere, falling from about 78% to under 30% even on familiar problems and from 33% to under 10% on new combinations.

Data table
Evaluation splitaligned traceno traceshuffled trace
seen-combination IID77.8%27.8%30.6%
format shift69.4%19.4%25%
recombination holdout33.3%8.3%10%

Numbers from reports/final_results.csv (repair_at_1 for hard_target_trace, hard_target_no_trace, hard_target_shuffled_trace)

Technical framing

Trace quality is decisive: aligned vs no-trace vs shuffled-trace training — Removing or scrambling the trace in the hard-target training data collapses repair on every split; the allocation effect rides on aligned trace evidence.

Concentrating examples opens combinations even spreading leaves stuck

How to read

Bars grouped by which never-seen skill-combination family; colored bars are spreading examples evenly, concentrating on the hardest families, and a control that concentrates on the easy families. Taller is better.

0%25%50%75%100%length_contains_codelength_contains_code0%25%8.3%modulo_sum_labelmodulo_sum_label0%0%0%tuple_branch_labeltuple_branch_label0%16.7%16.7%sorted_contains_countsorted_contains_count100%100%100%sorted_tuple_affinesorted_tuple_affine41.7%25%33.3%

Takeaway → Concentrating on hard families lifts two families off zero that even spreading never cracks, while the already-easy families stay high — one pinned at 100%, the other holding around a third.

Data table
Held-out recombination familyuniform 2/familyhard-target 8/8/8easy-target control
length_contains_code0%25%8.3%
modulo_sum_label0%0%0%
tuple_branch_label0%16.7%16.7%
sorted_contains_count100%100%100%
sorted_tuple_affine41.7%25%33.3%

Numbers from reports/final_results_by_family.csv (repair_at_1, recombination_holdout)

Technical framing

Holdout repair per held-out family: where the bridge examples went matters — Hard-targeting opens two of three hard families (first three bars) that uniform allocation leaves at 0%, while the easy families stay saturated or noisy.

Flooding one skill with examples is not the lever

How to read

Each bar is repair on never-seen combinations for spreading examples evenly versus pouring sixteen examples into a single skill family. Taller is better.

0%10%20%30%40%uniform 2/familyuniform 2/family28.3%modulo dosed to 16modulo dosed to 1633.3%length dosed to 16length dosed to 1620%tuple dosed to 16tuple dosed to 1628.3%

Takeaway → Overloading one family helps modestly at best — 33% for the remainder-math skill — and actually lowers repair for the length skill, so sheer volume is not what helps.

Data table
Conditiontrace-conditioned LoRA
uniform 2/family28.3%
modulo dosed to 1633.3%
length dosed to 1620%
tuple dosed to 1628.3%

Numbers from reports/final_results.csv (repair_at_1, recombination_holdout)

Technical framing

Single-family 16-example dose probes vs uniform baseline — Pouring 16 bridge examples into one hard family helps at most modestly (modulo) and can hurt aggregate holdout (length), so dose alone is not the lever.

Concentrating examples costs nothing on familiar problems

How to read

One group of bars per training setup, with two colors: repair on familiar problems and on reworded versions of them; the leftmost frozen pair is the untrained model. Taller is better.

0%25%50%75%100%FrozenFrozen0%0%Uniform 2Uniform 275%72.2%Uniform 4Uniform 472.2%83.3%Hard-targetHard-target77.8%69.4%Seen-preserving targetSeen-preserving target75%72.2%Easy-target controlEasy-target control80.6%80.6%Modulo-16 probeModulo-16 probe83.3%75%Length-16 probeLength-16 probe80.6%77.8%Tuple-16 probeTuple-16 probe75%69.4%

Takeaway → Every trained setup clusters in a 69-83% band, so concentrating the worked examples on hard cases sacrifices nothing on familiar or reworded problems.

Data table
Training conditionSeen-combination IIDFormat shift
Frozen0%0%
Uniform 275%72.2%
Uniform 472.2%83.3%
Hard-target77.8%69.4%
Seen-preserving target75%72.2%
Easy-target control80.6%80.6%
Modulo-16 probe83.3%75%
Length-16 probe80.6%77.8%
Tuple-16 probe75%69.4%

Numbers from report table (reports/targeted_bridge_allocation_report.md, Overall Results)

Technical framing

Targeting bridges does not hurt in-distribution repair — All 240-record allocations land in a 69-83% band on seen and format-shift splits, so concentrating bridges costs nothing in-distribution.

At test time the model still needs the worked solution in the prompt

How to read

Three test types along the bottom; colored bars give the model, inside its prompt, a correct step-by-step solution, none, or a scrambled one. Taller is better.

0%20%40%60%80%Seen-combination IIDSeen-combination IID77.8%19.4%8.3%Format shiftFormat shift69.4%16.7%5.6%Recombination holdoutRecombination holdout33.3%8.3%6.7%

Takeaway → Removing or scrambling the solution from the prompt crashes repair even on familiar problems, from 78% to under 20%.

Data table
Evaluation splitAligned trace promptNo-trace promptShuffled-trace prompt
Seen-combination IID77.8%19.4%8.3%
Format shift69.4%16.7%5.6%
Recombination holdout33.3%8.3%6.7%

Numbers from report table (reports/targeted_bridge_allocation_report.md, Prompt Ablations)

Technical framing

Eval-time prompt ablation: the hard-target adapter needs an aligned trace in the prompt — Removing or shuffling the trace at evaluation time collapses the trained hard-target adapter from 78% to under 20% even on seen data.

Repair depends heavily on the kind of task

How to read

One bar per task mechanism, sorted most to least repaired, for the hard-case model on never-seen combinations. Taller is better.

0%25%50%75%100%sequence_iterationsequence_iteration100%string_normalizationstring_normalization100%orderingordering62.5%string_matchstring_match62.5%aggregationaggregation50%string_formatstring_format35.4%lengthlength25%tuple_accesstuple_access20.8%arithmeticarithmetic16.7%modulomodulo0%

Takeaway → Repair splits sharply: step-through-a-sequence and text-cleanup tasks reach 100%, while remainder-arithmetic tasks stay flat at 0%.

Data table
Task factorHard-target trace, recombination holdout
sequence_iteration100%
string_normalization100%
ordering62.5%
string_match62.5%
aggregation50%
string_format35.4%
length25%
tuple_access20.8%
arithmetic16.7%
modulo0%

Numbers from report table (reports/targeted_bridge_allocation_report.md, Hard-Target Holdout By Factor)

Technical framing

Hard-target holdout repair splits sharply by task mechanism — Iteration and normalization factors are fully repaired while modulo stays at 0%; 10 of 11 factors shown (branching, 13.9%, omitted).

The total number of examples does not predict repair

How to read

Each dot is a training setup; left to right is the total worked examples used, up is repair on never-seen combinations. A rising trend would mean more examples help; higher is better.

15%20%25%30%35%1020

Takeaway → Setups with the exact same example count scatter from 20% to 33%, so how examples are allocated matters more than how many.

Data table
Total bridge examples in the 240-record budgetTrace-trained allocation conditions
1028.3%
2025%
2433.3%
2833.3%

Numbers from report table (reports/targeted_bridge_allocation_report.md, Allocation Plans + Overall Results)

Technical framing

Total bridge budget does not predict holdout repair — Conditions with identical bridge totals span 20-33% holdout repair, so allocation across families matters more than bridge count.

In the author’s words from the Overview · “Final Result Snapshot”

The full evaluation matrix completed 39/39 jobs. Main recombination-holdout results (table on the experiment page). Key readout: hard_target_trace tied the top aggregate holdout score while covering more target-family successes than the easy-target control. Trace quality was decisive: no-trace and shuffled-trace controls collapsed on the target families.

Overview

Standalone experiment package for testing whether fixed-budget targeted bridge allocation improves trace-conditioned recombination repair compared with uniform bridge allocation.

Design

Every trained condition has exactly 240 training records. Exact bridge examples are drawn from five held-out recombination families. Adding bridge examples removes seen-combination examples, so allocation strategy is the tested variable rather than total dataset size.

Core allocation conditions:

  • uniform2_trace: 2 exact bridge examples for each held-out family.
  • uniform4_trace: 4 exact bridge examples for each held-out family.
  • hard_target_trace: 8 bridge examples for each target family and 2 for each responsive-control family.
  • hard_target_seen_preserving_trace: 6 bridge examples for each target family and 1 for each responsive-control family, matching the uniform4 bridge total.
  • easy_target_control_trace: same bridge total as hard_target_trace, concentrated on responsive-control families.
  • modulo16_trace, length16_trace, tuple16_trace: light single-family high-dose probes, with one target family raised to 16 bridge examples and all other held-out families kept at 2.
  • hard_target_no_trace and hard_target_shuffled_trace: controls for whether the main targeted allocation depends on aligned trace evidence.

Layout

  • configs/experiment.json: fixed design, model, training, and evaluation settings.
  • data/: generated JSONL datasets and dataset manifest.
  • scripts/: dataset generation, LoRA training runner, final evaluation runner, evaluator, and report generator.
  • reports/: training manifests, final evaluation JSON, CSV summaries, and markdown report.
  • figures/: generated plots.
  • logs/experiment_log.md: chronological run log.
  • run_logs/: raw console logs for dataset build, training, evaluation, and report generation.

Large model artifacts are intentionally outside this directory:

The compact experiment directory should contain scripts, logs, data, reports, and figures only. Adapter weights and training checkpoints are separated so this directory can be downloaded without multi-GB model files.

Final Result Snapshot

The full evaluation matrix completed 39/39 jobs. Main recombination-holdout results:

ConditionRecombination Holdout
frozen_trace0.0% (0/60)
uniform2_trace28.3% (17/60)
uniform4_trace25.0% (15/60)
hard_target_trace33.3% (20/60)
hard_target_seen_preserving_trace25.0% (15/60)
easy_target_control_trace31.7% (19/60)
modulo16_trace33.3% (20/60)
length16_trace20.0% (12/60)
tuple16_trace28.3% (17/60)
hard_target_no_trace8.3% (5/60)
hard_target_shuffled_trace10.0% (6/60)

Key readout: hard_target_trace tied the top aggregate holdout score while covering more target-family successes than the easy-target control. Trace quality was decisive: no-trace and shuffled-trace controls collapsed on the target families.

Primary outputs:

Core Commands

Build data:

python experiments/targeted_bridge_allocation/scripts/build_allocation_dataset.py \
  --output-dir experiments/targeted_bridge_allocation/data

Train adapters:

python experiments/targeted_bridge_allocation/scripts/run_training.py --suite all

Run final evaluations:

python experiments/targeted_bridge_allocation/scripts/run_final_evaluations.py --suite all

Generate report:

python experiments/targeted_bridge_allocation/scripts/make_report.py

Report

Rendered from reports/targeted_bridge_allocation_report.md

Question

Does fixed-budget targeted bridge allocation improve trace-conditioned recombination repair compared with uniform bridge allocation?

Design

  • Every trained condition uses a fixed 240-record budget.
  • Uniform baselines allocate exact bridge examples evenly across five held-out recombination families.
  • Targeted conditions concentrate bridge examples on three target families while retaining some bridges for two responsive-control families.
  • A seen-preserving targeted condition keeps total bridge count equal to the uniform4 baseline.
  • An easy-target control spends the same bridge budget as the main targeted condition, but concentrates it on the responsive-control families.
  • Three light single-family probes raise one target family to 16 bridge examples while keeping every other held-out family at 2.
  • No-trace and shuffled-trace controls on the main targeted allocation test whether any allocation effect depends on aligned trace evidence.
  • Checkpoints and adapter weights are stored outside this experiment directory under large_artifacts/targeted_bridge_allocation/models/.

Allocation Plans

PlanBridge totalSeen totalmodulo_sum_labellength_contains_codetuple_branch_labelsorted_contains_countsorted_tuple_affine
Uniform 2 trace1023022222
Uniform 4 trace2022044444
Hard-target trace2821288822
Hard-target seen-preserving trace2022066611
Easy-target control trace282122221111
Modulo-16 trace24216162222
Length-16 trace24216216222
Tuple-16 trace24216221622

Overall Results

ConditionSeen-Combination IIDFormat ShiftRecombination Holdout
Frozen trace0.0% (0/36)0.0% (0/36)0.0% (0/60)
Uniform 2 trace75.0% (27/36)72.2% (26/36)28.3% (17/60)
Uniform 4 trace72.2% (26/36)83.3% (30/36)25.0% (15/60)
Hard-target trace77.8% (28/36)69.4% (25/36)33.3% (20/60)
Hard-target seen-preserving trace75.0% (27/36)72.2% (26/36)25.0% (15/60)
Easy-target control trace80.6% (29/36)80.6% (29/36)31.7% (19/60)
Modulo-16 trace83.3% (30/36)75.0% (27/36)33.3% (20/60)
Length-16 trace80.6% (29/36)77.8% (28/36)20.0% (12/60)
Tuple-16 trace75.0% (27/36)69.4% (25/36)28.3% (17/60)
Hard-target no-trace27.8% (10/36)19.4% (7/36)8.3% (5/60)
Hard-target shuffled-trace train30.6% (11/36)25.0% (9/36)10.0% (6/60)

Allocation Comparison

ConditionSeen-Combination IIDFormat ShiftRecombination Holdout
Uniform 2 trace75.0% (27/36)72.2% (26/36)28.3% (17/60)
Uniform 4 trace72.2% (26/36)83.3% (30/36)25.0% (15/60)
Hard-target trace77.8% (28/36)69.4% (25/36)33.3% (20/60)
Hard-target seen-preserving trace75.0% (27/36)72.2% (26/36)25.0% (15/60)
Easy-target control trace80.6% (29/36)80.6% (29/36)31.7% (19/60)

Light Single-Family Probes

ConditionSeen-Combination IIDFormat ShiftRecombination Holdout
Uniform 2 trace75.0% (27/36)72.2% (26/36)28.3% (17/60)
Modulo-16 trace83.3% (30/36)75.0% (27/36)33.3% (20/60)
Length-16 trace80.6% (29/36)77.8% (28/36)20.0% (12/60)
Tuple-16 trace75.0% (27/36)69.4% (25/36)28.3% (17/60)

Prompt Ablations

ConditionSeen-Combination IIDFormat ShiftRecombination Holdout
Hard-target trace77.8% (28/36)69.4% (25/36)33.3% (20/60)
Hard-target trace, no-trace prompt19.4% (7/36)16.7% (6/36)8.3% (5/60)
Hard-target trace, shuffled-trace prompt8.3% (3/36)5.6% (2/36)6.7% (4/60)

Recombination Holdout By Family

FamilyUniform 2 traceUniform 4 traceHard-target traceHard-target seen-preserving traceEasy-target control traceModulo-16 traceLength-16 traceTuple-16 trace
length_contains_code0.0% (0/12)16.7% (2/12)25.0% (3/12)0.0% (0/12)8.3% (1/12)0.0% (0/12)25.0% (3/12)0.0% (0/12)
modulo_sum_label0.0% (0/12)8.3% (1/12)0.0% (0/12)41.7% (5/12)0.0% (0/12)41.7% (5/12)0.0% (0/12)16.7% (2/12)
sorted_contains_count100.0% (12/12)58.3% (7/12)100.0% (12/12)66.7% (8/12)100.0% (12/12)66.7% (8/12)41.7% (5/12)58.3% (7/12)
sorted_tuple_affine41.7% (5/12)41.7% (5/12)25.0% (3/12)16.7% (2/12)33.3% (4/12)58.3% (7/12)33.3% (4/12)50.0% (6/12)
tuple_branch_label0.0% (0/12)0.0% (0/12)16.7% (2/12)0.0% (0/12)16.7% (2/12)0.0% (0/12)0.0% (0/12)16.7% (2/12)

Hard-Target Holdout By Factor

Factorrepair@1
aggregation50.0% (12/24)
arithmetic16.7% (8/48)
branching13.9% (5/36)
length25.0% (3/12)
modulo0.0% (0/12)
ordering62.5% (15/24)
sequence_iteration100.0% (12/12)
string_format35.4% (17/48)
string_match62.5% (15/24)
string_normalization100.0% (12/12)
tuple_access20.8% (5/24)

Readout

  • Best core recombination result: Hard-target trace at 33.3% (20/60).
  • Hard-target vs uniform2: 33.3% (20/60) vs 28.3% (17/60), delta 5.0%.
  • Seen-preserving target vs uniform4: 25.0% (15/60) vs 25.0% (15/60), delta 0.0%.
  • Hard-target vs easy-target budget control: 33.3% (20/60) vs 31.7% (19/60), delta 1.7%.

Figures

Artifact Layout

Experiment log 7

Show the running log (7 entries, 2026-06-22)

2026-06-22

  • Created standalone experiment directory at experiments/targeted_bridge_allocation/.
  • Created large-artifact directory at large_artifacts/targeted_bridge_allocation/models/.
  • Defined the central question: whether targeted allocation of a fixed bridge-example budget improves trace-conditioned recombination repair compared with uniform allocation.

  • Planned a fixed-budget allocation experiment:

    • every trained condition has 240 records,
    • uniform baselines allocate 2 or 4 exact bridge examples to every held-out family,
    • the main targeted condition allocates 8 bridge examples to each target family and 2 to each responsive-control family,
    • a seen-preserving targeted condition keeps the total bridge count equal to the uniform4 baseline,
    • an easy-target control spends the same bridge budget as the main targeted condition on responsive-control families,
    • three light single-family probes raise one target family at a time to 16 bridge examples,
    • no-trace and shuffled-trace controls test whether the targeted allocation depends on aligned trace evidence.

  • Implemented the experiment package:

    • scripts/build_allocation_dataset.py
    • scripts/eval_allocation.py
    • scripts/run_training.py
    • scripts/run_final_evaluations.py
    • scripts/make_report.py

  • The builder validates every generated record before writing:

    • wrong implementation fails visible tests,
    • target corrective diff applies,
    • target implementation passes visible and hidden tests,
    • visible and hidden inputs are disjoint.

Next step: build and validate the datasets, then inspect the manifest before starting training.

Dataset Build

  • Ran scripts/build_allocation_dataset.py with seed 20260622.
  • Completed executable validation for every generated record.

  • Wrote eight 240-record training files:

    • repair_train_uniform2.jsonl
    • repair_train_uniform4.jsonl
    • repair_train_hard_target.jsonl
    • repair_train_hard_target_seen_preserving.jsonl
    • repair_train_easy_target_control.jsonl
    • repair_train_modulo16.jsonl
    • repair_train_length16.jsonl
    • repair_train_tuple16.jsonl

  • Wrote three fixed evaluation splits:

    • repair_val_seen_iid.jsonl: 36 records.
    • repair_val_format_shift.jsonl: 36 records.
    • repair_val_recombination_holdout.jsonl: 60 records.

  • Verified allocation invariants from data/dataset_manifest.json:

    • uniform2: 2 bridge examples for every held-out factor pair.
    • uniform4: 4 bridge examples for every held-out factor pair.
    • hard_target: 8 bridge examples for aggregation+modulo, branching+tuple_access, and length+string_match; 2 each for ordering+string_match and ordering+tuple_access.
    • hard_target_seen_preserving: 6 target-family bridge examples and 1 responsive-control bridge example for each responsive-control pair; total bridge count matches uniform4.
    • easy_target_control: same bridge total as hard_target, but 11 bridge examples each for ordering+string_match and ordering+tuple_access.
    • modulo16: 16 bridge examples for aggregation+modulo; 2 for every other held-out factor pair.
    • length16: 16 bridge examples for length+string_match; 2 for every other held-out factor pair.
    • tuple16: 16 bridge examples for branching+tuple_access; 2 for every other held-out factor pair.
  • Compact experiment directory after dataset build: about 15 MB.
  • Large artifact directory remains empty before training.
  • Confirmed no .safetensors, .bin, .pt, or .pth files are present in the compact experiment directory.

Next step: train the allocation adapters and controls.

Training

  • Ran scripts/run_training.py --suite all.
  • Completed 10/10 LoRA training jobs.
  • Wrote adapter and checkpoint artifacts under large_artifacts/targeted_bridge_allocation/models/.

  • Training runtimes:

    • uniform2_trace: 429.947 seconds.
    • uniform4_trace: 442.551 seconds.
    • hard_target_trace: 446.382 seconds.
    • hard_target_seen_preserving_trace: 420.899 seconds.
    • easy_target_control_trace: 441.100 seconds.
    • modulo16_trace: 428.771 seconds.
    • length16_trace: 444.229 seconds.
    • tuple16_trace: 426.457 seconds.
    • hard_target_no_trace: 437.514 seconds.
    • hard_target_shuffled_trace: 416.731 seconds.

  • Final checkpoint eval-loss pattern:

    • hard_target_trace had the strongest loss profile among trained adapters.
    • uniform4_trace was close.
    • hard_target_no_trace and hard_target_shuffled_trace were much weaker, indicating that trace quality mattered during training.

Next step: run the final 39-job evaluation matrix.

Final Evaluation

  • Ran scripts/run_final_evaluations.py --suite all.
  • Completed 39/39 evaluation jobs.
  • Wrote per-job JSON results under reports/final/.
  • Wrote final job manifest at reports/final/final_evaluation_jobs.json.

  • Overall recombination-holdout results:

    • frozen_trace: 0/60.
    • uniform2_trace: 17/60.
    • uniform4_trace: 15/60.
    • hard_target_trace: 20/60.
    • hard_target_seen_preserving_trace: 15/60.
    • easy_target_control_trace: 19/60.
    • modulo16_trace: 20/60.
    • length16_trace: 12/60.
    • tuple16_trace: 17/60.
    • hard_target_no_trace: 5/60.
    • hard_target_shuffled_trace: 6/60.
    • hard_target_trace with no-trace prompt: 5/60.
    • hard_target_trace with shuffled-trace prompt: 4/60.

  • Main target-family holdout counts:

    • hard_target_trace: length_contains_code 3/12, modulo_sum_label 0/12, tuple_branch_label 2/12.
    • easy_target_control_trace: length_contains_code 1/12, modulo_sum_label 0/12, tuple_branch_label 2/12.
    • modulo16_trace: length_contains_code 0/12, modulo_sum_label 5/12, tuple_branch_label 0/12.
    • length16_trace: length_contains_code 3/12, modulo_sum_label 0/12, tuple_branch_label 0/12.
    • tuple16_trace: length_contains_code 0/12, modulo_sum_label 2/12, tuple_branch_label 2/12.
    • hard_target_no_trace: 0/12 for all three target families.
    • hard_target_shuffled_trace: 0/12 for all three target families.

  • Responsive-control holdout behavior was strong and often dominated aggregate scores:

    • easy_target_control_trace reached 12/12 on sorted_contains_count and 4/12 on sorted_tuple_affine.
    • hard_target_trace reached 12/12 on sorted_contains_count and 3/12 on sorted_tuple_affine.
    • modulo16_trace reached 8/12 on sorted_contains_count and 7/12 on sorted_tuple_affine.

Report Generation

Artifact Split Verification

  • Compact experiment directory: 24 MB.
  • Large artifact directory: 13 GB.
  • Confirmed no .safetensors, .bin, .pt, or .pth files in experiments/targeted_bridge_allocation/.

  • Large artifact tree contains:

    • 10 adapter directories.
    • 410 files.
    • 30 checkpoint directories.
  • Wrote reports/large_artifacts_manifest.md.

Interpretation

  • hard_target_trace is the strongest allocation choice for balanced recombination holdout behavior: it tied the top aggregate result at 20/60 and produced target-family successes on length and tuple while preserving strong responsive-control behavior.
  • easy_target_control_trace nearly matched the aggregate at 19/60, but most of its holdout score came from responsive-control families. It did not improve modulo and only added one length success.
  • modulo16_trace matched the top aggregate score and moved modulo_sum_label to 5/12, but it did not transfer to length or tuple.
  • length16_trace moved length_contains_code to 3/12 but lowered aggregate holdout to 12/60.
  • tuple16_trace did not improve tuple_branch_label beyond 2/12.
  • Trace alignment is essential. Removing traces, shuffling traces at inference, training without traces, and training on shuffled traces all collapsed target-family holdout repair.
  • The next high-impact direction is not simply increasing single-family bridge intensity. A better follow-up would preserve aligned traces while changing the training mix to support target-family mechanisms without sacrificing responsive-control coverage.

Figures 2

final repair by condition split
final repair by condition split · figures/
recombination holdout by family
recombination holdout by family · figures/

Data files 4

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗