Research log Small Model Experimentation
GitHub

Feature-Factorized Rule Diversity

Finished2026-06-21imported · line ZAlgorithmic Memory and RetrievalBenchmark GeneralizationEvidence-Conditioned SelectionGitHub ↗
Worked steps carry transfer, not the practice mix

The one idea you need

Imagine teaching an apprentice to fix broken programs by drilling repairs. You can drill single tricks, drill combined tricks, or split the same practice time between both. Then test on bug types never drilled. Which practice diet best prepares them for genuinely new combinations?

The question

Does splitting practice between single-skill and combined-skill repairs help a small model fix brand-new bug combinations better than drilling either kind of practice alone?

What we found

No. All three practice diets fixed only about 1 in 5 brand-new bug combinations, so mixing bought nothing over drilling either kind alone. What actually mattered was showing worked, step-by-step repair reasoning during training: strip it out and success collapsed to about 1 in 30. And most of that thin transfer came from a single lucky bug family, not broad skill.

Why it matters

When fine-tuning small models to reach unseen combinations, spend effort on including step-by-step worked solutions, not on balancing which skill mixtures you cover. And check transfer family-by-family, since one easy family can inflate an otherwise flat average.

Fixing brand-new combinationsabout 1 in 5single, combined, and mixed practice all scored 20-23%
Worked reasoning removed from training22% → 3%success on new combinations collapses without step-by-step traces
Mastering familiar problemsup to 92%each diet aces the problem kind it drilled on
New bug families actually cracked1 of 5one family solved every time; the other four near zero
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Question
    2. Design
    3. Overall Results
    4. Prompt Ablations
    5. Recombination Holdout By Family
    6. Mixed Trace Holdout By Factor
    7. Readout
    8. Figures
    9. Artifact Layout
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 6

Each practice diet aces its own problems but ties on brand-new ones

How to read

Grouped bars show how often each training recipe fixes bugs across three tests: familiar single-skill, familiar combined-skill, and brand-new combinations. Taller is better; the flat gray bar is the untrained model.

0%25%50%75%100%singleton IIDsingleton IIDcomposite IIDcomposite IIDrecombination holdoutrecombination holdout

Takeaway → On the brand-new-combination test all three trained recipes sit at nearly equal low height, so mixing practice adds no transfer over either recipe alone.

Data table
evaluation splitfrozen (no adapter)singleton factors, tracecomposite factors, tracemixed factors, trace
singleton IID0%85%10%57.5%
composite IID0%22.2%91.7%69.4%
recombination holdout0%20%23.3%21.7%

Numbers from reports/final_results.csv

Technical framing

Repair success by training coverage and eval split — Each coverage recipe masters its own distribution, but all three land at ~20-23% on held-out recombinations — mixing buys no extra transfer.

Worked reasoning is required, in both practice and solving

How to read

Bars show repair success on brand-new bug combinations across five setups that keep, remove, or scramble the step-by-step worked reasoning during practice or at solving time. Taller is better.

0%10%20%30%trace train + trace prompttrace train + trace prompt21.7%no-trace trainno-trace train3.3%shuffled-trace trainshuffled-trace train6.7%trace adapter, no-trace prompttrace adapter, no-trace prompt0%trace adapter, shuffled-trace prompttrace adapter, shuffled-trace pro…1.7%

Takeaway → Only the setup keeping intact reasoning throughout rises; every removed or scrambled version drops to near zero, so the reasoning is essential.

Data table
control conditionrepair@1
trace train + trace prompt21.7%
no-trace train3.3%
shuffled-trace train6.7%
trace adapter, no-trace prompt0%
trace adapter, shuffled-trace prompt1.7%

Numbers from reports/final_results.csv

Technical framing

Aligned traces are necessary: mixed-coverage controls on recombination holdout — Removing or corrupting traces at train or eval time collapses recombination repair from 21.7% to near zero.

Nearly all apparent transfer comes from one bug family

How to read

For each of five brand-new bug families, grouped bars show repair success under the three practice recipes, one color per recipe. Taller is better.

0%25%50%75%100%contains+length codecontains+length code0%0%0%parity offsetparity offset0%0%0%sorted joinsorted join100%100%100%sum parity shiftsum parity shift0%16.7%8.3%tuple max labeltuple max label0%0%0%

Takeaway → Only the sorted-join family reaches full height for every recipe while the other four stay near the floor, so transfer is confined to one family.

Data table
held-out bug familysingleton factors, tracecomposite factors, tracemixed factors, trace
contains+length code0%0%0%
parity offset0%0%0%
sorted join100%100%100%
sum parity shift0%16.7%8.3%
tuple max label0%0%0%

Numbers from reports/final_results_by_family.csv

Technical framing

Recombination transfer is concentrated in one family — All the apparent recombination transfer comes from the sorted-join family (12/12 for every adapter); the other four families stay near zero.

The worked reasoning must be present when solving, not just when practicing

How to read

Grouped bars compare one trained model tested with intact, removed, or scrambled worked reasoning, across familiar single-skill, familiar combined-skill, and brand-new tests. Taller is better.

0%20%40%60%80%Singleton IIDSingleton IID57.5%5%5%Composite IIDComposite IID69.4%13.9%8.3%Recombination holdoutRecombination holdout21.7%0%1.7%

Takeaway → Removing or scrambling the reasoning at solving time flattens every group to near zero, so the reasoning must accompany the actual problem.

Data table
Evaluation splitMixed trace adapter, trace promptSame adapter, no-trace promptSame adapter, shuffled-trace prompt
Singleton IID57.5%5%5%
Composite IID69.4%13.9%8.3%
Recombination holdout21.7%0%1.7%

Numbers from experiments/feature_factorized_rule_diversity/reports/feature_factorized_rule_diversity_report.md (Prompt Ablations table)

Technical framing

Traces must also be present at evaluation time, not just in training — Removing or shuffling the trace at eval time collapses the trained mixed adapter to near zero on every split.

Transfer to new problems is skill-specific, not broad

How to read

Bars show the mixed-diet model's repair success on brand-new combinations, one bar per underlying skill involved, sorted from highest to lowest. Taller is better.

0%25%50%75%100%sequence_iterationsequence_iteration100%sortingsorting100%string_normalizationstring_normalization100%string_formatstring_format33.3%aggregationaggregation4.2%modulomodulo4.2%arithmeticarithmetic2.1%branchingbranching2.1%string_matchstring_match0%tuple_accesstuple_access0%

Takeaway → A few skills reach full height while most sit near the floor, so the model's transfer is narrow and skill-specific rather than general.

Data table
Primitive factor in held-out ruleMixed factors, trace (recombination holdout)
sequence_iteration100%
sorting100%
string_normalization100%
string_format33.3%
aggregation4.2%
modulo4.2%
arithmetic2.1%
branching2.1%
string_match0%
tuple_access0%

Numbers from experiments/feature_factorized_rule_diversity/reports/feature_factorized_rule_diversity_report.md (Mixed Trace Holdout By Factor table)

Technical framing

Mixed adapter's holdout transfer is confined to a few primitive factors — Three factors transfer perfectly while most sit near zero; the length factor (0.0%, omitted) also fails, so transfer is factor-specific.

Failures are misreading the rule, not bad formatting

How to read

Paired bars per training recipe: one for how often the code fix is well-formed and applies cleanly, one for how often it passes the hidden tests. Taller is better.

0%25%50%75%100%Singleton factors, traceSingleton factors, trace100%20%Composite factors, traceComposite factors, trace100%23.3%Mixed factors, traceMixed factors, trace100%21.7%Mixed, no traceMixed, no trace96.7%3.3%Mixed, shuffled traceMixed, shuffled trace100%6.7%

Takeaway → The 'applies cleanly' bar is near full for every recipe while 'passes tests' stays low, so failures come from misunderstanding the rule, not formatting.

Data table
Training conditionPatch applies cleanlyHidden tests pass
Singleton factors, trace100%20%
Composite factors, trace100%23.3%
Mixed factors, trace100%21.7%
Mixed, no trace96.7%3.3%
Mixed, shuffled trace100%6.7%

Numbers from experiments/feature_factorized_rule_diversity/reports/final_results.csv (patch_apply_rate, hidden_pass_rate, recombination_holdout rows)

Technical framing

Holdout failures are semantic, not formatting: patches apply but tests fail — Every condition emits well-formed, applicable patches on the holdout, so the 20-point gap to hidden-test pass is pure rule misunderstanding.

In the author’s words from the Overview

The best recombination score was the composite-trace adapter at 14/60; mixed trace reached 13/60 and mostly transferred only the sorted_join_holdout family.

Overview

Standalone experiment package for testing whether trace-conditioned repair transfer is driven more by isolated primitive factor coverage, analogous multi-factor composition coverage, or a fixed-budget mixture of both.

Layout

  • configs/experiment.json: fixed design, model, and hyperparameter settings.
  • data/: generated JSONL datasets and dataset manifest.
  • scripts/: dataset generation, LoRA training runner, final evaluation runner, and report generator.
  • reports/: training manifests, final evaluation JSON, CSV summaries, and markdown report.
  • figures/: generated plots.
  • logs/experiment_log.md: chronological run log.

Final Report

  • Main report: reports/feature_factorized_rule_diversity_report.md
  • Detailed log: logs/experiment_log.md
  • Final results CSV: reports/final_results.csv
  • Final evaluation manifest: reports/final/final_evaluation_jobs.json

Key readout: aligned trace supervision is necessary, but fixed-budget mixed factor coverage did not improve broad recombination. The best recombination score was the composite-trace adapter at 14/60; mixed trace reached 13/60 and mostly transferred only the sorted_join_holdout family.

Large model artifacts are intentionally outside this directory:

Core Commands

Build data:

python experiments/feature_factorized_rule_diversity/scripts/build_factorized_dataset.py \
  --output-dir experiments/feature_factorized_rule_diversity/data

Train adapters:

python experiments/feature_factorized_rule_diversity/scripts/run_training.py --suite all

Run final evaluations:

python experiments/feature_factorized_rule_diversity/scripts/run_final_evaluations.py --suite all

Generate report:

python experiments/feature_factorized_rule_diversity/scripts/make_report.py

Report

Rendered from reports/feature_factorized_rule_diversity_report.md

Question

Does held-out rule repair improve more when training examples cover isolated primitive factors, analogous multi-factor compositions, or a fixed-budget mixture of both?

Design

  • Three trace-trained adapters use the same 240-record budget: singleton factors only, composite factors only, and a mixed singleton/composite allocation.
  • Two controls use the mixed allocation with traces removed or trace outputs shuffled during training.
  • Final evaluation uses singleton IID, composite IID, and recombination holdout splits.
  • Checkpoints and adapter weights are stored outside this experiment directory under large_artifacts/feature_factorized_rule_diversity/models/.

Overall Results

ConditionSingleton IIDComposite IIDRecombination Holdout
Frozen trace0.0% (0/40)0.0% (0/36)0.0% (0/60)
Singleton factors, trace85.0% (34/40)22.2% (8/36)20.0% (12/60)
Composite factors, trace10.0% (4/40)91.7% (33/36)23.3% (14/60)
Mixed factors, trace57.5% (23/40)69.4% (25/36)21.7% (13/60)
Mixed factors, no trace train/eval5.0% (2/40)16.7% (6/36)3.3% (2/60)
Mixed factors, shuffled trace train5.0% (2/40)13.9% (5/36)6.7% (4/60)

Prompt Ablations

ConditionSingleton IIDComposite IIDRecombination Holdout
Mixed factors, trace57.5% (23/40)69.4% (25/36)21.7% (13/60)
Mixed trace adapter, no trace prompt5.0% (2/40)13.9% (5/36)0.0% (0/60)
Mixed trace adapter, shuffled trace prompt5.0% (2/40)8.3% (3/36)1.7% (1/60)

Recombination Holdout By Family

FamilySingleton factors, traceComposite factors, traceMixed factors, trace
contains_length_code_holdout0.0% (0/12)0.0% (0/12)0.0% (0/12)
parity_offset_holdout0.0% (0/12)0.0% (0/12)0.0% (0/12)
sorted_join_holdout100.0% (12/12)100.0% (12/12)100.0% (12/12)
sum_parity_shift_holdout0.0% (0/12)16.7% (2/12)8.3% (1/12)
tuple_max_label_holdout0.0% (0/12)0.0% (0/12)0.0% (0/12)

Mixed Trace Holdout By Factor

Factorrepair@1
aggregation4.2% (1/24)
arithmetic2.1% (1/48)
branching2.1% (1/48)
length0.0% (0/12)
modulo4.2% (1/24)
sequence_iteration100.0% (12/12)
sorting100.0% (12/12)
string_format33.3% (12/36)
string_match0.0% (0/12)
string_normalization100.0% (12/12)
tuple_access0.0% (0/12)

Readout

  • Best core recombination result: Composite factors, trace at 23.3% (14/60).
  • Mixed trace vs controls on recombination: trace 21.7%, no trace 3.3%, shuffled-trace train 6.7%.

Figures

Artifact Layout

Experiment log 6

Show the running log (6 entries, 2026-06-21)

2026-06-21

  • Created standalone experiment directory at experiments/feature_factorized_rule_diversity/.
  • Created large-artifact directory at large_artifacts/feature_factorized_rule_diversity/models/.
  • Defined the central question: compare fixed-budget primitive factor coverage, analogous composition coverage, and a mixed allocation for trace-conditioned held-out recombination repair.

  • Implemented scripts/build_factorized_dataset.py.

    • The builder writes three 240-record training sets.
    • It writes three evaluation splits: singleton IID, composite IID, and recombination holdout.

    • Each generated record is validated before writing:

      • the wrong implementation fails visible tests,
      • the target corrective diff applies,
      • the target implementation passes visible and hidden tests,
      • hidden inputs are disjoint from visible inputs,
      • visible expected outputs appear in the failing trace.
  • Implemented scripts/eval_factorized.py with summary outputs by split, family, and factor tag.
  • Implemented scripts/run_training.py so all adapter training jobs and hyperparameters are captured in reports/training/training_jobs.json.
  • Implemented scripts/run_final_evaluations.py for the full final evaluation matrix.
  • Implemented scripts/make_report.py for CSV summaries, plots, markdown report, and large-artifact manifest refresh.

Next step: build and validate the dataset, then train the five planned adapters.

Dataset Build Pass 1

  • Started dataset generation with full pytest validation.
  • Interrupted after the run remained silent for too long during the first split.
  • Diagnosis: validation was active, but the builder had no progress logging. The test template also used a looped pytest test, which can hide later visible counterexamples after the first failure.

  • Fix applied:

    • converted generated tests to pytest.mark.parametrize cases,
    • added per-family progress logging to stderr.

Dataset Build Pass 2

  • Completed dataset generation and validation.

  • Wrote 856 total records:

    • train_singletons: 240 records, 10 singleton families x 24 records.
    • train_composites: 240 records, 6 composite families x 40 records.
    • train_mixed: 240 records, 8 mixed families x 30 records.
    • val_singleton_iid: 40 records.
    • val_composite_iid: 36 records.
    • val_recombination_holdout: 60 records, 5 held-out recombination families x 12 records.
  • Confirmed compact experiment directory size after dataset build: about 13 MB.
  • Confirmed large_artifacts/feature_factorized_rule_diversity/ remains empty before training.

Next step: train five LoRA adapters: singleton trace, composite trace, mixed trace, mixed no-trace, and mixed shuffled-trace.

Training

  • Completed all five planned LoRA training jobs.
  • Training jobs and exact commands are recorded in reports/training/training_jobs.json.
  • Console output is recorded in run_logs/training_console.log.

  • Adapter directories:

  • Approximate adapter directory size after training: 1.3 GB each.

  • Final checkpoint eval losses:

    • singleton trace on singleton IID: about 0.0071.
    • composite trace on composite IID: about 0.0018.
    • mixed trace on recombination holdout: about 0.0886.
    • mixed no-trace on recombination holdout: about 0.2108.
    • mixed shuffled-trace on recombination holdout: about 0.1782.

Next step: run final generation evaluations for frozen baseline, trace adapters, controls, and prompt ablations.

Final Generation Evaluation

  • Completed the full 24-job final evaluation suite.
  • Final evaluation jobs and result paths are recorded in reports/final/final_evaluation_jobs.json.
  • Console output is recorded in run_logs/final_evaluation_console.log.

  • Generated CSV summaries:

    • reports/final_results.csv
    • reports/final_results_by_family.csv
    • reports/final_results_by_factor.csv

  • Generated report and figures:

Core repair@1 results:

ConditionSingleton IIDComposite IIDRecombination Holdout
Frozen trace0/400/360/60
Singleton factors, trace34/408/3612/60
Composite factors, trace4/4033/3614/60
Mixed factors, trace23/4025/3613/60
Mixed factors, no trace train/eval2/406/362/60
Mixed factors, shuffled trace train2/405/364/60

Prompt ablations for the mixed trace adapter:

Prompt conditionSingleton IIDComposite IIDRecombination Holdout
Normal aligned trace prompt23/4025/3613/60
No-trace prompt2/405/360/60
Shuffled-trace prompt2/403/361/60

Recombination holdout family pattern for the three main trace adapters:

FamilySingleton traceComposite traceMixed trace
contains_length_code_holdout0/120/120/12
parity_offset_holdout0/120/120/12
sorted_join_holdout12/1212/1212/12
sum_parity_shift_holdout0/122/121/12
tuple_max_label_holdout0/120/120/12

Interpretation:

  • The strongest in-distribution behavior is narrow specialization: singleton trace is best on singleton IID, and composite trace is best on composite IID.
  • The fixed-budget mixed trace adapter improves breadth relative to single-domain adapters, but it does not improve the recombination holdout. It reaches 13/60, slightly below composite trace at 14/60 and only slightly above singleton trace at 12/60.
  • The apparent recombination success is mostly not broad recombination. All three trace adapters solve sorted_join_holdout perfectly, but they fail three of five holdout families completely. Composite trace gets only 2/12 on sum_parity_shift_holdout; mixed trace gets only 1/12.
  • Aligned traces are clearly important. Removing traces during training collapses mixed performance to 2/40, 6/36, and 2/60. Shuffling training traces also collapses to 2/40, 5/36, and 4/60.
  • The aligned mixed trace adapter is also highly dependent on the inference prompt contract. With no trace prompt it drops to 2/40, 5/36, and 0/60. With shuffled trace prompts it drops to 2/40, 3/36, and 1/60.

Decision readout:

  • This experiment does not support simple fixed-budget mixing as the next scaling direction for broad recombination.
  • The most useful positive signal is that aligned traces carry real supervision. The most important failure is that traces and examples do not yet teach reusable factor abstractions for length/string-match/tuple/modulo recombinations.
  • The next experiment should preserve aligned trace supervision, but change the data design toward factor-balanced recombination coverage and trace consistency diagnostics instead of simply mixing singleton and composite examples.

Artifact Verification

Figures 2

final repair by condition split
final repair by condition split · figures/
recombination holdout by family
recombination holdout by family · figures/

Data files 4

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗