Rule-Family Diversity Scaling Experiment
The one idea you need
Give a mechanic the same 240 practice repairs. Spread across only 3 engine types, they just memorize those engines. Spread across 12 types, they instead learn to read the diagnostic printout itself, and can then fix an engine model they have never seen before.
The question
With a fixed budget of practice examples for teaching a small model to fix buggy code, does covering more kinds of problems help it fix brand-new kinds?
What we found
Yes. Holding training at 240 examples but spreading them across 12 kinds of problems instead of 3 lifted fixes on completely unseen problem types from 0% to 29%, while fixes on familiar problems stayed high. But it only works because the model reads the failing-run evidence: strip that evidence and transfer drops to 0%. And the gain is lopsided, with one unseen type hitting 92% while two others never worked at all.
Why it matters
When fine-tuning a small model to generalize, spend a fixed example budget on breadth of problem types, not depth on a few. And feed it real execution-failure evidence at both training and answer time, or transfer to new types vanishes.
On this page
Results at a glance 7
How to read
Horizontal axis is how many problem-type varieties were in training (3, 6, 12); vertical axis is the share of buggy tasks fixed, higher is better. One line tracks familiar problems, one the same problems in new formats, one brand-new problem types.
Takeaway → The brand-new-type line barely lifts off zero, going 0% then just 4%, until 12 varieties, where it jumps to 29%, while familiar-problem fixes stay high throughout.
Data table
| training rule families | in-distribution (base IID) | format holdout | held-out rule families |
|---|---|---|---|
| 3 | 94.4% | 50% | 0% |
| 6 | 77.8% | 52.8% | 4.2% |
| 12 | 86.1% | 44.4% | 29.2% |
Numbers from reports/final_core_results.csv
Technical framing
Repair success vs number of training rule families — Transfer to unseen rule families only emerges at 12 training families (0% -> 4% -> 29%), while in-distribution repair stays high throughout.
How to read
Each bar is a setup's share of never-seen-type bugs fixed, taller is better. First bar is the full trained model with real evidence; the rest drop or scramble that evidence at answer or training time, or use the untouched base model.
Takeaway → Only the full setup with real evidence reaches 29%; every stripped or scrambled variant collapses to 4% or below, so the evidence does the work.
Data table
| condition | repair@1 |
|---|---|
| trace-trained + trace prompt | 29.2% |
| trace-trained, no-trace prompt | 0% |
| trace-trained, shuffled-trace prompt | 4.2% |
| no-trace training | 2.1% |
| shuffled-trace training | 2.1% |
| frozen base | 0% |
Numbers from reports/final_core_results.csv, reports/final_ablation_results.csv
Technical framing
Held-out rule-family repair: 12-family trace training vs controls and prompt ablations — The 29% transfer requires real traces at both training and test time: removing or shuffling the trace at either stage collapses it to near zero.
How to read
Each bar is one never-seen problem type, with 12 tasks each; height is the share fixed, higher is better.
Takeaway → One type is nearly solved at 92% and another partly at 25%, while two get zero fixes, so the gain is narrow, not broad.
Data table
| held-out rule family | 12-family trace-trained |
|---|---|
| sorted_join | 91.7% |
| quadratic_shift | 25% |
| parity_offset | 0% |
| tuple_max | 0% |
Numbers from reports/final_trace_by_family.csv
Technical framing
Where the 12-family model's transfer comes from: held-out families, one by one — Transfer is concentrated: one held-out family is nearly solved (92%), one partially (25%), and two show zero transfer.
How to read
Bars show the share of familiar bugs fixed for three bug types, grouped by 3, 6, and 12 training varieties; higher is better.
Takeaway → Two bug types stay perfectly fixed at every variety level, and all the movement lands on the linear-formula bug type.
Data table
| Bug family (base IID split) | 3 families | 6 families | 12 families |
|---|---|---|---|
| affine_int | 83.3% | 33.3% | 58.3% |
| slug_affix | 100% | 100% | 100% |
| threshold_label | 100% | 100% | 100% |
Numbers from reports/final_trace_by_family.csv
Technical framing
Trained-family repair on in-distribution bugs as diversity grows — Adding families leaves slug and threshold repairs perfect; all in-distribution interference lands on affine_int.
How to read
Bars show the share of same-problems-in-new-formats bugs fixed for three bug types, grouped by 3, 6, and 12 training varieties; higher is better.
Takeaway → The threshold-labeling bug type stays near-perfect across formats, the linear-formula type is stuck near the bottom, and the text-slug type slips at the widest variety.
Data table
| Bug family (format holdout split) | 3 families | 6 families | 12 families |
|---|---|---|---|
| affine_int | 8.3% | 8.3% | 8.3% |
| slug_affix | 50% | 58.3% | 25% |
| threshold_label | 91.7% | 91.7% | 100% |
Numbers from reports/final_trace_by_family.csv
Technical framing
Format-holdout repair by bug family: robustness is family-specific — Under new surface formats threshold_label stays near-perfect, affine_int is stuck at 1/12, and slug_affix degrades at 12 families.
How to read
Horizontal axis is training variety (3, 6, 12); vertical axis is a share on new-type problems, higher is better. One line is how often the model names the correct fix rule, the other how often it fully fixes the bug.
Takeaway → The correct-label line climbs steadily while full fixes stay flat until 12 varieties, so partial understanding builds before complete repairs arrive.
Data table
| Number of training rule families | Correct rule marker present | Repair@1 |
|---|---|---|
| 3 | 22.9% | 0% |
| 6 | 33.3% | 4.2% |
| 12 | 45.8% | 29.2% |
Numbers from reports/final_core_results.csv
Technical framing
On held-out families, partial rule-marker progress precedes full repairs — Marker presence climbs steadily with diversity while full repairs only catch up at 12 families, hinting transfer builds gradually.
How to read
Bars show median seconds to produce one attempt on new-type problems for each setup; there is no better direction, this just checks whether some setups simply ran longer.
Takeaway → Every trained setup takes about eight seconds regardless of variety, so the widest-variety gain is not explained by more time per attempt.
Data table
| Model condition (rule holdout split) | Median seconds per attempt |
|---|---|
| Frozen base | 3.98 |
| 3-family trace | 8.07 |
| 6-family trace | 7.75 |
| 12-family trace | 7.92 |
| 12-family no-trace | 7.78 |
| 12-family shuffled-trace | 7.75 |
Numbers from reports/final_core_results.csv
Technical framing
Generation time on held-out families: transfer is not bought with extra decoding — All trained adapters spend about 8s per attempt regardless of diversity, so the 12-family transfer gain is not extra test-time compute.
In the author’s words from the Report · “Abstract”
This experiment tests whether increasing rule-family diversity in trace-conditioned repair training improves transfer to unseen rule structures when the total number of training records is fixed. Three trace-conditioned LoRA adapters were trained on 240 examples each, using 3, 6, or 12 rule families. The evaluation separates base-IID repair, format holdout, and fully held-out rule-family transfer. Controls test frozen-model behavior, no-trace training, shuffled-trace training, and prompt-time trace ablations.
Overview
This directory contains the small, download-friendly artifacts for a standalone rule-family diversity scaling experiment.
Large generated artifacts such as model adapters and checkpoints are stored separately in:
/workspace/large_artifacts/rule_family_diversity_scaling/
Contents
configs/: experiment configuration.data/: generated JSONL datasets and dataset manifest.figures/: report figures generated byscripts/make_report.py.logs/: detailed experiment log.reports/: evaluation JSON, CSV summaries, and final paper.scripts/: dataset builder, evaluator, final evaluation runner, and report generator.
Main Report
Read reports/rule_family_diversity_scaling_paper.md after final report generation.
Headline final results:
- Best base-IID repair among trace diversity scales:
scale3_trace, 34/36. - Best format-holdout repair among trace diversity scales:
scale6_trace, 19/36. - Best held-out rule-family repair among trace diversity scales:
scale12_trace, 14/48. scale12_tracerule-holdout transfer collapses under prompt ablations: no-trace prompt 0/48, shuffled-trace prompt 2/48.- Scale12 no-trace training and shuffled-trace training controls each reached only 1/48 on rule holdout.
Large Artifacts
Adapters and checkpoints live in:
/workspace/large_artifacts/rule_family_diversity_scaling/models/
Do not include that directory when downloading only the small package.
Report
Rendered from reports/rule_family_diversity_scaling_paper.md
Date: 2026-06-21
Abstract
This experiment tests whether increasing rule-family diversity in trace-conditioned repair training improves transfer to unseen rule structures when the total number of training records is fixed. Three trace-conditioned LoRA adapters were trained on 240 examples each, using 3, 6, or 12 rule families. The evaluation separates base-IID repair, format holdout, and fully held-out rule-family transfer. Controls test frozen-model behavior, no-trace training, shuffled-trace training, and prompt-time trace ablations.
Design
- Base model:
Qwen/Qwen2.5-Coder-3B-Instruct. - Revision:
488639f1ff808d1d3d0ba301aef8c11461451ec5. - Training method: QLoRA, rank 32, alpha 64, dropout 0.05.
- Training budget: 240 records per diversity scale, 3 epochs.
- Evaluation metric:
repair@1, requiring both visible and hidden tests to pass. - Validation splits: 36 base-IID records, 36 format-holdout records, and 48 held-out rule-family records.
Training family counts:
- 3-family scale: 80 records per family.
- 6-family scale: 40 records per family.
- 12-family scale: 20 records per family.
Held-out rule families:
parity_offset_holdoutquadratic_shift_holdouttuple_max_holdoutsorted_join_holdout
Core Results
| Condition | Base IID | Format Holdout | Rule Holdout |
|---|---|---|---|
| Frozen trace | 0.0% (0/36) | 0.0% (0/36) | 0.0% (0/48) |
| 3 families, trace | 94.4% (34/36) | 50.0% (18/36) | 0.0% (0/48) |
| 6 families, trace | 77.8% (28/36) | 52.8% (19/36) | 4.2% (2/48) |
| 12 families, trace | 86.1% (31/36) | 44.4% (16/36) | 29.2% (14/48) |
| 12 families, no trace train/eval | 0.0% (0/36) | 0.0% (0/36) | 2.1% (1/48) |
| 12 families, shuffled trace train | 0.0% (0/36) | 0.0% (0/36) | 2.1% (1/48) |
Diversity Scale Results
| Condition | Base IID | Format Holdout | Rule Holdout |
|---|---|---|---|
| 3 families, trace | 94.4% (34/36) | 50.0% (18/36) | 0.0% (0/48) |
| 6 families, trace | 77.8% (28/36) | 52.8% (19/36) | 4.2% (2/48) |
| 12 families, trace | 86.1% (31/36) | 44.4% (16/36) | 29.2% (14/48) |
Trace Control And Ablation Results
| Condition | Base IID | Format Holdout | Rule Holdout |
|---|---|---|---|
| 12 families, trace | 86.1% (31/36) | 44.4% (16/36) | 29.2% (14/48) |
| 12-family trace adapter, no trace prompt | 0.0% (0/36) | 0.0% (0/36) | 0.0% (0/48) |
| 12-family trace adapter, shuffled trace prompt | 0.0% (0/36) | 0.0% (0/36) | 4.2% (2/48) |
Held-Out Rule Results By Family
| Condition | Family | repair@1 | Successes |
|---|---|---|---|
| 3 families, trace | parity_offset_holdout | 0.0% | 0/12 |
| 3 families, trace | quadratic_shift_holdout | 0.0% | 0/12 |
| 3 families, trace | sorted_join_holdout | 0.0% | 0/12 |
| 3 families, trace | tuple_max_holdout | 0.0% | 0/12 |
| 6 families, trace | parity_offset_holdout | 0.0% | 0/12 |
| 6 families, trace | quadratic_shift_holdout | 16.7% | 2/12 |
| 6 families, trace | sorted_join_holdout | 0.0% | 0/12 |
| 6 families, trace | tuple_max_holdout | 0.0% | 0/12 |
| 12 families, trace | parity_offset_holdout | 0.0% | 0/12 |
| 12 families, trace | quadratic_shift_holdout | 25.0% | 3/12 |
| 12 families, trace | sorted_join_holdout | 91.7% | 11/12 |
| 12 families, trace | tuple_max_holdout | 0.0% | 0/12 |
Interpretation
- Best held-out rule-family repair among the diversity-scale adapters was
scale12_traceat 29.2% (14/48). - Best format-holdout repair among the diversity-scale adapters was
scale6_traceat 52.8% (19/36). - Best base-IID repair among the diversity-scale adapters was
scale3_traceat 94.4% (34/36). - On held-out rule families, the 12-family trace adapter scored 29.2%, while the no-trace control scored 2.1% and the shuffled-trace-trained control scored 2.1%.
The central comparison is the diversity-scale curve on the rule-holdout split. A useful positive result is not simply high base-IID repair; it is improved rule-holdout repair under the same 240-record training budget. The control rows indicate whether any held-out repair depends on valid trace evidence or can be explained by patch priors learned from the training distribution.
Figures
experiments/rule_family_diversity_scaling/figures/final_repair_by_condition_split.pngexperiments/rule_family_diversity_scaling/figures/diversity_scale_curve.pngexperiments/rule_family_diversity_scaling/figures/scale12_trace_ablation.png
Artifacts
- Dataset manifest:
experiments/rule_family_diversity_scaling/data/dataset_manifest.json. - Final JSON results:
experiments/rule_family_diversity_scaling/reports/final/. - CSV summaries:
experiments/rule_family_diversity_scaling/reports/*.csv. - Figures:
experiments/rule_family_diversity_scaling/figures/. - Large adapter artifacts:
large_artifacts/rule_family_diversity_scaling/models/.
Experiment log 12
Show the running log (12 entries, 2026-06-21)
2026-06-21 Setup
Objective: run a standalone rule-family diversity scaling experiment that tests whether trace-conditioned repair learns a transferable counterexample-to-rule procedure as training family diversity increases.
Directory policy:
- Small, downloadable experiment package:
/workspace/experiments/rule_family_diversity_scaling/. - Large artifacts excluded from the small package:
/workspace/large_artifacts/rule_family_diversity_scaling/. - Model adapters and checkpoints go under
large_artifacts/rule_family_diversity_scaling/models/. - Reports, logs, configs, figures, and compact JSON/JSONL result summaries go under
experiments/rule_family_diversity_scaling/.
Initial hypothesis:
If trace-conditioned repair is learning a general counterexample-to-rule procedure, then adapters trained on more diverse rule families should improve on held-out rule families even when total training record count is held fixed.
Design constraints:
- The paper and artifacts must be standalone.
- Total training records are fixed at 240 for every diversity scale.
- The exact target rule is not stated in issue text.
- Visible failure traces contain concrete counterexamples.
- Hidden tests use inputs not present in the visible trace.
- Final evaluation must separately report trained-family IID, trained-family format holdout, and fully held-out rule-family transfer.
2026-06-21 Dataset Builder Smoke Test
Builder:
experiments/rule_family_diversity_scaling/scripts/build_diversity_dataset.py
Smoke command:
python experiments/rule_family_diversity_scaling/scripts/build_diversity_dataset.py --output-dir /tmp/rfds_smoke --total-train-records 12 --base-iid-per-family 1 --format-per-family 1 --holdout-per-family 1 --seed 20260621
Smoke result:
- Build completed successfully.
- Each diversity scale had 12 training records.
- Validation smoke splits had 3 base-IID records, 3 format-holdout records, and 4 held-out-family records.
- The builder initially exposed a useful edge case: numeric wrong rules can coincidentally pass one visible case, which would omit that expected output from the failure trace. The builder now rejects and resamples any record that does not satisfy the trace-evidence invariant.
2026-06-21 Full Dataset Build
Command:
python experiments/rule_family_diversity_scaling/scripts/build_diversity_dataset.py --output-dir experiments/rule_family_diversity_scaling/data --total-train-records 240 --base-iid-per-family 12 --format-per-family 12 --holdout-per-family 12 --seed 20260621
Result:
train_scale3: 240 records from 3 families, 80 records per family.train_scale6: 240 records from 6 families, 40 records per family.train_scale12: 240 records from 12 families, 20 records per family.val_base_iid: 36 records from the 3 base families.val_format_holdout: 36 records from the 3 base families with shifted numeric ranges and token formats.val_rule_holdout: 48 records from 4 families absent from every training scale.repair_all: 840 total records.
Training-family scales:
scale3:affine_int,threshold_label,slug_affix.scale6:affine_int,threshold_label,slug_affix,abs_shift,clamp_offset,tuple_linear.scale12:affine_int,threshold_label,slug_affix,abs_shift,clamp_offset,tuple_linear,length_label,contains_label,prefix_switch,modulo_label,sign_piece,replace_wrap.
Held-out rule families:
parity_offset_holdoutquadratic_shift_holdouttuple_max_holdoutsorted_join_holdout
All builder invariants passed.
2026-06-21 Frozen Evaluation Smoke
Command:
python experiments/rule_family_diversity_scaling/scripts/eval_diversity.py --data experiments/rule_family_diversity_scaling/data/repair_val_base_iid.jsonl --output experiments/rule_family_diversity_scaling/reports/frozen_trace_base_iid_pilot6.json --condition trace --max-records 6 --max-new-tokens 256
Result:
- Records: 6 base-IID examples.
- Repair@1: 0/6.
- Patch apply rate: 0/6.
- Visible pass rate: 0/6.
Decision:
- The evaluator works on the new record schema.
- Proceed with LoRA training for diversity scales and controls.
2026-06-21 Training: scale3_trace_lora
Command:
python scripts/train_repair_lora.py --train experiments/rule_family_diversity_scaling/data/repair_train_scale3.jsonl --eval experiments/rule_family_diversity_scaling/data/repair_val_base_iid.jsonl --mode trace --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/rule_family_diversity_scaling/models/scale3_trace_lora --max-length 3072 --epochs 3 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30
Result:
- Completed successfully.
- Output directory:
large_artifacts/rule_family_diversity_scaling/models/scale3_trace_lora. - Trainable parameters: 59,867,136 (1.9031%).
- Training steps: 90.
- Train runtime: 411 seconds.
- Final train loss: 0.01668.
- Evaluation loss by epoch on
val_base_iid: epoch 1 = 0.004631, epoch 2 = 0.003665, epoch 3 = 0.001845. - Checkpoints and adapter weights were written under
large_artifacts/, not under the downloadable experiment package.
2026-06-21 Training: scale6_trace_lora
Command:
python scripts/train_repair_lora.py --train experiments/rule_family_diversity_scaling/data/repair_train_scale6.jsonl --eval experiments/rule_family_diversity_scaling/data/repair_val_base_iid.jsonl --mode trace --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/rule_family_diversity_scaling/models/scale6_trace_lora --max-length 3072 --epochs 3 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30
Result:
- Completed successfully.
- Output directory:
large_artifacts/rule_family_diversity_scaling/models/scale6_trace_lora. - Trainable parameters: 59,867,136 (1.9031%).
- Training steps: 90.
- Train runtime: 415.4 seconds.
- Final train loss: 0.02493.
- Evaluation loss by epoch on
val_base_iid: epoch 1 = 0.01205, epoch 2 = 0.01092, epoch 3 = 0.0101. - Checkpoints and adapter weights were written under
large_artifacts/, not under the downloadable experiment package.
2026-06-21 Training: scale12_trace_lora
Command:
python scripts/train_repair_lora.py --train experiments/rule_family_diversity_scaling/data/repair_train_scale12.jsonl --eval experiments/rule_family_diversity_scaling/data/repair_val_base_iid.jsonl --mode trace --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/rule_family_diversity_scaling/models/scale12_trace_lora --max-length 3072 --epochs 3 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30
Result:
- Completed successfully.
- Output directory:
large_artifacts/rule_family_diversity_scaling/models/scale12_trace_lora. - Trainable parameters: 59,867,136 (1.9031%).
- Training steps: 90.
- Train runtime: 393.7 seconds.
- Final train loss: 0.02588.
- Evaluation loss by epoch on
val_base_iid: epoch 1 = 0.0199, epoch 2 = 0.01371, epoch 3 = 0.01347. - Checkpoints and adapter weights were written under
large_artifacts/, not under the downloadable experiment package.
2026-06-21 Training: scale12_no_trace_lora
Command:
python scripts/train_repair_lora.py --train experiments/rule_family_diversity_scaling/data/repair_train_scale12.jsonl --eval experiments/rule_family_diversity_scaling/data/repair_val_base_iid.jsonl --mode no_trace --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/rule_family_diversity_scaling/models/scale12_no_trace_lora --max-length 3072 --epochs 3 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30
Result:
- Completed successfully.
- Output directory:
large_artifacts/rule_family_diversity_scaling/models/scale12_no_trace_lora. - Trainable parameters: 59,867,136 (1.9031%).
- Training steps: 90.
- Train runtime: 391 seconds.
- Final train loss: 0.2446.
- Evaluation loss by epoch on
val_base_iid: epoch 1 = 0.2427, epoch 2 = 0.2351, epoch 3 = 0.2393. - Checkpoints and adapter weights were written under
large_artifacts/, not under the downloadable experiment package.
2026-06-21 Training: scale12_shuffled_trace_lora
Command:
python scripts/train_repair_lora.py --train experiments/rule_family_diversity_scaling/data/repair_train_scale12.jsonl --eval experiments/rule_family_diversity_scaling/data/repair_val_base_iid.jsonl --mode trace --shuffle-traces --model-id Qwen/Qwen2.5-Coder-3B-Instruct --revision 488639f1ff808d1d3d0ba301aef8c11461451ec5 --output-dir large_artifacts/rule_family_diversity_scaling/models/scale12_shuffled_trace_lora --max-length 3072 --epochs 3 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30
Result:
- Completed successfully.
- Output directory:
large_artifacts/rule_family_diversity_scaling/models/scale12_shuffled_trace_lora. - Trainable parameters: 59,867,136 (1.9031%).
- Training steps: 90.
- Train runtime: 394.4 seconds.
- Final train loss: 0.2452.
- Evaluation loss by epoch on
val_base_iid: epoch 1 = 0.2418, epoch 2 = 0.2307, epoch 3 = 0.2331. - Checkpoints and adapter weights were written under
large_artifacts/, not under the downloadable experiment package.
2026-06-21 Final Evaluation Suite
Command:
python experiments/rule_family_diversity_scaling/scripts/run_final_evaluations.py --force
Evaluation protocol:
- Deterministic generation with
max_new_tokens=256. repair@1requires both visible and hidden tests to pass.- Full splits were evaluated: 36 base-IID records, 36 format-holdout records, and 48 held-out rule-family records.
- Final outputs were written to
experiments/rule_family_diversity_scaling/reports/final/. - The final suite produced 18 core result JSONs, 6 ablation result JSONs, and
final_evaluation_jobs.json.
Core repair@1 results:
| Condition | Base IID | Format Holdout | Rule Holdout |
|---|---|---|---|
| Frozen trace | 0/36 | 0/36 | 0/48 |
| scale3 trace | 34/36 | 18/36 | 0/48 |
| scale6 trace | 28/36 | 19/36 | 2/48 |
| scale12 trace | 31/36 | 16/36 | 14/48 |
| scale12 no-trace train/eval | 0/36 | 0/36 | 1/48 |
| scale12 shuffled-trace train, real trace eval | 0/36 | 0/36 | 1/48 |
Scale12 trace adapter prompt ablations:
| Prompt condition | Base IID | Format Holdout | Rule Holdout |
|---|---|---|---|
| Real trace | 31/36 | 16/36 | 14/48 |
| No trace | 0/36 | 0/36 | 0/48 |
| Shuffled trace | 0/36 | 0/36 | 2/48 |
Held-out family detail for scale12_trace:
parity_offset_holdout: 0/12.quadratic_shift_holdout: 3/12.sorted_join_holdout: 11/12.tuple_max_holdout: 0/12.
Interpretation:
- Increasing trace-training diversity from 3 to 12 rule families improved held-out rule-family repair from 0/48 to 14/48 under the same 240-record training budget.
- The transfer gain was not uniform across held-out families; it was concentrated in
sorted_join_holdoutand, to a smaller extent,quadratic_shift_holdout. - Valid trace evidence was necessary for the observed transfer: no-trace training, shuffled-trace training, no-trace prompting, and shuffled-trace prompting all scored at or near zero on the same held-out split.
- The broadest diversity condition did not dominate every split:
scale3_tracewas best on base-IID andscale6_tracewas best on format holdout.
2026-06-21 Report Generation
Command:
python experiments/rule_family_diversity_scaling/scripts/make_report.py
Generated compact artifacts:
reports/rule_family_diversity_scaling_paper.mdreports/rule_family_diversity_scaling_summary.mdreports/final_core_results.csvreports/final_ablation_results.csvreports/final_scale_by_split.csvreports/final_trace_by_family.csvreports/pilot_results.csvfigures/final_repair_by_condition_split.pngfigures/diversity_scale_curve.pngfigures/scale12_trace_ablation.png
Packaging checks:
- Compact experiment directory size after report generation: 15 MB.
- Large artifact directory size after training: 6.4 GB.
- No
*.safetensors,*.bin,*.pt, or*.pthfiles were found underexperiments/rule_family_diversity_scaling/. - No files larger than 50 MB were found under
experiments/rule_family_diversity_scaling/. - Adapter weights and checkpoints are stored under
large_artifacts/rule_family_diversity_scaling/models/.
2026-06-21 Final Verification
Verification commands checked:
- Final evaluation manifest: 24 jobs, 24 completed.
- Final result JSONs: 18 core JSON files and 6 ablation JSON files.
- Generated reports, CSVs, and figures are present.
- No
__pycache__or.ipynb_checkpointsdirectories remain in the compact package. - No active
train_repair_lora.py,eval_diversity.py, orrun_final_evaluations.pyprocesses remain. - No model weight/checkpoint files or files larger than 50 MB were found under
experiments/rule_family_diversity_scaling/. - No references to earlier experiment directory names were found under
experiments/rule_family_diversity_scaling/.
Final package sizes:
- Compact package:
experiments/rule_family_diversity_scaling/= 15 MB. - Large artifacts:
large_artifacts/rule_family_diversity_scaling/= 6.4 GB.
Conclusion:
- The experiment is complete and packaged with small artifacts separated from adapters/checkpoints.
Figures 3
Data files 4
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/final_ablation_results.csv2.5 kBreports/final_core_results.csv6.2 kBreports/pilot_results.csv138 Breports/final/final_evaluation_jobs.json32 kB
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.