Qwen 3.5 4B Unsaturated Frontier Active Bridge
The one idea you need
Picture sixty tutoring sessions to split across ten tricky exam topics. Piling them onto the topic you flunk worst leaves shakier topics with too few sessions, and those quietly collapse on test day; splitting the sessions evenly covers every topic.
The question
You have a fixed budget of extra training examples to fix a model's weak spots. Should you spread them evenly, or pile them onto its worst failures?
What we found
Spread evenly. Giving each of ten problem types the same six extra correction examples let the model fully fix 98% of hard cases. Piling those same examples onto whichever types it failed most reached only 85%, and starved one type all the way to zero. Chasing the loudest failures robbed quieter ones that needed just a little help.
Why it matters
With a small budget of corrective training examples, spread it evenly across every skill you care about. Concentrating it on the worst-scoring failures can silently starve borderline skills until they collapse entirely.
Even spread of the extra examples98%hard cases fully fixed (118 of 120)
Piling examples onto worst failures85%best either targeting plan reached (102 of 120)
No extra practice on the hard cases52%starting point before any correction examples (62 of 120)
One neglected type under targeting100% → 0%fully solved under even spread, wiped out under targeting
On this page
Results at a glance 3
How to read
Four training recipes along the bottom: no extra practice, evenly-spread practice, and two ways of concentrating practice on the model's own mistakes. Bar height is the share of unseen tasks fully fixed; taller is better. Two bars per recipe: one try versus best of three tries.
Takeaway → The evenly-spread bars tower near the top while both concentrate-on-mistakes recipes sit well below and no-practice lowest, so spreading broadly beat targeting weak spots.
Data table
| Adapter (240 training records each) | greedy@1 | reranked (3 samples) |
|---|---|---|
| seed only | 46.7% | 51.7% |
| static bridge | 99.2% | 98.3% |
| seed-mined bridge | 76.7% | 84.2% |
| adaptive bridge | 85% | 85% |
Numbers from reports/qwen35_4b_unsaturated_frontier_active_bridge_report.md (Main Results table)
Technical framing
Frontier hidden all-pass: bridge data allocation strategies — Uniform static bridge coverage (98.3%) beats both active mining variants (~85%) on the broad frontier; seed-only reaches 51.7%.
How to read
Three bars for the same trained model, differing only in the worked-out solution steps shown alongside the broken program: correct steps, no steps, or deliberately jumbled steps. Height is share of tasks fully fixed; taller is better.
Takeaway → Accuracy drops sharply when the steps are removed and collapses when they are jumbled, so the model genuinely reads and relies on the steps rather than ignoring them.
Data table
| Prompt condition | adaptive bridge adapter |
|---|---|
| aligned trace | 85% |
| no trace | 58.3% |
| shuffled trace | 17.5% |
Numbers from reports/qwen35_4b_unsaturated_frontier_active_bridge_report.md (Main Results table)
Technical framing
Trace controls: adaptive bridge adapter on the frontier — The execution trace carries real signal: removing it costs ~27 points and a shuffled (misleading) trace collapses accuracy to 17.5%.
How to read
Five hardest task types along the bottom, four colored bars each for the recipes: no extra practice, evenly-spread, and two concentrate-on-mistakes versions. Height is share fully fixed on unseen inputs; taller is better.
Takeaway → Evenly-spread bars stay tall across every type, while a concentrate-on-mistakes bar drops to zero on at least one type, exposing the blind spots targeting creates.
Data table
| Frontier task family (12 items each) | seed only | static bridge | seed-mined bridge | adaptive bridge |
|---|---|---|---|---|
| length_contains_code | 8.3% | 100% | 100% | 83.3% |
| sum_offset_mod_label | 0% | 100% | 100% | 83.3% |
| sorted_index_offset_label | 0% | 91.7% | 100% | 100% |
| not_contains_length_code | 0% | 100% | 58.3% | 0% |
| tuple_branch_label | 100% | 100% | 41.7% | 100% |
Numbers from reports/qwen35_4b_unsaturated_frontier_active_bridge_report.md (Frontier By Family table)
Technical framing
Hardest frontier families: where each allocation strategy breaks — Active mining fixes the families it targets but starves others: the adaptive adapter drops to 0% on not_contains_length_code while static coverage holds 100%.
In the author’s words from the Overview
Seed adapter frontier reranked hidden all-pass: 62/120 = 51.7%. Static bridge adapter frontier reranked hidden all-pass: 118/120 = 98.3%. Seed-mined bridge adapter frontier reranked hidden all-pass: 101/120 = 84.2%. Adaptive bridge adapter frontier reranked hidden all-pass: 102/120 = 85.0%. All four adapters retained 60/60 hidden all-pass on the IID eval split. Adaptive trace controls: aligned trace 102/120, no trace 70/120, shuffled trace 21/120.
Overview
This standalone experiment tests active bridge allocation on a broad executable-DSL frontier suite.
The model receives:
- an input schema,
- a current wrong DSL program,
- visible execution cases,
- and must output one corrected DSL expression.
The experiment trains four fixed-budget adapters:
seed_lora: 240 base-family random-trace records.static_bridge_lora: 180 base-family records plus 60 uniformly allocated static frontier bridge records.seed_mined_bridge_lora: 180 base-family records plus 60 uniformly allocated bridge records selected against seed-adapter wrong programs.adaptive_bridge_lora: 180 base-family records plus 60 bridge records adaptively allocated from wrong programs generated after static bridge training.
Layout
configs/experiment.json: fixed design and hyperparameters.src/: standalone DSL, data, prompt, and model utilities.scripts/: dataset generation, mining, training, evaluation, and report entry points.data/: generated JSONL datasets and manifests.reports/: mining JSON, evaluation JSON files, and final report.logs/andrun_logs/: experiment notebook and command output.large_artifacts_manifest.md: pointers to adapter directories stored outside this compact directory.
Main Readout
- Final report:
reports/qwen35_4b_unsaturated_frontier_active_bridge_report.md. - Seed adapter frontier reranked hidden all-pass: 62/120 = 51.7%.
- Static bridge adapter frontier reranked hidden all-pass: 118/120 = 98.3%.
- Seed-mined bridge adapter frontier reranked hidden all-pass: 101/120 = 84.2%.
- Adaptive bridge adapter frontier reranked hidden all-pass: 102/120 = 85.0%.
- All four adapters retained 60/60 hidden all-pass on the IID eval split.
- Adaptive trace controls: aligned trace 102/120, no trace 70/120, shuffled trace 21/120.
Large adapters and checkpoints are intentionally outside this directory:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/
Report
Rendered from reports/qwen35_4b_unsaturated_frontier_active_bridge_report.md
Question
Can active bridge allocation outperform uniform static bridge coverage on a frontier suite broad enough that static bridge examples do not automatically saturate the target space?
Design
- Base model:
Qwen/Qwen3.5-4B. - Model output: one executable DSL expression.
- Training: 4-bit NF4 QLoRA adapters.
- Training budget: 240 records per trained adapter.
- Seed adapter: 240 base-family random-trace records.
- Static bridge adapter: 180 base-family records plus 60 uniformly allocated static frontier bridge records.
- Seed-mined bridge adapter: 180 base-family records plus 60 uniformly allocated bridge records selected against seed-adapter wrong programs.
- Adaptive bridge adapter: 180 base-family records plus 60 bridge records allocated toward wrong programs generated after static bridge training.
- Evaluation: parse and execute generated programs on visible and hidden cases.
- Candidate selection: choose the valid candidate with the most visible-case passes.
- Large adapter/checkpoint files are stored outside the compact experiment directory.
Dataset
- Seed train records: 240.
- Static bridge train records: 240.
- Bridge total: 60.
- Frontier families: 10.
- Frontier eval records: 120.
- IID eval records: 60.
- Mining pool records: 240.
- Visible cases per record: 6.
- Hidden cases per record: 18.
Seed-Adapter Mining Summary
- Allocation mode:
fixed. - Bridge allocation:
{'contains_count_length_code': 6, 'length_contains_code': 6, 'length_mod_contains_code': 6, 'modulo_sum_label': 6, 'not_contains_length_code': 6, 'sorted_index_offset_label': 6, 'sum_length_branch_label': 6, 'sum_offset_mod_label': 6, 'tuple_branch_label': 6, 'tuple_sum_gate_label': 6}. contains_count_length_code: 6/6 selected records had model-generated wrong programs; wrong-candidate score 20; 3 unique wrong programs. Top: 10x(if (and (contains tokens needle) (gt (count_eq tokens needle) threshold)) "MANY_LONG" "MISS"); 6x(if (and (contains tokens needle) (and (gt (count_eq tokens needle) threshold) (gt (len needle) min_len))) "MANY_LONG" "MISS"); 4x(if (and (contains tokens needle) (gt (count_eq tokens needle) threshold)) "MANY_LONG" (if (gt (len tokens) min_len) "MANY_LONG" "MISS")).length_contains_code: 6/6 selected records had model-generated wrong programs; wrong-candidate score 67; 1 unique wrong programs. Top: 67x(if (and (contains text needle) (gt (count_eq text needle) threshold)) "MATCH_LONG" "MISS").length_mod_contains_code: 6/6 selected records had model-generated wrong programs; wrong-candidate score 23; 4 unique wrong programs. Top: 19x(if (and (contains text needle) (eq (mod (count_eq text needle) modulus) target)) "HIT_MOD" "MISS"); 2x(if (and (contains text needle) (gt (count_eq text needle) target)) "HIT_MOD" "MISS"); 1x(if (and (contains text needle) (gt (count_eq text needle) target)) (format "HIT_MOD" (mod (count_eq text needle) modulus)) "MISS").modulo_sum_label: 0/6 selected records had model-generated wrong programs; wrong-candidate score 0; 0 unique wrong programs. Top: none.not_contains_length_code: 6/6 selected records had model-generated wrong programs; wrong-candidate score 72; 2 unique wrong programs. Top: 69x(if (and (contains text needle) (gt (count_eq text needle) threshold)) "ABSENT_LONG" "OTHER"); 3x(if (and (contains text needle) (gt (count_eq text needle) threshold)) "OTHER" "ABSENT_LONG").sorted_index_offset_label: 6/6 selected records had model-generated wrong programs; wrong-candidate score 55; 3 unique wrong programs. Top: 49x(format "SI{}" (add (tuple_get values index) offset)); 4x(format "SI{}" (add (tuple_get values (mod index (len values))) offset)); 2x(format "SI{}" (sub (add (tuple_get values index) offset) 0)).sum_length_branch_label: 0/6 selected records had model-generated wrong programs; wrong-candidate score 0; 0 unique wrong programs. Top: none.sum_offset_mod_label: 6/6 selected records had model-generated wrong programs; wrong-candidate score 36; 2 unique wrong programs. Top: 24x(format "OM{}" (add (mod (sum values) modulus) offset)); 12x(format "OM{}" (add (sum values) offset)).tuple_branch_label: 1/6 selected records had model-generated wrong programs; wrong-candidate score 1; 1 unique wrong programs. Top: 1x(if (if (gt (tuple_get item index) threshold) high_label low_label) high_label low_label).tuple_sum_gate_label: 1/6 selected records had model-generated wrong programs; wrong-candidate score 1; 1 unique wrong programs. Top: 1x(if (or (gt (tuple_get item index) threshold) (gt (sum item) sum_threshold)) high_label low_label).
Static-Adapter Adaptive Mining Summary
- Allocation mode:
adaptive. - Bridge allocation:
{'contains_count_length_code': 5, 'length_contains_code': 9, 'length_mod_contains_code': 10, 'modulo_sum_label': 2, 'not_contains_length_code': 2, 'sorted_index_offset_label': 24, 'sum_length_branch_label': 2, 'sum_offset_mod_label': 2, 'tuple_branch_label': 2, 'tuple_sum_gate_label': 2}. contains_count_length_code: 1/5 selected records had model-generated wrong programs; wrong-candidate score 1; 1 unique wrong programs. Top: 1x(if (and (contains tokens needle) (gt (count_eq tokens needle) threshold) (gt (len tokens) min_len)) "FOUND" "MISS").length_contains_code: 1/9 selected records had model-generated wrong programs; wrong-candidate score 2; 1 unique wrong programs. Top: 2x(if (and (contains text needle) (gt (count_eq text needle) threshold)) "MATCH_LONG" "MISS").length_mod_contains_code: 1/10 selected records had model-generated wrong programs; wrong-candidate score 2; 1 unique wrong programs. Top: 2x(if (and (contains text needle) (eq (mod (len text) modulus) target)) "HIT" "MISS").modulo_sum_label: 0/2 selected records had model-generated wrong programs; wrong-candidate score 0; 0 unique wrong programs. Top: none.not_contains_length_code: 0/2 selected records had model-generated wrong programs; wrong-candidate score 0; 0 unique wrong programs. Top: none.sorted_index_offset_label: 4/24 selected records had model-generated wrong programs; wrong-candidate score 8; 1 unique wrong programs. Top: 8x(format "SI{}" (add (sum values) offset)).sum_length_branch_label: 0/2 selected records had model-generated wrong programs; wrong-candidate score 0; 0 unique wrong programs. Top: none.sum_offset_mod_label: 0/2 selected records had model-generated wrong programs; wrong-candidate score 0; 0 unique wrong programs. Top: none.tuple_branch_label: 0/2 selected records had model-generated wrong programs; wrong-candidate score 0; 0 unique wrong programs. Top: none.tuple_sum_gate_label: 0/2 selected records had model-generated wrong programs; wrong-candidate score 0; 0 unique wrong programs. Top: none.
Main Results
| Condition | Data | Prompt | Samples | Greedy Hidden | Rerank Hidden |
|---|---|---|---|---|---|
| Seed adapter | dsl_eval_challenge.jsonl | trace | 3 | 46.7% (56/120) | 51.7% (62/120) |
| Static bridge adapter | dsl_eval_challenge.jsonl | trace | 3 | 99.2% (119/120) | 98.3% (118/120) |
| Seed-mined bridge adapter | dsl_eval_challenge.jsonl | trace | 3 | 76.7% (92/120) | 84.2% (101/120) |
| Adaptive bridge adapter | dsl_eval_challenge.jsonl | trace | 3 | 85.0% (102/120) | 85.0% (102/120) |
| Adaptive bridge adapter, no trace | dsl_eval_challenge.jsonl | no_trace | 0 | 58.3% (70/120) | 58.3% (70/120) |
| Adaptive bridge adapter, shuffled trace | dsl_eval_challenge.jsonl | shuffled_trace | 0 | 17.5% (21/120) | 17.5% (21/120) |
| Seed adapter, IID | dsl_eval_iid.jsonl | trace | 0 | 100.0% (60/60) | 100.0% (60/60) |
| Static bridge adapter, IID | dsl_eval_iid.jsonl | trace | 0 | 100.0% (60/60) | 100.0% (60/60) |
| Seed-mined bridge adapter, IID | dsl_eval_iid.jsonl | trace | 0 | 100.0% (60/60) | 100.0% (60/60) |
| Adaptive bridge adapter, IID | dsl_eval_iid.jsonl | trace | 0 | 100.0% (60/60) | 100.0% (60/60) |
Frontier By Family
| Family | Seed adapter | Static bridge adapter | Seed-mined bridge adapter | Adaptive bridge adapter |
|---|---|---|---|---|
modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
length_contains_code | 8.3% (1/12) | 100.0% (12/12) | 100.0% (12/12) | 83.3% (10/12) |
tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 41.7% (5/12) | 100.0% (12/12) |
sum_offset_mod_label | 0.0% (0/12) | 100.0% (12/12) | 100.0% (12/12) | 83.3% (10/12) |
length_mod_contains_code | 50.0% (6/12) | 91.7% (11/12) | 100.0% (12/12) | 100.0% (12/12) |
sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
sorted_index_offset_label | 0.0% (0/12) | 91.7% (11/12) | 100.0% (12/12) | 100.0% (12/12) |
contains_count_length_code | 83.3% (10/12) | 100.0% (12/12) | 83.3% (10/12) | 83.3% (10/12) |
tuple_sum_gate_label | 75.0% (9/12) | 100.0% (12/12) | 58.3% (7/12) | 100.0% (12/12) |
not_contains_length_code | 0.0% (0/12) | 100.0% (12/12) | 58.3% (7/12) | 0.0% (0/12) |
Readout
- Frontier reranked hidden all-pass: seed 51.7% (62/120), static bridge 98.3% (118/120), seed-mined bridge 84.2% (101/120), adaptive bridge 85.0% (102/120).
- Static bridge greedy hidden all-pass: 99.2% (119/120).
- Adaptive bridge greedy hidden all-pass: 85.0% (102/120).
- Adaptive bridge prompt controls: aligned trace 85.0% (102/120), no trace 58.3% (70/120), shuffled trace 17.5% (21/120).
Failure Signatures
modulo_sum_labelseed greedy top: 12x(format "M{}" (mod (sum values) modulus)); adaptive greedy top: 12x(format "M{}" (mod (sum values) modulus)).length_contains_codeseed greedy top: 12x(if (and (contains text needle) (gt (count_eq text needle) threshold)) "MATCH_LONG" "MISS"); adaptive greedy top: 9x(if (and (contains text needle) (gt (len text) threshold)) "MATCH_LONG" "MISS"); 2x(if (and (gt (len text) threshold) (gt (len needle) 0)) "MATCH_LONG" "MISS").tuple_branch_labelseed greedy top: 12x(if (gt (tuple_get item index) threshold) high_label low_label); adaptive greedy top: 12x(if (gt (tuple_get item index) threshold) high_label low_label).sum_offset_mod_labelseed greedy top: 10x(format "OM{}" (add (mod (sum values) modulus) offset)); 1x(format "OM{}" (add (sum values) offset) (mod . 0)); adaptive greedy top: 10x(format "OM{}" (mod (add (sum values) offset) modulus)); 2x(format "OM{}" (mod (sum values) offset)).length_mod_contains_codeseed greedy top: 6x(if (and (contains text needle) (eq (mod (len text) modulus) target)) "HIT_MOD" "MISS"); 4x(if (and (contains text needle) (eq (mod (count_eq text needle) modulus) target)) "HIT_MOD" "MISS"); adaptive greedy top: 11x(if (and (contains text needle) (eq (mod (len text) modulus) target)) "HIT_MOD" "MISS"); 1x(if (and (gt (len text) needle) (eq (mod (len text) modulus) target)) "HIT_MOD" "MISS").sum_length_branch_labelseed greedy top: 12x(if (and (gt (sum values) threshold) (gt (len text) min_len)) high_label low_label); adaptive greedy top: 12x(if (and (gt (sum values) threshold) (gt (len text) min_len)) high_label low_label).sorted_index_offset_labelseed greedy top: 4x(format "SI{}" (add (tuple_get values index) (add (sum values) offset)); 3x(format "SI{}" (add (tuple_get values index) offset)); adaptive greedy top: 12x(format "SI{}" (add (tuple_get (sort values) index) offset)).contains_count_length_codeseed greedy top: 4x(if (and (contains tokens needle) (and (gt (count_eq tokens needle) threshold) (gt (len tokens) min_len))) "MANY_LONG" "MISS"); 3x(if (and (contains tokens needle) (gt (count_eq tokens needle) threshold)) "MANY_LONG" "MISS"); adaptive greedy top: 6x(if (and (gt (len tokens) min_len) (gt (count_eq tokens needle) threshold)) "MANY_LONG" "MISS"); 5x(if (and (contains tokens needle) (gt (count_eq tokens needle) threshold) (gt (len tokens) min_len)) "MANY_LONG" "MISS").tuple_sum_gate_labelseed greedy top: 7x(if (and (gt (tuple_get item index) threshold) (gt (sum item) sum_threshold)) high_label low_label); 5x(if (and (gt (sum item) sum_threshold) (gt (get item index) threshold)) high_label low_label); adaptive greedy top: 12x(if (and (gt (tuple_get item index) threshold) (gt (sum item) sum_threshold)) high_label low_label).not_contains_length_codeseed greedy top: 12x(if (and (contains text needle) (gt (count_eq text needle) threshold)) "ABSENT_LONG" "OTHER"); adaptive greedy top: 10x(if (and (contains text needle) (gt (len text) threshold)) "ABSENT_LONG" "OTHER"); 2x(if (and (gt (len text) threshold) (contains text needle)) "ABSENT_LONG" "OTHER").
Per-Condition Details
seed_lora_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/seed_lora. - Data:
data/eval/dsl_eval_challenge.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 46.7% (56/120).
- Rerank hidden all-pass: 51.7% (62/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 58.3% (7/12) | 83.3% (10/12) | 58.3% (7/12) | 91.7% (11/12) |
| length_contains_code | 0.0% (0/12) | 8.3% (1/12) | 8.3% (1/12) | 16.7% (2/12) |
| length_mod_contains_code | 50.0% (6/12) | 50.0% (6/12) | 66.7% (8/12) | 66.7% (8/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 0.0% (0/12) | 0.0% (0/12) | 8.3% (1/12) | 8.3% (1/12) |
| sorted_index_offset_label | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 58.3% (7/12) | 75.0% (9/12) | 58.3% (7/12) | 75.0% (9/12) |
static_bridge_lora_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/static_bridge_lora. - Data:
data/eval/dsl_eval_challenge.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 99.2% (119/120).
- Rerank hidden all-pass: 98.3% (118/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_mod_contains_code | 100.0% (12/12) | 91.7% (11/12) | 100.0% (12/12) | 100.0% (12/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sorted_index_offset_label | 91.7% (11/12) | 91.7% (11/12) | 91.7% (11/12) | 91.7% (11/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
seed_mined_bridge_lora_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/seed_mined_bridge_lora. - Data:
data/eval/dsl_eval_challenge.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 76.7% (92/120).
- Rerank hidden all-pass: 84.2% (101/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 58.3% (7/12) | 83.3% (10/12) | 58.3% (7/12) | 83.3% (10/12) |
| length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_mod_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 8.3% (1/12) | 58.3% (7/12) | 8.3% (1/12) | 58.3% (7/12) |
| sorted_index_offset_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_branch_label | 41.7% (5/12) | 41.7% (5/12) | 41.7% (5/12) | 41.7% (5/12) |
| tuple_sum_gate_label | 58.3% (7/12) | 58.3% (7/12) | 75.0% (9/12) | 75.0% (9/12) |
adaptive_bridge_lora_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/adaptive_bridge_lora. - Data:
data/eval/dsl_eval_challenge.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 85.0% (102/120).
- Rerank hidden all-pass: 85.0% (102/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 91.7% (11/12) | 83.3% (10/12) | 91.7% (11/12) | 91.7% (11/12) |
| length_contains_code | 83.3% (10/12) | 83.3% (10/12) | 91.7% (11/12) | 91.7% (11/12) |
| length_mod_contains_code | 91.7% (11/12) | 100.0% (12/12) | 91.7% (11/12) | 100.0% (12/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| sorted_index_offset_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 83.3% (10/12) | 83.3% (10/12) | 83.3% (10/12) | 83.3% (10/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
adaptive_bridge_lora_no_trace_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/adaptive_bridge_lora. - Data:
data/eval/dsl_eval_challenge.jsonl. - Prompt mode:
no_trace. - Samples: 0.
- Greedy hidden all-pass: 58.3% (70/120).
- Rerank hidden all-pass: 58.3% (70/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| length_contains_code | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) |
| length_mod_contains_code | 50.0% (6/12) | 50.0% (6/12) | 50.0% (6/12) | 50.0% (6/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| sorted_index_offset_label | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 83.3% (10/12) | 83.3% (10/12) | 83.3% (10/12) | 83.3% (10/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
adaptive_bridge_lora_shuffled_trace_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/adaptive_bridge_lora. - Data:
data/eval/dsl_eval_challenge.jsonl. - Prompt mode:
shuffled_trace. - Samples: 0.
- Greedy hidden all-pass: 17.5% (21/120).
- Rerank hidden all-pass: 17.5% (21/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 16.7% (2/12) | 16.7% (2/12) | 16.7% (2/12) | 16.7% (2/12) |
| length_contains_code | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) |
| length_mod_contains_code | 41.7% (5/12) | 41.7% (5/12) | 41.7% (5/12) | 41.7% (5/12) |
| modulo_sum_label | 8.3% (1/12) | 8.3% (1/12) | 8.3% (1/12) | 8.3% (1/12) |
| not_contains_length_code | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| sorted_index_offset_label | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| sum_length_branch_label | 41.7% (5/12) | 41.7% (5/12) | 41.7% (5/12) | 41.7% (5/12) |
| sum_offset_mod_label | 8.3% (1/12) | 8.3% (1/12) | 8.3% (1/12) | 8.3% (1/12) |
| tuple_branch_label | 33.3% (4/12) | 33.3% (4/12) | 33.3% (4/12) | 33.3% (4/12) |
| tuple_sum_gate_label | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
seed_lora_iid
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/seed_lora. - Data:
data/eval/dsl_eval_iid.jsonl. - Prompt mode:
trace. - Samples: 0.
- Greedy hidden all-pass: 100.0% (60/60).
- Rerank hidden all-pass: 100.0% (60/60).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_and_count_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_count_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_and_mod_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_mod_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| mod_scalar_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| scalar_branch_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sorted_first_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_add_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_and_scalar_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_threshold_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_get_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
static_bridge_lora_iid
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/static_bridge_lora. - Data:
data/eval/dsl_eval_iid.jsonl. - Prompt mode:
trace. - Samples: 0.
- Greedy hidden all-pass: 100.0% (60/60).
- Rerank hidden all-pass: 100.0% (60/60).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_and_count_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_count_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_and_mod_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_mod_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| mod_scalar_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| scalar_branch_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sorted_first_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_add_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_and_scalar_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_threshold_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_get_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
seed_mined_bridge_lora_iid
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/seed_mined_bridge_lora. - Data:
data/eval/dsl_eval_iid.jsonl. - Prompt mode:
trace. - Samples: 0.
- Greedy hidden all-pass: 100.0% (60/60).
- Rerank hidden all-pass: 100.0% (60/60).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_and_count_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_count_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_and_mod_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_mod_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| mod_scalar_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| scalar_branch_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sorted_first_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_add_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_and_scalar_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_threshold_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_get_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
adaptive_bridge_lora_iid
- Adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/adaptive_bridge_lora. - Data:
data/eval/dsl_eval_iid.jsonl. - Prompt mode:
trace. - Samples: 0.
- Greedy hidden all-pass: 100.0% (60/60).
- Rerank hidden all-pass: 100.0% (60/60).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_and_count_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_count_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_and_mod_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_mod_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| mod_scalar_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| scalar_branch_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sorted_first_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_add_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_and_scalar_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_threshold_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_get_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
Artifact Layout
- Compact artifacts:
/workspace/experiments/qwen35_4b_unsaturated_frontier_active_bridge/. - Large artifacts:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/. - Dataset manifest:
data/dataset_manifest.json. - Mining reports:
reports/mining/. - Evaluation JSON files:
reports/eval/.
Experiment log 12
Show the running log (12 entries, 2026-06-22)
2026-06-22
- Created standalone experiment directory.
- Selected
Qwen/Qwen3.5-4Brevision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Defined the central question: whether active bridge allocation can outperform uniform static bridge coverage on a broad executable-DSL frontier suite.
- Fixed the adapter training budget at 240 records per condition.
Planned conditions:
- seed adapter: 240 base-family random-trace records,
- static bridge adapter: 180 base-family records plus 60 uniformly allocated static frontier bridge records,
- seed-mined bridge adapter: 180 base-family records plus 60 uniformly allocated bridge records selected against seed-adapter wrong programs,
- adaptive bridge adapter: 180 base-family records plus 60 bridge records allocated toward wrong programs still produced after static bridge training.
Frontier suite:
- 10 held-out compositional families,
- 12 eval records per family,
- 24 mining records per family,
- 6 visible cases and 18 hidden cases per ordinary record.
- Large model artifacts will be stored under
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/.
Next step: build and validate the datasets.
Dataset Build
- Ran
python scripts/build_dataset.py. - Wrote seed train split: 240 records.
- Wrote static bridge train split: 240 records.
- Wrote base anchor split for mined/adaptive bridge training: 180 records.
- Wrote static frontier bridge records: 60 records.
- Wrote IID eval split: 60 records.
- Wrote frontier eval split: 120 records, 12 per frontier family.
- Wrote mining pool: 240 records, 24 per frontier family.
- Each ordinary record has 6 visible cases and 18 hidden cases.
- Each mining record has a 96-case pool for identifying executable wrong programs.
- Static bridge allocation: 6 records for each of the 10 frontier families.
- Dataset manifest:
data/dataset_manifest.json.
Next step: train the seed and static bridge adapters, then use them for fixed and adaptive mining.
Seed Adapter Training
- Trained
seed_loraondata/seed/dsl_train.jsonl. - Training records: 240.
- Eval records used during training: first 24 IID records from
data/eval/dsl_eval_iid.jsonl. - Epochs: 2.
- LoRA rank/alpha/dropout: 32/64/0.05.
- Final IID eval loss: 0.0001375.
- Training runtime: about 855 seconds.
- Train loss: about 0.1087.
- Adapter output:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/seed_lora.
Next step: train the uniform static bridge adapter under the same record budget and hyperparameters.
Static Bridge Adapter Training
- Trained
static_bridge_loraondata/static_bridge/dsl_train.jsonl. - Training records: 240.
Composition:
- 180 base-family records,
- 60 frontier-family static bridge records,
- 6 bridge records for each frontier family.
- Epochs: 2.
- Final IID eval loss: 0.0005135.
- Training runtime: about 893 seconds.
- Train loss: about 0.1119.
- Adapter output:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/static_bridge_lora.
Next step: mine executable wrong programs from the seed adapter and build the seed-mined bridge training split.
Seed-Adapter Mining
- First mining pass exposed a data-quality issue: syntactically parseable but non-executable candidate programs could be counted as wrong programs.
- Patched
scripts/mine_model_counterexamples.pyso a candidate must parse and execute on every case-pool input before it can count as a wrong program for bridge selection. - Removed stale seed-mined outputs and reran mining with the strict executable filter.
Ran strict seed mining with:
- adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/seed_lora, - mining records: 240,
- candidates per record: 1 greedy + 2 sampled,
- max new tokens: 64,
- allocation mode: fixed,
- trace strategy:
seed_mined.
- adapter:
- Wrote mining report:
reports/mining/seed_mined_mining.json. - Wrote seed-mined bridge records:
data/seed_mined/bridge_records.jsonl. - Wrote seed-mined train split:
data/seed_mined/dsl_train.jsonl. Seed-mined train split:
- 240 records total,
- 180 base-family records,
- 60 bridge records.
- Fixed bridge allocation remained 6 records per frontier family.
Strict executable model-wrong record counts:
contains_count_length_code: 6/6 selected records had executable model wrong programs,length_contains_code: 6/6,length_mod_contains_code: 6/6,modulo_sum_label: 0/6, fallback static,not_contains_length_code: 6/6,sorted_index_offset_label: 6/6,sum_length_branch_label: 0/6, fallback static,sum_offset_mod_label: 6/6,tuple_branch_label: 1/6,tuple_sum_gate_label: 1/6.
Dominant strict executable wrong-program clusters:
- count-vs-length substitutions in string and token tasks,
- use of unsorted values in the sorted-index task,
- adding offset after modulo rather than applying modulo after offset,
orinstead ofandin tuple-sum gate.
Next step: train the seed-mined bridge adapter under the same 240-record budget.
Seed-Mined Bridge Adapter Training
- Trained
seed_mined_bridge_loraondata/seed_mined/dsl_train.jsonl. - Training records: 240.
Composition:
- 180 base-family records,
- 60 frontier-family bridge records selected from strict executable seed-adapter wrong programs or static fallback where no executable wrong program was available.
- Epochs: 2.
- Final IID eval loss: 0.00008227.
- Training runtime: about 853 seconds.
- Train loss: about 0.1110.
- Adapter output:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/seed_mined_bridge_lora.
Next step: mine residual executable wrong programs from the static bridge adapter with adaptive family allocation.
Static-Adapter Adaptive Mining
Ran adaptive mining with:
- adapter:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/static_bridge_lora, - mining records: 240,
- candidates per record: 1 greedy + 2 sampled,
- max new tokens: 64,
- allocation mode: adaptive,
- trace strategy:
adaptive_mined.
- adapter:
- Wrote mining report:
reports/mining/adaptive_mining.json. - Wrote adaptive bridge records:
data/adaptive/bridge_records.jsonl. - Wrote adaptive train split:
data/adaptive/dsl_train.jsonl. Adaptive train split:
- 240 records total,
- 180 base-family records,
- 60 bridge records.
Adaptive bridge allocation:
sorted_index_offset_label: 24 records,length_mod_contains_code: 10 records,length_contains_code: 9 records,contains_count_length_code: 5 records,modulo_sum_label: 2 records,not_contains_length_code: 2 records,sum_length_branch_label: 2 records,sum_offset_mod_label: 2 records,tuple_branch_label: 2 records,tuple_sum_gate_label: 2 records.
Trace strategy counts:
- 48 records used
adaptive_mined, - 12 records used
adaptive_mined_fallback_static.
- 48 records used
Strict executable model-wrong record counts:
sorted_index_offset_label: 4/24 selected records had executable static-adapter wrong programs; wrong score 8; unique wrong program count 1,length_mod_contains_code: 1/10; wrong score 2; unique wrong program count 1,length_contains_code: 1/9; wrong score 2; unique wrong program count 1,contains_count_length_code: 1/5; wrong score 1; unique wrong program count 1,- all other selected families: 0 executable static-adapter wrong programs, static fallback.
- Dominant residual executable wrong-program cluster:
sorted_index_offset_labelcollapsed to(format "SI{}" (add (sum values) offset)), replacing sorted indexed extraction with a sum-based shortcut.
Next step: train the adaptive bridge adapter under the same 240-record budget.
Adaptive Bridge Adapter Training
- Trained
adaptive_bridge_loraondata/adaptive/dsl_train.jsonl. - Training records: 240.
Composition:
- 180 base-family records,
- 60 frontier-family bridge records allocated by residual executable wrong-program mining from the static bridge adapter.
- Epochs: 2.
- Final IID eval loss: 0.0005400.
- Training runtime: about 845 seconds.
- Train loss: about 0.1192.
- Adapter output:
/workspace/large_artifacts/qwen35_4b_unsaturated_frontier_active_bridge/models/adaptive_bridge_lora.
Next step: run frontier executable evaluations for seed, static bridge, seed-mined bridge, and adaptive bridge adapters; then run trace controls and IID retention checks.
Frontier Evaluations
- Ran matched frontier evaluations on
data/eval/dsl_eval_challenge.jsonlwith trace prompts, 3 sampled candidates, max new tokens 64. Seed adapter:
- greedy hidden all-pass: 56/120 = 46.7%,
- reranked hidden all-pass: 62/120 = 51.7%.
Static bridge adapter:
- greedy hidden all-pass: 119/120 = 99.2%,
- reranked hidden all-pass: 118/120 = 98.3%.
Seed-mined bridge adapter:
- greedy hidden all-pass: 92/120 = 76.7%,
- reranked hidden all-pass: 101/120 = 84.2%.
Adaptive bridge adapter:
- greedy hidden all-pass: 102/120 = 85.0%,
- reranked hidden all-pass: 102/120 = 85.0%.
- Static bridge was the strongest condition by a wide margin. The adaptive bridge allocation repaired
sorted_index_offset_labelbut lost full coverage onnot_contains_length_code,length_contains_code,contains_count_length_code, andsum_offset_mod_label.
Adaptive Trace Controls
- Ran greedy-only adaptive controls on
data/eval/dsl_eval_challenge.jsonl. Aligned trace result from the main adaptive run:
- greedy hidden all-pass: 102/120 = 85.0%.
No-trace control:
- greedy hidden all-pass: 70/120 = 58.3%.
Shuffled-trace control:
- greedy hidden all-pass: 21/120 = 17.5%.
- Interpretation: aligned trace content carries substantial task information. Shuffled traces are actively harmful, so the effect is not just extra prompt length or generic in-context formatting.
IID Retention Evaluations
- Ran greedy-only IID evaluations on
data/eval/dsl_eval_iid.jsonl. - Seed adapter: 60/60 = 100.0% hidden all-pass.
- Static bridge adapter: 60/60 = 100.0% hidden all-pass.
- Seed-mined bridge adapter: 60/60 = 100.0% hidden all-pass.
- Adaptive bridge adapter: 60/60 = 100.0% hidden all-pass.
- No adapter showed measurable IID retention loss on this 60-record IID eval split.
Report
- Generated final standalone report:
reports/qwen35_4b_unsaturated_frontier_active_bridge_report.md. - Key conclusion: on this frontier suite, uniformly allocated static bridge records were more impactful than either seed-model mined bridges or adaptive residual mining. The strongest next direction is not more adaptive allocation on the current residual-mining rule; it is to preserve broad bridge coverage while improving bridge record construction and reranking robustness.
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.