Qwen 3.5 4B Balanced Discriminative Bridge
The one idea you need
A small model fixes buggy programs written in a tiny custom language, guided by worked examples showing each test's expected-versus-actual output. To teach it ten brand-new program types on a fixed lesson budget, an even mix of examples taught far better than a diet of only the trickiest look-alike cases.
The question
Given a fixed budget of practice examples to teach a small model new kinds of programs, should you feed it the hardest, most confusing cases or an even mix?
What we found
An even mix, clearly. Sixty evenly-spread ordinary examples across ten new program types lifted the model's success on unseen hard problems from 60% (with no examples at all) to 99% fully solved. Hand-picking only the trickiest look-alike cases dropped it to 90%, and targeting the model's own past mistakes dropped it further to 83%. Cleverer selection consistently hurt.
Why it matters
When fine-tuning a small model on a tight example budget, spend it on even, representative coverage of the target skills, not curated hard cases. Mining for difficulty concentrates odd, unrepresentative examples and can quietly erase the gains you paid for.
Even mix vs no examples at all60% → 99%unseen hard problems fully solved
Hand-picking only tricky look-alike cases90%worse than the even mix
Targeting the model's own past mistakes83%most elaborate selection, worst result
Feeding it scrambled worked examples99% → 16%proof the model actually reads the examples
Results at a glance 3
How to read
Four training setups along the bottom: a plain starter with no new examples, the even mix, and two hand-picked-hard sets. Paired bars show easier versus tougher new tasks; height is share of tasks fully solved, and taller is better.
Takeaway → The even-mix bars tower near 99% while both hand-picked-hard setups sit lower and the plain starter is lowest, so clever difficulty selection hurt rather than helped.
Data table
| training condition (240 records each) | normal frontier | hard frontier |
|---|---|---|
| seed only | 62.5% | 60% |
| static bridge | 98.3% | 99.2% |
| alias-discriminative bridge | 88.3% | 90% |
| model-discriminative bridge | 78.3% | 82.5% |
Numbers from report table (reports/qwen35_4b_balanced_discriminative_bridge_report.md, Normal/Hard Frontier Results)
Technical framing
Held-out frontier accuracy: plain bridge data beats discriminative selection — Equal-family bridge records lift hard-frontier accuracy from 60% to 99%; hard-case discriminative selection only hurts.
How to read
Three ways of showing the worked examples along the bottom: correct examples, no examples, and scrambled examples. Height is share of tough new tasks fully solved, and taller is better.
Takeaway → Removing the examples costs only about eight points, but scrambling them collapses success to 16%, showing the model depends on the examples' actual content.
Data table
| visible-trace condition | static bridge, hard frontier |
|---|---|
| correct trace | 99.2% |
| no trace | 90.8% |
| shuffled trace | 15.8% |
Numbers from report table (reports/qwen35_4b_balanced_discriminative_bridge_report.md, Trace Control Results)
Technical framing
Trace controls: the bridge adapter really reads the execution trace — Removing traces costs 8 points but shuffling them collapses accuracy to 16%, so the model is semantically dependent on trace content.
How to read
Five different program types along the bottom, each with three bars for the even mix and the two hand-picked-hard sets. Height is share fully solved on tough tasks, and taller is better.
Takeaway → The even mix stays near the top on every type, while the hand-picked sets crater on some, dropping as low as 17%.
Data table
| frontier family | static bridge | alias-discriminative | model-discriminative |
|---|---|---|---|
| length_contains_code | 100% | 100% | 66.7% |
| sum_offset_mod_label | 100% | 100% | 66.7% |
| sorted_index_offset_label | 100% | 50% | 83.3% |
| contains_count_length_code | 100% | 50% | 100% |
| not_contains_length_code | 91.7% | 100% | 16.7% |
Numbers from report table (reports/qwen35_4b_balanced_discriminative_bridge_report.md, Hard Frontier By Family)
Technical framing
Where discriminative selection lost ground (hard frontier, by family) — Hard-case selection trades gains on a few families for large regressions on others; the plain static bridge stays near-perfect everywhere.
In the author’s words from the Overview
This standalone experiment tests whether equal frontier-family bridge coverage improves when visible traces are selected to discriminate against hard alias programs and seed-adapter mistakes. The model receives: an input schema, a current wrong DSL program, visible execution cases with expected and got values, and must output one corrected executable DSL expression. The experiment trains four fixed-budget adapters: seed_lora: 240 base-family random-trace records. static_bridge_lora: 180 base-family records plus 60 equally allocated normal frontier bridge records. alias_discriminative_bridge_lora: 180 base-family records plus 60 equally allocated hard-case frontier records selected against an expanded alias bank. … Read the full result →
Overview
This standalone experiment tests whether equal frontier-family bridge coverage improves when visible traces are selected to discriminate against hard alias programs and seed-adapter mistakes.
The model receives:
- an input schema,
- a current wrong DSL program,
- visible execution cases with expected and got values,
- and must output one corrected executable DSL expression.
The experiment trains four fixed-budget adapters:
seed_lora: 240 base-family random-trace records.static_bridge_lora: 180 base-family records plus 60 equally allocated normal frontier bridge records.alias_discriminative_bridge_lora: 180 base-family records plus 60 equally allocated hard-case frontier records selected against an expanded alias bank.model_discriminative_bridge_lora: 180 base-family records plus 60 equally allocated hard-case frontier records selected against seed-adapter wrong programs plus the alias bank.
Layout
configs/experiment.json: fixed design and hyperparameters.src/: standalone DSL, data, prompt, and model utilities.scripts/: dataset generation, mining, training, evaluation, and report entry points.data/: generated JSONL datasets and manifests.reports/: mining JSON, evaluation JSON files, and final report.logs/andrun_logs/: experiment notebook and command output.large_artifacts_manifest.md: pointers to adapter directories stored outside this compact directory.
Main Readout
The strongest condition was static_bridge_lora: 118/120 normal frontier hidden all-pass and 119/120 hard frontier hidden all-pass.
Hard-case discriminative trace selection did not improve this setting. alias_discriminative_bridge_lora reached 108/120 hard frontier rerank hidden all-pass, and model_discriminative_bridge_lora reached 99/120.
All four adapters retained 60/60 IID hidden all-pass. Trace controls for static_bridge_lora on the hard frontier were: correct trace 119/120, no trace 109/120, shuffled trace 19/120.
Final report:
reports/qwen35_4b_balanced_discriminative_bridge_report.md
Large adapters and checkpoints are intentionally outside this directory:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/
Report
Rendered from reports/qwen35_4b_balanced_discriminative_bridge_report.md
Question
Can equal frontier-family coverage improve when visible traces are chosen to discriminate against harder aliases and seed-adapter mistakes, while keeping the same 240-record posttraining budget?
Design
- Base model:
Qwen/Qwen3.5-4B. - Model output: one executable DSL expression.
- Training: 4-bit NF4 QLoRA adapters.
- Training budget: 240 records per trained adapter.
- Seed adapter: 240 base-family random-trace records.
- Static bridge adapter: 180 base-family records plus 60 equally allocated normal frontier bridge records.
- Alias-discriminative bridge adapter: 180 base-family records plus 60 equally allocated hard-case frontier records selected against an expanded alias bank.
- Model-discriminative bridge adapter: 180 base-family records plus 60 equally allocated hard-case frontier records selected against seed-adapter wrong programs plus the alias bank.
- Evaluation: normal frontier, harder frontier, trace controls, and IID retention.
- Candidate selection: choose the valid candidate with the most visible-case passes.
- Large adapter/checkpoint files are stored outside the compact experiment directory.
Dataset
- Seed train records: 240.
- Bridge anchor records per bridge condition: 180.
- Bridge records per bridge condition: 60.
- Static bridge train records: 240.
- Alias-discriminative train records: 240.
- Frontier eval records: 120.
- Hard frontier eval records: 120.
- IID eval records: 60.
- Mining pool records: 240.
- Frontier families: 10.
- Visible cases per record: 6.
- Hidden cases per record: 18.
Model-Discriminative Mining
- Allocation mode:
fixed. - Selector case mode:
hard. - Bridge allocation:
{'contains_count_length_code': 6, 'length_contains_code': 6, 'length_mod_contains_code': 6, 'modulo_sum_label': 6, 'not_contains_length_code': 6, 'sorted_index_offset_label': 6, 'sum_length_branch_label': 6, 'sum_offset_mod_label': 6, 'tuple_branch_label': 6, 'tuple_sum_gate_label': 6}. contains_count_length_code: 6/6 selected records had seed-adapter wrong programs; wrong-candidate score 33; 1 unique model wrong programs. Top: 33x(if (and (gt (len tokens) min_len) (contains tokens needle)) "MANY_LONG" "MISS").length_contains_code: 4/6 selected records had seed-adapter wrong programs; wrong-candidate score 6; 1 unique model wrong programs. Top: 6x(if (and (contains text needle) (gt (len text) threshold)) "MATCH" "MISS").length_mod_contains_code: 6/6 selected records had seed-adapter wrong programs; wrong-candidate score 23; 1 unique model wrong programs. Top: 23x(if (and (contains text needle) (gt (mod (len text) modulus) target)) "HIT_MOD" "MISS").modulo_sum_label: 0/6 selected records had seed-adapter wrong programs; wrong-candidate score 0; 0 unique model wrong programs. Top: none.not_contains_length_code: 6/6 selected records had seed-adapter wrong programs; wrong-candidate score 72; 6 unique model wrong programs. Top: 60x(if (and (contains text needle) (gt (len text) threshold)) "ABSENT_LONG" "OTHER"); 5x(if (and (contains text needle) (gt (len text) threshold)) "OTHER" "ABSENT_LONG"); 3x(if (and (contains text needle) (gt (count_eq text needle) threshold)) "ABSENT_LONG" "OTHER").sorted_index_offset_label: 6/6 selected records had seed-adapter wrong programs; wrong-candidate score 35; 3 unique model wrong programs. Top: 17x(format "SI{}" (add (tuple_get values index) offset)); 15x(format "SI{}" (sub (tuple_get values index) offset)); 3x(format "SI{}" (add (sum values) offset)).sum_length_branch_label: 0/6 selected records had seed-adapter wrong programs; wrong-candidate score 0; 0 unique model wrong programs. Top: none.sum_offset_mod_label: 6/6 selected records had seed-adapter wrong programs; wrong-candidate score 66; 2 unique model wrong programs. Top: 60x(format "OM{}" (add (sum values) (mod offset modulus))); 6x(format "OM{}" (add (sum values) offset)).tuple_branch_label: 6/6 selected records had seed-adapter wrong programs; wrong-candidate score 8; 3 unique model wrong programs. Top: 3x(if (and (gt (tuple_get item index) threshold) (gt (sum item) threshold)) high_label low_label); 3x(if (and (gt (sum item) threshold) (gt (tuple_get item index) threshold)) high_label low_label); 2x(if (and (gt (sum item) threshold) (gt (tuple_get item index) 0)) high_label low_label).tuple_sum_gate_label: 6/6 selected records had seed-adapter wrong programs; wrong-candidate score 33; 4 unique model wrong programs. Top: 16x(if (and (gt (sum item) sum_threshold) (gt (len item) threshold)) high_label low_label); 7x(if (and (gt (sum item) sum_threshold) (gt (sum item) threshold)) high_label low_label); 7x(if (and (gt (sum item) sum_threshold) (gt threshold 0)) high_label low_label).
Normal Frontier Results
| Condition | Data | Prompt | Samples | Greedy Hidden | Rerank Hidden |
|---|---|---|---|---|---|
| Seed adapter, frontier | dsl_eval_frontier.jsonl | trace | 3 | 56.7% (68/120) | 62.5% (75/120) |
| Static bridge adapter, frontier | dsl_eval_frontier.jsonl | trace | 3 | 98.3% (118/120) | 98.3% (118/120) |
| Alias-discriminative bridge adapter, frontier | dsl_eval_frontier.jsonl | trace | 3 | 88.3% (106/120) | 88.3% (106/120) |
| Model-discriminative bridge adapter, frontier | dsl_eval_frontier.jsonl | trace | 3 | 72.5% (87/120) | 78.3% (94/120) |
Hard Frontier Results
| Condition | Data | Prompt | Samples | Greedy Hidden | Rerank Hidden |
|---|---|---|---|---|---|
| Seed adapter, hard frontier | dsl_eval_hard_frontier.jsonl | trace | 3 | 56.7% (68/120) | 60.0% (72/120) |
| Static bridge adapter, hard frontier | dsl_eval_hard_frontier.jsonl | trace | 3 | 99.2% (119/120) | 99.2% (119/120) |
| Alias-discriminative bridge adapter, hard frontier | dsl_eval_hard_frontier.jsonl | trace | 3 | 89.2% (107/120) | 90.0% (108/120) |
| Model-discriminative bridge adapter, hard frontier | dsl_eval_hard_frontier.jsonl | trace | 3 | 79.2% (95/120) | 82.5% (99/120) |
Trace Control Results
| Condition | Data | Prompt | Samples | Greedy Hidden | Rerank Hidden |
|---|---|---|---|---|---|
| Static bridge adapter, no trace hard frontier | dsl_eval_hard_frontier.jsonl | no_trace | 0 | 90.8% (109/120) | 90.8% (109/120) |
| Static bridge adapter, shuffled trace hard frontier | dsl_eval_hard_frontier.jsonl | shuffled_trace | 0 | 15.8% (19/120) | 15.8% (19/120) |
IID Retention Results
| Condition | Data | Prompt | Samples | Greedy Hidden | Rerank Hidden |
|---|---|---|---|---|---|
| Seed adapter, IID | dsl_eval_iid.jsonl | trace | 0 | 100.0% (60/60) | 100.0% (60/60) |
| Static bridge adapter, IID | dsl_eval_iid.jsonl | trace | 0 | 100.0% (60/60) | 100.0% (60/60) |
| Alias-discriminative bridge adapter, IID | dsl_eval_iid.jsonl | trace | 0 | 100.0% (60/60) | 100.0% (60/60) |
| Model-discriminative bridge adapter, IID | dsl_eval_iid.jsonl | trace | 0 | 100.0% (60/60) | 100.0% (60/60) |
Normal Frontier By Family
| Family | Seed adapter, frontier | Static bridge adapter, frontier | Alias-discriminative bridge adapter, frontier | Model-discriminative bridge adapter, frontier |
|---|---|---|---|---|
modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 41.7% (5/12) |
tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
sum_offset_mod_label | 16.7% (2/12) | 100.0% (12/12) | 100.0% (12/12) | 66.7% (8/12) |
length_mod_contains_code | 66.7% (8/12) | 100.0% (12/12) | 100.0% (12/12) | 75.0% (9/12) |
sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
sorted_index_offset_label | 50.0% (6/12) | 91.7% (11/12) | 41.7% (5/12) | 75.0% (9/12) |
contains_count_length_code | 33.3% (4/12) | 100.0% (12/12) | 41.7% (5/12) | 100.0% (12/12) |
tuple_sum_gate_label | 58.3% (7/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
not_contains_length_code | 0.0% (0/12) | 91.7% (11/12) | 100.0% (12/12) | 25.0% (3/12) |
Hard Frontier By Family
| Family | Seed adapter, hard frontier | Static bridge adapter, hard frontier | Alias-discriminative bridge adapter, hard frontier | Model-discriminative bridge adapter, hard frontier |
|---|---|---|---|---|
modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 66.7% (8/12) |
tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
sum_offset_mod_label | 16.7% (2/12) | 100.0% (12/12) | 100.0% (12/12) | 66.7% (8/12) |
length_mod_contains_code | 75.0% (9/12) | 100.0% (12/12) | 100.0% (12/12) | 91.7% (11/12) |
sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
sorted_index_offset_label | 33.3% (4/12) | 100.0% (12/12) | 50.0% (6/12) | 83.3% (10/12) |
contains_count_length_code | 8.3% (1/12) | 100.0% (12/12) | 50.0% (6/12) | 100.0% (12/12) |
tuple_sum_gate_label | 66.7% (8/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
not_contains_length_code | 0.0% (0/12) | 91.7% (11/12) | 100.0% (12/12) | 16.7% (2/12) |
Readout
- Hard frontier reranked hidden all-pass: seed 60.0% (72/120), static bridge 99.2% (119/120), alias-discriminative bridge 90.0% (108/120), model-discriminative bridge 82.5% (99/120).
- Hard frontier greedy hidden all-pass: seed 56.7% (68/120), static bridge 99.2% (119/120), alias-discriminative bridge 89.2% (107/120), model-discriminative bridge 79.2% (95/120).
- Static bridge trace controls on hard frontier: correct trace 99.2% (119/120), no trace 90.8% (109/120), shuffled trace 15.8% (19/120).
Next Experiment Options
- Recommended: run a static-normal bridge ceiling breaker. Keep
Qwen/Qwen3.5-4B, keep equal family allocation, and replace selector hardness with harder held-out family construction: more unseen compositions, longer inputs, adversarial edge cases, and trace controls. This directly tests whether the 119/120 result is a real bridge-interface gain or an evaluation ceiling. - Run a bridge-budget and case-count ablation around the static recipe: 20/40/60/80 bridge records and 2/4/6/8 visible cases per record. This identifies whether the gain is coming from family coverage, trace density, or sheer bridge-token exposure.
- Run a mild hard-case mixture instead of fully hard discriminative selection: 75% normal static records and 25% hard selector records within each family. This tests whether the regression came from hard-case distribution shift rather than discriminative selection itself.
- Run trace-semantic regularization only after the ceiling breaker: train with a small fraction of corrupted or missing traces labeled by the correct program. This is higher risk, but the shuffled-trace collapse shows the interface is semantically sensitive enough to justify a targeted robustness experiment.
Failure Signatures
modulo_sum_labelseed greedy top: 12x(format "M{}" (mod (sum values) modulus)); model-discriminative greedy top: 12x(format "M{}" (mod (sum values) modulus)).length_contains_codeseed greedy top: 12x(if (and (contains text needle) (gt (len text) threshold)) "MATCH_LONG" "MISS"); model-discriminative greedy top: 8x(if (and (contains text needle) (gt (len text) threshold)) "MATCH_LONG" "MISS"); 4x(if (and (contains text needle) (gt (count_eq text needle) threshold)) "MATCH_LONG" "MISS").tuple_branch_labelseed greedy top: 8x(if (gt (tuple_get item index) threshold) high_label low_label); 3x(if (and (gt (sum item) threshold) (gt (tuple_get item index) 0)) high_label low_label); model-discriminative greedy top: 12x(if (gt (tuple_get item index) threshold) high_label low_label).sum_offset_mod_labelseed greedy top: 10x(format "OM{}" (add (sum values) (mod offset modulus))); 1x(format "OM{}" (add (sum values) offset)); model-discriminative greedy top: 6x(format "OM{}" (mod (add (sum values) offset) modulus)); 3x(format "OM{}" (add (mod (sum values) offset) modulus)).length_mod_contains_codeseed greedy top: 9x(if (and (contains text needle) (eq (mod (len text) modulus) target)) "HIT_MOD" "MISS"); 3x(if (and (contains text needle) (gt (mod (len text) modulus) target)) "HIT_MOD" "MISS"); model-discriminative greedy top: 10x(if (and (contains text needle) (eq (mod (len text) modulus) target)) "HIT_MOD" "MISS"); 2x(if (and (contains text needle) (eq (mod (count_eq text needle) modulus) target)) "HIT_MOD" "MISS").sum_length_branch_labelseed greedy top: 7x(if (and (gt (sum values) threshold) (gt (len text) min_len)) high_label low_label); 5x(if (and (gt (len text) min_len) (gt (sum values) threshold)) high_label low_label); model-discriminative greedy top: 12x(if (and (gt (sum values) threshold) (gt (len text) min_len)) high_label low_label).sorted_index_offset_labelseed greedy top: 7x(format "SI{}" (add (tuple_get values index) offset)); 4x(format "SI{}" (add (tuple_get (sort values) index) offset)); model-discriminative greedy top: 10x(format "SI{}" (add (tuple_get (sort values) index) offset)); 1x(format "SI{}" (tuple_get (add (sort values) offset) index)).contains_count_length_codeseed greedy top: 7x(if (and (gt (len tokens) min_len) (contains tokens needle)) "MANY_LONG" "MISS"); 5x(if (and (contains tokens needle) (and (gt (count_eq tokens needle) threshold) (gt (len tokens) min_len)) "MANY_LONG" "MISS"); model-discriminative greedy top: 12x(if (and (contains tokens needle) (gt (count_eq tokens needle) threshold) (gt (len tokens) min_len)) "MANY_LONG" "MISS").tuple_sum_gate_labelseed greedy top: 7x(if (and (gt (tuple_get item index) threshold) (gt (sum item) sum_threshold)) high_label low_label); 2x(if (and (gt (sum item) sum_threshold) (gt (len item) threshold)) high_label low_label); model-discriminative greedy top: 12x(if (and (gt (tuple_get item index) threshold) (gt (sum item) sum_threshold)) high_label low_label).not_contains_length_codeseed greedy top: 11x(if (and (contains text needle) (gt (len text) threshold)) "ABSENT_LONG" "OTHER"); 1x(if (and (contains text needle) (gt (len text) threshold)) "FOUND" "ABSENT_LONG"); model-discriminative greedy top: 8x(if (and (contains text needle) (gt (len text) threshold)) "ABSENT_LONG" "OTHER"); 3x(if (and (contains text needle) (gt (count_eq text needle) threshold)) "ABSENT_LONG" "OTHER").
Per-Condition Details
seed_lora_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/seed_lora. - Data:
data/eval/dsl_eval_frontier.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 56.7% (68/120).
- Rerank hidden all-pass: 62.5% (75/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 16.7% (2/12) | 33.3% (4/12) | 41.7% (5/12) | 58.3% (7/12) |
| length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_mod_contains_code | 58.3% (7/12) | 66.7% (8/12) | 58.3% (7/12) | 66.7% (8/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| sorted_index_offset_label | 25.0% (3/12) | 50.0% (6/12) | 25.0% (3/12) | 50.0% (6/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 8.3% (1/12) | 16.7% (2/12) | 8.3% (1/12) | 16.7% (2/12) |
| tuple_branch_label | 91.7% (11/12) | 100.0% (12/12) | 91.7% (11/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 66.7% (8/12) | 58.3% (7/12) | 75.0% (9/12) | 75.0% (9/12) |
static_bridge_lora_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/static_bridge_lora. - Data:
data/eval/dsl_eval_frontier.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 98.3% (118/120).
- Rerank hidden all-pass: 98.3% (118/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_mod_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 91.7% (11/12) | 91.7% (11/12) | 91.7% (11/12) | 91.7% (11/12) |
| sorted_index_offset_label | 91.7% (11/12) | 91.7% (11/12) | 91.7% (11/12) | 91.7% (11/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
alias_discriminative_bridge_lora_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/alias_discriminative_bridge_lora. - Data:
data/eval/dsl_eval_frontier.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 88.3% (106/120).
- Rerank hidden all-pass: 88.3% (106/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 41.7% (5/12) | 41.7% (5/12) | 66.7% (8/12) | 66.7% (8/12) |
| length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_mod_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sorted_index_offset_label | 41.7% (5/12) | 41.7% (5/12) | 41.7% (5/12) | 41.7% (5/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
model_discriminative_bridge_lora_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/model_discriminative_bridge_lora. - Data:
data/eval/dsl_eval_frontier.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 72.5% (87/120).
- Rerank hidden all-pass: 78.3% (94/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_contains_code | 33.3% (4/12) | 41.7% (5/12) | 33.3% (4/12) | 41.7% (5/12) |
| length_mod_contains_code | 58.3% (7/12) | 75.0% (9/12) | 58.3% (7/12) | 75.0% (9/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 8.3% (1/12) | 25.0% (3/12) | 25.0% (3/12) | 41.7% (5/12) |
| sorted_index_offset_label | 66.7% (8/12) | 75.0% (9/12) | 66.7% (8/12) | 75.0% (9/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 58.3% (7/12) | 66.7% (8/12) | 58.3% (7/12) | 66.7% (8/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
seed_lora_hard_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/seed_lora. - Data:
data/eval/dsl_eval_hard_frontier.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 56.7% (68/120).
- Rerank hidden all-pass: 60.0% (72/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 0.0% (0/12) | 8.3% (1/12) | 0.0% (0/12) | 8.3% (1/12) |
| length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_mod_contains_code | 75.0% (9/12) | 75.0% (9/12) | 75.0% (9/12) | 75.0% (9/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| sorted_index_offset_label | 33.3% (4/12) | 33.3% (4/12) | 33.3% (4/12) | 33.3% (4/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 8.3% (1/12) | 16.7% (2/12) | 8.3% (1/12) | 16.7% (2/12) |
| tuple_branch_label | 75.0% (9/12) | 100.0% (12/12) | 75.0% (9/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 75.0% (9/12) | 66.7% (8/12) | 91.7% (11/12) | 91.7% (11/12) |
static_bridge_lora_hard_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/static_bridge_lora. - Data:
data/eval/dsl_eval_hard_frontier.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 99.2% (119/120).
- Rerank hidden all-pass: 99.2% (119/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_mod_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 91.7% (11/12) | 91.7% (11/12) | 91.7% (11/12) | 91.7% (11/12) |
| sorted_index_offset_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
alias_discriminative_bridge_lora_hard_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/alias_discriminative_bridge_lora. - Data:
data/eval/dsl_eval_hard_frontier.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 89.2% (107/120).
- Rerank hidden all-pass: 90.0% (108/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 50.0% (6/12) | 50.0% (6/12) | 58.3% (7/12) | 58.3% (7/12) |
| length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_mod_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 91.7% (11/12) | 100.0% (12/12) | 91.7% (11/12) | 100.0% (12/12) |
| sorted_index_offset_label | 50.0% (6/12) | 50.0% (6/12) | 50.0% (6/12) | 50.0% (6/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
model_discriminative_bridge_lora_hard_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/model_discriminative_bridge_lora. - Data:
data/eval/dsl_eval_hard_frontier.jsonl. - Prompt mode:
trace. - Samples: 3.
- Greedy hidden all-pass: 79.2% (95/120).
- Rerank hidden all-pass: 82.5% (99/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_contains_code | 66.7% (8/12) | 66.7% (8/12) | 66.7% (8/12) | 66.7% (8/12) |
| length_mod_contains_code | 83.3% (10/12) | 91.7% (11/12) | 83.3% (10/12) | 91.7% (11/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 8.3% (1/12) | 16.7% (2/12) | 25.0% (3/12) | 33.3% (4/12) |
| sorted_index_offset_label | 83.3% (10/12) | 83.3% (10/12) | 83.3% (10/12) | 83.3% (10/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 50.0% (6/12) | 66.7% (8/12) | 50.0% (6/12) | 66.7% (8/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
static_bridge_lora_no_trace_hard_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/static_bridge_lora. - Data:
data/eval/dsl_eval_hard_frontier.jsonl. - Prompt mode:
no_trace. - Samples: 0.
- Greedy hidden all-pass: 90.8% (109/120).
- Rerank hidden all-pass: 90.8% (109/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_contains_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| length_mod_contains_code | 58.3% (7/12) | 58.3% (7/12) | 58.3% (7/12) | 58.3% (7/12) |
| modulo_sum_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| not_contains_length_code | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sorted_index_offset_label | 50.0% (6/12) | 50.0% (6/12) | 50.0% (6/12) | 50.0% (6/12) |
| sum_length_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| sum_offset_mod_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_branch_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
| tuple_sum_gate_label | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) | 100.0% (12/12) |
static_bridge_lora_shuffled_trace_hard_frontier
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/static_bridge_lora. - Data:
data/eval/dsl_eval_hard_frontier.jsonl. - Prompt mode:
shuffled_trace. - Samples: 0.
- Greedy hidden all-pass: 15.8% (19/120).
- Rerank hidden all-pass: 15.8% (19/120).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_count_length_code | 16.7% (2/12) | 16.7% (2/12) | 16.7% (2/12) | 16.7% (2/12) |
| length_contains_code | 33.3% (4/12) | 33.3% (4/12) | 41.7% (5/12) | 41.7% (5/12) |
| length_mod_contains_code | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) |
| modulo_sum_label | 8.3% (1/12) | 8.3% (1/12) | 8.3% (1/12) | 8.3% (1/12) |
| not_contains_length_code | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| sorted_index_offset_label | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
| sum_length_branch_label | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) | 25.0% (3/12) |
| sum_offset_mod_label | 16.7% (2/12) | 16.7% (2/12) | 16.7% (2/12) | 16.7% (2/12) |
| tuple_branch_label | 33.3% (4/12) | 33.3% (4/12) | 33.3% (4/12) | 33.3% (4/12) |
| tuple_sum_gate_label | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) | 0.0% (0/12) |
seed_lora_iid
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/seed_lora. - Data:
data/eval/dsl_eval_iid.jsonl. - Prompt mode:
trace. - Samples: 0.
- Greedy hidden all-pass: 100.0% (60/60).
- Rerank hidden all-pass: 100.0% (60/60).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_and_count_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_count_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_and_mod_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_mod_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| mod_scalar_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| scalar_branch_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sorted_first_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_add_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_and_scalar_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_threshold_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_get_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
static_bridge_lora_iid
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/static_bridge_lora. - Data:
data/eval/dsl_eval_iid.jsonl. - Prompt mode:
trace. - Samples: 0.
- Greedy hidden all-pass: 100.0% (60/60).
- Rerank hidden all-pass: 100.0% (60/60).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_and_count_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_count_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_and_mod_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_mod_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| mod_scalar_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| scalar_branch_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sorted_first_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_add_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_and_scalar_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_threshold_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_get_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
alias_discriminative_bridge_lora_iid
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/alias_discriminative_bridge_lora. - Data:
data/eval/dsl_eval_iid.jsonl. - Prompt mode:
trace. - Samples: 0.
- Greedy hidden all-pass: 100.0% (60/60).
- Rerank hidden all-pass: 100.0% (60/60).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_and_count_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_count_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_and_mod_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_mod_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| mod_scalar_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| scalar_branch_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sorted_first_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_add_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_and_scalar_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_threshold_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_get_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
model_discriminative_bridge_lora_iid
- Adapter:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/model_discriminative_bridge_lora. - Data:
data/eval/dsl_eval_iid.jsonl. - Prompt mode:
trace. - Samples: 0.
- Greedy hidden all-pass: 100.0% (60/60).
- Rerank hidden all-pass: 100.0% (60/60).
| Family | Greedy Hidden | Rerank Hidden | Greedy Visible | Rerank Visible |
|---|---|---|---|---|
| contains_and_count_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| contains_count_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_and_mod_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| length_mod_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| mod_scalar_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| scalar_branch_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sorted_first_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_add_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_and_scalar_code | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| sum_threshold_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_get_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
| tuple_sum_label | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) | 100.0% (4/4) |
Artifact Layout
- Compact artifacts:
/workspace/experiments/qwen35_4b_balanced_discriminative_bridge/. - Large artifacts:
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/. - Dataset manifest:
data/dataset_manifest.json. - Mining reports:
reports/mining/. - Evaluation JSON files:
reports/eval/.
Experiment log 5
Show the running log (5 entries)
Objective
Test whether a balanced frontier bridge curriculum becomes more useful when the trace cases are deliberately discriminative against hard aliases and seed-adapter mistakes, while preserving the same 240-record training budget and using only Qwen/Qwen3.5-4B.
Design Commitments
- Keep all trained conditions at 240 records.
- Keep bridge-family allocation equal: 6 records for each of 10 frontier families.
- Compare random base training, normal static bridge training, alias-discriminative bridge training, and model-discriminative bridge training.
- Evaluate on both normal frontier records and hard frontier records.
- Store trained adapters and checkpoints under
/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models. - Keep the compact experiment directory downloadable without large model artifacts.
Starting Hypotheses
- Equal frontier coverage is likely more important than reallocating bridge records across families.
- The useful knob is within-family trace selection: hard visible cases should remove shortcut programs that normal random cases leave viable.
- Seed-adapter wrong programs provide useful selectors only if combined with a broader alias bank, because sampled model errors can be sparse and idiosyncratic.
- A harder frontier split should reveal whether a near-perfect normal frontier result is genuine compositional robustness or just eval saturation.
Planned Runs
- Build deterministic datasets from seed
20260627. - Train
seed_lora. - Train
static_bridge_lora. - Train
alias_discriminative_bridge_lora. - Mine seed-adapter wrong programs on the hard mining pool.
- Train
model_discriminative_bridge_lora. - Evaluate all adapters on normal frontier, hard frontier, and IID retention.
- Evaluate trace controls for bridge adapters on hard frontier.
- Generate a final report and large-artifact manifest.
Step Log
- Initialized standalone directory and large artifact directory.
- Added hard-case input generation for all frontier families.
- Added alias-selector banks for frontier families.
- Added alias-discriminative bridge dataset generation.
- Added normal and hard frontier eval splits.
- Updated model-discriminative mining to use fixed equal allocation and alias-bank fallback.
- Built datasets with seed
20260627. - Dataset counts: seed train 240, static bridge train 240, alias-discriminative bridge train 240, base anchor 180, static bridge records 60, alias-discriminative bridge records 60, IID eval 60, normal frontier eval 120, hard frontier eval 120, mining pool 240.
- Bridge allocation audit: every frontier family has 6 records in both static and alias-discriminative bridge sets.
- Selector audit: static bridge records used normal cases and eliminated 4.10 wrong programs on average; alias-discriminative bridge records used hard cases and eliminated 5.10 wrong programs on average.
- Trained
seed_lorafor 2 epochs / 60 optimizer steps on 240 seed records. seed_loratraining summary: runtime 847.3s, train loss 0.1115, eval loss 0.0001022 on the 24-record training-time eval subset.- Saved
seed_loraunder/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/seed_lora. - Mined seed-adapter generations on the 240-record hard mining pool with greedy plus 2 samples per record.
- Mining runtime: 49m46s.
- Mining found 276 executable wrong candidates across the pool.
- Model-error bridge coverage: 46/60 selected bridge rows had source-row seed-adapter wrong programs; 48/60 rows had model wrong programs after adding family-level wrong selectors; 12/60 rows used alias fallback.
- Model-discriminative bridge audit: 6 records per frontier family, hard selector cases, average 6.90 selector programs per record, average 6.80 eliminated selector programs per record.
- Trained
static_bridge_lorafor 2 epochs / 60 optimizer steps on 240 records. static_bridge_loratraining summary: runtime 905.3s, train loss 0.1112, eval loss 0.0001177 on the 24-record training-time eval subset.- Saved
static_bridge_loraunder/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/static_bridge_lora. - Trained
alias_discriminative_bridge_lorafor 2 epochs / 60 optimizer steps on 240 records. alias_discriminative_bridge_loratraining summary: runtime 930.5s, train loss 0.1040, eval loss 0.0005526 on the 24-record training-time eval subset.- Saved
alias_discriminative_bridge_loraunder/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/alias_discriminative_bridge_lora. - Trained
model_discriminative_bridge_lorafor 2 epochs / 60 optimizer steps on 240 records. model_discriminative_bridge_loratraining summary: runtime 888.3s, train loss 0.1062, eval loss 0.0001196 on the 24-record training-time eval subset.- Saved
model_discriminative_bridge_loraunder/workspace/large_artifacts/qwen35_4b_balanced_discriminative_bridge/models/model_discriminative_bridge_lora. - Updated evaluation generation to request sampled return sequences in one call per prompt while preserving the same greedy plus sampled candidate semantics.
Evaluated normal frontier split with greedy plus 3 sampled candidates per record:
seed_lora: greedy hidden 68/120, rerank hidden 75/120.static_bridge_lora: greedy hidden 118/120, rerank hidden 118/120.alias_discriminative_bridge_lora: greedy hidden 106/120, rerank hidden 106/120.model_discriminative_bridge_lora: greedy hidden 87/120, rerank hidden 94/120.
Evaluated hard frontier split with greedy plus 3 sampled candidates per record:
seed_lora: greedy hidden 68/120, rerank hidden 72/120.static_bridge_lora: greedy hidden 119/120, rerank hidden 119/120.alias_discriminative_bridge_lora: greedy hidden 107/120, rerank hidden 108/120.model_discriminative_bridge_lora: greedy hidden 95/120, rerank hidden 99/120.
Evaluated IID retention split greedily:
seed_lora: 60/60 hidden all-pass.static_bridge_lora: 60/60 hidden all-pass.alias_discriminative_bridge_lora: 60/60 hidden all-pass.model_discriminative_bridge_lora: 60/60 hidden all-pass.
Ran hard-frontier trace controls for
static_bridge_lora:- Correct trace prompt: 119/120 hidden all-pass.
- No trace prompt: 109/120 hidden all-pass.
- Shuffled trace prompt: 19/120 hidden all-pass and 20/120 visible all-pass.
- Main readout: normal static bridge records were the strongest condition. Hard-case alias-discriminative and model-discriminative trace selection reduced frontier performance despite preserving IID retention.
- Trace-control readout: the static adapter learned substantial parametric structure, but correct visible traces still carried test-time information; semantically wrong traces strongly misled the learned bridge interface.
- Generated final report with next-experiment options at
reports/qwen35_4b_balanced_discriminative_bridge_report.md. Final audit:
python -m compileall src scriptspassed.- Removed generated
__pycache__directories after compile audit. - Stale-name scan found no matches.
- Compact directory large-file scan found no files over 50M.
- No train/eval/mining processes were running.
- Compact directory size: 13M.
- Large adapter directory size: 1.8G.
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.