Research log Small Model Experimentation
GitHub

Qwen 3.5 4B GraphIR Self Repair

Finishedimported 2026-07-12imported · line ZBenchmark GeneralizationEvidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safety+1 moreGitHub ↗
A fix-it pass catches noise, not real mistakes

The one idea you need

The model writes a program two ways: a dense one-line formula, or an editable step-by-step diagram. A separate fix-it pass proofreads that diagram, but it practiced only on randomly scrambled versions, so it catches obvious damage while missing a clean-looking step that quietly computes the wrong answer.

The question

If a small model writes code as an editable step-by-step diagram, then runs a learned fix-it pass over its failures, does that beat a plain one-liner?

What we found

No. The plain one-line formula fully solved 29% of brand-new tasks; the step-by-step diagram managed only 22%, and the fix-it pass recovered part of that gap to 24% — still behind. That pass genuinely works: on randomly scrambled diagrams it lifted success from 3% to 27%, an eightfold jump. But the model's own errors are confident, consistent wrong logic, not obvious damage, so repair rarely helps.

Why it matters

Do not assume a fix-it loop that aces synthetically broken inputs fixes a model's real errors — it patches random damage, not systematic wrong logic. Test repair on the model's own failures, and check whether a fancier program format earns its complexity.

Plain formula vs diagram-plus-repair on new tasks29% vs 24%brand-new tasks fully solved (35 vs 29 of 120)
Fix-it pass on randomly scrambled diagrams3% → 27%fully solved before vs after repair, an eightfold jump (4 to 32 of 120)
Repair's lift on the real pipeline22% → 24%diagram alone to diagram-plus-repair, still under the 29% baseline
Familiar-style tasksabout 100%every approach solves seen-style tasks perfectly (60 of 60)
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Question
    2. Design
    3. Dataset
    4. Ceiling Results
    5. Support Results
    6. IID Results
    7. Repair Diagnostic
    8. Ceiling By Family
    9. Readout
    10. Figures
    11. Failure Signatures
    12. Per-Condition Details
  4. Experiment log
  5. Figures
  6. Reproduce
  7. Related

Results at a glance 3

On new tasks, the plain formula still beats the diagram pipeline

How to read

Three bars give the share of 120 brand-new tasks whose every hidden test passes: the plain one-line formula, the step-by-step diagram alone, and the diagram plus the fix-it pass. Taller is better.

0%10%20%30%DSL baselineDSL baseline29.2%GraphIR constructGraphIR construct21.7%GraphIR construct + repairGraphIR construct + repair24.2%

Takeaway → The plain formula is tallest at 29%; the fix-it pass lifts the diagram from 22% to 24% but never closes the gap — the extra machinery does not pay off.

Data table
conditionhidden all-pass
DSL baseline29.2%
GraphIR construct21.7%
GraphIR construct + repair24.2%

Numbers from report table (Ceiling Results), reports/qwen35_4b_graphir_self_repair_report.md

Technical framing

Held-out ceiling: GraphIR pipeline does not beat the DSL baseline — Repair lifts GraphIR from 21.7% to 24.2%, but the plain DSL baseline still wins at 29.2% (out of 120 tasks).

The fix-it pass really does repair randomly broken diagrams

How to read

Two bars show the share of 120 tasks whose every hidden test passes on diagrams damaged by random scrambling: before the fix-it pass runs, and after it. Taller is better; the leap measures raw repair skill.

0%10%20%30%corrupted input (before repair)corrupted input (before repair)3.3%after repairafter repair26.7%

Takeaway → Success jumps from 3% to 27%, an eightfold rise, proving the repair skill is real — yet it barely helps the full pipeline because real mistakes differ from random damage.

Data table
stagehidden all-pass
corrupted input (before repair)3.3%
after repair26.7%

Numbers from report table (Repair Diagnostic), reports/qwen35_4b_graphir_self_repair_report.md

Technical framing

Repair skill exists: synthetic corrupted graphs before vs after repair — On synthetic corrupted GraphIR, repair jumps 3.3% to 26.7% — the skill is real but does not transfer to the model's own construction errors.

Which task families each approach wins and loses

How to read

Six task families along the bottom; three bars each give the share solved on every hidden test — plain formula, diagram, and diagram-plus-repair. Taller is better; families are ordered by how much the approaches diverge.

0%25%50%75%100%token_count_mod_lengthtoken_count_mod_length0%83.3%83.3%text_absent_modtext_absent_mod100%0%0%text_value_gatetext_value_gate100%100%100%sum_len_modsum_len_mod41.7%33.3%33.3%tuple_value_modtuple_value_mod25%0%0%tuple_sum_mod_gatetuple_sum_mod_gate25%0%8.3%

Takeaway → Gains and losses cancel: the diagram unlocks one family the formula fails completely (83% vs 0%) but collapses to zero on families the formula solves perfectly.

Data table
task familyDSL baselineGraphIR constructGraphIR construct + repair
token_count_mod_length0%83.3%83.3%
text_absent_mod100%0%0%
text_value_gate100%100%100%
sum_len_mod41.7%33.3%33.3%
tuple_value_mod25%0%0%
tuple_sum_mod_gate25%0%8.3%

Numbers from report table (Ceiling By Family), reports/qwen35_4b_graphir_self_repair_report.md

Technical framing

Ceiling families where representations diverge most — Gains and losses are family-specific: GraphIR unlocks one family DSL fails (83.3% vs 0%) but collapses on others DSL solves.

In the author’s words from the Overview · “Result”

The main ceiling result was negative for the GraphIR pipeline: DSL baseline rerank: 35/120 hidden all-pass. GraphIR construction: 26/120 hidden all-pass. GraphIR construction plus repair: 29/120 hidden all-pass. Direct synthetic corrupted-GraphIR repair diagnostic: 4/120 input hidden all-pass to 32/120 repaired hidden all-pass. The repair adapter learned useful behavior on synthetic corrupted graphs and fixed the two GraphIR support failures, but actual construction errors on held-out ceiling families did not transfer well enough for GraphIR plus repair to beat the DSL baseline.

Overview

This standalone experiment tests whether Qwen 3.5 4B can improve held-out executable repair by configuring a typed register graph and then applying a verifier-guided graph repair step.

The model receives:

  • an input schema,
  • a current wrong program or candidate graph,
  • visible execution cases with expected and got values,
  • and must output either one DSL expression or a GraphIR register program.

The trained conditions are:

  • dsl_static60_lora: DSL baseline trained on 180 base records plus 60 support bridge records.
  • graphir_construct_lora: same source records, target is typed GraphIR.
  • graphir_repair_lora: corrupted candidate GraphIR plus visible mismatches, target is corrected GraphIR.

Large adapters and checkpoints are intentionally outside this compact directory:

/workspace/large_artifacts/qwen35_4b_graphir_self_repair/

Layout

  • configs/experiment.json: fixed design and hyperparameters.
  • src/: standalone DSL, GraphIR, data, prompt, and model utilities.
  • scripts/: dataset generation, training, evaluation, and report entry points.
  • data/: generated JSONL datasets and manifest.
  • reports/: evaluation JSON files and final report.
  • figures/: generated charts.
  • logs/ and run_logs/: experiment notebook and command output.
  • large_artifacts_manifest.md: pointers to adapter directories stored outside this compact directory.

Report

Final report:

reports/qwen35_4b_graphir_self_repair_report.md

Result

The main ceiling result was negative for the GraphIR pipeline:

  • DSL baseline rerank: 35/120 hidden all-pass.
  • GraphIR construction: 26/120 hidden all-pass.
  • GraphIR construction plus repair: 29/120 hidden all-pass.
  • Direct synthetic corrupted-GraphIR repair diagnostic: 4/120 input hidden all-pass to 32/120 repaired hidden all-pass.

The repair adapter learned useful behavior on synthetic corrupted graphs and fixed the two GraphIR support failures, but actual construction errors on held-out ceiling families did not transfer well enough for GraphIR plus repair to beat the DSL baseline.

Report

Rendered from reports/qwen35_4b_graphir_self_repair_report.md

Question

Can a Qwen 3.5 4B adapter improve held-out executable repair by configuring a typed register graph, then applying a verifier-guided repair step?

Design

  • Base model: Qwen/Qwen3.5-4B.
  • Training: 4-bit NF4 QLoRA adapters.
  • Fixed budget: 240 records per adapter.
  • DSL baseline: emits one prefix DSL expression.
  • GraphIR construct adapter: emits typed register assignments ending in out.
  • GraphIR repair adapter: receives a candidate graph plus visible execution mismatches and emits a corrected graph.
  • Inference policy: generate configured construction candidates, execute visible cases, keep the best graph, optionally repair it, and score hidden cases.
  • Large adapter/checkpoint files are stored outside the compact experiment directory.

Dataset

  • Base train records: 180.
  • Support bridge train records: 60.
  • Train records per adapter: 240.
  • IID eval records: 60.
  • Support eval records: 120.
  • Ceiling eval records: 120.
  • Visible cases per record: 6.
  • Hidden cases per record: 18.
  • Support bridge families: 10.
  • Ceiling families: 10.

Ceiling Results

ConditionDataPromptSamplesMain HiddenSecondary Hidden
DSL baseline, ceilingdsl_ceiling.jsonltrace129.2% (35/120)27.5% (33/120)
GraphIR construct, ceilinggraph_ceiling.jsonltrace121.7% (26/120)21.7% (26/120)
GraphIR construct+repair, ceilinggraph_ceiling.jsonltrace124.2% (29/120)21.7% (26/120)

Support Results

ConditionDataPromptSamplesMain HiddenSecondary Hidden
DSL baseline, supportdsl_support.jsonltrace3100.0% (120/120)100.0% (120/120)
GraphIR construct, supportgraph_support.jsonltrace098.3% (118/120)98.3% (118/120)
GraphIR construct+repair, supportgraph_support.jsonltrace1100.0% (120/120)98.3% (118/120)

IID Results

ConditionDataPromptSamplesMain HiddenSecondary Hidden
DSL baseline, IIDdsl_iid.jsonltrace0100.0% (60/60)100.0% (60/60)
GraphIR construct, IIDgraph_iid.jsonltrace0100.0% (60/60)100.0% (60/60)
GraphIR construct+repair, IIDgraph_iid.jsonltrace0100.0% (60/60)100.0% (60/60)

Repair Diagnostic

ConditionDataPromptSamplesMain HiddenSecondary Hidden
GraphIR repair, corrupted ceilinggraph_repair_ceiling_corrupt.jsonltrace026.7% (32/120)3.3% (4/120)

Ceiling By Family

FamilyDSL baseline, ceilingGraphIR construct, ceilingGraphIR construct+repair, ceiling
sum_length_mod_gate_label0.0% (0/12)0.0% (0/12)0.0% (0/12)
sorted_index_sum_branch_label0.0% (0/12)0.0% (0/12)8.3% (1/12)
token_absent_length_code0.0% (0/12)0.0% (0/12)8.3% (1/12)
token_count_mod_length_code0.0% (0/12)83.3% (10/12)83.3% (10/12)
text_value_gate_label100.0% (12/12)100.0% (12/12)100.0% (12/12)
tuple_value_mod_label25.0% (3/12)0.0% (0/12)0.0% (0/12)
sorted_join_contains_code0.0% (0/12)0.0% (0/12)0.0% (0/12)
text_absent_mod_code100.0% (12/12)0.0% (0/12)0.0% (0/12)
sum_len_mod_label41.7% (5/12)33.3% (4/12)33.3% (4/12)
tuple_sum_mod_gate_label25.0% (3/12)0.0% (0/12)8.3% (1/12)

Readout

  • Ceiling hidden all-pass: DSL baseline 29.2% (35/120), GraphIR construction 21.7% (26/120), GraphIR construction plus repair 24.2% (29/120).
  • GraphIR construction greedy ceiling hidden all-pass: 21.7% (26/120).
  • GraphIR pipeline construction-only selected ceiling hidden all-pass: 21.7% (26/120).
  • The GraphIR repair stage improved the actual ceiling pipeline from 21.7% (26/120) to 24.2% (29/120), but did not beat the DSL baseline.
  • On synthetic corrupted ceiling GraphIR candidates, repair improved hidden all-pass from 3.3% (4/120) to 26.7% (32/120), indicating repair skill exists but does not transfer enough to actual construction errors.

Figures

Failure Signatures

  • sum_length_mod_gate_label DSL top: 6x (if (and (gt (sum values) target) (eq (mod (len text) modulus) target)) high_label low_label); 3x (if (and (gt (sum values) target) (eq (mod (sum values) modulus) target)) high_label low_label); GraphIR construct top: 4x `r0 = SUM values

r1 = LEN text r2 = MOD r1 modulus r3 = EQ r2 target r4 = AND r3 GT r0 0 r5 = IF r4 high_label low_label out = r5; GraphIR pipeline top: 4xr0 = SUM values r1 = LEN text r2 = MOD r1 modulus r3 = EQ r2 target r4 = AND r3 GT r0 0 r5 = IF r4 high_label low_label out = r5`.

  • sorted_index_sum_branch_label DSL top: 6x (if (gt (tuple_get (sort values) index) threshold) high_label low_label); 6x (if (gt (tuple_get values index) threshold) high_label low_label); GraphIR construct top: 12x `r0 = GET values index

r1 = GT r0 threshold r2 = IF r1 high_label low_label out = r2; GraphIR pipeline top: 11xr0 = GET values index r1 = GT r0 threshold r2 = IF r1 high_label low_label out = r2`.

  • token_absent_length_code DSL top: 9x (if (and (not (contains tokens needle)) (gt (count_eq tokens needle) min_len)) "ABSENT_LONG" "OTHER"); 2x (if (and (contains tokens needle) (gt (count_eq tokens needle) min_len)) "ABSENT_LONG" "OTHER"); GraphIR construct top: 11x `r0 = CONTAINS tokens needle

r1 = LEN tokens r2 = GT r1 min_len r3 = AND r0 r2 r4 = IF r3 "ABSENT_LONG" "OTHER" out = r4; GraphIR pipeline top: 11xr0 = CONTAINS tokens needle r1 = LEN tokens r2 = GT r1 min_len r3 = AND r0 r2 r4 = IF r3 "ABSENT_LONG" "OTHER" out = r4`.

  • token_count_mod_length_code DSL top: 7x (if (and (contains tokens needle) (gt (count_eq tokens needle) min_len) (eq (mod (count_eq tokens needle) modulus) target)) "COUNT_MOD_LONG" "MISS"); 3x (if (and (contains tokens needle) (gt (count_eq tokens needle) target) (gt (mod (count_eq tokens needle) modulus) 0) (gt (len tokens) min_len)) "COUNT_MOD_LONG" "MISS"); GraphIR construct top: 9x `r0 = CONTAINS tokens needle

r1 = COUNT_EQ tokens needle r2 = MOD r1 modulus r3 = EQ r2 target r4 = LEN tokens r5 = GT r4 min_len r6 = AND r0 r3 r5 r7 = IF r6 "COUNT_MOD_LONG" "MISS" out = r7; GraphIR pipeline top: 9xr0 = CONTAINS tokens needle r1 = COUNT_EQ tokens needle r2 = MOD r1 modulus r3 = EQ r2 target r4 = LEN tokens r5 = GT r4 min_len r6 = AND r0 r3 r5 r7 = IF r6 "COUNT_MOD_LONG" "MISS" out = r7`.

  • text_value_gate_label DSL top: 12x (if (and (contains text needle) (gt (sum values) threshold) (gt (len text) min_len)) high_label low_label); GraphIR construct top: 12x `r0 = CONTAINS text needle

r1 = LEN text r2 = GT r1 min_len r3 = SUM values r4 = GT r3 threshold r5 = AND r0 r2 r4 r6 = IF r5 high_label low_label out = r6; GraphIR pipeline top: 12xr0 = CONTAINS text needle r1 = LEN text r2 = GT r1 min_len r3 = SUM values r4 = GT r3 threshold r5 = AND r0 r2 r4 r6 = IF r5 high_label low_label out = r6`.

  • tuple_value_mod_label DSL top: 8x (format "TV{}" (mod (add (tuple_get item index) (sum values)) modulus)); 3x (format "TV{}" (add (tuple_get item index) (mod (sum values) modulus))); GraphIR construct top: 6x `r0 = GET item index

r1 = MOD r0 modulus r2 = FORMAT "TV{}" r1 out = r2; GraphIR pipeline top: 5xr0 = GET item index r1 = MOD r0 modulus r2 = FORMAT "TV{}" r1 out = r2`.

  • sorted_join_contains_code DSL top: 12x (if (contains tokens needle) "JOIN_HAS" "JOIN_MISS"); GraphIR construct top: 8x `r0 = CONTAINS tokens needle

r1 = IF r0 "JOIN_HAS" "JOIN_MISS" out = r1; GraphIR pipeline top: 8xr0 = CONTAINS tokens needle r1 = IF r0 "JOIN_HAS" "JOIN_MISS" out = r1`.

  • text_absent_mod_code DSL top: 12x (if (and (not (contains text needle)) (eq (mod (len text) modulus) target)) "ABSENT_MOD" "OTHER"); GraphIR construct top: 12x `r0 = CONTAINS text needle

r1 = LEN text r2 = MOD r1 modulus r3 = EQ r2 target r4 = AND r0 r3 r5 = IF r4 "ABSENT_MOD" "OTHER" out = r5; GraphIR pipeline top: 12xr0 = CONTAINS text needle r1 = LEN text r2 = MOD r1 modulus r3 = EQ r2 target r4 = AND r0 r3 r5 = IF r4 "ABSENT_MOD" "OTHER" out = r5`.

  • sum_len_mod_label DSL top: 7x (format "SL{}" (mod (sum values) modulus)); 5x (format "SL{}" (mod (add (sum values) (len text)) modulus)); GraphIR construct top: 6x `r0 = SUM values

r1 = LEN text r2 = MOD r1 modulus r3 = FORMAT "SL{}" r2 out = r3; GraphIR pipeline top: 5xr0 = SUM values r1 = LEN text r2 = MOD r1 modulus r3 = FORMAT "SL{}" r2 out = r3`.

  • tuple_sum_mod_gate_label DSL top: 7x (if (and (gt (tuple_get item index) threshold) (eq (mod (tuple_get item index) modulus) target)) high_label low_label); 3x (if (and (gt (tuple_get item index) threshold) (eq (mod (sum item) modulus) target)) high_label low_label); GraphIR construct top: 9x `r0 = GET item index

r1 = GT r0 threshold r2 = MOD r0 modulus r3 = EQ r2 target r4 = AND r1 r3 r5 = IF r4 high_label low_label out = r5; GraphIR pipeline top: 8xr0 = GET item index r1 = GT r0 threshold r2 = MOD r0 modulus r3 = EQ r2 target r4 = AND r1 r3 r5 = IF r4 high_label low_label out = r5`.

Per-Condition Details

dsl_lora_ceiling

  • Data: data/eval/dsl_ceiling.jsonl.
  • Prompt mode: trace.
  • Samples: 1.
FamilyMain HiddenMain Visible
sorted_index_sum_branch_label0.0% (0/12)0.0% (0/12)
sorted_join_contains_code0.0% (0/12)0.0% (0/12)
sum_len_mod_label41.7% (5/12)41.7% (5/12)
sum_length_mod_gate_label0.0% (0/12)8.3% (1/12)
text_absent_mod_code100.0% (12/12)100.0% (12/12)
text_value_gate_label100.0% (12/12)100.0% (12/12)
token_absent_length_code0.0% (0/12)8.3% (1/12)
token_count_mod_length_code0.0% (0/12)16.7% (2/12)
tuple_sum_mod_gate_label25.0% (3/12)50.0% (6/12)
tuple_value_mod_label25.0% (3/12)25.0% (3/12)

dsl_lora_iid

  • Data: data/eval/dsl_iid.jsonl.
  • Prompt mode: trace.
  • Samples: 0.
FamilyMain HiddenMain Visible
contains_and_count_code100.0% (4/4)100.0% (4/4)
contains_code100.0% (4/4)100.0% (4/4)
contains_count_label100.0% (4/4)100.0% (4/4)
length_and_mod_code100.0% (4/4)100.0% (4/4)
length_label100.0% (4/4)100.0% (4/4)
length_mod_label100.0% (4/4)100.0% (4/4)
mod_scalar_label100.0% (4/4)100.0% (4/4)
scalar_branch_label100.0% (4/4)100.0% (4/4)
sorted_first_label100.0% (4/4)100.0% (4/4)
sum_add_label100.0% (4/4)100.0% (4/4)
sum_and_scalar_code100.0% (4/4)100.0% (4/4)
sum_label100.0% (4/4)100.0% (4/4)
sum_threshold_label100.0% (4/4)100.0% (4/4)
tuple_get_label100.0% (4/4)100.0% (4/4)
tuple_sum_label100.0% (4/4)100.0% (4/4)

dsl_lora_support

  • Data: data/eval/dsl_support.jsonl.
  • Prompt mode: trace.
  • Samples: 3.
FamilyMain HiddenMain Visible
contains_count_length_code100.0% (12/12)100.0% (12/12)
length_contains_code100.0% (12/12)100.0% (12/12)
length_mod_contains_code100.0% (12/12)100.0% (12/12)
modulo_sum_label100.0% (12/12)100.0% (12/12)
not_contains_length_code100.0% (12/12)100.0% (12/12)
sorted_index_offset_label100.0% (12/12)100.0% (12/12)
sum_length_branch_label100.0% (12/12)100.0% (12/12)
sum_offset_mod_label100.0% (12/12)100.0% (12/12)
tuple_branch_label100.0% (12/12)100.0% (12/12)
tuple_sum_gate_label100.0% (12/12)100.0% (12/12)

graphir_construct_ceiling

  • Data: data/eval/graph_ceiling.jsonl.
  • Prompt mode: trace.
  • Samples: 1.
FamilyMain HiddenMain Visible
sorted_index_sum_branch_label0.0% (0/12)0.0% (0/12)
sorted_join_contains_code0.0% (0/12)0.0% (0/12)
sum_len_mod_label33.3% (4/12)33.3% (4/12)
sum_length_mod_gate_label0.0% (0/12)25.0% (3/12)
text_absent_mod_code0.0% (0/12)0.0% (0/12)
text_value_gate_label100.0% (12/12)100.0% (12/12)
token_absent_length_code0.0% (0/12)8.3% (1/12)
token_count_mod_length_code83.3% (10/12)100.0% (12/12)
tuple_sum_mod_gate_label0.0% (0/12)8.3% (1/12)
tuple_value_mod_label0.0% (0/12)0.0% (0/12)

graphir_construct_iid

  • Data: data/eval/graph_iid.jsonl.
  • Prompt mode: trace.
  • Samples: 0.
FamilyMain HiddenMain Visible
contains_and_count_code100.0% (4/4)100.0% (4/4)
contains_code100.0% (4/4)100.0% (4/4)
contains_count_label100.0% (4/4)100.0% (4/4)
length_and_mod_code100.0% (4/4)100.0% (4/4)
length_label100.0% (4/4)100.0% (4/4)
length_mod_label100.0% (4/4)100.0% (4/4)
mod_scalar_label100.0% (4/4)100.0% (4/4)
scalar_branch_label100.0% (4/4)100.0% (4/4)
sorted_first_label100.0% (4/4)100.0% (4/4)
sum_add_label100.0% (4/4)100.0% (4/4)
sum_and_scalar_code100.0% (4/4)100.0% (4/4)
sum_label100.0% (4/4)100.0% (4/4)
sum_threshold_label100.0% (4/4)100.0% (4/4)
tuple_get_label100.0% (4/4)100.0% (4/4)
tuple_sum_label100.0% (4/4)100.0% (4/4)

graphir_construct_support

  • Data: data/eval/graph_support.jsonl.
  • Prompt mode: trace.
  • Samples: 0.
FamilyMain HiddenMain Visible
contains_count_length_code100.0% (12/12)100.0% (12/12)
length_contains_code100.0% (12/12)100.0% (12/12)
length_mod_contains_code83.3% (10/12)83.3% (10/12)
modulo_sum_label100.0% (12/12)100.0% (12/12)
not_contains_length_code100.0% (12/12)100.0% (12/12)
sorted_index_offset_label100.0% (12/12)100.0% (12/12)
sum_length_branch_label100.0% (12/12)100.0% (12/12)
sum_offset_mod_label100.0% (12/12)100.0% (12/12)
tuple_branch_label100.0% (12/12)100.0% (12/12)
tuple_sum_gate_label100.0% (12/12)100.0% (12/12)

graphir_pipeline_ceiling

  • Data: data/eval/graph_ceiling.jsonl.
  • Prompt mode: trace.
  • Samples: 1.
FamilyMain HiddenMain Visible
sorted_index_sum_branch_label8.3% (1/12)8.3% (1/12)
sorted_join_contains_code0.0% (0/12)0.0% (0/12)
sum_len_mod_label33.3% (4/12)33.3% (4/12)
sum_length_mod_gate_label0.0% (0/12)25.0% (3/12)
text_absent_mod_code0.0% (0/12)0.0% (0/12)
text_value_gate_label100.0% (12/12)100.0% (12/12)
token_absent_length_code8.3% (1/12)16.7% (2/12)
token_count_mod_length_code83.3% (10/12)100.0% (12/12)
tuple_sum_mod_gate_label8.3% (1/12)33.3% (4/12)
tuple_value_mod_label0.0% (0/12)0.0% (0/12)

graphir_pipeline_iid

  • Data: data/eval/graph_iid.jsonl.
  • Prompt mode: trace.
  • Samples: 0.
FamilyMain HiddenMain Visible
contains_and_count_code100.0% (4/4)100.0% (4/4)
contains_code100.0% (4/4)100.0% (4/4)
contains_count_label100.0% (4/4)100.0% (4/4)
length_and_mod_code100.0% (4/4)100.0% (4/4)
length_label100.0% (4/4)100.0% (4/4)
length_mod_label100.0% (4/4)100.0% (4/4)
mod_scalar_label100.0% (4/4)100.0% (4/4)
scalar_branch_label100.0% (4/4)100.0% (4/4)
sorted_first_label100.0% (4/4)100.0% (4/4)
sum_add_label100.0% (4/4)100.0% (4/4)
sum_and_scalar_code100.0% (4/4)100.0% (4/4)
sum_label100.0% (4/4)100.0% (4/4)
sum_threshold_label100.0% (4/4)100.0% (4/4)
tuple_get_label100.0% (4/4)100.0% (4/4)
tuple_sum_label100.0% (4/4)100.0% (4/4)

graphir_pipeline_support

  • Data: data/eval/graph_support.jsonl.
  • Prompt mode: trace.
  • Samples: 1.
FamilyMain HiddenMain Visible
contains_count_length_code100.0% (12/12)100.0% (12/12)
length_contains_code100.0% (12/12)100.0% (12/12)
length_mod_contains_code100.0% (12/12)100.0% (12/12)
modulo_sum_label100.0% (12/12)100.0% (12/12)
not_contains_length_code100.0% (12/12)100.0% (12/12)
sorted_index_offset_label100.0% (12/12)100.0% (12/12)
sum_length_branch_label100.0% (12/12)100.0% (12/12)
sum_offset_mod_label100.0% (12/12)100.0% (12/12)
tuple_branch_label100.0% (12/12)100.0% (12/12)
tuple_sum_gate_label100.0% (12/12)100.0% (12/12)

graphir_repair_corrupt_ceiling

  • Data: data/eval/graph_repair_ceiling_corrupt.jsonl.
  • Prompt mode: trace.
  • Samples: 0.
FamilyMain HiddenMain Visible
sorted_index_sum_branch_label33.3% (4/12)66.7% (8/12)
sorted_join_contains_code0.0% (0/12)16.7% (2/12)
sum_len_mod_label0.0% (0/12)0.0% (0/12)
sum_length_mod_gate_label0.0% (0/12)16.7% (2/12)
text_absent_mod_code25.0% (3/12)25.0% (3/12)
text_value_gate_label83.3% (10/12)100.0% (12/12)
token_absent_length_code33.3% (4/12)41.7% (5/12)
token_count_mod_length_code50.0% (6/12)75.0% (9/12)
tuple_sum_mod_gate_label33.3% (4/12)50.0% (6/12)
tuple_value_mod_label8.3% (1/12)8.3% (1/12)

Experiment log 5

Show the running log (5 entries)

Objective

Test whether fixed-budget posttraining can improve held-out executable repair by moving from freeform DSL generation to typed register-graph configuration plus a verifier-guided repair step.

Design Commitments

  • Use only Qwen/Qwen3.5-4B.
  • Keep each trained adapter at 240 records.
  • Keep the compact experiment directory downloadable by storing adapters and checkpoints outside it.
  • Train a DSL baseline, a GraphIR construction adapter, and a GraphIR repair adapter.
  • Evaluate support-family generalization, held-out ceiling-family generalization, IID retention, and direct graph repair.
  • Keep trace controls optional unless the main pipeline result remains positive or ambiguous.
  • Generate a final markdown report and charts.

Starting Hypotheses

  1. GraphIR should reduce syntactic and parenthesization failures by forcing one operation per register assignment.
  2. GraphIR should make composition errors more visible to the verifier because each intermediate is executable.
  3. A repair adapter should improve over construction-only by editing the selected graph after visible-case execution.
  4. Shuffled traces should hurt if the model is using execution evidence rather than only memorizing output shape.

Planned Runs

  1. Build deterministic datasets from seed 20260701.
  2. Train dsl_static60_lora.
  3. Train graphir_construct_lora.
  4. Train graphir_repair_lora.
  5. Evaluate DSL baseline on IID, support, and ceiling splits.
  6. Evaluate GraphIR construction and construction+repair on IID, support, and ceiling splits.
  7. Optionally run GraphIR trace controls on the ceiling split if the main result needs disambiguation.
  8. Run direct corrupted-GraphIR repair diagnostic on the ceiling split.
  9. Generate charts and final report.
  10. Audit compact artifact size and large artifact separation.

Step Log

  • Initialized standalone experiment directory and large artifact directory.
  • Added GraphIR compiler, parser, executor, and visible-case scoring.
  • Added task-specific prompts for DSL baseline, GraphIR construction, and GraphIR repair.
  • Added dataset builder for aligned DSL, GraphIR construction, and GraphIR repair records.
  • Added generic QLoRA trainer and evaluation scripts.
  • Added report generator scaffold.
  • python -m compileall src scripts passed before dataset generation.
  • Built deterministic datasets with seed 20260701.
  • Dataset counts: DSL train 240, GraphIR construct train 240, GraphIR repair train 240, IID eval 60, support eval 120, ceiling eval 120, corrupted ceiling repair eval 120.
  • Training mix per adapter: 180 base-family records plus 60 support bridge records.
  • Confirmed that no held-out ceiling family appears in any training set.
  • Confirmed target GraphIR executes correctly on train and ceiling eval records.
  • Corrupted GraphIR repair candidates are nontrivial: train candidates pass all visible cases in 9/240 records and all hidden cases in 6/240 records; ceiling diagnostic candidates pass all visible cases in 19/120 records and all hidden cases in 4/120 records.
  • Trained dsl_static60_lora for 2 epochs / 60 optimizer steps on 240 DSL records.
  • dsl_static60_lora training summary: runtime 877.9s, train loss 0.1078, eval loss 0.0003974 on the 24-record training-time eval subset.
  • Saved dsl_static60_lora under /workspace/large_artifacts/qwen35_4b_graphir_self_repair/models/dsl_static60_lora.
  • Trained graphir_construct_lora for 2 epochs / 60 optimizer steps on 240 GraphIR construction records.
  • graphir_construct_lora training summary: runtime 890.7s, train loss 0.05635, eval loss 0.0003681 on the 24-record training-time eval subset.
  • Saved graphir_construct_lora under /workspace/large_artifacts/qwen35_4b_graphir_self_repair/models/graphir_construct_lora.
  • Trained graphir_repair_lora for 2 epochs / 60 optimizer steps on 240 corrupted-candidate GraphIR repair records.
  • graphir_repair_lora training summary: runtime 918.7s, train loss 0.05824, eval loss 0.4679 on the 24-record corrupted-ceiling training-time eval subset.
  • Saved graphir_repair_lora under /workspace/large_artifacts/qwen35_4b_graphir_self_repair/models/graphir_repair_lora.
  • Evaluation token-budget adjustment: target GraphIR outputs were measured before full ceiling/support evaluation. Tokenized target lengths were max 69 tokens on support and max 77 tokens on ceiling, so GraphIR support/ceiling and repair evaluations were capped at 96 new tokens. This preserved margin while preventing non-answer tails from dominating runtime.
  • Added scripts/eval_graphir_cached_repair.py so construction results can be repaired without rerunning construction. This preserves the same selected construct graph while avoiding duplicate GPU work.
  • DSL IID evaluation: 60/60 greedy hidden all-pass and 60/60 rerank hidden all-pass, num_samples=0, max_new_tokens=96.
  • GraphIR construct IID evaluation: 60/60 construction hidden all-pass, num_samples=0, max_new_tokens=160.
  • GraphIR pipeline IID evaluation: 60/60 construction hidden all-pass and 60/60 repair hidden all-pass, num_samples=0, max_new_tokens=160, visible-pass repair skip enabled.
  • DSL support evaluation: 120/120 greedy hidden all-pass and 120/120 rerank hidden all-pass, num_samples=3, max_new_tokens=96.
  • DSL ceiling evaluation: 33/120 greedy hidden all-pass and 35/120 rerank hidden all-pass, num_samples=1, max_new_tokens=96.
  • GraphIR support construction evaluation: 118/120 hidden all-pass, num_samples=0, max_new_tokens=96.
  • GraphIR support cached repair evaluation: improved support from 118/120 to 120/120 hidden all-pass, num_samples=1, max_new_tokens=96, visible-pass repair skip enabled.
  • GraphIR ceiling construction evaluation: 26/120 hidden all-pass, num_samples=1, max_new_tokens=96.
  • GraphIR ceiling cached repair evaluation: improved actual pipeline from 26/120 to 29/120 hidden all-pass, num_samples=1, max_new_tokens=96, visible-pass repair skip enabled.
  • Direct corrupted-GraphIR repair diagnostic: corrupted candidate input scored 4/120 hidden all-pass; greedy repair improved it to 32/120 hidden all-pass, num_samples=0, max_new_tokens=96.
  • Main conclusion: GraphIR plus repair did not beat the DSL baseline on held-out ceiling families. The actual pipeline ended at 29/120 hidden all-pass versus DSL rerank at 35/120.
  • Interpretation: the repair adapter learned some repair behavior on synthetic corrupted GraphIR candidates, but it did not transfer enough to the actual construction adapter's error distribution.
  • Trace controls were not run. The main result was below baseline and the direct repair diagnostic resolved the key ambiguity: repair skill exists, but actual construction errors remain the bottleneck.
  • Generated final report at reports/qwen35_4b_graphir_self_repair_report.md.
  • Generated charts: figures/ceiling_hidden_success.png and figures/ceiling_by_family.png.

Figures 2

ceiling by family
ceiling by family · figures/
ceiling hidden success
ceiling hidden success · figures/

Reproduce

Runnable scripts exist in the experiment folder, but the exact invocation was not written down.

Browse the experiment folder on GitHub ↗