Research log Small Model Experimentation
GitHub

Qwen35 4B Exec-Trace Install

Finished2026-07-17Agentic Breadth InstallationStructured Execution and CompilersGitHub ↗
Null: teaching the model to trace code did not make it code better (but did not hurt it either)

The one idea you need

The team is trying to turn a small model that can write one function into one that can carry a task across many steps. A cheap proxy benchmark they'd hoped would predict real coding turned out not to (it moved exactly with chance). So the new plan is to install the missing skill directly and prove it on real coding tests. This experiment's idea: make the model an accurate 'mental interpreter'. They generate 400 random little Python programs, actually RUN each one to record exactly what every line does to every variable, and train the model to reproduce that step-by-step trace and the final printout. Crucially the training material looks nothing like the exams (the exams ask 'write code from a description'; this teaches 'read this code and say what it does'), and none of it is copied from the exams, so any improvement is real transfer, not memorization. There is one real danger, designed around: teaching a model to always narrate code can make it forget how to WRITE code, so the dose is deliberately small and a fallback 'also practice writing code' blend is prepared.

The question

Does teaching the 4B to trace the execution of random Python programs, step by step, make it better at real coding (HumanEval + MBPP) without making it forget how to write functions?

What we found

The first bet at installing coding cognition came back flat. We trained the 4B on 400 self-generated, execution-verified program traces to install an accurate 'mental interpreter,' the idea being that a model that can simulate code should write and debug it better. It did not pan out: HumanEval nudged +1 problem (noise, it is near ceiling), MBPP dropped 3, and the real target — a multi-step agentic coding harness — landed EXACTLY where the base did, 8 of 35, just solving a different 5 tasks. The good news is retention held: teaching the model to trace did not wreck its ability to write code, which was the main risk. But the capability was reshuffled, not raised — the same pattern the menagerie work showed. The lesson: installing a passive skill (understanding execution) does not convert to active coding ability. The next bet targets the actual failure mode we see — the model takes one shot at a multi-step task and stops instead of planning, verifying, and repairing — by training the agentic loop directly.

Why it matters

This is the program's first real attempt to INSTALL coding cognition (not just measure it) after the cheap proxy was shown not to transfer. State-tracking across steps is the specific thing separating the 4B's 76% function-writing from its 23% agent score. If tracing execution transfers even a little to real pass@1 without breaking function-writing, it validates the whole 'design a contamination-free curriculum and prove it by transfer' method for the rest of the program; if it fails, the pre-committed rule tells the team exactly which fallback to fund next instead of leaving them guessing.

Agentic8/35 -> 8/35exactly flat, 5v5 discordant
HumanEval+1 problem76.2 -> 76.8%, noise near ceiling
MBPP-3 problems56.5 -> 55.0%
Retentionheldno catastrophic forgetting - design worked
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Next Experiments
    9. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 1

Bet #1 (execution tracing): flat on every coding signal (NULL) First cognitive-core coding bet: install a 'mental interpreter' by training on 400 execution-verified program traces (self-generated, no teacher, contamination-clean). Result: NULL. HumanEval +1 problem (76.2->76.8%, noise near ceiling), MBPP -3 (56.5->55.0%), and the real target — the agentic eval — EXACTLY FLAT (8/35 vs 8/35, with 5 base-only and 5 exec_trace-only discordant scenarios). Retention held (the catastrophic-forgetting risk was real and the distinct-instruction + moderate-dose design beat it), but capability was RESHUFFLED not RAISED — the same signature the menagerie composites showed. install!=convert extends to coding: installing a passive cognitive primitive does not convert to coding capability. Bet #2 targets the agentic loop (plan-act-verify-repair) directly, the observed failure mode, rather than a passive component.

pass rate · coding fitness signal →

00.20.40.60.8HumanEvalHumanEval0.7620.768MBPPMBPP0.5650.55Agentic (of 35)Agentic (of 35)0.2290.229
Data table
coding fitness signalbaseexec_trace
HumanEval0.7620.768
MBPP0.5650.55
Agentic (of 35)0.2290.229

Numbers from experiments/qwen35_4b_exec_trace_install/reports/measure_review.md

In the author’s words from the Overview · “Results”

When run, runs/measure/transfer_summary.json records all four pass@1 numbers (base/treatment x HE/MBPP), the per-problem paired deltas, and the frozen verdict. Separate deployable evidence (transfer) from the retention guard.

Overview

Research Program

  • Program: agentic_breadth_installation (cognitive-core coding sub-program)
  • Program question: can real, transferable coding capability be INSTALLED into base Qwen/Qwen3.5-4B by designed, contamination-free curricula, proven by transfer to held-out coding — after the menagerie proxy was shown NOT to transfer (McNemar p=1.00 vs base)?
  • Prior anchors: base coding baselines from experiments/qwen35_4b_coding_fitness_harness — HumanEval 76.2% (strong function coder), MBPP 56.5%, agentic duet-eval 23% (weak agent). The gap is agentic/multi-step cognition (state-tracking across steps).

Question

Does installing an accurate "mental interpreter" via EXECUTION TRACING — train the model to reproduce the execution-verified, step-by-step running-state trace of random terminating Python programs — TRANSFER to real coding (HumanEval + MBPP pass@1) without catastrophically shifting the model into trace-mode (forgetting)?

Hypothesis

The base 4B can write a single function but cannot track state across steps. Tracing concrete execution drills exactly that latent gap. Because the training data is code -> trace (nothing like spec -> code), any HumanEval/MBPP movement is genuine TRANSFER, not memorization. A moderate dose (400 rows, 1 epoch) with an explicit distinct instruction should install state-tracking while retaining function-writing. Honest prior on a strict pass@1 gain with retention: ~25-35% (a real bet, not a sure thing — the forgetting risk is real, cf. answer-only SFT 0.72->0.09).

Setup

  • Model: only Qwen/Qwen3.5-4B (rev 851bf6e8…); ONE fresh r32/a64 QLoRA adapter trained from the base_reserialized composite in a single stage.
  • Dataset/task source: data/sft_exec_trace.jsonl — 400 random terminating Python programs + execution-verified traces, built by scripts/gen_exec_trace_curriculum.py (construction seed 90210). Every trace is triple-verified (primary interpreter, real-CPython sys.settrace re-execution, safety/termination caps).
  • Train/eval split: training is code -> trace; evaluation is the held-out spec -> code HumanEval (164) + MBPP (200) — deliberately disjoint surfaces.
  • Baseline: base Qwen/Qwen3.5-4B on the same shared harness.
  • Controls: contamination firewall (whole-word banned-benchmark-name audit, zero hits; distinctive n-gram overlap, zero); the base composite is authenticated fail-closed (tree + weights) before training.
  • Primary metric: greedy pass@1 on HumanEval + MBPP (shared fitness harness, experiments/qwen35_4b_coding_fitness_harness/scripts/eval_pass1.py, referenced not copied).
  • Oracle-only metrics: none gate here; the agentic duet-eval is a follow-on confirm, not part of this cell's frozen consequence.
  • Hidden-label boundary: the frozen two-directional consequence (INSTALLED_CODING / RETENTION_FAIL / NULL) is read once from the four pass@1 numbers; benchmarks are executed, never read as data.

Run

Smoke (no GPU, no writes):

python scripts/run.py --smoke

GPU stages (each gated behind a staged adversarial review; see reports/preregistration.md for the exact commands and checkpoint order):

python scripts/run.py --stage train     # r32/a64, 1 epoch, seed 90211
python scripts/run.py --stage merge     # vendored composite merger
python scripts/run.py --stage measure   # shared HumanEval+MBPP harness, both arms

Results

Not yet run. The install/merge/measure stages are gated behind staged reviews. When run, runs/measure/transfer_summary.json records all four pass@1 numbers (base/treatment x HE/MBPP), the per-problem paired deltas, and the frozen verdict. Separate deployable evidence (transfer) from the retention guard.

Interpretation

Pending the sealed measurement. INSTALLED_CODING makes exec_trace the program's reference composite and funds the agentic confirm; RETENTION_FAIL realizes the forgetting risk and funds the --mix-retention re-run; NULL is a preserved boundary finding that funds a larger/redesigned dose.

Knowledgebase Update

  • Program evidence updated: pending measurement.
  • Program backlog updated: pending measurement.
  • Claim ledger updated: pending measurement.

Artifacts

  • scripts/ — curriculum generator + tracer, contamination module, vendored trainer/merger, fail-closed train_trial.py, measure_transfer.py, run.py.
  • data/sft_exec_trace.jsonl — the 400-row curriculum + curriculum_receipt.json; data/provenance/base_reserialized.json; data/contamination/.
  • configs/, reports/, tests/
  • reports/artifact_manifest.yaml

Report

Rendered from reports/report.md

Design-frozen report. The model-free construction is complete and verified; the train/merge/measure GPU stages are gated behind staged adversarial reviews and have not run. Results will be appended to the Results section once the sealed measurement is read.

Summary

Lifecycle 32 — the FIRST curriculum bet of the cognitive-core coding program. After the menagerie proxy was shown NOT to transfer to real coding (McNemar p=1.00 vs base), this cell installs an accurate "mental interpreter" into base Qwen/Qwen3.5-4B via EXECUTION TRACING: a fresh r32/a64 LoRA trains (1 epoch, seed 90211) on 400 random terminating Python programs paired with their execution-verified, step-by-step running-state traces, merges onto base, and is measured for TRANSFER + RETENTION on the shared HumanEval + MBPP fitness harness under a frozen two-directional consequence (INSTALLED_CODING / RETENTION_FAIL / NULL). The training signal is SELF-GENERATED and EXECUTION-VERIFIED (no larger teacher) and looks like nothing in the benchmarks (code -> trace, not spec -> code), so any movement is genuine transfer.

Research Program Fit

The program's target is the base 4B's agentic/multi-step cognition gap: HumanEval 76.2% (strong function coder) vs duet-eval 23% (weak agent). The missing sub-skill is state-tracking across steps. Execution tracing drills that directly. HumanEval/MBPP serve as the fast transfer + retention signal here; the agentic duet-eval is the eventual real target (a follow-on confirm, not gated by this cell).

Method

  • Curriculum (scripts/gen_exec_trace_curriculum.py, seed 90210). Random terminating Python over integer/float/string/list/dict assignments, arithmetic, augmented assignment, bounded for/while loops, if/elif/else, list append + index-set, dict update, string concat/repeat + upper/lower, simple function defs+calls and shallow bounded recursion. Difficulty is a FROZEN mixed schedule biased to medium/long: short 80, medium 160, long 160 (400 total). Each row: prompt = a distinct "Trace the following program's execution" instruction + the code; think = the running-state trace, one line per state CHANGE (<stmt> -> <var>=<val>); answer = FINAL: <printed output>.
  • Triple truth audit (never ship an unverified trace). (1) A hand-written diff-emitting primary interpreter produces the trace/output/final-state. (2) The program is rendered to real Python and executed by REAL CPython under sys.settrace in a restricted namespace with a step cap; the trace/output/final-state are reconstructed from actual execution and byte-compared against the primary — a mismatch aborts and the program is discarded. (3) Safety/termination: restricted builtins (no imports/I/O/ filesystem), bounded loops + recursion, a step cap that aborts runaway programs. The committed corpus additionally re-executes end-to-end via --verify-corpus (used in smoke).
  • Contamination firewall (scripts/contamination.py). A committed banned set of all 668 HumanEval + MBPP function names (Python keywords/emitted builtins whitelisted) — zero whole-word hits. A present-only n-gram aid — zero shared 7-grams carrying a distinctive (non-idiom) token with benchmark solution code. Row-level uniqueness (400 unique prompts/task-ids/programs).
  • Install (scripts/train_trial.py → vendored scripts/train_think.py). One fresh r32/a64 adapter, epochs 1, lr 1e-5, batch 1, grad-accum 8, max-length 4096, w_think 0.2, w_close 0.2, seed 90211 (50 optimizer steps), from the base_reserialized composite. The base is authenticated FAIL-CLOSED (in-cell provenance copy + full tree manifest + full 9 GB weights hash) before training.
  • Merge (vendored scripts/merge_adapter.py) with --base-model = the base composite → merged/exec_trace.
  • Measure (scripts/measure_transfer.py → SHARED harness, referenced not copied). Base and exec_trace, HumanEval 164 + MBPP 200, greedy pass@1, identical vLLM path; all four numbers + per-problem paired deltas + the frozen verdict recorded.

Results

Pending the sealed measurement. runs/measure/transfer_summary.json will carry pass_at_1{base,exec_trace}{humaneval,mbpp}, the McNemar b/c paired deltas per dataset, and the frozen consequence. Deployable transfer evidence is the strict pass@1 gain; the retention guard is the paired dataset staying within tolerance.

Construction facts already established (model-free):

  • Corpus sha 7c5b77ea87438f4fb46b1d6d1b468edb275feee4e915161c9180c109d410e32e, 400 rows, all exec_trace; tiers short 80 / medium 160 / long 160; steps 5-47, mean ≈ 23; 400 unique programs.
  • Contamination: 668 banned benchmark names, 0 whole-word hits; 0 distinctive shared 7-grams (34 shared spans, all pure control-flow idioms with no distinctive token).
  • 51 unit tests green (2 present-only cache aids skip without the HF cache).

Controls

  • Contamination firewall (banned-name audit + distinctive n-gram overlap), both zero, so a benchmark movement cannot be memorization.
  • Base composite authenticated fail-closed (tree + weights) before training and merge; a swapped composite aborts.
  • Identical measurement path for both arms (the shared harness), so base and treatment pass@1 are directly comparable.

Oracle Versus Deployable Evidence

Deployable evidence = the strict HumanEval/MBPP pass@1 gain (real, held-out spec -> code generation). The retention guard (the other dataset within 0.02) is a control on the forgetting risk, not a capability claim. The agentic duet-eval is the eventual deployable target but is a follow-on confirm, not gated here. No metric here uses hidden labels beyond the one-shot frozen verdict read.

Interpretation

Pending measurement. INSTALLED_CODING: the mental-interpreter curriculum transfers; exec_trace becomes the program reference and funds the agentic confirm. RETENTION_FAIL: the forgetting risk is realized; re-run with --mix-retention. NULL: a preserved boundary finding pricing this dose, funding a larger/redesigned one.

Next Experiments

  • If INSTALLED_CODING: run the agentic duet-eval confirm on the exec_trace composite; consider a larger/deeper trace dose.
  • If RETENTION_FAIL: re-run the --mix-retention variant (retention-mixed code-completion blend) and re-measure.
  • If NULL: scale the dose (rows/epochs) or redesign the trace surface (e.g., longer programs, more control-flow depth) and re-measure.

Artifact Manifest

See artifact_manifest.yaml — the trained adapter and merged composite live under large_artifacts/ (omitted from git); the curriculum, contamination fixture, base provenance copy, and receipts are in-repo and reproducibility- critical.

Experiment log 2

Show the running log (2 entries, 2026-07-17 → 18)

2026-07-17 — model-free construction frozen

Lifecycle 32, the FIRST curriculum bet of the cognitive-core coding program. Mission: install real coding capability into base Qwen/Qwen3.5-4B via a designed, contamination-free curriculum, proven by transfer — after the menagerie proxy was shown NOT to transfer (McNemar p=1.00 vs base).

The bet: install an accurate "mental interpreter" via EXECUTION TRACING (train code -> execution-verified running-state trace), self-generated and execution-verified (no larger teacher), disjoint from HumanEval/MBPP.

Built and verified (no GPU, no commit):

  • scripts/gen_exec_trace_curriculum.py (seed 90210) — 400 random terminating Python programs + traces, TRIPLE-verified (primary interpreter + real-CPython sys.settrace re-execution + safety/termination caps). Corpus sha 7c5b77ea87438f4fb46b1d6d1b468edb275feee4e915161c9180c109d410e32e; tiers short 80 / medium 160 / long 160; steps 5-47, mean ≈ 23; 400 unique programs. A --mix-retention R switch (default OFF) is prepared as the forgetting guard.
  • scripts/contamination.py + data/contamination/banned_function_names.json (668 benchmark function names) — 0 whole-word hits; 0 distinctive shared 7-grams with benchmark solution code.
  • Vendored scripts/train_think.py (sha e0eca2a2…) and scripts/merge_adapter.py (sha cb9af8b4…), byte-identical to the chain trainer/merger.
  • scripts/train_trial.py — fail-closed base authentication (in-cell provenance copy + tree manifest + full weights hash); recipe r32/a64, 1 epoch, lr 1e-5, seed 90211 (50 optimizer steps).
  • scripts/measure_transfer.py — invokes the shared fitness harness for both arms x both datasets; frozen INSTALLED_CODING / RETENTION_FAIL / NULL consequence.
  • scripts/run.py — checkpointed --smoke | --stage train | --stage merge | --stage measure; each GPU stage gated behind a staged adversarial review.
  • 51 unit tests green (2 present-only HF-cache aids skip without the cache); run.py --smoke green; boundary drills refuse.

Grep-fresh note: construction seed 90210 is fully fresh repo-wide; training seed 90211's only other occurrence is an unrelated menagerie confidence-gym sampling-seed default (qwen35_4b_gauntlet_frontier/scripts/gym_confidence.py), a different seed context — no training-seed collision in this chain.

GPU stages (train/merge/measure) are pending their staged reviews.

2026-07-18 — Transfer measurement complete: NULL (bet #1 closed)

  • exec_trace vs base: HumanEval 0.7622->0.7683 (+1 problem), MBPP 0.5650->0.5500 (-3), agentic 8/35->8/35 (5v5 discordant). Retention held (no catastrophic forgetting - the distinct-instruction + moderate-dose design worked). No real coding improvement on any signal. The frozen fast-rule technically fired INSTALLED_CODING on the +1 HumanEval, but that is noise (near ceiling) and the agentic real target is exactly flat -> honest verdict NULL.
  • LAW: install!=convert extends to coding. Installing a passive cognitive primitive (execution-tracing / state-tracking) reshuffles WHICH coding tasks are solved, not HOW MANY - the same signature the menagerie composites showed. Two independent curriculum families now reshuffle-without-raising, suggesting narrow static-SFT skill-install has a structural limit for coding capability.
  • NEXT (bet #2): target the AGENTIC LOOP directly (plan-act-verify- repair / persistence + self-correction), the observed failure mode (model does one pass and stops), rather than a passive component; and tighten the transfer rule to a meaningful-delta / significance bar.

Reproduce

Smoke test

python scripts/run.py --smoke

Full run

python scripts/run.py --stage train && python scripts/run.py --stage merge && python scripts/run.py --stage measure

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗