Research log Small Model Experimentation
GitHub

Real-Repo Agentic Instrument

In progress2026-07-25Agentic Breadth InstallationStructured Execution and CompilersGitHub ↗

In progress since — task generation and baseline measurement under way

INSTRUMENT BUILT: 200 execution-verified stub-a-function tasks across 15 real OSS libraries (138 train / 62 held-out, firewalled by repo, scored per-test), replacing an 11-task holdout that sat at 0.818 with every task already solvable. Also established the cause of eight WSL VM deaths that had been blocking the program.

The one idea you need

To tell whether training made a coding agent better, you need tasks it can be scored on fairly. The previous test set had 11 synthetic tasks, and the model could already solve every one of them given a few tries - so no experiment could show an improvement. This builds a bigger, harder one out of real open-source Python libraries: check out a library, delete the body of one function, and ask the agent to write it back; the library's OWN test suite decides whether it worked. Libraries are split into a training group and an evaluation group, so held-out tasks come from codebases the model was never trained on.

The question

Can we build a real-codebase agentic coding test set with enough headroom to detect an improvement at all?

What we found

Yes: 200 tasks over 15 libraries, each one verified two ways - the library's suite passes untouched, and deleting the target function actually breaks specific named tests. Three design traps had to be avoided, and each would have produced confident nonsense. Demanding a perfectly green test suite threw away 11 of 24 libraries over one or two unrelated environment failures, so scoring moved to named per-test sets, which ignores tests that were already failing and makes 'edit the test instead of the code' worthless. A standard editable install made the tests import the ORIGINAL source instead of the agent's copy, which would have handed every episode a free pass. And one common test runner mode hides per-test results, which silently recorded a 1,317-test library as having zero tests.

Why it matters

Everything the program wants to try next - training on harvested trajectories, reinforcement learning with execution rewards, confidence-gated sampling - was unmeasurable on the old eval, and one required control (how the untrained base model actually performs in the real agent harness) had never been run at all. This makes both possible. Separately, the crash investigation it forced turned an eight-times-repeated 'the machine just dies' into a diagnosed, fixed engineering problem, which recovers the ability to run long jobs at all.

Held-out tasks62over 6 libraries never used for training; 138 more on the train side
Prior eval ceiling0.818 selected11 tasks, all solvable -> could not resolve a +0.05 effect
Libraries recovered18 of 24 usableper-test scoring instead of whole-suite-green; +8 recovered by fixing the test-runner parser
Crash cause~31GB in one process15.3GB resident + 16.1GB swapped -> global OOM condemns /init.scope, i.e. all of WSL
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Next Experiments
    9. Artifact Manifest
  4. Experiment log
  5. Reproduce
  6. Related

Results at a glance 2

A measurable instrument: 200 firewalled real-repo tasks replace an 11-task saturated holdout The prior eval sat at 0.818 execution-selected with 11/11 tasks solvable, so it could not resolve a +0.05 change. The new set spans 15 real OSS libraries, split by REPO so held-out tasks live in codebases never harvested, and is scored per-test (each task's fail_to_pass set plus the repo's baseline as a regression guard).

number of tasks · task set →

050100150old pi holdout (synthetic)old pi holdout (synthetic)11new: train reposnew: train repos138new: held-out reposnew: held-out repos62
Data table
task settasks
old pi holdout (synthetic)11
new: train repos138
new: held-out repos62

Numbers from

Libraries recovered by per-test scoring instead of whole-suite-green Requiring a globally green suite discarded 11 of 24 libraries over one or two environment-dependent failures that behave identically before and after an agent's edit. Per-test sets exclude them by construction; adding -rA (pytest-xdist hides per-test lines from -v) recovered 8 more.

number of libraries · stage →

0102030candidatescandidates24usable after baseliningusable after baselining18yielding tasksyielding tasks15
Data table
stagelibraries
candidates24
usable after baselining18
yielding tasks15

Numbers from

In the author’s words from the Report · “Summary”

Built a measurable replacement for this program's saturated eval: 200 execution-verified stub-a-function tasks across 15 real OSS Python libraries, 138 train / 62 held-out, firewalled by repo, scored per-test against each task's fail_to_pass set with the repo's baseline as a regression guard. The verifier is the repository's own pytest. The raw-base pi baseline — a control this program has never had — is measuring now.

Overview

Build a broad, firewalled real-codebase agentic-coding instrument for Qwen3.5-4B in pi-coding-agent — and measure the raw-base baseline this program has never had.

Research Program

  • Program: agentic_breadth_installation (cognitive-core coding sub-program).
  • Program question: can real-codebase agentic coding capability be INSTALLED into Qwen3.5-4B on one 24GB GPU, and can any installed gain beat matched-compute sampling?
  • Prior anchors: C63 (execution-selected best-of-N is the deployable lift: 0.606 → 0.818 with zero training), C64 (the 4B already does real-repo agentic coding: 0.70 single-shot / 0.91 selected on toolz via pi), C62 (four LoRA edits of that warm-start each REGRESSED deployment), C60 (authored think traces crater a near-ceiling coder; only the model's own harvested traces are retention-safe).

Question

Can we measure an agentic-coding improvement at all? Everything queued behind this experiment — harvest-and-retrain, RLVR with a world-model auxiliary loss, confidence-gated selection — is unmeasurable on the current eval, and one required control is missing entirely.

Hypothesis

Not a capability hypothesis. Two instrument claims, each falsifiable:

  1. A broad, firewalled real-repo task set (many libraries, difficulty-stratified, held-out repos never trained on) has enough headroom and resolution to detect a ≥0.05 change in deployed solve rate — unlike the 11-task holdout, which sits at 0.818 selected with 11/11 tasks solvable.
  2. The raw base measured in pi is materially below the SFT warm-start. If it is not, then the program's deployment baseline was never the warm-start's achievement and every subsequent attribution needs revisiting.

Setup

  • Model: Qwen/Qwen3.5-4B only (pinned revision), served through vLLM to pi-coding-agent.
  • Task source: 24 candidate pure-Python OSS libraries (scripts/repos.py) cloned at recorded SHAs. A task = one function body replaced by raise NotImplementedError; the verifier is the repository's own pytest — the property that made C64 credible.
  • Train/eval split: repo-level firewall, each repo's side fixed in repos.py before any task was generated or any number measured. Held-out tasks live in libraries whose code, conventions, and test style are absent from any training corpus. (Splitting within a repo would leak: harvesting funcy teaches its idioms, its test layout, and often the neighbouring functions its held-out functions call.) split_tasks.py asserts no repo appears on both sides.
  • Baseline: raw base through pi. Controls: merged SFT warm-start; execution-selected best-of-k on both arms (the "sample more" bar any training must beat, per C63); a closed-book memorization probe.
  • Primary metric: single-shot solve rate and execution-selected best-of-k on the held-out split.
  • Oracle-only: none. Partial reward is reported but the headline is binary solve.
  • Hidden-label boundary: test-group repos are eval-only, forever; harvesting or training on them invalidates the instrument.

Scoring: per-test, not whole-suite

Each task carries the fail_to_pass set its stub breaks; each repo carries its baseline passing set, whose remainder is pass_to_pass. Reward 1.0 = all broken tests pass and nothing regressed; partial credit is the fraction of broken tests fixed, halved if anything regressed.

Requiring a globally green suite is wrong — it dropped 11 of 24 repos over one or two environment-dependent failures (a package-metadata check; a test wanting >6 GB). Those tests fail identically before and after an agent's edit, so they carry zero information about the agent, yet they would have silently capped every episode at partial credit. Per-test sets exclude them by construction, and pass_to_pass makes "edit the test instead of the code" unrewardable. Dependency installation is fair game (environment); editing a repo's test code never is — a suite we repaired is no longer "the repository's own pytest".

Three traps this harness is built to avoid

The editable-install trap gives free rewards. uv pip install -e . + copy-per-episode is silently broken for src/ layouts: the editable .pth resolves imports to the ORIGINAL checkout, so pytest in the episode's copy imports unstubbed source, every test passes, and every episode scores 1.0 — an instrument reporting a perfect agent that never wrote a line. Layered mitigation: install editable once to resolve deps then uninstall the distribution; point PYTHONPATH at the copy's own roots; and admit a task only if stubbing it in a copy actually breaks tests, so an import leak yields zero tasks instead of free reward.

-v alone cannot read a suite that uses pytest-xdist. With -n auto in a repo's own addopts the per-test progress lines vanish and the parser sees nothing — wcwidth baselined at "0 passed" until -rA was added. The -rA summary section is authoritative in every execution mode.

Generation and evaluation must share one code path. Both call env_util.prepare_copy / run_tests / score_task. This program's three worst measurement failures were harness mismatches (real-repo ~0.00 vs pi 0.70; synthetic 0.486 vs 0.810; seven "absent" tasks that were a dict-default bug), so a baseline captured with a different command than the episode is scored with is the same class of bug — which is why changing the test command requires --rebaseline.

Operational contribution: the WSL crash cause, established

This experiment's first runs killed the WSL VM, which forced the diagnosis the program had been missing through seven prior deaths (docs/wsl_stability.md, rewritten). Replicated signature:

Out of memory: Killed process (python) total-vm:32964064kB, anon-rss:15671808kB
oom-kill: ... global_oom, task_memcg=/init.scope
init.scope: Failed with result 'oom-kill'   [15.1G mem peak, 15.8G swap peak]

One python process reaches ~31 GB of anonymous memory (15.3 GB resident + 16.1 GB swapped), saturating the 16 GB VM cap and 16 GB swap; the kernel's global OOM condemns /init.scope, which under WSL2 contains all of WSL. Not host-memory exhaustion (falsified at crash #7 with 7.3 GB host free) and not GPU-specific (crash #8 had zero GPU work). Two mechanisms produce it, both live in this repo: unbounded third-party test allocation, and — the general one — subprocess.run(..., stdout=PIPE) buffering a child's entire output in the parent's RAM, which is how every driver here captured output, including pi episode runners that stream --mode json events for a 600 s wall.

Verified fixes: run pipelines in a memory-limited cgroup scope (scripts/guard.sh — a deliberate 3 GB runaway at MemoryMax=1G is killed with the VM surviving); spool child output to disk and stream it (env_util.run_spooled); cap untrusted children with ulimit -v (confirmed honoured by dash); kill process groups, since a killed pytest's children keep printing.

Run

Smoke (two repos, a handful of candidates, no GPU):

scripts/guard.sh .venv/bin/python scripts/fetch_repos.py --only toolz,mergedeep --jobs 1
scripts/guard.sh .venv/bin/python scripts/gen_tasks.py --only toolz --jobs 2

Full:

scripts/guard.sh .venv/bin/python scripts/fetch_repos.py --jobs 2      # -> data/repo_*.json
scripts/guard.sh .venv/bin/python scripts/gen_tasks.py --jobs 6        # -> data/tasks_all.json
.venv/bin/python scripts/split_tasks.py                                # -> data/tasks_split.json
scripts/serve.sh Qwen/Qwen3.5-4B                                       # readiness-gated vLLM
.venv/bin/python scripts/pi_episode_repo.py --label base --split test --k 3

Every heavy step runs under scripts/guard.sh; every runner checkpoints and resumes from its own partial output (this box has died eight times and no completed episode has ever been lost).

Results

Pending. Report single-shot, execution-selected best-of-k, and mean partial reward per split and stratum, with the raw-base vs warm-start contrast as the headline.

Interpretation

Pending.

Knowledgebase Update

  • Program evidence updated: pending
  • Program backlog updated: pending
  • Claim ledger updated: pending (an instrument cell may close without a capability claim; the raw-base-vs-warm-start contrast is claim-bearing if it lands)

Artifacts

  • scripts/ — repo registry, fetch/baseline, task generation, splits, serving, pi episode runner
  • data/ — repo_manifest.json, repo_baselines.json, tasks_all.json, tasks_split.json (small and committed: this is the reproduction path)
  • large_artifacts/qwen35_4b_realrepo_agentic_instrument/ — episode records, pi trajectories (retained deliberately as harvest substrate for the Line-1 successor), serve/run logs
  • reports/artifact_manifest.yaml

Report

Rendered from reports/report.md

Summary

Built a measurable replacement for this program's saturated eval: 200 execution-verified stub-a-function tasks across 15 real OSS Python libraries, 138 train / 62 held-out, firewalled by repo, scored per-test against each task's fail_to_pass set with the repo's baseline as a regression guard. The verifier is the repository's own pytest.

The raw-base pi baseline — a control this program has never had — is measuring now.

Research Program Fit

agentic_breadth_installation closed its prior cell with a deployment result and an instrument problem. Execution-selected best-of-N reaches 0.818 on the 11-task synthetic holdout (C63) and 0.909 on toolz (C64), with 11/11 holdout tasks solvable, so per-run noise is the size of the effects now being chased. Four LoRA edits of the warm-start each regressed deployment (C62). Nothing queued behind that — harvest-and-retrain, RLVR with a world-model auxiliary loss, confidence-gated selection — is measurable on the old eval, and "the warm-start improved pi deployment" was inferred from a TRL-env engagement contrast rather than measured against the base.

Method

Each candidate repo is cloned at a recorded SHA, given its own test venv, and baselined per test in a throwaway copy. A task replaces one function body with raise NotImplementedError; it is admitted only if stubbing it in a copy makes tests that pass in the baseline fail. Episodes run through pi-coding-agent with cwd set to a fresh copy, scored by env_util.score_task.

Reward: 1.0 when every fail_to_pass test passes and nothing in pass_to_pass regressed; partial credit is the fraction of broken tests fixed, halved if anything regressed; 0.0 if the stub is still present (engagement gate).

Firewall: by REPO, with each repo's side fixed in repos.py before any task was generated or any number measured. split_tasks.py asserts no repo appears on both sides.

Results

count
candidate libraries24
usable after baselining18
libraries yielding tasks15
train tasks138 (9 repos)
held-out tasks62 (6 repos)
candidates validated310 (241 admitted)

Held-out strata: 23 small / 30 medium / 9 large by body size; median 9 body lines. Reject reasons: 25 not test-covered, 21 suite unrunnable, 23 stub failed.

End-to-end validation on two held-out tasks with the raw base: bidict.popitem solved first try (reward 1.0); BidictBase.equals_order_sensitive reached 25/27 target tests with zero regressions before timing out (reward 0.563, pi_exit 124). That second episode reproduces the C62/C63 termination failure mode independently on a new substrate — the model edits, passes most tests, and fails to finish.

Controls

  • Two-side task validation: the suite passes in an untouched copy, and stubbing the target breaks specific named tests. This is also the detector for import leakage — a leaking repo yields zero tasks rather than free rewards.
  • pass_to_pass regression guard makes "edit the test instead of the code" unrewardable.
  • Repo-level firewall, pre-registered in repos.py.
  • Closed-book memorization probe (planned, not yet run): can the model emit the function from name and docstring alone with no repo access? These are public libraries, so a solve rate on that probe bounds how much of any score is recall rather than agentic capability.

Oracle Versus Deployable Evidence

Every number is deployable-side: episodes run in the real agent scaffold, and rewards come from the repository's own tests. There are no oracle arms. The one caveat to state plainly is contamination — these are public repos, so absolute solve rates may include memorised implementations; the firewall protects training claims, and the memorization probe is what would bound the absolute numbers.

Interpretation

Three design decisions were each load-bearing, and the naive alternative would have produced confident nonsense in a different direction:

  1. Whole-suite-green is the wrong admission gate. It discarded 11 of 24 libraries over one or two environment-dependent failures whose outcome is identical before and after an agent's edit — while silently capping every episode at partial credit. Per-test sets exclude them by construction.
  2. The editable-install trap gives free rewards. For src/ layouts an editable install resolves imports to the original checkout, so the stub has no effect and every episode scores 1.0. Guarded by uninstalling the distribution, pointing PYTHONPATH at the copy, and the stub-breaks-tests gate.
  3. -v cannot read a pytest-xdist suite. wcwidth baselined at "0 passed" until -rA was added; that fix recovered 8 libraries.

Generation and evaluation share one code path, because this program's three worst failures were harness mismatches — which is why changing the test command requires --rebaseline.

Next Experiments

  1. Finish the raw-base held-out baseline (in flight), then the merged SFT warm-start on the same tasks: the first direct measurement of what that fine-tune bought in pi.
  2. Execution-selected best-of-k on both arms — the "sample more" bar any training must beat (C63).
  3. The closed-book memorization probe, to bound contamination in the absolute numbers.
  4. Line 1 (successor cell): harvest execution-verified trajectories from the 138 train tasks, train ONE warm-start from base rather than editing the existing policy (0/4 base rate, C62), then GRPO with an ECHO-style auxiliary loss on environment-observation tokens.

Artifact Manifest

data/ holds the reproduction path and is committed: repo_manifest.json (resolved SHAs, install strategy, dependency repairs, baseline counts), repo_baselines.json (exact passing/failing node ids per repo), tasks_all.json, tasks_split.json. Repo checkouts, per-episode records and pi trajectories live under large_artifacts/ — see artifact_manifest.yaml.

Experiment log 1

Show the running log (1 entry)

Scaffold

Created as a new experiment scaffold.

Reproduce

Smoke test

scripts/fetch_repos.py --only toolz,mergedeep --jobs 1 && scripts/gen_tasks.py --only toolz --jobs 2

Full run

scripts/fetch_repos.py --jobs 2 && scripts/gen_tasks.py --jobs 4 && scripts/split_tasks.py

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗