Coding Fitness Harness (cognitive-core program)
The one idea you need
After proving the menagerie proxy does not transfer to real coding, the mission became installing real coding ability into the 4B, measured directly. This is the measuring stick: a fast HumanEval+MBPP pass@1 harness that runs actual tests against the model's code, and can grade any trained version the same way.
The question
How good is the raw 4B at coding, on signals we can iterate against fast?
What we found
Surprisingly good at writing single functions (HumanEval 76.2%, MBPP 56.5%) but weak at driving a multi-step coding task in a real agent loop (23%). The harness is validated: it agrees with an independent run to within 1 problem in 164, is deterministic, and correctly passes canonical solutions while failing wrong ones. The 76-vs-23 gap tells us exactly where to aim: not function-writing, which is nearly maxed, but the agentic cognition — planning, tracking state across edits, debugging — that turns a function writer into a coding agent.
Why it matters
You cannot install a capability you cannot measure. This fixes the fitness signal to the real target (running code) instead of a proxy, and pinpoints the gap worth attacking.
On this page
Results at a glance 1
pass@1 · benchmark →
Data table
| benchmark | thinking-OFF (broken, 512 tok) | thinking-ON (fixed, 8192 tok) |
|---|---|---|
| HumanEval | 0.762 | 0.896 |
| MBPP | 0.565 | 0.755 |
Numbers from experiments/qwen35_4b_coding_fitness_harness/runs/base_humaneval.json
In the author’s words from the Overview
Shared measurement infrastructure for the cognitive-core coding program (owner pivot, 2026-07-17): after the menagerie aggregate was shown NOT to transfer to real agentic coding (McNemar p=1.00 vs base), the mission became installing REAL coding capability into base Qwen/Qwen3.5-4B via designed contamination-free curricula, proven by TRANSFER to held-out coding. This cell is the fast fitness signal every curriculum bet is scored against: a standalone, adapter-evaluable HumanEval + MBPP pass@1 harness (greedy, execution-graded) reusing the C46 record/prompt/execute machinery through the pinned model_override vLLM runner, so base and every trained composite are graded on the identical path. … Read the full result →
Overview
Shared measurement infrastructure for the cognitive-core coding program (owner pivot, 2026-07-17): after the menagerie aggregate was shown NOT to transfer to real agentic coding (McNemar p=1.00 vs base), the mission became installing REAL coding capability into base Qwen/Qwen3.5-4B via designed contamination-free curricula, proven by TRANSFER to held-out coding. This cell is the fast fitness signal every curriculum bet is scored against: a standalone, adapter-evaluable HumanEval + MBPP pass@1 harness (greedy, execution-graded) reusing the C46 record/prompt/execute machinery through the pinned model_override vLLM runner, so base and every trained composite are graded on the identical path.
Base Qwen/Qwen3.5-4B baselines (greedy pass@1, this harness)
CORRECTED 2026-07-18 — thinking-on, 8192-token budget (the harness had been mistakenly measuring thinking-OFF at 512 tokens, suppressing base ~13-19pp; all prior coding-program numbers are superseded):
- HumanEval: 0.8963 (147/164) — thinking-on; NEAR CEILING (little SFT headroom). Was 0.7622 thinking-off.
- MBPP (full test, first 200): 0.7550 (151/200) — thinking-on; some headroom (53% of traces hit the 8k cap / force-close). Was 0.5650.
- Agentic (duet-eval gen4, external, measurement-only): 8/35 = 0.229 — already thinking-on; the base is a WEAK multi-step coding agent.
The 90% vs 23% gap is the program's target: the 4B writes functions well but cannot drive a multi-step coding task. Function completion is nearly maxed; the agentic gap is the real prize (RLVR). Curricula are measured here thinking-on for retention + fast signal, and on the agentic eval for the real target.
Interface
.venv-vllm/bin/python scripts/eval_pass1.py --dataset {humaneval,mbpp} --n N [--model-override /path/to/merged/composite] --out FILE (greedy, seed 0, execution-graded; --model-override fingerprint-checks a merged Qwen3.5-4B composite so trained arms are graded identically).
Caveat: numbers are vLLM-greedy; never compare to an HF run — keep every arm on this harness.
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
.venv-vllm/bin/python experiments/qwen35_4b_coding_fitness_harness/scripts/eval_pass1.py --dataset humaneval --smoke --out /tmp/smoke.jsonFull run
.venv-vllm/bin/python experiments/qwen35_4b_coding_fitness_harness/scripts/eval_pass1.py --dataset {humaneval,mbpp} --n N [--model-override /path/to/merged/composite] --out FILERun steps are documented inside the experiment folder (README and scripts).