Research log Small Model Experimentation
GitHub

Self-Written Verifier Fidelity

In progress2026-07-27 → 28Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safety+1 moreGitHub ↗

In the author’s words from the Overview · “Results”

Fill this after the run. Separate deployable evidence from oracle/hidden evaluation.

On this page
  1. Overview
  2. Report
    1. Summary
    2. Research Program Fit
    3. Method
    4. Results
    5. Controls
    6. Oracle Versus Deployable Evidence
    7. Interpretation
    8. Next Experiments
    9. Artifact Manifest
  3. Experiment log
  4. Reproduce
  5. Related

Overview

Can the 4B write executable checks from a spec alone that predict a hidden real-repo test suite's verdict? Converts weak self-judgment (0.679 AUROC) into strong execution; the load-bearing question for verifier-gated search on tasks with no test suite.

Research Program

  • Program: evidence_conditioned_selection
  • Program question:
  • Prior anchors:

Question

What specific uncertainty does this experiment resolve?

Hypothesis

State the mechanism you expect to work and why it should beat the baseline.

Setup

  • Model:
  • Dataset/task source:
  • Train/eval split:
  • Baseline:
  • Controls:
  • Primary metric:
  • Oracle-only metrics:
  • Hidden-label boundary:

Run

Smoke:

python scripts/run.py --smoke

Full:

python scripts/run.py

Results

Fill this after the run. Separate deployable evidence from oracle/hidden evaluation.

Interpretation

What changed after this result? What is now more likely, less likely, or still unknown?

Knowledgebase Update

  • Program evidence updated:
  • Program backlog updated:
  • Claim ledger updated:

Artifacts

  • src/
  • scripts/
  • configs/
  • data/
  • runs/
  • analysis/
  • reports/
  • reports/artifact_manifest.yaml

Report

Rendered from reports/report.md

Summary

Research Program Fit

Method

Results

Controls

Oracle Versus Deployable Evidence

Interpretation

Next Experiments

Artifact Manifest

Update artifact_manifest.yaml before considering this result complete.

Experiment log 1

Show the running log (1 entry)

Scaffold

Created as a new experiment scaffold.

Reproduce

Smoke test

python scripts/run.py --smoke

Full run

python scripts/run.py

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗