Evaluate each adapter before you activate it.

Choose the operation that answers your question

Eval generation shares the inference side of the GPU lock: it can run beside ordinary inference but waits while training has exclusive accelerator ownership. maintenance rejects eval admission. The default stable profile supports saved adapters, training, and gated promotion in one process.

By default, Kiln stores suites, datasets, and judgments under <adapter-dir>/.eval/{suites,datasets,judgments}/. To choose another root, set eval_dir = "/path/to/eval" in [eval]; eval.root is not an accepted field. See the complete Eval guide for wire contracts and Architecture › Eval pipeline for the worker flow.

Choose a scorer with kind

Each scorer compares a generated completion with the example target and metadata, then records a numeric score and a pass, fail, invalid, or error outcome. The suite or individual example selects a scorer through kind.

auto_detect checks the target in this order: multiple-choice label, number, tool call, code, valid JSON, short exact text, then long-form key phrases with contains. It never selects llm_judge implicitly.

Turn an SFT or GRPO file into a graded suite

Upload one corpus and Kiln creates content-addressed, group-aware train, validation, and holdout views. Named training defaults to train; suite synthesis defaults to holdout. Split assignment, not the shared JSONL file, determines whether a row is training or held-out data. The reservoir-sampled max_examples limit bounds memory use for large sources.

curl -fsS http://localhost:8420/v1/eval/datasets/upload \
  -F "file=@/tmp/sft-trajectories.jsonl" \
  -F "name=trajectories-v1" \
  -F "format=sft_chat" \
  | python3 -m json.tool

curl -fsS http://localhost:8420/v1/eval/datasets/trajectories-v1/synthesize \
  -H "Content-Type: application/json" \
  -d '{
    "suite_name": "trajectories-suite",
    "source_split": "holdout",
    "strategy": "tool_call_predict",
    "scorer": {"kind": "auto_detect"},
    "sampling": {"max_examples": 256}
  }' \
  | python3 -m json.tool

The persisted split manifest keeps normalized duplicates and declared groups or sessions together. Inspect or replace it through GET/PUT /v1/eval/datasets/{name}/split. The complete identity, migration, contamination, and provenance contract is in Dataset splits and train/eval separation.

Grade an adapter against a registered or inline suite

POST /v1/eval/run queues a job and returns its job_id and immutable effective_seed immediately. The worker resolves the registered suite or validates its inline copy, runs the requested base model or adapter, and persists each outcome and aggregate metrics. Open /ui › Evals to watch progress and copy the job and per-completion seeds.

Admission also snapshots the resident kiln.base-weight-shards.v1 manifest and startup-owned kiln.execution-provenance.v1 record. Job detail and raw JSON retain every shard plus the complete backend/device, runtime, executable/source, model/tokenizer/template, precision, kernel, and effective-configuration envelope. The dashboard and kiln-eval show compact exact-copy summaries; downloaded outcome JSONL carries both complete records on every standalone row, and terminal archives validate them again on restart. See the base-weight and execution provenance contracts.

curl -fsS http://localhost:8420/v1/eval/run \
  -H "Content-Type: application/json" \
  -d '{
    "suite": "trajectories-suite",
    "adapter": "my-coder-v3",
    "seed": 42
  }' \
  | python3 -m json.tool

JOB_ID=replace-with-returned-job-id
curl -fsS "http://localhost:8420/v1/eval/jobs/${JOB_ID}" \
  | python3 -m json.tool

Re-run only failures from a completed job with POST /v1/eval/jobs/{id}/rerun. The re-run retains the original effective seed by default; send {"seed": 73} only when changing it deliberately. A failure rerun is a subset diagnostic and makes no byte-reproduction claim; use strict replay for that contract.

Replay one run and compare exact decoder bytes

Every newly completed server eval run carries a self-validating kiln.eval-replay.v1 record. It binds the exact suite, generation overrides, effective and per-completion seeds, resolved thinking budgets, scorer configurations, candidate and judge targets, base weights, execution environment, and content hashes for every raw decoder continuation and normalized scoring text.

The examples use illustrative job IDs. Replace them with IDs from a completed run or compare job. The kiln-eval command also assumes that you built and placed that client on your PATH.

kiln-eval replay --job eval_123

curl -fsS -X POST http://localhost:8420/v1/eval/jobs/eval_123/replay \
  -H "Content-Type: application/json" \
  -d '{}'

kiln-eval replay --job eval_compare_123 --run-index 1 --json

Admission refuses legacy or incomplete sources, changed execution or base-weight digests, and changed candidate or judge adapter bytes before the replay enters the queue. The executor rechecks the loaded candidate before decoding and each loaded judge before scoring. Terminal replay_verdict.status is matched, mismatch, or error; the CLI waits for a terminal state and exits nonzero unless every bound identity and raw continuation matches byte-for-byte. The Evals dashboard offers the same operation and copyable hashes for the selected run.

A match proves reproduction for that declared run and environment, not driver correctness or cross-backend determinism. A mismatch is retained evidence and blocks the claim; nondeterministic backend operations may still be the cause. New terminal archives require and validate replay records, while legacy archives remain readable but cannot be strict-replayed. The complete identity, race-boundary, archive, and verdict contract is in EVAL_GUIDE.md.

Head-to-head adapter comparison

POST /v1/eval/compare materializes one job seed, derives stable seeds from each example_id and completion index, and runs every target on those same sampling inputs. Empty or duplicate resolved example IDs are rejected so seed and result rows cannot alias. An empty adapter string selects the no-adapter baseline. Omit seed for a recorded random value, or set it at the top level without replacing other suite generation settings.

curl -fsS http://localhost:8420/v1/eval/compare \
  -H "Content-Type: application/json" \
  -d '{
    "suite": "trajectories-suite",
    "adapters": ["", "my-coder-v2", "my-coder-v3", "my-coder-v4"],
    "seed": 42,
    "generation": { "temperature": 0.0, "max_tokens": 256 }
  }' \
  | python3 -m json.tool

Results encode effective_seed and each outcome's generation_seed as decimal strings so browser clients preserve all 64 bits. seed_derivation: "kiln.eval-seed.v1" versions the mapping. Seeded inputs support same-environment comparison and audit; they do not imply byte-identical output across different model revisions, binaries, drivers, devices, kernels, or precision policies.

A compare job is diagnostic: it never activates an adapter. Use a training request with post_eval.min_accuracy when held-out evidence must gate activation of the exact trained revision.

Grade training against declared held-out data

The default stable profile supports this live train → eval → promotion workflow, including coordinated adapter-weight transitions. maintenance admits drained training but rejects every eval request because inference is disabled.

An SFT or GRPO request can carry a post_eval block whose data_scope defaults to held-out. Before publishing the training job, Kiln rejects exact, normalized, source-row, group, or session overlap with the admitted corpus. After training, it queues the adapter eval and optional baseline comparison and links both from training detail.

curl -fsS http://localhost:8420/v1/train/sft \
  -H "Content-Type: application/json" \
  -d '{
    "dataset": "trajectories-v1",
    "dataset_split": "train",
    "config": { "output_name": "my-coder-v4", "auto_load": true },
    "post_eval": {
      "suite": "trajectories-suite",
      "data_scope": "held-out",
      "include_baseline": true,
      "min_accuracy": 0.85,
      "generation": { "temperature": 0.0, "max_tokens": 256, "seed": 42 }
    }
  }' \
  | python3 -m json.tool

The same post_eval shape is supported on /v1/train/grpo. Intentional training-data measurement must use data_scope: "train-set-eval"; that diagnostic label cannot be combined with min_accuracy. See Dataset splits and train/eval separation and the generated control-plane schema for the full contract.

Fail-closed promotion evidence

min_accuracy enables the fixed paired_wilson_v1 policy. Kiln compares the candidate with the previously active adapter, or the base model, under the same versioned suite, generation identity, seed derivation, and reduced per-example aggregation. It requires at least 20 independent reduced examples, runs a two-sided exact sign test at alpha=0.05, and requires the candidate’s 95% Wilson lower bound to reach min_accuracy. A point estimate never promotes an adapter.

The 20-example minimum and alpha are fixed rather than caller-configurable. The accuracy threshold is a floor, not a point-estimate check: 20 passes from 20 examples have a Wilson lower bound of about 0.84, so they cannot prove a 0.90 requirement. Multi-sample suites reduce to one independent result per example before this calculation.

Training status responses persist post_eval_gate_evidence[] with the suite and generation hashes, paired sample and flip counts, exact p-value, baseline and candidate Wilson intervals, all configured confidence thresholds, and the final classification. The dashboard surfaces n, p, and the candidate lower bound without parsing verdict prose.

Automatic promotion accepts one versioned held-out suite. Compose required domains into that suite and use tags for slices; Kiln does not average independent suites or accept a passing suite as an OR over a failing one. include_baseline controls an additional browseable base result, not the gate’s mandatory paired comparison.

distill_refresh enforces the same boundary across gated post_eval, if_eval_suite, and new_knowledge_eval_suite fields. Automatic loading with more than one configured gate is rejected before training; set config.auto_load=false for independent diagnostic evidence or compose one held-out suite.

Multiple diagnostic evidence rows are retained individually, while the training summary keeps the strongest fail-closed outcome. A later pass cannot hide an earlier error, regression, demotion, or inconclusive result.

A/B preferences → SFT → judge LoRA

The judgment workflow turns human A/B/Tie/Skip preferences into a local judge LoRA that a future suite can use as an llm_judge scorer. In /ui › Judgments, each choice is written to a judgment dataset. Compile those choices into SFT JSONL, train the judge under the default stable profile, and validate it against a held-out slice before use.

{
  "name": "code-quality-suite",
  "examples": [{
    "id": "sum-47-138",
    "messages": [{
      "role": "user",
      "content": "What is 47 + 138? Explain briefly."
    }],
    "target": "185"
  }],
  "default_scorer": {
    "kind": "llm_judge",
    "judge_adapter": "my-judge-v1",
    "template": "Question: {question}\nAnswer: {answer}\nScore 1 if the answer correctly solves the question, else 0.\nScore:"
  }
}

The eval worker loads the named judge through the same adapter scheduler used for inference and binds that adapter's attested content identity into the run.

kiln-eval CLI

kiln-eval wraps the HTTP API for scripts: list or register suites, run or compare targets, probe one prompt, and strict-replay a completed run. It is not included in the current prebuilt server archives. Build this client from source with the command below, or use the HTTP examples on this page. Dataset-to-suite synthesis remains HTTP-only at POST /v1/eval/datasets/<name>/synthesize. See the CLI Reference and complete Eval guide.

cargo build --release --locked \
  -p kiln-server --bin kiln-eval

./target/release/kiln-eval --help

./target/release/kiln-eval run \
  --suite trajectories-suite \
  --adapter my-coder-v3

./target/release/kiln-eval replay \
  --job eval_123 \
  --json