Start with a working request, then find the contract you need.

Use GET /v1/config to inspect the immutable startup configuration, effective backend policy, graph state, capacity resolution, and live memory snapshot. Each typed setting reports its resolved value, source, canonical environment name, redaction state, and restart requirement. Kiln keeps configuration intent separate from backend-selected execution policy and reports unsupported or unavailable measurements as null rather than inventing values.

curl -fsS http://localhost:8420/v1/config | python3 -m json.tool
curl -fsS http://localhost:8420/v1/config | jq '.effective_configuration'

See the Configuration Reference for common settings, the Complete Configuration Reference for every field and precedence rule, and the Observability API Schema for the exact response shape.

Attribute first-token latency

Set include_performance: true on a chat request to receive terminal metadata.performance. Batching-engine responses partition TTFT into actor_queue_ms (API enqueue to admission start), actor_admission_ms (slot preparation), and actor_prefill_wall_ms (admission completion to first sampled-token readiness), alongside accumulated model prefill_ms, decode_ms, TTFT, and total latency. resident_prefill_used is request-scoped route evidence: it is true only after a successful native multi-row resident-prefill forward included that request, false for batching requests that never entered the route, and null for direct paths. It is not inferred from process-global counters, and a mutation-free native decline remains false. Production currently advertises resident_prefill_enabled=false, so all batching requests must report resident_prefill_used=false. Non-batching paths return the actor fields as null. Both real-model streaming paths publish a performance object on the terminal chat chunk; explicit opt-in also emits a separate kiln.token_timing SSE object for each model token with a bounded path source, exact accepted token_id, plus producer-ready/delivered, handler-received, and response-queue times. The token ID remains observable when a special token decodes to an empty text fragment, without enabling logprobs or behavior capture. Direct streams retain the same request summary while leaving actor-only phases null and classifying otherwise unpartitioned gaps as unexplained. Batching-engine streams also propagate exact, invocation-owned gpu_lock_wait_ms and synchronization_ms candidates. ROCm sampled paged decode also reports its distinct post-transformer final-norm, LM-head, filtering, selection, and token-return tail as sampling_ms; behavior-logprob capture reports the same boundary across backends. Qualified ROCm W8 sampling splits the existing token-index transfer into readback_ms without adding work and subtracts it from sampling to prevent double-counting; other readback routes remain null until their owner exposes the exact boundary. Greedy or native fused-forward routes leave sampling null when no independent boundary exists. Qualified ROCm direct and batching graph routes also report request-owned graph_capture_ms and graph_replay_ms. One graph-runner lock encloses the before/call/after phase snapshot, capture time survives eager fallback, and replay launch excludes the separately reported external-yield synchronization. A shared batched invocation is attached only to the ready requests whose rows participated, so request totals are overlap evidence and may duplicate device work by at most observed decode width; lifetime graph telemetry remains the process-work total, and ROCm qualification enforces that conservation bound. Other-backend graph, resize/trim, adapter, and training phases remain null until their owning runtime returns correlated timings; unlocked process-global counter deltas are never presented as per-request evidence. Mixed-load, development-soak, and endurance qualification receipts retain each phase as a measured-window millisecond total plus an observation count, so unsupported null remains distinct from a measured zero. A successful measured request without terminal phase metadata fails qualification, and broad actor and narrower backend totals must be ranked within their layer rather than summed as a critical path. The complete phase, overlap, reason-set, nullability, rolling-statistics, and Prometheus contract is documented in Latency Observability.

The ROCm mixed-load qualification runs an isolated fixed-seed sampled profile before its greedy measurement window: eight concurrent requests, 32 tokens each, temperature 0.7, top-p 0.9, top-k 40, and min-p 0.0. Dedicated sampled_profile_* metrics retain aggregate completion-window throughput, median per-request throughput, broad decode and sampled-tail costs, nullable readback population, and batch-width evidence. The driver drains the actor and captures a new health baseline before ordinary load, so the profile cannot contaminate established mixed-load counters. A pass requires sampled attribution for every request, no separately claimed readback timing, and proven multi-row decode.

Greedy measured requests bind prompt identity variant_invariant_fixed_output_v5. They request exactly 64 ascending six-digit integers while stating that server truncation before the target is expected. At roughly seven model tokens per integer, the target exceeds every 32-, 128-, and 256-token measured cap without the output-limit refusals caused by million- and 1,024-item measured requests. The prompt requires immediate sequence output, forbids explanation, refusal, summaries, and output-limit discussion, and uses only nonnumeric itemNN padding that is explicitly unrelated to output length. The separate unvalidated stalled socket retains a 1,024-integer fill target for its 4,096-token cap. Receipts record the prompt identity, response_oracle_target_integer_count = 64, slow_response_target_integer_count = 1024, and long_prefill_marker_role = "long-prefill". The diagnostic and acceptance paths therefore tokenize the same named long request for exact A/B configuration identity.

Slow-consumer containment uses a 256-token pressure peer, twice the ordinary response length. It dispatches that peer before opening the stalled socket and waits for that peer's first producer-ready token. Acceptance then requires additional producer-ready tokens during and after the request-attributed backpressure window. The typed schedule records pressure_peer_dispatch = "before_slow_start_after_first_token", proving continuity of an already-decoding request instead of comparing a queued peer with a pressure interval that began first.

{
  "triggered": true,
  "trigger": "tokens",
  "closed": true,
  "thinking_tokens": 128,
  "thinking_time_ms": 742
}

This outcome appears on each non-streaming choices[].thinking_budget and completions[].thinking_budget. trigger is tokens, time, or max_tokens; a natural close reports triggered=false with closed=true. Chat and batch metadata.thinking_budget also report effective limits and each dimension's source: request, server_default, request_unlimited, or unlimited. Batch outcomes remain per completion rather than being aggregated at the root. In SSE streams, the chunk containing finish_reason carries chat metadata and the final outcome before the optional usage chunk and [DONE]. Cached responses preserve their original outcome while reporting provenance for the current request.

Each /v1/stats/recent-requests row includes the same effective pair in thinking_budget, plus applied and any final triggered, trigger, closed, thinking_tokens, and thinking_time_ms fields. Outcome fields are absent for inert or not-yet-resolved budgets instead of fabricating a natural close. The dashboard request drill renders the limits, independent sources, application state, and outcome.

Prometheus uses only closed label sets: kiln_thinking_budget_source_total counts each dimension's provenance, kiln_thinking_budget_outcomes_total counts unconfigured, inert, natural-close, forced-close, unclosed, interrupted, and unresolved outcomes, and kiln_thinking_budget_effective_tokens plus kiln_thinking_budget_effective_seconds are fixed-bucket histograms. Numeric request limits never become labels.

Native SFT renders TRL’s prefix-preserving Qwen3.5 training template with add_generation_prompt=false. Loss covers the thinking/answer/tool-call body, <|im_end|>, and trailing newline; assistant role headers plus system, user, and tool-response turns use the -100 ignore label. Exact rendered text, token IDs, and labels are checked against source-pinned Hugging Face and TRL sources for plain, thinking, tool-call, tool-response, and multi-turn examples. See SFT Tokenization and Loss for the normative contract and reproduction command.

Native SFT accepts one fixed profile: native_online_lora_v1. One conversation is one optimizer update at a constant learning rate; accumulation is 1, with no warmup, decay, or gradient clipping. Unknown profile names and general-trainer fields return structured training_invalid_request errors. Server SFT always normalizes omitted train_mtp to false and rejects explicit true before queue publication; the deferred MTP alignment phase remains offline-only until it participates in server GPU coordination, memory admission, cancellation, and settlement. A hybrid Vulkan server deliberately returns training_backend_unsupported before dataset admission: its serving weights use CPU-host handles, while the multi-GiB full-model resident Vulkan training path has not passed the production qualification gates. There is no environment bypass; use HF/TRL export/import until that substrate is proven. See the Native SFT Profile for update, precision, optimizer-state, checkpoint, and MTP semantics. Use the HF/TRL route for broader training: SFT and recorded-GRPO exports snapshot their canonical corpus, exact model/tokenizer/template provenance, optional split, and optional adapter into one private immutable registry, embed the task-aware pinned runner, and revalidate every byte before download. GRPO accepts exactly one provenance-complete inline group array or server-local canonical JSONL path. kiln train hf export-sft and export-grpo wrap creation, redirect-free streamed download, strict single-root archive validation, manifest verification, atomic no-clobber local publication, and default server cleanup; list and delete manage retained exports. kiln train hf import-peft verifies the completed local bundle before connecting, streams its deterministic corpus-free envelope through bounded memory, and accepts success only when HTTP 201, JSON, strong ETag, import digest, PEFT content revision, installed byte count, six-file count, task, and source identities match values derived locally; it retains the source bundle on every outcome. The raw PEFT import API accepts only the derived ten-file envelope, caps compressed/expanded/metadata resources, compares the complete current resident identity, validates every LoRA A/B shape, shares the adapter mutation barrier and strict disk quota, and stores a self-verifying kiln.hf-trl-import.v1 receipt before kernel-enforced no-replace publication. The versioned identity model, 256 MiB inline-body ceiling, 256-export registry cap, 2 GiB import-body ceiling, and lifecycle are documented in HF/TRL Interoperability.

config.optimizer is tagged by kind; omission selects Muon, and {"kind":"adam_w"} or {"kind":"muon"} selects that optimizer's defaults. Expanded forms may set beta1, beta2, eps, and weight_decay for AdamW or momentum, nesterov, ns_iters, and weight_decay for Muon. SGD accepts only {"kind":"sgd"}, and unknown fields are rejected. AdamW betas and Muon momentum must be finite in [0,1); epsilon must be finite and positive; both weight-decay values must be finite and non-negative; and Muon iterations must be in 1..=20. An explicit learning rate must remain finite and positive after F32 conversion. Omitted SFT rates resolve to 1e-3 for Muon and 1e-4 for AdamW/SGD; GRPO/OPD resolve 2e-3 and 1e-5, respectively. The cheap per-workload gate and resident optimizer tuple are checked before checkpoint or corpus materialization, at dequeue before memory reservation, and before device residency. This applies to direct training, the intent front door, every recipe step, judge/self-improve, DistillRefresh, and OPD-backed distillation. Cheap teacher-alias validation and metadata pinning may occur first; checkpoint loading, remote/local teacher materialization, corpus scanning, memory preflight, and GPU reservation occur only after the static workload check. Invalid optimizer kind, rank, or hyperparameters return structured training_invalid_request; unsupported base dtype, backend/device identity, Marlin-packed weights, serving-profile ownership, or authoritative tape/loss substrate return training_backend_unsupported. Neither changes route mid-run. A recipe descriptor is a static preview rather than a memory reservation; run submission preflights every step again before preparing any of them.

{
  "recipes": [{
    "name": "frontier-pump",
    "description": "Distill a frontier teacher into a local adapter",
    "num_steps": 1,
    "admission": {
      "supported": false,
      "unavailable_reason": "step 1 (opd) is unavailable: resident backend tuple does not support OPD training"
    }
  }]
}

description may be null. unavailable_reason is null exactly when the descriptor is supported; a false descriptor reports the first failing step. The descriptor is advisory process-lifetime admission, so clients must still handle a run-time rejection caused by changed lock health or live memory. Any descriptor containing a DistillRefresh step is currently unsupported with the stable two-phase planning reason above.

Every SFT source uses one row-admission contract. config.invalid_row_policy is fail by default; explicit skip queues only valid rows and records ordered kept/rejected SHA-256 identities under train_receipt.json → data.sft_ingestion. Server-local JSONL is revalidated against its submit-time manifest before GPU ownership. See SFT Ingestion and Row Identity.

New exact checkpoints, train_receipt.json, and adapter_manifest.json persist the full validated base-weight shard manifest and kiln.execution-provenance.v1 process/runtime envelope. Exact resume first passes the current workload and resident tuple gates, then compares shard bytes and the canonical execution digest before GPU ownership. It separately validates the backend/device, optimizer kind and state, rank, concrete parameter/activation/gradient dtypes, immutable rounding mode, and checkpoint/tape/loss routes. Server training records only {"mode":"round_to_nearest"}. A legacy checkpoint recording stochastic rounding fails closed on precision mismatch before GPU ownership; Kiln does not discard its seed or continue under a different update rule. A changed native identity, newly Marlin-packed weight, unavailable authoritative route, aggregate-only checkpoint, or partial-runtime checkpoint is likewise not exact-resumable.

Queued resume admission fully validates the checkpoint manifest plus every declared artifact size and SHA-256, then retains only the checkpoint ID, a digest of that validated manifest (whose entries cover the artifact hashes), and the effective seed. Before memory reservation at dequeue, the worker fully reloads the checkpoint and requires both the recomputed identity and effective seed to match. This is revalidation, not a filesystem snapshot: the queue does not copy or pin checkpoint files, and it cannot eliminate a mutation race after the reload. Keep an exact checkpoint directory immutable for its entire use.

Native SFT checkpoint-boundary replay is also startup-authoritative. [training].recompute_checkpoint_boundaries defaults to auto, whose inclusive sequence threshold defaults to 8,192 tokens. checkpoint_boundary_anchor_stride defaults to auto and derives one positive stride from the admitted shape and the default 6 GiB checkpoint_boundary_cache_gb target; explicit positive strides bypass only that shape calculation. Canonical environment names derive mechanically as KILN_TRAINING_<FIELD>; the four historical unsectioned names are deprecated, parsed strictly, and must agree with a present canonical value. The same pure policy functions drive admission and execution, and no runtime path re-reads those names. Exact SFT checkpoints additionally bind the admitted backend loss route as sft_loss_route under kiln.training-checkpoint-planning.v4; prior SFT v3 checkpoints fail closed as planning drift. GRPO and OPD continue to retain every segment boundary and remain under kiln.training-checkpoint-planning.v3. A v2 checkpoint or any applicable policy change is planning drift and cannot resume exactly; the outer checkpoint envelope remains schema v1. Restart the server to change startup policy. See the complete configuration reference and Native Training Checkpoints.

# Start exact SFT checkpoints every 25 committed steps
curl -fsS http://localhost:8420/v1/train/sft \
  -H 'content-type: application/json' \
  -d '{"dataset_path":"/data/corrections.jsonl","config":{"training_profile":"native_online_lora_v1","output_name":"support-bot","epochs":3,"checkpoint_interval":25}}'

# Continue with identical data/config and the basename reported by job detail
curl -fsS http://localhost:8420/v1/train/sft \
  -H 'content-type: application/json' \
  -d '{"dataset_path":"/data/corrections.jsonl","config":{"training_profile":"native_online_lora_v1","output_name":"support-bot","epochs":3,"checkpoint_interval":25,"resume_checkpoint":"support-bot-checkpoint-step-00000025.kiln-checkpoint"}}'

# The same exact contract applies to streamed GRPO optimizer groups
curl -fsS http://localhost:8420/v1/train/grpo \
  -H 'content-type: application/json' \
  -d '{"dataset_path":"/data/scored-groups.jsonl","config":{"output_name":"reward-bot","checkpoint_interval":25}}'

curl -fsS http://localhost:8420/v1/train/grpo \
  -H 'content-type: application/json' \
  -d '{"dataset_path":"/data/scored-groups.jsonl","config":{"output_name":"reward-bot","checkpoint_interval":25,"resume_checkpoint":"reward-bot-checkpoint-step-00000025.kiln-checkpoint"}}'

# OPD defaults to an exact checkpoint every 25 committed optimizer steps
curl -fsS http://localhost:8420/v1/train/opd \
  -H 'content-type: application/json' \
  -d '{"prompts":[{"messages":[{"role":"user","content":"Explain why the sky is blue."}]}],"teacher":"qwen35@vllm","config":{"output_name":"distilled-bot","checkpoint_interval":25}}'

# Resume with identical prompts/config and the exact registered teacher revision
curl -fsS http://localhost:8420/v1/train/opd \
  -H 'content-type: application/json' \
  -d '{"prompts":[{"messages":[{"role":"user","content":"Explain why the sky is blue."}]}],"teacher":"qwen35@vllm","config":{"output_name":"distilled-bot","checkpoint_interval":25,"resume_checkpoint":"distilled-bot-checkpoint-step-00000025.kiln-checkpoint"}}'

SFT, GRPO, and OPD checkpoints are immutable .kiln-checkpoint directories published directly beneath the adapter registry while training runs. Cancellation checkpoints at the next committed SFT step, GRPO group, or settled OPD candidate boundary. Resume requires the same output name, data bytes and route, effective configuration, precision, model/base weights, tokenizer, and backend; OPD additionally binds the exact teacher content revision and keeps its optimizer-step and candidate cursors separate. The server validates the complete adapter, optimizer, reference/EMA where applicable, loop-state artifact set, and checksums before continuation. Job detail reports training_kind, data_source_kind, the next epoch/group/candidate cursor, validated effective configuration and data identity, plus OPD teacher identity. PEFT snapshots remain serving-only. See Native Training Checkpoints for the exact contract.

Training writes every artifact beneath a hidden staging root and captures the target’s starting content revision. Publication compares that revision again: if another operation won, the job fails with adapter_revision_conflict and preserves the newer adapter. An ungated same-name adapter that is already loaded is reloaded atomically with the directory replacement, even when auto_load=false. A gated post_eval.min_accuracy rewrite of a physically loaded same-name target is rejected before GPU work; unload it or choose a versioned config.output_name.

A GRPO run that reaches the training loop records grpo.policy_audit in the adapter receipt returned by GET /v1/adapters/{name}/receipt. The versioned kiln.grpo-policy-audit.v1 object reports behavior-policy importance ratios, independently referenced K1/K3 metrics, entropy-mask coverage, clipping tails, and content-addressed rollout-source identities. No-correction runs report unit ratios; they never substitute the KL reference for missing behavior probabilities. Token PPO and sequence GSPO use clip_epsilon/clip_eps_high; CISPO uses the separate absolute upper-only cispo_max_weight cap and has no lower clip count. See the GRPO audit guide for denominator and count semantics.