Concept
What GRPO does in Kiln
Group Relative Policy Optimization (GRPO) fits tasks where you can
score model outputs more reliably than you can write one ideal answer.
Each group contains the original chat-format messages and
several completions, each with generated text
and a finite numeric reward.
Kiln derives relative advantages within each group, applies the selected
GRPO-family loss to LoRA parameters, and atomically publishes the completed
adapter. The request enters a background queue, but GPU training takes exclusive
accelerator ownership; inference resumes only after that ownership is released.
The default profile runs the complete loop
The default stable profile supports generation, training,
evaluation, and coordinated adapter-weight transitions in one process.
maintenance admits drained mutation but disables generation and
evaluation. experimental is reserved for backend development and
is not needed for speed or training. See
Serving profiles.
Before you run the commands
Start Kiln normally with real Qwen/Qwen3.5-4B weights. The CLI
examples assume that the extracted kiln binary is on your
PATH; otherwise replace kiln with its local path.
Generate and score data from a trusted client because Kiln does not
authenticate its HTTP API.
Rollouts
Generate multiple completions per prompt
Use kiln rollout-generate when training with recorded importance
correction. It requests one seeded, non-streaming choice at a time,
validates the server-issued token and behavior-policy provenance before
scoring, and atomically publishes trainer-ready GRPO JSONL only after every
rollout passes.
Provenance-bound rollout dataset
kiln rollout-generate \
--adapter base \
--thinking false \
--tasks tasks.jsonl \
--seeds 8 \
--seed-start 42 \
--request-template request.json \
--scorer ./score_math.py \
--output math.rollouts.jsonl \
--summary-output math.rollouts.summary.json
Submit the resulting dataset with config.behavior_policy: "recorded". The trainer revalidates the exact prompt, sampled actions, tokenizer/template identity, behavior model revision, sampling controls, and selected-token log-probabilities before the first forward.
Text-only batch alternative
curl -fsS http://localhost:8420/v1/completions/batch \
-H "Content-Type: application/json" \
-d '{"prompts": [[{"role": "user", "content": "What is 47 + 138?"}]],
"n": 8, "temperature": 0.9, "max_tokens": 64, "seed": 42}' \
| python3 -m json.tool
This endpoint does not emit per-token behavior probabilities or a
trainer-ready scored-group file. Score its completions, reshape them into
groups, and train only with
behavior_policy: "no_importance_correction". Kiln will not
substitute the KL reference for the missing behavior denominator.
Training
Submit scored completions to /v1/train/grpo
The minimal request is {"groups": [...]}; every
config field has a server default. Set
output_name for a predictable artifact name. Although
auto_load defaults to true, the safer
train-then-evaluate workflow sets it to false and activates the
adapter only after held-out comparison.
First GRPO training request
curl -fsS http://localhost:8420/v1/train/grpo \
-H "Content-Type: application/json" \
-d '{
"groups": [{
"messages": [{"role": "user", "content": "What is 47 + 138? Reply with just the number."}],
"completions": [
{"text": "185", "reward": 1.0},
{"text": "The answer is 184", "reward": 0.0},
{"text": "185.", "reward": 1.0},
{"text": "47 + 138 = 185", "reward": 0.8}
]
}],
"config": {
"behavior_policy": "no_importance_correction",
"kl_coeff": 0.1,
"clip_epsilon": 0.2,
"lora_rank": 16,
"output_name": "math-correctness",
"auto_load": false
}
}' \
| python3 -m json.tool
Common config fields
kl_coeff tunes the independent frozen-reference penalty; learning_rate is resolved per optimizer when omitted (Muon default: 2e-3).is_level selects token PPO (default), sequence GSPO, or CISPO. PPO/GSPO use clip_epsilon plus optional clip_eps_high.- CISPO instead uses the absolute upper-only
cispo_max_weight cap (default 5.0); it has no PPO lower floor.
behavior_policy selects recorded rollout probabilities or an explicit fixed-one ratio; kl_reference_policy independently selects the frozen KL anchor.lora_rank and lora_alpha control adapter capacity.base_adapter continues from an existing adapter.seed selects the run seed. When omitted, Kiln assigns one before queueing the job and returns it as an exact decimal string.output_name names the saved adapter.auto_load defaults to true; set it to false when evaluation should gate activation.checkpoint_interval writes exact state every positive N committed optimizer groups.resume_checkpoint continues from the immutable basename reported by job detail.adapter_smoke_test enables the post-training canary; adapter_smoke_prompts supplies request-local replacement prompts.shared_prefix_reference defaults to true and reuses qualified prompt-side reference state. Disable it only for an explicit comparison.detect_anomaly is an off-by-default, request-local backward diagnostic. It scans each operation's returned gradients and fails at the first NaN or Inf; use kiln train grpo --detect-anomaly to enable it.
Payload guardrails
- Put options under
config, not at the top level.
- Use
groups, messages, completions, text, and reward.
- GRPO does not use SFT's
epochs field.
- Use at least two completions with different rewards; equal rewards produce zero normalized advantage.
Checkpoint and resume exact GRPO state
kiln train grpo \
--file scored-groups.jsonl \
--adapter math-correctness \
--checkpoint-interval 25
kiln train grpo \
--file scored-groups.jsonl \
--adapter math-correctness \
--checkpoint-interval 25 \
--resume-checkpoint math-correctness-checkpoint-step-00000025.kiln-checkpoint
A .jsonl file uses the memory-bounded streamed route; JSON containing groups uses the inline route. Resume requires identical source bytes, route, adapter, and effective configuration. Before GPU setup, Kiln validates and restores adapter, optimizer, reference or EMA, cursor and RNG, loss, and diagnostic state. Cancellation settles at the next group boundary. A crash can lose the in-flight group, but not the newest committed checkpoint. Dashboard job details can copy the checkpoint basename or prepare the matching form. See Native training checkpoints for the fail-closed contract.
Submission and status report the immutable effective_seed as a decimal string. Exact resume inherits the checkpoint's original LoRA-initialization seed and rejects a conflicting request. Fresh runs that reuse a seed are comparable only when weights, data, tokenizer, build, backend, runtime, and precision also match.
Monitoring
Watch the background training job
/v1/train/grpo returns a queued job immediately. Poll /v1/train/status to see pending, running, completed, or failed jobs. GET /v1/train/jobs/{job_id} reports the latest exact checkpoint basename, inline/JSONL route, and next group cursor; then confirm the trained adapter through the adapter APIs or the web UI.
If the kiln CLI is on your PATH, run kiln train status for the same status as a readable summary (use --url http://host:8420 for a trusted remote server). The curl command below is the equivalent HTTP probe for CI, scripts, or environments without the CLI.
Training status from the CLI
kiln train status
Training status
curl -fsS http://localhost:8420/v1/train/status | python3 -m json.tool
After a job is completed, confirm that Kiln saved the new
adapter. kiln adapters list shows every saved adapter and
identifies the active one. With the recommended
auto_load: false, math-correctness should be
present but inactive. The curl command below calls the same
endpoint for CI or scripts.
Confirm the trained adapter from the CLI
kiln adapters list
Confirm the trained adapter
curl -fsS http://localhost:8420/v1/adapters | python3 -m json.tool
Manual review for an ungated artifact
Compare math-correctness with the no-adapter baseline on a
held-out suite in the Evals guide. A
manual comparison does not create the atomic promotion gate above. If an
operator nevertheless approves activation, recheck the adapter detail and
receipt immediately before loading it by name, then record the
content_revision returned by this request:
curl -fsS http://localhost:8420/v1/adapters/load \
-H "Content-Type: application/json" \
-d '{"name": "math-correctness"}' \
| python3 -m json.tool
This is an operator-controlled action under the default
stable profile, not evidence-bound automatic promotion.
If a GRPO request is rejected or stays queued longer than expected, start with the troubleshooting guide and the full endpoint map in API training.
Audit
Verify which policy comparisons trained the adapter
Every non-dry run that reaches the training loop writes grpo.policy_audit into train_receipt.json. The same versioned object is returned by the adapter receipt API and reports behavior-policy importance ratios separately from frozen-reference KL metrics.
Read the policy audit
# This filtered view requires jq.
ADAPTER=math-correctness
curl -fsS http://localhost:8420/v1/adapters/$ADAPTER/receipt \
| jq '.grpo.policy_audit'
importance_sampling reports ratios against recorded behavior probabilities, or exact unit ratios in no-correction mode. CISPO’s below-clip count is always zero; its upper count is measured against cispo_max_weight.kl_reference reports K1/K3 observations against the independently selected reference, including entropy-mask coverage.recorded_provenance binds counts to content-addressed behavior model, adapter revision, tokenizer/template, sampling, and generation-backend identities.
Counts are observations across the actual training loop, so repeated epochs count a completion each time it is trained. See the full guide for sequence-versus-token ratio and masked-KL denominator semantics.