Eval and judgment API schema — Kiln Documentation

JSON Schema for eval suites, jobs and results; uploaded datasets and deterministic synthesis; and pairwise judgments. Each request shape states whether unrecognized fields are ignored. Server-emitted objects are closed unless a definition explicitly says otherwise.

Schema identity

Property	Value
Title	Kiln Eval, Dataset Synthesis, and Judgment API
`$id`	`https://ericflo.github.io/kiln/contracts/kiln-evals-v1.schema.json`
Dialect	`https://json-schema.org/draft/2020-12/schema`
Root type	Union of 33 public entrypoints
Root object	Defined by the selected entrypoint

Entrypoints

Choose the request or response shape for the operation you are implementing. Each name links to its complete field table and constraints.

Entrypoint	Purpose	Shape	Unknown fields
`AppendJudgmentBody`	Store one human preference over two exact responses to the same prompt.	9 fields	Unknown fields accepted and ignored
`AppendJudgmentResponse`	Assigned row ID flattened with the updated judgment manifest.	9 fields	Unknown fields rejected
`CancelEvalJobResponse`	Queued cancellation, cooperative running cancellation, or terminal-record deletion result.	0 fields	Varies by composed variant
`CompileJudgmentBody`	Compile judgment rows to a swap-augmented SFT dataset with a holdout.	3 fields	Unknown fields accepted and ignored
`CompileJudgmentResponse`	The swap-augmented SFT output, reserved holdout count, source-row boundary, dataset manifest, and warnings.	6 fields	Unknown fields rejected
`CreateJudgmentBody`	Create an append-only judgment dataset.	2 fields	Unknown fields accepted and ignored
`DatasetListResponse`	All uploaded eval/training datasets.	1 field	Unknown fields rejected
`DatasetManifest`	Persisted content-addressed dataset identity and partition summary.	16 fields	Unknown fields rejected
`DatasetSplitConfig`	Deterministic group-aware split policy. Train plus validation must be below 100.	3 fields	Unknown fields rejected
`DatasetSplitManifest`	Persisted deterministic split manifest; normalized duplicates and connected group/session rows share one partition.	7 fields	Unknown fields rejected
`DatasetUploadMultipart`	Multipart dataset upload. Unrecognized multipart fields are read and ignored.	4 fields	Unknown fields accepted and ignored
`DeleteDatasetResponse`	Confirmation that a dataset was deleted.	2 fields	Unknown fields rejected
`DeleteJudgmentResponse`	Confirmation that a judgment dataset was deleted.	2 fields	Unknown fields rejected
`DeleteSuiteResponse`	Confirmation that a suite was deleted.	2 fields	Unknown fields rejected
`EvalCompareSpec`	Run one registered suite against one to eight ordered adapters. Empty adapter names select the base model.	4 fields	Unknown fields accepted and ignored
`EvalJobListResponse`	All retained eval jobs in descending submission order.	1 field	Unknown fields rejected
`EvalResult`	Top-level retained result for a single- or multi-adapter eval job.	12 fields	Unknown fields rejected
`EvalRunRequest`	Submit exactly one registered or inline suite for evaluation.	5 fields	Unknown fields accepted and ignored
`EvalRunResponse`	Immutable admission receipt for a queued eval job.	4 fields	Unknown fields rejected
`EvalSuite`	A named set of examples with shared scoring, generation, aggregation, prompt, and tool settings. Omitted settings use documented defaults.	9 fields	Unknown fields accepted and ignored
`JudgmentListResponse`	All retained pairwise-judgment datasets.	1 field	Unknown fields rejected
`JudgmentManifest`	Persisted pairwise-judgment dataset identity and compile split.	8 fields	Unknown fields rejected
`PromoteJudgmentBody`	Queue held-out validation for a trained judge adapter.	2 fields	Unknown fields accepted and ignored
`RenderJudgmentPromptResponse`	Exact pairwise prompt used by both judge training and inference.	1 field	Unknown fields rejected
`ReplayBody`	Select one retained run for strict identity-checked byte replay.	1 field	Unknown fields rejected
`RerunBody`	Select outcomes and adapter identity for a failure-focused re-run.	4 fields	Unknown fields accepted and ignored
`SuiteListResponse`	All registered evaluation suites.	1 field	Unknown fields rejected
`SuiteSaveResponse`	Suite persistence confirmation.	3 fields	Unknown fields rejected
`SynthesisPreview`	Non-persisted synthesis preview including its persisted source partition and completion reduction.	7 fields	Unknown fields rejected
`SynthesisPreviewBody`	Preview a suite synthesized from one persisted partition without saving it. Source defaults to holdout.	12 fields	Unknown fields accepted and ignored
`SynthesizeBody`	Persist a suite synthesized from one partition and optionally queue it against adapters. Source defaults to holdout.	13 fields	Unknown fields accepted and ignored
`SynthesizeDatasetResponse`	Persisted suite summary plus any auto-queued eval job IDs.	3 fields	Unknown fields rejected
`ValidateJudgmentResponse`	Held-out judge-validation admission receipt.	5 fields	Unknown fields rejected

Composition and conditional rules

Show exact composition rules

{
  "oneOf": [
    {
      "$ref": "#/$defs/AppendJudgmentBody"
    },
    {
      "$ref": "#/$defs/AppendJudgmentResponse"
    },
    {
      "$ref": "#/$defs/CancelEvalJobResponse"
    },
    {
      "$ref": "#/$defs/CompileJudgmentBody"
    },
    {
      "$ref": "#/$defs/CompileJudgmentResponse"
    },
    {
      "$ref": "#/$defs/CreateJudgmentBody"
    },
    {
      "$ref": "#/$defs/DatasetListResponse"
    },
    {
      "$ref": "#/$defs/DatasetManifest"
    },
    {
      "$ref": "#/$defs/DatasetSplitConfig"
    },
    {
      "$ref": "#/$defs/DatasetSplitManifest"
    },
    {
      "$ref": "#/$defs/DatasetUploadMultipart"
    },
    {
      "$ref": "#/$defs/DeleteDatasetResponse"
    },
    {
      "$ref": "#/$defs/DeleteJudgmentResponse"
    },
    {
      "$ref": "#/$defs/DeleteSuiteResponse"
    },
    {
      "$ref": "#/$defs/EvalCompareSpec"
    },
    {
      "$ref": "#/$defs/EvalJobListResponse"
    },
    {
      "$ref": "#/$defs/EvalResult"
    },
    {
      "$ref": "#/$defs/EvalRunRequest"
    },
    {
      "$ref": "#/$defs/EvalRunResponse"
    },
    {
      "$ref": "#/$defs/EvalSuite"
    },
    {
      "$ref": "#/$defs/JudgmentListResponse"
    },
    {
      "$ref": "#/$defs/JudgmentManifest"
    },
    {
      "$ref": "#/$defs/PromoteJudgmentBody"
    },
    {
      "$ref": "#/$defs/RenderJudgmentPromptResponse"
    },
    {
      "$ref": "#/$defs/ReplayBody"
    },
    {
      "$ref": "#/$defs/RerunBody"
    },
    {
      "$ref": "#/$defs/SuiteListResponse"
    },
    {
      "$ref": "#/$defs/SuiteSaveResponse"
    },
    {
      "$ref": "#/$defs/SynthesisPreview"
    },
    {
      "$ref": "#/$defs/SynthesisPreviewBody"
    },
    {
      "$ref": "#/$defs/SynthesizeBody"
    },
    {
      "$ref": "#/$defs/SynthesizeDatasetResponse"
    },
    {
      "$ref": "#/$defs/ValidateJudgmentResponse"
    }
  ]
}

Kiln contract annotations

Show exact Kiln annotations and examples

{
  "x-kiln-entrypoints": [
    "AppendJudgmentBody",
    "AppendJudgmentResponse",
    "CancelEvalJobResponse",
    "CompileJudgmentBody",
    "CompileJudgmentResponse",
    "CreateJudgmentBody",
    "DatasetListResponse",
    "DatasetManifest",
    "DatasetSplitConfig",
    "DatasetSplitManifest",
    "DatasetUploadMultipart",
    "DeleteDatasetResponse",
    "DeleteJudgmentResponse",
    "DeleteSuiteResponse",
    "EvalCompareSpec",
    "EvalJobListResponse",
    "EvalResult",
    "EvalRunRequest",
    "EvalRunResponse",
    "EvalSuite",
    "JudgmentListResponse",
    "JudgmentManifest",
    "PromoteJudgmentBody",
    "RenderJudgmentPromptResponse",
    "ReplayBody",
    "RerunBody",
    "SuiteListResponse",
    "SuiteSaveResponse",
    "SynthesisPreview",
    "SynthesisPreviewBody",
    "SynthesizeBody",
    "SynthesizeDatasetResponse",
    "ValidateJudgmentResponse"
  ],
  "x-kiln-examples": {
    "AppendJudgmentBody": [
      {
        "prompt": [
          {
            "role": "user",
            "content": "Explain the answer."
          }
        ],
        "adapter_a": null,
        "adapter_b": "concise-v1",
        "response_a": "A long answer",
        "response_b": "A concise answer",
        "winner": "b",
        "tags": [
          "style"
        ]
      }
    ],
    "AppendJudgmentResponse": [
      {
        "judgment_id": "judgment-1",
        "name": "style-prefs",
        "description": "Concise-answer preferences",
        "num_rows": 1,
        "created_at": "2026-07-14T12:00:00Z",
        "updated_at": "2026-07-14T12:01:00Z",
        "winner_histogram": {
          "a": 1
        }
      }
    ],
    "CancelEvalJobResponse": [
      {
        "status": "cancelled",
        "job_id": "eval-1",
        "was_in_queue": true
      },
      {
        "status": "cancelling",
        "job_id": "eval-2",
        "note": "running job will exit at the next example boundary"
      },
      {
        "status": "deleted",
        "job_id": "eval-3",
        "removed_archive_file": true
      }
    ],
    "CompileJudgmentBody": [
      {
        "output_dataset": "style-prefs-sft",
        "include_skips": false,
        "holdout_n": 1
      }
    ],
    "CompileJudgmentResponse": [
      {
        "status": "compiled",
        "rows": 2,
        "holdout_n": 1,
        "train_validation_split": 1,
        "dataset": {
          "schema_version": 2,
          "name": "math-sft",
          "format": "sft_chat",
          "description": "Arithmetic conversations",
          "num_rows": 1,
          "size_bytes": 64,
          "created_at": "2026-07-14T12:00:00Z",
          "updated_at": "2026-07-14T12:00:00Z",
          "corpus_sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
          "normalized_corpus_sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
          "split_manifest_sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333",
          "split_config": {
            "seed": "0",
            "train_percent": 80,
            "validation_percent": 10
          },
          "split_counts": {
            "train": 1,
            "validation": 0,
            "holdout": 0
          },
          "num_groups": 0,
          "num_sessions": 0,
          "stats": {
            "num_assistant_turns": 1,
            "num_tool_messages": 0,
            "num_with_tool_calls": 0,
            "max_messages_per_conv": 2,
            "max_content_chars": 8,
            "avg_messages_per_conv": 2,
            "sample_role_patterns": [
              "user>assistant"
            ]
          }
        },
        "warnings": []
      }
    ],
    "CreateJudgmentBody": [
      {
        "name": "style-prefs",
        "description": "Concise-answer preferences"
      }
    ],
    "DatasetListResponse": [
      {
        "datasets": [
          {
            "schema_version": 2,
            "name": "math-sft",
            "format": "sft_chat",
            "description": "Arithmetic conversations",
            "num_rows": 1,
            "size_bytes": 64,
            "created_at": "2026-07-14T12:00:00Z",
            "updated_at": "2026-07-14T12:00:00Z",
            "corpus_sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
            "normalized_corpus_sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
            "split_manifest_sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333",
            "split_config": {
              "seed": "0",
              "train_percent": 80,
              "validation_percent": 10
            },
            "split_counts": {
              "train": 1,
              "validation": 0,
              "holdout": 0
            },
            "num_groups": 0,
            "num_sessions": 0,
            "stats": {
              "num_assistant_turns": 1,
              "num_tool_messages": 0,
              "num_with_tool_calls": 0,
              "max_messages_per_conv": 2,
              "max_content_chars": 8,
              "avg_messages_per_conv": 2,
              "sample_role_patterns": [
                "user>assistant"
              ]
            }
          }
        ]
      }
    ],
    "DatasetManifest": [
      {
        "schema_version": 2,
        "name": "math-sft",
        "format": "sft_chat",
        "description": "Arithmetic conversations",
        "num_rows": 1,
        "size_bytes": 64,
        "created_at": "2026-07-14T12:00:00Z",
        "updated_at": "2026-07-14T12:00:00Z",
        "corpus_sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
        "normalized_corpus_sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
        "split_manifest_sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333",
        "split_config": {
          "seed": "0",
          "train_percent": 80,
          "validation_percent": 10
        },
        "split_counts": {
          "train": 1,
          "validation": 0,
          "holdout": 0
        },
        "num_groups": 0,
        "num_sessions": 0,
        "stats": {
          "num_assistant_turns": 1,
          "num_tool_messages": 0,
          "num_with_tool_calls": 0,
          "max_messages_per_conv": 2,
          "max_content_chars": 8,
          "avg_messages_per_conv": 2,
          "sample_role_patterns": [
            "user>assistant"
          ]
        }
      }
    ],
    "DatasetSplitConfig": [
      {
        "seed": "0",
        "train_percent": 80,
        "validation_percent": 10
      }
    ],
    "DatasetSplitManifest": [
      {
        "schema": "kiln.dataset-split.v1",
        "dataset_name": "math-sft",
        "corpus_sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
        "normalized_corpus_sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
        "config": {
          "seed": "0",
          "train_percent": 80,
          "validation_percent": 10
        },
        "counts": {
          "train": 1,
          "validation": 0,
          "holdout": 0
        },
        "rows": [
          {
            "row_number": 1,
            "content_sha256": "sha256:4444444444444444444444444444444444444444444444444444444444444444",
            "normalized_sha256": "sha256:5555555555555555555555555555555555555555555555555555555555555555",
            "split": "train"
          }
        ]
      }
    ],
    "DatasetUploadMultipart": [
      {
        "name": "math-sft",
        "format": "sft_chat",
        "description": "Arithmetic",
        "file": "<binary JSONL body>"
      }
    ],
    "DeleteDatasetResponse": [
      {
        "status": "deleted",
        "name": "math-sft"
      }
    ],
    "DeleteJudgmentResponse": [
      {
        "status": "deleted",
        "name": "style-prefs"
      }
    ],
    "DeleteSuiteResponse": [
      {
        "status": "deleted",
        "name": "math-smoke"
      }
    ],
    "EvalCompareSpec": [
      {
        "suite": "math-smoke",
        "adapters": [
          "",
          "math-v1"
        ],
        "seed": 42
      }
    ],
    "EvalJobListResponse": [
      {
        "jobs": [
          {
            "schema_version": 2,
            "job_id": "eval-1",
            "suite_name": "math-smoke",
            "adapters": [
              null
            ],
            "submission_kind": "on_demand",
            "effective_seed": "42",
            "state": "completed",
            "progress": {
              "examples_completed": 1,
              "examples_total": 1,
              "running_accuracy": 1,
              "running_mean_score": 1
            },
            "finished_runs": [
              {
                "suite_name": "math-smoke",
                "adapter": null,
                "aggregation": {
                  "kind": "single"
                },
                "metrics": {
                  "num_examples": 1,
                  "num_completions": 1,
                  "num_pass": 1,
                  "num_fail": 0,
                  "num_invalid": 0,
                  "num_error": 0,
                  "accuracy": 1,
                  "accuracy_confidence_interval": {
                    "confidence_level": 0.95,
                    "lower": 0.2,
                    "upper": 1
                  },
                  "mean_score": 1,
                  "weighted_mean_score": 1,
                  "latency": {
                    "p50_ms": 10,
                    "p90_ms": 10,
                    "p99_ms": 10,
                    "mean_ms": 10,
                    "max_ms": 10
                  },
                  "total_prompt_tokens": 8,
                  "total_completion_tokens": 1,
                  "elapsed_secs": 0.01,
                  "pass_rate_by_tag": {
                    "math": 1
                  },
                  "tag_breakdown": {
                    "math": {
                      "num_examples": 1,
                      "num_pass": 1,
                      "pass_rate": 1,
                      "confidence_interval": {
                        "confidence_level": 0.95,
                        "lower": 0.2,
                        "upper": 1
                      }
                    }
                  },
                  "by_scorer": [
                    {
                      "scorer_kind": "exact_match",
                      "num_examples": 1,
                      "mean_score": 1,
                      "pass_rate": 1
                    }
                  ],
                  "reasoning_length": {
                    "num_with_thinking": 0,
                    "mean_chars": 0,
                    "p50_chars": 0,
                    "p90_chars": 0,
                    "max_chars": 0
                  },
                  "num_unclosed_thinking": 0,
                  "num_non_xml_tool_calls": 0,
                  "num_schema_missing_required": 0,
                  "num_schema_extra_unknown": 0
                },
                "aggregated_outcomes": [
                  {
                    "example_id": "two-plus-two",
                    "kind": "pass",
                    "score": 1,
                    "completion_indices": [
                      0
                    ],
                    "representative_completion_index": 0,
                    "num_pass": 1,
                    "num_fail": 0,
                    "num_invalid": 0,
                    "num_error": 0,
                    "tags": [
                      "math"
                    ]
                  }
                ],
                "outcomes": [
                  {
                    "example_id": "two-plus-two",
                    "completion_index": 0,
                    "generation_seed": "43",
                    "completion_text": "4",
                    "kind": "pass",
                    "score": 1,
                    "tags": [
                      "math"
                    ]
                  }
                ],
                "started_at": "2026-07-14T12:00:00Z",
                "finished_at": "2026-07-14T12:00:01Z",
                "suite_hash": "sha256:6666666666666666666666666666666666666666666666666666666666666666"
              }
            ],
            "headline_accuracy": 1,
            "error": null,
            "source_training_job_id": null,
            "submitted_at_iso": "2026-07-14T12:00:00Z",
            "started_at_iso": "2026-07-14T12:00:00Z",
            "finished_at_iso": "2026-07-14T12:00:01Z"
          }
        ]
      }
    ],
    "EvalResult": [
      {
        "schema_version": 2,
        "job_id": "eval-1",
        "state": "completed",
        "effective_seed": "42",
        "seed_derivation": "kiln.eval-seed.v1",
        "runs": [
          {
            "suite_name": "math-smoke",
            "adapter": null,
            "aggregation": {
              "kind": "single"
            },
            "metrics": {
              "num_examples": 1,
              "num_completions": 1,
              "num_pass": 1,
              "num_fail": 0,
              "num_invalid": 0,
              "num_error": 0,
              "accuracy": 1,
              "accuracy_confidence_interval": {
                "confidence_level": 0.95,
                "lower": 0.2,
                "upper": 1
              },
              "mean_score": 1,
              "weighted_mean_score": 1,
              "latency": {
                "p50_ms": 10,
                "p90_ms": 10,
                "p99_ms": 10,
                "mean_ms": 10,
                "max_ms": 10
              },
              "total_prompt_tokens": 8,
              "total_completion_tokens": 1,
              "elapsed_secs": 0.01,
              "pass_rate_by_tag": {
                "math": 1
              },
              "tag_breakdown": {
                "math": {
                  "num_examples": 1,
                  "num_pass": 1,
                  "pass_rate": 1,
                  "confidence_interval": {
                    "confidence_level": 0.95,
                    "lower": 0.2,
                    "upper": 1
                  }
                }
              },
              "by_scorer": [
                {
                  "scorer_kind": "exact_match",
                  "num_examples": 1,
                  "mean_score": 1,
                  "pass_rate": 1
                }
              ],
              "reasoning_length": {
                "num_with_thinking": 0,
                "mean_chars": 0,
                "p50_chars": 0,
                "p90_chars": 0,
                "max_chars": 0
              },
              "num_unclosed_thinking": 0,
              "num_non_xml_tool_calls": 0,
              "num_schema_missing_required": 0,
              "num_schema_extra_unknown": 0
            },
            "aggregated_outcomes": [
              {
                "example_id": "two-plus-two",
                "kind": "pass",
                "score": 1,
                "completion_indices": [
                  0
                ],
                "representative_completion_index": 0,
                "num_pass": 1,
                "num_fail": 0,
                "num_invalid": 0,
                "num_error": 0,
                "tags": [
                  "math"
                ]
              }
            ],
            "outcomes": [
              {
                "example_id": "two-plus-two",
                "completion_index": 0,
                "generation_seed": "43",
                "completion_text": "4",
                "kind": "pass",
                "score": 1,
                "tags": [
                  "math"
                ]
              }
            ],
            "started_at": "2026-07-14T12:00:00Z",
            "finished_at": "2026-07-14T12:00:01Z",
            "suite_hash": "sha256:6666666666666666666666666666666666666666666666666666666666666666"
          }
        ]
      }
    ],
    "EvalRunRequest": [
      {
        "suite": "math-smoke",
        "adapter": "math-v1",
        "seed": 42
      }
    ],
    "EvalRunResponse": [
      {
        "job_id": "eval-1",
        "state": "queued",
        "effective_seed": "42",
        "message": "Queued eval against suite `math-smoke`"
      }
    ],
    "EvalSuite": [
      {
        "name": "math-smoke",
        "description": "One deterministic arithmetic case",
        "default_scorer": {
          "kind": "exact_match",
          "case_sensitive": false,
          "strip_whitespace": true
        },
        "generation": {
          "temperature": 0,
          "top_p": 1,
          "top_k": 0,
          "max_tokens": 32,
          "n": 1,
          "stop": [],
          "seed": null
        },
        "aggregation": {
          "kind": "single"
        },
        "examples": [
          {
            "id": "two-plus-two",
            "messages": [
              {
                "role": "user",
                "content": "2 + 2?"
              }
            ],
            "target": "4",
            "weight": 1
          }
        ],
        "schema_version": 2
      }
    ],
    "JudgmentListResponse": [
      {
        "judgments": [
          {
            "name": "style-prefs",
            "description": "Concise-answer preferences",
            "num_rows": 1,
            "created_at": "2026-07-14T12:00:00Z",
            "updated_at": "2026-07-14T12:01:00Z",
            "winner_histogram": {
              "a": 1
            }
          }
        ]
      }
    ],
    "JudgmentManifest": [
      {
        "name": "style-prefs",
        "description": "Concise-answer preferences",
        "num_rows": 1,
        "created_at": "2026-07-14T12:00:00Z",
        "updated_at": "2026-07-14T12:01:00Z",
        "winner_histogram": {
          "a": 1
        }
      }
    ],
    "PromoteJudgmentBody": [
      {
        "adapter": "judge-v1",
        "holdout_n": 20
      }
    ],
    "RenderJudgmentPromptResponse": [
      {
        "prompt": "Compare the following two assistant replies..."
      }
    ],
    "ReplayBody": [
      {
        "run_index": 0
      }
    ],
    "RerunBody": [
      {
        "adapter": "math-v2",
        "outcome_kinds": [
          "fail",
          "invalid",
          "error"
        ],
        "include_pass": false,
        "seed": 42
      }
    ],
    "SuiteListResponse": [
      {
        "suites": [
          {
            "name": "math-smoke",
            "description": "One deterministic arithmetic case",
            "num_examples": 1,
            "completions_per_example": 1,
            "aggregation": {
              "kind": "single"
            },
            "schema_version": 2,
            "default_scorer_kind": "exact_match",
            "tags": {
              "math": 1
            }
          }
        ]
      }
    ],
    "SuiteSaveResponse": [
      {
        "name": "math-smoke",
        "path": "/srv/kiln/.eval/suites/math-smoke/suite.json",
        "status": "created"
      }
    ],
    "SynthesisPreview": [
      {
        "examples": [
          {
            "id": "two-plus-two",
            "messages": [
              {
                "role": "user",
                "content": "2 + 2?"
              }
            ],
            "target": "4",
            "weight": 1
          }
        ],
        "stats": {
          "trajectories_seen": 1,
          "trajectories_used": 1,
          "examples_generated": 1,
          "skipped_no_target": 0,
          "skipped_prompt_too_long": 0,
          "skipped_target_too_long": 0,
          "skipped_duplicate": 0,
          "skipped_strategy_match": 0,
          "sample_kept": 1,
          "effective_seed": "42",
          "auto_scorer_histogram": {
            "exact_match": 1
          }
        },
        "suite_name": "math-smoke",
        "default_scorer_kind": "exact_match",
        "aggregation": {
          "kind": "single"
        },
        "completions_per_example": 1,
        "source_split": "holdout"
      }
    ],
    "SynthesisPreviewBody": [
      {
        "suite_name": "math-smoke",
        "strategy": "final_assistant",
        "scorer": {
          "kind": "fixed",
          "scorer": {
            "kind": "exact_match"
          }
        },
        "aggregation": {
          "kind": "single"
        },
        "sampling": {
          "max_examples": 100,
          "seed": 42
        },
        "source_split": "holdout",
        "head_n": 5
      }
    ],
    "SynthesizeBody": [
      {
        "suite_name": "math-smoke",
        "strategy": "final_assistant",
        "scorer": {
          "kind": "fixed",
          "scorer": {
            "kind": "exact_match"
          }
        },
        "aggregation": {
          "kind": "single"
        },
        "sampling": {
          "max_examples": 100,
          "seed": 42
        },
        "source_split": "holdout",
        "force": true,
        "run_against": [
          ""
        ]
      }
    ],
    "SynthesizeDatasetResponse": [
      {
        "suite": {
          "name": "math-smoke",
          "description": "One deterministic arithmetic case",
          "num_examples": 1,
          "completions_per_example": 1,
          "aggregation": {
            "kind": "single"
          },
          "schema_version": 2,
          "default_scorer_kind": "exact_match",
          "tags": {
            "math": 1
          }
        },
        "stats": {
          "trajectories_seen": 1,
          "trajectories_used": 1,
          "examples_generated": 1,
          "skipped_no_target": 0,
          "skipped_prompt_too_long": 0,
          "skipped_target_too_long": 0,
          "skipped_duplicate": 0,
          "skipped_strategy_match": 0,
          "sample_kept": 1,
          "effective_seed": "42",
          "auto_scorer_histogram": {
            "exact_match": 1
          }
        },
        "queued_eval_job_ids": [
          "eval-1"
        ]
      }
    ],
    "ValidateJudgmentResponse": [
      {
        "status": "queued",
        "eval_job_id": "eval-judge-1",
        "effective_seed": "42",
        "validation_suite": "judge-validate-style-prefs",
        "warnings": []
      }
    ]
  },
  "x-kiln-external-contracts": [
    "kiln-observability-v1.schema.json",
    "thinking-budget-v1.schema.json"
  ],
  "x-kiln-field-schema-status": "complete"
}

Definitions

AggregateMetricsComplete aggregate metrics for one suite/adapter run.24 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`num_examples`	yes	`NonNegativeInteger`	-	Num examples. A whole number greater than or equal to zero.
`num_completions`	yes	`NonNegativeInteger`	-	Num completions. A whole number greater than or equal to zero.
`num_pass`	yes	`NonNegativeInteger`	-	Num pass. A whole number greater than or equal to zero.
`num_fail`	yes	`NonNegativeInteger`	-	Num fail. A whole number greater than or equal to zero.
`num_invalid`	yes	`NonNegativeInteger`	-	Num invalid. A whole number greater than or equal to zero.
`num_error`	yes	`NonNegativeInteger`	-	Num error. A whole number greater than or equal to zero.
`accuracy`	yes	`UnitInterval`	-	Accuracy. A score or rate from 0 through 1, inclusive.
`accuracy_confidence_interval`	yes	`PassRateConfidenceInterval`	-	Accuracy confidence interval. Wilson confidence interval for a pass rate.
`mean_score`	yes	`UnitInterval`	-	Mean score. A score or rate from 0 through 1, inclusive.
`weighted_mean_score`	yes	`UnitInterval`	-	Weighted mean score. A score or rate from 0 through 1, inclusive.
`latency`	yes	`LatencyStats`	-	Latency. Latency distribution summary in milliseconds.
`total_prompt_tokens`	yes	`NonNegativeInteger`	-	Total prompt tokens. A whole number greater than or equal to zero.
`total_completion_tokens`	yes	`NonNegativeInteger`	-	Total completion tokens. A whole number greater than or equal to zero.
`elapsed_secs`	yes	`FiniteNumber`	-	Elapsed secs. A finite JSON number; NaN and infinities are not valid JSON.
`pass_rate_by_tag`	yes	`object`	-	Pass rate by tag.
`tag_breakdown`	no	`object`	-	Tag breakdown.
`by_scorer`	yes	`array<ScorerBreakdown>`	-	By scorer.
`pass_rate_by_tool`	no	`object`	-	Pass rate by tool.
`reasoning_length`	yes	`ReasoningLengthStats`	-	Reasoning length. Reasoning-block character-length summary.
`num_unclosed_thinking`	yes	`NonNegativeInteger`	-	Num unclosed thinking. A whole number greater than or equal to zero.
`num_non_xml_tool_calls`	yes	`NonNegativeInteger`	-	Num non xml tool calls. A whole number greater than or equal to zero.
`confusion_by_tool`	no	`object`	-	Confusion by tool.
`num_schema_missing_required`	yes	`NonNegativeInteger`	-	Num schema missing required. A whole number greater than or equal to zero.
`num_schema_extra_unknown`	yes	`NonNegativeInteger`	-	Num schema extra unknown. A whole number greater than or equal to zero.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::AggregateMetrics"
}

AggregatedExampleOutcomeOne independent example statistic reduced from its complete raw completion group.11 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`example_id`	yes	`NonEmptyString`	-	Example ID. Text containing at least one character.
`kind`	yes	`EvalOutcomeKind`	-	Kind. Per-completion scoring outcome.
`score`	yes	`UnitInterval`	-	Score. A score or rate from 0 through 1, inclusive.
`completion_indices`	yes	`array<NonNegativeInteger>`	minimum items 1; maximum items 128	Completion indices.
`representative_completion_index`	yes	`NonNegativeInteger`	-	Representative completion index. A whole number greater than or equal to zero.
`num_pass`	yes	`NonNegativeInteger`	-	Num pass. A whole number greater than or equal to zero.
`num_fail`	yes	`NonNegativeInteger`	-	Num fail. A whole number greater than or equal to zero.
`num_invalid`	yes	`NonNegativeInteger`	-	Num invalid. A whole number greater than or equal to zero.
`num_error`	yes	`NonNegativeInteger`	-	Num error. A whole number greater than or equal to zero.
`tags`	no	`array<String>`	-	Tags.
`metadata`	no	`AnyJson`	-	Metadata. Any JSON value; Kiln does not interpret its internal shape at this boundary.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::AggregatedExampleOutcome",
  "x-kiln-semantic-constraints": [
    "completion_indices are exactly 0..k",
    "raw kind counts sum to k",
    "representative_completion_index is a member of completion_indices"
  ]
}

AnyJsonAny JSON value; Kiln does not interpret its internal shape at this boundary.0 fields

Type: any.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "serde_json::Value"
}

AppendJudgmentBodyStore one human preference over two exact responses to the same prompt.9 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`id`	no	`String \| null`	-	ID.
`prompt`	yes	`array<EvalChatMessage>`	-	Prompt.
`adapter_a`	no	`String \| null`	-	Adapter a.
`adapter_b`	no	`String \| null`	-	Adapter b.
`response_a`	yes	`String`	-	Response a. Text, including an empty string.
`response_b`	yes	`String`	-	Response b. Text, including an empty string.
`winner`	yes	`JudgmentWinner`	-	Winner. Human pairwise preference label.
`note`	no	`String \| null`	-	Note.
`tags`	no	`array<String>`	-	Tags.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "AppendJudgmentBody",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

AppendJudgmentResponseAssigned row ID flattened with the updated judgment manifest.9 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`judgment_id`	yes	`NonEmptyString`	-	Judgment ID. Text containing at least one character.
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`description`	yes	`String \| null`	-	Description.
`num_rows`	yes	`NonNegativeInteger`	-	Num rows. A whole number greater than or equal to zero.
`created_at`	yes	`Rfc3339Timestamp`	-	Created at. An RFC 3339 timestamp.
`updated_at`	yes	`Rfc3339Timestamp`	-	Updated at. An RFC 3339 timestamp.
`winner_histogram`	yes	`object`	-	Winner histogram.
`last_compiled_split`	no	`NonNegativeInteger`	-	Last compiled split. A whole number greater than or equal to zero.
`last_compiled_at`	no	`Rfc3339Timestamp`	-	Last compiled at. An RFC 3339 timestamp.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "AppendJudgmentResponse"
}

ArgsScoringTool-argument scoring policy, including recursive per-key scorers.0 fields

Type: object | object | object | object.

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "oneOf": [
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "keys_only"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "structural"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "auto"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind",
        "scorers"
      ],
      "properties": {
        "kind": {
          "const": "per_key"
        },
        "scorers": {
          "type": "object",
          "additionalProperties": {
            "$ref": "#/$defs/Scorer"
          }
        },
        "extra_key_penalty": {
          "$ref": "#/$defs/FiniteNumber"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    }
  ]
}

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::scorers::ArgsScoring"
}

BooleanEither true or false.0 fields

Type: boolean.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "bool"
}

CancelEvalJobResponseQueued cancellation, cooperative running cancellation, or terminal-record deletion result.0 fields

Type: object | object | object.

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "oneOf": [
    {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "status",
        "job_id",
        "was_in_queue"
      ],
      "properties": {
        "status": {
          "const": "cancelled"
        },
        "job_id": {
          "$ref": "#/$defs/NonEmptyString"
        },
        "was_in_queue": {
          "$ref": "#/$defs/Boolean"
        }
      }
    },
    {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "status",
        "job_id",
        "note"
      ],
      "properties": {
        "status": {
          "const": "cancelling"
        },
        "job_id": {
          "$ref": "#/$defs/NonEmptyString"
        },
        "note": {
          "const": "running job will exit at the next example boundary"
        }
      }
    },
    {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "status",
        "job_id",
        "removed_archive_file"
      ],
      "properties": {
        "status": {
          "const": "deleted"
        },
        "job_id": {
          "$ref": "#/$defs/NonEmptyString"
        },
        "removed_archive_file": {
          "$ref": "#/$defs/Boolean"
        }
      }
    }
  ]
}

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "CancelEvalJobResponse"
}

CodeStyleHow the code scorer compares extracted code. Omitted variant settings use the scorer defaults.0 fields

Type: object | object | object | object.

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "oneOf": [
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "any_block"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "exact_block"
        },
        "strip_comments": {
          "$ref": "#/$defs/Boolean"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "token_similarity"
        },
        "min_jaccard": {
          "$ref": "#/$defs/FiniteNumber"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "line_coverage"
        },
        "min_coverage": {
          "$ref": "#/$defs/FiniteNumber"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    }
  ]
}

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::scorers::CodeStyle"
}

CompileJudgmentBodyCompile judgment rows to a swap-augmented SFT dataset with a holdout.3 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`output_dataset`	yes	`EvalResourceName`	-	Output dataset. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`include_skips`	no	`Boolean`	-	Include skips. Either true or false.
`holdout_n`	no	`NonNegativeInteger \| null`	-	Holdout n.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "CompileJudgmentBody",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

CompileJudgmentResponseThe swap-augmented SFT output, reserved holdout count, source-row boundary, dataset manifest, and warnings.6 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`status`	yes	`any`	const "compiled"	Status.
`rows`	yes	`PositiveInteger`	-	Rows. A whole number greater than zero.
`holdout_n`	yes	`NonNegativeInteger`	-	Holdout n. A whole number greater than or equal to zero.
`train_validation_split`	yes	`PositiveInteger`	-	Train validation split. A whole number greater than zero.
`dataset`	yes	`DatasetManifest`	-	Dataset. Persisted content-addressed dataset identity and partition summary.
`warnings`	yes	`array<String>`	-	Warnings.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "CompileJudgmentResponse"
}

ContainsModeHow phrase containment matches combine.0 fields

Type: string. Constraints: enum "any", "all", "none".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "ContainsMode"
}

CreateJudgmentBodyCreate an append-only judgment dataset.2 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`description`	no	`String \| null`	-	Description.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "CreateJudgmentBody",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

DatasetFormatCanonical persisted dataset format.0 fields

Type: string. Constraints: enum "sft_chat", "grpo_groups", "raw".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DatasetFormat"
}

DatasetListResponseAll uploaded eval/training datasets.1 field

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`datasets`	yes	`array<DatasetManifest>`	-	Datasets.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DatasetListResponse"
}

DatasetManifestPersisted content-addressed dataset identity and partition summary.16 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`schema_version`	yes	`any`	const 2	Schema version.
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`format`	yes	`DatasetFormat`	-	Format. Canonical persisted dataset format.
`description`	yes	`String \| null`	-	Description.
`num_rows`	yes	`NonNegativeInteger`	-	Num rows. A whole number greater than or equal to zero.
`size_bytes`	yes	`NonNegativeInteger`	-	Size bytes. A whole number greater than or equal to zero.
`created_at`	yes	`Rfc3339Timestamp`	-	Created at. An RFC 3339 timestamp.
`updated_at`	yes	`Rfc3339Timestamp`	-	Updated at. An RFC 3339 timestamp.
`corpus_sha256`	yes	`Sha256Digest`	-	Corpus SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`normalized_corpus_sha256`	yes	`Sha256Digest`	-	Normalized corpus SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`split_manifest_sha256`	yes	`Sha256Digest`	-	Split manifest SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`split_config`	yes	`DatasetSplitConfig`	-	Split config. Deterministic group-aware split policy. Train plus validation must be below 100.
`split_counts`	yes	`DatasetSplitCounts`	-	Split counts. Row counts in each persisted partition.
`num_groups`	yes	`NonNegativeInteger`	-	Num groups. A whole number greater than or equal to zero.
`num_sessions`	yes	`NonNegativeInteger`	-	Num sessions. A whole number greater than or equal to zero.
`stats`	yes	`DatasetStats`	-	Stats. Bounded structural statistics computed from an uploaded dataset.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DatasetManifest"
}

DatasetSplitPersisted dataset partition.0 fields

Type: string. Constraints: enum "train", "validation", "holdout".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::DatasetSplit"
}

DatasetSplitConfigDeterministic group-aware split policy. Train plus validation must be below 100.3 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`seed`	yes	`DecimalU64`	-	Seed. An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.
`train_percent`	yes	`integer`	minimum 1; maximum 98	Train percent.
`validation_percent`	yes	`integer`	minimum 0; maximum 98	Validation percent.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DatasetSplitConfig",
  "x-kiln-semantic-constraints": [
    "train_percent + validation_percent < 100"
  ]
}

DatasetSplitCountsRow counts in each persisted partition.3 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`train`	yes	`NonNegativeInteger`	-	Train. A whole number greater than or equal to zero.
`validation`	yes	`NonNegativeInteger`	-	Validation. A whole number greater than or equal to zero.
`holdout`	yes	`NonNegativeInteger`	-	Holdout. A whole number greater than or equal to zero.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::DatasetSplitCounts"
}

DatasetSplitManifestPersisted deterministic split manifest; normalized duplicates and connected group/session rows share one partition.7 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`schema`	yes	`any`	const "kiln.dataset-split.v1"	Schema.
`dataset_name`	yes	`EvalResourceName`	-	Dataset name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`corpus_sha256`	yes	`Sha256Digest`	-	Corpus SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`normalized_corpus_sha256`	yes	`Sha256Digest`	-	Normalized corpus SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`config`	yes	`DatasetSplitConfig`	-	Config. Deterministic group-aware split policy. Train plus validation must be below 100.
`counts`	yes	`DatasetSplitCounts`	-	Counts. Row counts in each persisted partition.
`rows`	yes	`array<DatasetSplitRow>`	minimum items 1	Rows.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DatasetSplitManifest"
}

DatasetSplitRowContent-addressed row assignment with preserved group/session identity.6 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`row_number`	yes	`PositiveInteger`	-	Row number. A whole number greater than zero.
`content_sha256`	yes	`Sha256Digest`	-	Content SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`normalized_sha256`	yes	`Sha256Digest`	-	Normalized SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`split`	yes	`DatasetSplit`	-	Split. Persisted dataset partition.
`group_id`	no	`String`	-	Group ID. Text, including an empty string.
`session_id`	no	`String`	-	Session ID. Text, including an empty string.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::DatasetSplitRow"
}

DatasetStatsBounded structural statistics computed from an uploaded dataset.7 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`num_assistant_turns`	yes	`NonNegativeInteger`	-	Num assistant turns. A whole number greater than or equal to zero.
`num_tool_messages`	yes	`NonNegativeInteger`	-	Num tool messages. A whole number greater than or equal to zero.
`num_with_tool_calls`	yes	`NonNegativeInteger`	-	Num with tool calls. A whole number greater than or equal to zero.
`max_messages_per_conv`	yes	`NonNegativeInteger`	-	Max messages per conv. A whole number greater than or equal to zero.
`max_content_chars`	yes	`NonNegativeInteger`	-	Max content chars. A whole number greater than or equal to zero.
`avg_messages_per_conv`	yes	`FiniteNumber`	-	Avg messages per conv. A finite JSON number; NaN and infinities are not valid JSON.
`sample_role_patterns`	yes	`array<String>`	-	Sample role patterns.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DatasetStats"
}

DatasetUploadFormatAccepted multipart dataset format including request aliases.0 fields

Type: string. Constraints: enum "sft_chat", "sft", "grpo_groups", "grpo", "raw".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DatasetUploadMultipart::format"
}

DatasetUploadMultipartMultipart dataset upload. Unrecognized multipart fields are read and ignored.4 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`format`	no	`DatasetUploadFormat`	-	Format. Accepted multipart dataset format including request aliases.
`description`	no	`String`	-	Description. Text, including an empty string.
`file`	yes	`string`	format binary; content media type application/jsonl	File.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-default-format": "sft_chat",
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DatasetUploadMultipart",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

DecimalU64An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.0 fields

Type: string. Constraints: pattern ^(0|[1-9][0-9]*)$.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "u64"
}

DeleteDatasetResponseConfirmation that a dataset was deleted.2 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`status`	yes	`any`	const "deleted"	Status.
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DeleteDatasetResponse"
}

DeleteJudgmentResponseConfirmation that a judgment dataset was deleted.2 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`status`	yes	`any`	const "deleted"	Status.
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DeleteJudgmentResponse"
}

DeleteSuiteResponseConfirmation that a suite was deleted.2 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`status`	yes	`any`	const "deleted"	Status.
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "DeleteSuiteResponse"
}

EvalAggregationExplicit reduction from k raw completions to one independent example statistic.0 fields

Type: object | object | object | object.

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "oneOf": [
    {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "single"
        }
      }
    },
    {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "kind",
        "k"
      ],
      "properties": {
        "kind": {
          "const": "mean_at_k"
        },
        "k": {
          "type": "integer",
          "minimum": 1,
          "maximum": 128
        }
      }
    },
    {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "kind",
        "k"
      ],
      "properties": {
        "kind": {
          "const": "pass_at_k"
        },
        "k": {
          "type": "integer",
          "minimum": 1,
          "maximum": 128
        }
      }
    },
    {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "kind",
        "k"
      ],
      "properties": {
        "kind": {
          "const": "majority_at_k"
        },
        "k": {
          "type": "integer",
          "minimum": 1,
          "maximum": 128
        }
      }
    }
  ]
}

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalAggregation",
  "x-kiln-semantic-constraints": [
    "majority_at_k requires odd k"
  ]
}

EvalChatMessageChat message shared by inference, training, and evals. Kiln normalizes supported structured content into its internal message form.5 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`role`	yes	`String`	-	Role. Text, including an empty string.
`content`	no	`String \| null \| array<AnyJson>`	-	Content.
`tool_calls`	no	`array<AnyJson> \| null`	-	Tool calls.
`name`	no	`String \| null`	-	Name.
`tool_call_id`	no	`String \| null`	-	Tool call ID.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_core::tokenizer::ChatMessage",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

EvalCompareSpecRun one registered suite against one to eight ordered adapters. Empty adapter names select the base model.4 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`suite`	yes	`EvalResourceName`	-	Suite. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`adapters`	yes	`array<String>`	minimum items 1; maximum items 8	Adapters.
`seed`	no	`NonNegativeInteger`	-	Seed. A whole number greater than or equal to zero.
`generation`	no	`EvalGenerationParams`	-	Generation. Eval decode parameters. Omitted thinking budgets inherit; explicit null is unlimited; finite values include zero.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "EvalCompareSpec",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

EvalExampleOne prompt/target pair with optional scoring and generation overrides.10 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`id`	no	`String`	-	ID. Text, including an empty string.
`messages`	yes	`array<EvalChatMessage>`	minimum items 1	Messages.
`target`	no	`String`	-	Target. Text, including an empty string.
`aliases`	no	`array<String>`	-	Aliases.
`tags`	no	`array<String>`	-	Tags.
`metadata`	no	`AnyJson`	-	Metadata. Any JSON value; Kiln does not interpret its internal shape at this boundary.
`scorer`	no	`Scorer`	-	Scorer. Select one built-in scorer. Each variant accepts and ignores unrecognized input fields; use only documented fields so mistakes remain visible in review.
`generation`	no	`EvalGenerationParams`	-	Generation. Eval decode parameters. Omitted thinking budgets inherit; explicit null is unlimited; finite values include zero.
`weight`	no	`number`	minimum 0	Weight.
`tools`	no	`array<AnyJson>`	-	Tools.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalExample",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

EvalGenerationParamsEval decode parameters. Omitted thinking budgets inherit; explicit null is unlimited; finite values include zero.10 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`temperature`	no	`FiniteNumber`	-	Temperature. A finite JSON number; NaN and infinities are not valid JSON.
`top_p`	no	`FiniteNumber`	-	Top p. A finite JSON number; NaN and infinities are not valid JSON.
`top_k`	no	`NonNegativeInteger`	-	Top k. A whole number greater than or equal to zero.
`max_tokens`	no	`NonNegativeInteger`	-	Max tokens. A whole number greater than or equal to zero.
`n`	no	`integer`	minimum 1; maximum 128	N.
`stop`	no	`array<String>`	-	Stop.
`seed`	no	`NonNegativeInteger \| null`	-	Seed.
`thinking_budget_tokens`	no	`NonNegativeInteger \| null`	-	Thinking budget tokens.
`thinking_budget_ms`	no	`NonNegativeInteger \| null`	-	Thinking budget ms.
`chat_template_kwargs`	no	`object`	-	Chat template kwargs.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalGenerationParams",
  "x-kiln-thinking-budget-contract": "thinking-budget-v1.schema.json",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

EvalJobInfoList-view record for a queued, running, or retained eval job. Process-local timers and cancellation handles are not part of the wire response.20 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`schema_version`	yes	`any`	const 2	Schema version.
`job_id`	yes	`NonEmptyString`	-	Job ID. Text containing at least one character.
`suite_name`	yes	`EvalResourceName`	-	Suite name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`adapters`	yes	`array<String \| null>`	-	Adapters.
`submission_kind`	yes	`EvalSubmissionKind`	-	Submission kind. Admission path that created an eval job.
`base_weight_shard_manifest`	no	`BaseWeightShardManifest`	-	Base weight shard manifest. See kiln-observability-v1.schema.json#/$defs/BaseWeightShardManifest.
`execution_provenance`	no	`ExecutionProvenanceV1`	-	Execution provenance. See kiln-observability-v1.schema.json#/$defs/ExecutionProvenanceV1.
`effective_seed`	no	`DecimalU64`	-	Effective seed. An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.
`state`	yes	`EvalJobState`	-	State. Evaluation job lifecycle state.
`progress`	yes	`EvalProgress`	-	Progress. Live progress for the active adapter run.
`finished_runs`	yes	`array<SuiteResult>`	-	Finished runs.
`headline_accuracy`	yes	`FiniteNumber \| null`	-	Headline accuracy.
`error`	yes	`String \| null`	-	Error.
`source_training_job_id`	yes	`String \| null`	-	Source training job ID.
`submitted_at_iso`	yes	`Rfc3339Timestamp`	-	Submitted at iso. An RFC 3339 timestamp.
`started_at_iso`	yes	`Rfc3339Timestamp \| null`	-	Started at iso.
`finished_at_iso`	yes	`Rfc3339Timestamp \| null`	-	Finished at iso.
`post_eval_gate`	no	`PostEvalGate`	-	Post eval gate. Fail-closed post-training promotion settings retained with the paired eval job.
`replay_expectation`	no	`EvalReplayExpectationV1`	-	Replay expectation. Immutable source and digest target installed before a strict replay becomes queue-visible.
`replay_verdict`	no	`EvalReplayVerdict`	-	Replay verdict. Terminal comparison between a replay expectation and actual identity and raw-byte sets.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "EvalJobInfo"
}

EvalJobListResponseAll retained eval jobs in descending submission order.1 field

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`jobs`	yes	`array<EvalJobInfo>`	-	Jobs.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "EvalJobListResponse"
}

EvalJobStateEvaluation job lifecycle state.0 fields

Type: string. Constraints: enum "queued", "running", "completed", "failed", "cancelled".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalJobState"
}

EvalModelTargetIdentityExact base-model or named-adapter byte identity used for generation or judge scoring.2 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`adapter`	no	`NonEmptyString`	-	Adapter. Text containing at least one character.
`adapter_content_sha256`	no	`Sha256Digest`	-	Adapter content SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalModelTargetIdentity",
  "x-kiln-semantic-constraints": [
    "adapter and adapter_content_sha256 are either both present or both absent"
  ]
}

EvalOutcomeKindPer-completion scoring outcome.0 fields

Type: string. Constraints: enum "pass", "fail", "invalid", "error".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalOutcomeKind"
}

EvalProgressLive progress for the active adapter run.4 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`examples_completed`	yes	`NonNegativeInteger`	-	Examples completed. A whole number greater than or equal to zero.
`examples_total`	yes	`NonNegativeInteger`	-	Examples total. A whole number greater than or equal to zero.
`running_accuracy`	yes	`UnitInterval`	-	Running accuracy. A score or rate from 0 through 1, inclusive.
`running_mean_score`	yes	`UnitInterval`	-	Running mean score. A score or rate from 0 through 1, inclusive.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalProgress"
}

EvalRawCompletionReferenceContent-addressed pointer from a replay record to exact raw and normalized completion bytes.9 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`outcome_index`	yes	`NonNegativeInteger`	-	Outcome index. A whole number greater than or equal to zero.
`example_id`	yes	`NonEmptyString`	-	Example ID. Text containing at least one character.
`completion_index`	yes	`NonNegativeInteger`	-	Completion index. A whole number greater than or equal to zero.
`generation_seed`	no	`DecimalU64`	-	Generation seed. An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.
`raw_completion_pointer`	yes	`NonEmptyString`	-	Raw completion pointer. Text containing at least one character.
`raw_completion_sha256`	no	`Sha256Digest`	-	Raw completion SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`raw_completion_bytes`	no	`NonNegativeInteger`	-	Raw completion bytes. A whole number greater than or equal to zero.
`normalized_completion_sha256`	yes	`Sha256Digest`	-	Normalized completion SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`normalized_completion_bytes`	yes	`NonNegativeInteger`	-	Normalized completion bytes. A whole number greater than or equal to zero.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalRawCompletionReference"
}

EvalReplayExpectationV1Immutable source and digest target installed before a strict replay becomes queue-visible.5 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`expectation_type`	yes	`any`	const "kiln.eval-replay-expectation.v1"	Expectation type.
`source_job_id`	yes	`NonEmptyString`	-	Source job ID. Text containing at least one character.
`source_run_index`	yes	`NonNegativeInteger`	-	Source run index. A whole number greater than or equal to zero.
`expected_record_sha256`	yes	`Sha256Digest`	-	Expected record SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`expected_raw_completion_set_sha256`	yes	`Sha256Digest`	-	Expected raw completion set SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalReplayExpectationV1"
}

EvalReplayRecordV1Self-validating immutable suite, generation, model, scorer, environment, and raw-completion identity for one run.17 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`schema_version`	yes	`any`	const 1	Schema version.
`record_type`	yes	`any`	const "kiln.eval-replay.v1"	Record type.
`suite`	yes	`EvalSuite`	-	Suite. A named set of examples with shared scoring, generation, aggregation, prompt, and tool settings. Omitted settings use documented defaults.
`generation_override`	no	`EvalGenerationParams`	-	Generation override. Eval decode parameters. Omitted thinking budgets inherit; explicit null is unlimited; finite values include zero.
`effective_seed`	yes	`DecimalU64`	-	Effective seed. An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.
`seed_derivation`	yes	`any`	const "kiln.eval-seed.v1"	Seed derivation.
`resolved_thinking_budgets`	yes	`array<record>`	-	Resolved thinking budgets.
`model_target`	no	`EvalModelTargetIdentity`	-	Model target. Exact base-model or named-adapter byte identity used for generation or judge scoring.
`judge_targets`	no	`array<EvalModelTargetIdentity>`	-	Judge targets.
`scorer_identities`	yes	`array<EvalScorerIdentity>`	-	Scorer identities.
`raw_completions`	yes	`array<EvalRawCompletionReference>`	-	Raw completions.
`suite_sha256`	yes	`Sha256Digest`	-	Suite SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`effective_generation_sha256`	yes	`Sha256Digest`	-	Effective generation SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`raw_completion_set_sha256`	yes	`Sha256Digest`	-	Raw completion set SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`execution_provenance_sha256`	no	`Sha256Digest`	-	Execution provenance SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`base_weight_manifest_sha256`	no	`Sha256Digest`	-	Base weight manifest SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`record_sha256`	yes	`Sha256Digest`	-	Record SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalReplayRecordV1"
}

EvalReplayStatusTerminal strict-replay classification.0 fields

Type: string. Constraints: enum "matched", "mismatch", "error".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalReplayStatus"
}

EvalReplayVerdictTerminal comparison between a replay expectation and actual identity and raw-byte sets.8 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`status`	yes	`EvalReplayStatus`	-	Status. Terminal strict-replay classification.
`source_job_id`	yes	`NonEmptyString`	-	Source job ID. Text containing at least one character.
`source_run_index`	yes	`NonNegativeInteger`	-	Source run index. A whole number greater than or equal to zero.
`expected_record_sha256`	yes	`Sha256Digest`	-	Expected record SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`actual_record_sha256`	no	`Sha256Digest`	-	Actual record SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`expected_raw_completion_set_sha256`	yes	`Sha256Digest`	-	Expected raw completion set SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`actual_raw_completion_set_sha256`	no	`Sha256Digest`	-	Actual raw completion set SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`message`	yes	`NonEmptyString`	-	Message. Text containing at least one character.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalReplayVerdict"
}

EvalResourceNameA non-blank suite, dataset, or judgment name without path separators or the sequence '..'.0 fields

Type: string. Constraints: minimum length 1; pattern ^(?![\s\S](?:/|\|..))(?=[\s\S]\S)[\s\S]+$.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "String"
}

EvalResultTop-level retained result for a single- or multi-adapter eval job.12 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`schema_version`	yes	`any`	const 2	Schema version.
`job_id`	yes	`NonEmptyString`	-	Job ID. Text containing at least one character.
`state`	yes	`EvalJobState`	-	State. Evaluation job lifecycle state.
`base_weight_shard_manifest`	no	`BaseWeightShardManifest`	-	Base weight shard manifest. See kiln-observability-v1.schema.json#/$defs/BaseWeightShardManifest.
`execution_provenance`	no	`ExecutionProvenanceV1`	-	Execution provenance. See kiln-observability-v1.schema.json#/$defs/ExecutionProvenanceV1.
`effective_seed`	no	`DecimalU64`	-	Effective seed. An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.
`seed_derivation`	no	`String`	-	Seed derivation. Text, including an empty string.
`replay_expectation`	no	`EvalReplayExpectationV1`	-	Replay expectation. Immutable source and digest target installed before a strict replay becomes queue-visible.
`replay_verdict`	no	`EvalReplayVerdict`	-	Replay verdict. Terminal comparison between a replay expectation and actual identity and raw-byte sets.
`runs`	yes	`array<SuiteResult>`	-	Runs.
`progress`	no	`EvalProgress`	-	Progress. Live progress for the active adapter run.
`error`	no	`String`	-	Error. Text, including an empty string.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "EvalResult"
}

EvalRunRequestSubmit exactly one registered or inline suite for evaluation.5 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`suite`	no	`EvalResourceName`	-	Suite. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`inline_suite`	no	`EvalSuite`	-	Inline suite. A named set of examples with shared scoring, generation, aggregation, prompt, and tool settings. Omitted settings use documented defaults.
`adapter`	no	`String \| null`	-	Adapter.
`seed`	no	`NonNegativeInteger`	-	Seed. A whole number greater than or equal to zero.
`generation`	no	`EvalGenerationParams`	-	Generation. Eval decode parameters. Omitted thinking budgets inherit; explicit null is unlimited; finite values include zero.

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "oneOf": [
    {
      "required": [
        "suite"
      ],
      "not": {
        "required": [
          "inline_suite"
        ]
      }
    },
    {
      "required": [
        "inline_suite"
      ],
      "not": {
        "required": [
          "suite"
        ]
      }
    }
  ]
}

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "EvalRunRequest",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

EvalRunResponseImmutable admission receipt for a queued eval job.4 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`job_id`	yes	`NonEmptyString`	-	Job ID. Text containing at least one character.
`state`	yes	`any`	const "queued"	State.
`effective_seed`	yes	`DecimalU64`	-	Effective seed. An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.
`message`	yes	`NonEmptyString`	-	Message. Text containing at least one character.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "EvalRunResponse"
}

EvalScorerIdentityContent identity of the fully resolved scorer selected for one stable example ID.5 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`example_id`	yes	`NonEmptyString`	-	Example ID. Text containing at least one character.
`kind`	yes	`NonEmptyString`	-	Kind. Text containing at least one character.
`config_sha256`	yes	`Sha256Digest`	-	Config SHA-256. A lowercase SHA-256 identity with an explicit algorithm prefix.
`requires_judge`	yes	`Boolean`	-	Requires judge. Either true or false.
`judge_adapter`	no	`String`	-	Judge adapter. Text, including an empty string.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalScorerIdentity"
}

EvalSubmissionKindAdmission path that created an eval job.0 fields

Type: string. Constraints: enum "on_demand", "post_training", "compare".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "EvalSubmissionKind"
}

EvalSuiteA named set of examples with shared scoring, generation, aggregation, prompt, and tool settings. Omitted settings use documented defaults.9 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`description`	no	`String \| null`	-	Description.
`default_scorer`	yes	`Scorer`	-	Default scorer. Select one built-in scorer. Each variant accepts and ignores unrecognized input fields; use only documented fields so mistakes remain visible in review.
`generation`	no	`EvalGenerationParams`	-	Generation. Eval decode parameters. Omitted thinking budgets inherit; explicit null is unlimited; finite values include zero.
`aggregation`	no	`EvalAggregation`	-	Aggregation. Explicit reduction from k raw completions to one independent example statistic.
`system_prompt`	no	`String \| null`	-	System prompt.
`examples`	yes	`array<EvalExample>`	minimum items 1	Examples.
`schema_version`	no	`integer`	enum 1, 2	Schema version.
`tools`	no	`array<AnyJson>`	-	Tools.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "EvalSuite",
  "x-kiln-semantic-constraints": [
    "resolved example IDs are unique",
    "weights are finite and non-negative",
    "tool entries have a non-empty function.name or name",
    "every effective generation.n equals aggregation.k",
    "schema version 1 permits only single aggregation and n=1"
  ],
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

EvalSuiteSummaryList-view projection of a registered suite.8 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`description`	yes	`String \| null`	-	Description.
`num_examples`	yes	`NonNegativeInteger`	-	Num examples. A whole number greater than or equal to zero.
`completions_per_example`	yes	`integer`	minimum 1; maximum 128	Completions per example.
`aggregation`	yes	`EvalAggregation`	-	Aggregation. Explicit reduction from k raw completions to one independent example statistic.
`schema_version`	yes	`integer`	enum 1, 2	Schema version.
`default_scorer_kind`	yes	`NonEmptyString`	-	Default scorer kind. Text containing at least one character.
`tags`	yes	`object`	-	Tags.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::EvalSuiteSummary"
}

ExampleOutcomeOne scored completion with optional raw reasoning, latency, seed, and thinking-budget evidence.16 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`example_id`	yes	`NonEmptyString`	-	Example ID. Text containing at least one character.
`completion_index`	yes	`NonNegativeInteger`	-	Completion index. A whole number greater than or equal to zero.
`generation_seed`	no	`DecimalU64`	-	Generation seed. An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.
`completion_text`	yes	`String`	-	Completion text. Text, including an empty string.
`raw_completion_text`	no	`String`	-	Raw completion text. Text, including an empty string.
`thinking_budget`	no	`record`	-	Thinking budget. See thinking-budget-v1.schema.json#/$defs/record.
`kind`	yes	`EvalOutcomeKind`	-	Kind. Per-completion scoring outcome.
`score`	yes	`UnitInterval`	-	Score. A score or rate from 0 through 1, inclusive.
`detail`	no	`String`	-	Detail. Text, including an empty string.
`prompt_tokens`	no	`NonNegativeInteger`	-	Prompt tokens. A whole number greater than or equal to zero.
`completion_tokens`	no	`NonNegativeInteger`	-	Completion tokens. A whole number greater than or equal to zero.
`latency_ms`	no	`FiniteNumber`	-	Latency ms. A finite JSON number; NaN and infinities are not valid JSON.
`tags`	no	`array<String>`	-	Tags.
`metadata`	no	`AnyJson`	-	Metadata. Any JSON value; Kiln does not interpret its internal shape at this boundary.
`reasoning_text`	no	`String`	-	Reasoning text. Text, including an empty string.
`unclosed_thinking`	no	`Boolean`	-	Unclosed thinking. Either true or false.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::ExampleOutcome"
}

FiniteNumberA finite JSON number; NaN and infinities are not valid JSON.0 fields

Type: number.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "f32 | f64"
}

JudgmentListResponseAll retained pairwise-judgment datasets.1 field

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`judgments`	yes	`array<JudgmentManifest>`	-	Judgments.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "JudgmentListResponse"
}

JudgmentManifestPersisted pairwise-judgment dataset identity and compile split.8 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`description`	yes	`String \| null`	-	Description.
`num_rows`	yes	`NonNegativeInteger`	-	Num rows. A whole number greater than or equal to zero.
`created_at`	yes	`Rfc3339Timestamp`	-	Created at. An RFC 3339 timestamp.
`updated_at`	yes	`Rfc3339Timestamp`	-	Updated at. An RFC 3339 timestamp.
`winner_histogram`	yes	`object`	-	Winner histogram.
`last_compiled_split`	no	`NonNegativeInteger`	-	Last compiled split. A whole number greater than or equal to zero.
`last_compiled_at`	no	`Rfc3339Timestamp`	-	Last compiled at. An RFC 3339 timestamp.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "JudgmentManifest"
}

JudgmentWinnerHuman pairwise preference label.0 fields

Type: string. Constraints: enum "a", "b", "tie", "skip".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "JudgmentWinner"
}

LatencyStatsLatency distribution summary in milliseconds.5 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`p50_ms`	yes	`FiniteNumber`	-	P50 ms. A finite JSON number; NaN and infinities are not valid JSON.
`p90_ms`	yes	`FiniteNumber`	-	P90 ms. A finite JSON number; NaN and infinities are not valid JSON.
`p99_ms`	yes	`FiniteNumber`	-	P99 ms. A finite JSON number; NaN and infinities are not valid JSON.
`mean_ms`	yes	`FiniteNumber`	-	Mean ms. A finite JSON number; NaN and infinities are not valid JSON.
`max_ms`	yes	`FiniteNumber`	-	Max ms. A finite JSON number; NaN and infinities are not valid JSON.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::LatencyStats"
}

NameMatchTool-name matching policy.0 fields

Type: object | object | object.

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "oneOf": [
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "exact"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "case_insensitive"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind",
        "allowed"
      ],
      "properties": {
        "kind": {
          "const": "one_of"
        },
        "allowed": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/String"
          }
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    }
  ]
}

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::scorers::NameMatch"
}

NonEmptyStringText containing at least one character.0 fields

Type: string. Constraints: minimum length 1.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "String"
}

NonNegativeIntegerA whole number greater than or equal to zero.0 fields

Type: integer. Constraints: minimum 0.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "u64 | u32 | usize"
}

PassRateConfidenceIntervalWilson confidence interval for a pass rate.3 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`confidence_level`	yes	`UnitInterval`	-	Confidence level. A score or rate from 0 through 1, inclusive.
`lower`	yes	`UnitInterval`	-	Lower. A score or rate from 0 through 1, inclusive.
`upper`	yes	`UnitInterval`	-	Upper. A score or rate from 0 through 1, inclusive.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::PassRateConfidenceInterval"
}

PositiveIntegerA whole number greater than zero.0 fields

Type: integer. Constraints: minimum 1.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "u64 | u32 | usize"
}

PostEvalGateFail-closed post-training promotion settings retained with the paired eval job.6 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`min_accuracy`	yes	`UnitInterval`	-	Min accuracy. A score or rate from 0 through 1, inclusive.
`relative_recovery`	no	`FiniteNumber`	-	Relative recovery. A finite JSON number; NaN and infinities are not valid JSON.
`absolute_gain`	no	`FiniteNumber`	-	Absolute gain. A finite JSON number; NaN and infinities are not valid JSON.
`adapter_name`	yes	`String`	-	Adapter name. Text, including an empty string.
`training_job_id`	yes	`NonEmptyString`	-	Training job ID. Text containing at least one character.
`auto_load_on_pass`	yes	`Boolean`	-	Auto load on pass. Either true or false.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "PostEvalGate"
}

PromoteJudgmentBodyQueue held-out validation for a trained judge adapter.2 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`adapter`	yes	`String`	-	Adapter. Text, including an empty string.
`holdout_n`	no	`NonNegativeInteger`	-	Holdout n. A whole number greater than or equal to zero.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "PromoteJudgmentBody",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

ReasoningLengthStatsReasoning-block character-length summary.5 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`num_with_thinking`	yes	`NonNegativeInteger`	-	Num with thinking. A whole number greater than or equal to zero.
`mean_chars`	yes	`FiniteNumber`	-	Mean chars. A finite JSON number; NaN and infinities are not valid JSON.
`p50_chars`	yes	`NonNegativeInteger`	-	P50 chars. A whole number greater than or equal to zero.
`p90_chars`	yes	`NonNegativeInteger`	-	P90 chars. A whole number greater than or equal to zero.
`max_chars`	yes	`NonNegativeInteger`	-	Max chars. A whole number greater than or equal to zero.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::ReasoningLengthStats"
}

RenderJudgmentPromptResponseExact pairwise prompt used by both judge training and inference.1 field

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`prompt`	yes	`String`	-	Prompt. Text, including an empty string.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "RenderJudgmentPromptResponse"
}

ReplayBodySelect one retained run for strict identity-checked byte replay.1 field

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`run_index`	no	`NonNegativeInteger`	-	Run index. A whole number greater than or equal to zero.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "ReplayBody"
}

RerunBodySelect outcomes and adapter identity for a failure-focused re-run.4 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`adapter`	no	`String \| null`	-	Adapter.
`outcome_kinds`	no	`array<EvalOutcomeKind>`	-	Outcome kinds.
`include_pass`	no	`Boolean`	-	Include pass. Either true or false.
`seed`	no	`NonNegativeInteger`	-	Seed. A whole number greater than or equal to zero.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "RerunBody",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

Rfc3339TimestampAn RFC 3339 timestamp.0 fields

Type: string. Constraints: format date-time.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "String"
}

SamplingSynthesis sampling and filtering controls.5 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`max_examples`	no	`NonNegativeInteger \| null`	-	Max examples.
`max_prompt_chars`	no	`NonNegativeInteger`	-	Max prompt chars. A whole number greater than or equal to zero.
`max_target_chars`	no	`NonNegativeInteger`	-	Max target chars. A whole number greater than or equal to zero.
`seed`	no	`NonNegativeInteger \| null`	-	Seed.
`dedupe`	no	`Boolean`	-	Dedupe. Either true or false.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::synthesis::Sampling",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

ScorerSelect one built-in scorer. Each variant accepts and ignores unrecognized input fields; use only documented fields so mistakes remain visible in review.0 fields

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "oneOf": [
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "exact_match"
        },
        "case_sensitive": {
          "$ref": "#/$defs/Boolean"
        },
        "strip_whitespace": {
          "$ref": "#/$defs/Boolean"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind",
        "phrases"
      ],
      "properties": {
        "kind": {
          "const": "contains"
        },
        "phrases": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/String"
          }
        },
        "mode": {
          "$ref": "#/$defs/ContainsMode"
        },
        "case_sensitive": {
          "$ref": "#/$defs/Boolean"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind",
        "pattern"
      ],
      "properties": {
        "kind": {
          "const": "regex"
        },
        "pattern": {
          "$ref": "#/$defs/String"
        },
        "capture_group": {
          "anyOf": [
            {
              "$ref": "#/$defs/NonNegativeInteger"
            },
            {
              "type": "null"
            }
          ]
        },
        "case_sensitive": {
          "$ref": "#/$defs/Boolean"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "json_validity"
        },
        "require_object": {
          "$ref": "#/$defs/Boolean"
        },
        "required_paths": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/String"
          }
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "multiple_choice"
        },
        "choices": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/String"
          }
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "numeric_tolerance"
        },
        "atol": {
          "$ref": "#/$defs/FiniteNumber"
        },
        "rtol": {
          "$ref": "#/$defs/FiniteNumber"
        },
        "integer_only": {
          "$ref": "#/$defs/Boolean"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "llm_judge"
        },
        "judge_adapter": {
          "anyOf": [
            {
              "$ref": "#/$defs/String"
            },
            {
              "type": "null"
            }
          ]
        },
        "template": {
          "$ref": "#/$defs/String"
        },
        "score_regex": {
          "$ref": "#/$defs/String"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "tool_call"
        },
        "name_match": {
          "$ref": "#/$defs/NameMatch"
        },
        "args": {
          "$ref": "#/$defs/ArgsScoring"
        },
        "weights": {
          "$ref": "#/$defs/ToolCallWeights"
        },
        "require_xml_format": {
          "$ref": "#/$defs/Boolean"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "code"
        },
        "language": {
          "anyOf": [
            {
              "$ref": "#/$defs/String"
            },
            {
              "type": "null"
            }
          ]
        },
        "style": {
          "$ref": "#/$defs/CodeStyle"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "python_doctest"
        },
        "timeout_seconds": {
          "$ref": "#/$defs/FiniteNumber"
        },
        "python_bin": {
          "anyOf": [
            {
              "$ref": "#/$defs/String"
            },
            {
              "type": "null"
            }
          ]
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind",
        "scorers"
      ],
      "properties": {
        "kind": {
          "const": "all"
        },
        "scorers": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/Scorer"
          }
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind",
        "scorers"
      ],
      "properties": {
        "kind": {
          "const": "any"
        },
        "scorers": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/Scorer"
          }
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    }
  ]
}

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::scorers::Scorer"
}

ScorerBreakdownAggregate metrics for one scorer kind.4 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`scorer_kind`	yes	`NonEmptyString`	-	Scorer kind. Text containing at least one character.
`num_examples`	yes	`NonNegativeInteger`	-	Num examples. A whole number greater than or equal to zero.
`mean_score`	yes	`UnitInterval`	-	Mean score. A score or rate from 0 through 1, inclusive.
`pass_rate`	yes	`UnitInterval`	-	Pass rate. A score or rate from 0 through 1, inclusive.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::ScorerBreakdown"
}

ScorerChoiceSynthesis scorer policy. Fixed scorers use a nested scorer field so both tagged unions remain unambiguous.0 fields

Type: object | object | object.

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "oneOf": [
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "auto_detect"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind",
        "scorer"
      ],
      "properties": {
        "kind": {
          "const": "fixed"
        },
        "scorer": {
          "$ref": "#/$defs/Scorer"
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    },
    {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "kind"
      ],
      "properties": {
        "kind": {
          "const": "judge"
        },
        "judge_adapter": {
          "anyOf": [
            {
              "$ref": "#/$defs/String"
            },
            {
              "type": "null"
            }
          ]
        }
      },
      "x-kiln-unknown-field-policy": "accepted_and_ignored"
    }
  ]
}

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::synthesis::ScorerChoice"
}

Sha256DigestA lowercase SHA-256 identity with an explicit algorithm prefix.0 fields

Type: string. Constraints: pattern ^sha256:[0-9a-f]{64}$.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "String"
}

StringText, including an empty string.0 fields

Type: string.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "String"
}

SuiteListResponseAll registered evaluation suites.1 field

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`suites`	yes	`array<EvalSuiteSummary>`	-	Suites.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "SuiteListResponse"
}

SuiteResultComplete result for one suite and adapter, retaining independent reductions, raw completions, and a replay identity record.11 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`suite_name`	yes	`EvalResourceName`	-	Suite name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`adapter`	yes	`String \| null`	-	Adapter.
`aggregation`	yes	`EvalAggregation`	-	Aggregation. Explicit reduction from k raw completions to one independent example statistic.
`metrics`	yes	`AggregateMetrics`	-	Metrics. Complete aggregate metrics for one suite/adapter run.
`aggregated_outcomes`	yes	`array<AggregatedExampleOutcome>`	-	Aggregated outcomes.
`outcomes`	yes	`array<ExampleOutcome>`	-	Outcomes.
`started_at`	yes	`Rfc3339Timestamp`	-	Started at. An RFC 3339 timestamp.
`finished_at`	yes	`Rfc3339Timestamp`	-	Finished at. An RFC 3339 timestamp.
`suite_hash`	yes	`Sha256Digest`	-	Suite hash. A lowercase SHA-256 identity with an explicit algorithm prefix.
`effective_generation_hash`	no	`Sha256Digest`	-	Effective generation hash. A lowercase SHA-256 identity with an explicit algorithm prefix.
`replay_record`	no	`EvalReplayRecordV1`	-	Replay record. Self-validating immutable suite, generation, model, scorer, environment, and raw-completion identity for one run.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::SuiteResult"
}

SuiteSaveResponseSuite persistence confirmation.3 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`name`	yes	`EvalResourceName`	-	Name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`path`	yes	`NonEmptyString`	-	Path. Text containing at least one character.
`status`	yes	`any`	enum "created", "overwritten"	Status.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "SuiteSaveResponse"
}

SynthesisPreviewNon-persisted synthesis preview including its persisted source partition and completion reduction.7 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`examples`	yes	`array<EvalExample>`	-	Examples.
`stats`	yes	`SynthesisStats`	-	Stats. Synthesis input, filtering, sampling, output, scorer, and effective-seed counts.
`suite_name`	yes	`EvalResourceName`	-	Suite name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`default_scorer_kind`	yes	`NonEmptyString`	-	Default scorer kind. Text containing at least one character.
`aggregation`	yes	`EvalAggregation`	-	Aggregation. Explicit reduction from k raw completions to one independent example statistic.
`completions_per_example`	yes	`integer`	minimum 1; maximum 128	Completions per example.
`source_split`	yes	`DatasetSplit`	-	Source split. Persisted dataset partition.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "SynthesisPreview"
}

SynthesisPreviewBodyPreview a suite synthesized from one persisted partition without saving it. Source defaults to holdout.12 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`suite_name`	yes	`EvalResourceName`	-	Suite name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`description`	no	`String \| null`	-	Description.
`strategy`	no	`SynthesisStrategy`	-	Strategy. How a conversation is decomposed into eval examples.
`scorer`	no	`ScorerChoice`	-	Scorer. Synthesis scorer policy. Fixed scorers use a nested scorer field so both tagged unions remain unambiguous.
`generation`	no	`EvalGenerationParams`	-	Generation. Eval decode parameters. Omitted thinking budgets inherit; explicit null is unlimited; finite values include zero.
`aggregation`	no	`EvalAggregation`	-	Aggregation. Explicit reduction from k raw completions to one independent example statistic.
`sampling`	no	`Sampling`	-	Sampling. Synthesis sampling and filtering controls.
`system_prompt`	no	`String \| null`	-	System prompt.
`strip_system_prompt`	no	`Boolean`	-	Strip system prompt. Either true or false.
`suite_tools`	no	`array<AnyJson>`	-	Suite tools.
`head_n`	no	`NonNegativeInteger`	-	Head n. A whole number greater than or equal to zero.
`source_split`	no	`DatasetSplit`	-	Source split. Persisted dataset partition.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "SynthesisPreviewBody",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

SynthesisStatsSynthesis input, filtering, sampling, output, scorer, and effective-seed counts.11 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`trajectories_seen`	yes	`NonNegativeInteger`	-	Trajectories seen. A whole number greater than or equal to zero.
`trajectories_used`	yes	`NonNegativeInteger`	-	Trajectories used. A whole number greater than or equal to zero.
`examples_generated`	yes	`NonNegativeInteger`	-	Examples generated. A whole number greater than or equal to zero.
`skipped_no_target`	yes	`NonNegativeInteger`	-	Skipped no target. A whole number greater than or equal to zero.
`skipped_prompt_too_long`	yes	`NonNegativeInteger`	-	Skipped prompt too long. A whole number greater than or equal to zero.
`skipped_target_too_long`	yes	`NonNegativeInteger`	-	Skipped target too long. A whole number greater than or equal to zero.
`skipped_duplicate`	yes	`NonNegativeInteger`	-	Skipped duplicate. A whole number greater than or equal to zero.
`skipped_strategy_match`	yes	`NonNegativeInteger`	-	Skipped strategy match. A whole number greater than or equal to zero.
`sample_kept`	yes	`NonNegativeInteger`	-	Sample kept. A whole number greater than or equal to zero.
`effective_seed`	yes	`DecimalU64`	-	Effective seed. An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.
`auto_scorer_histogram`	yes	`object`	-	Auto scorer histogram.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::synthesis::SynthesisStats"
}

SynthesisStrategyHow a conversation is decomposed into eval examples.0 fields

Type: string. Constraints: enum "final_assistant", "first_assistant_turn", "every_assistant_turn", "tool_call_predict", "tool_response_followup", "end_of_trajectory_answer".

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "SynthesisStrategy"
}

SynthesizeBodyPersist a suite synthesized from one partition and optionally queue it against adapters. Source defaults to holdout.13 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`suite_name`	yes	`EvalResourceName`	-	Suite name. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`description`	no	`String \| null`	-	Description.
`strategy`	no	`SynthesisStrategy`	-	Strategy. How a conversation is decomposed into eval examples.
`scorer`	no	`ScorerChoice`	-	Scorer. Synthesis scorer policy. Fixed scorers use a nested scorer field so both tagged unions remain unambiguous.
`generation`	no	`EvalGenerationParams`	-	Generation. Eval decode parameters. Omitted thinking budgets inherit; explicit null is unlimited; finite values include zero.
`aggregation`	no	`EvalAggregation`	-	Aggregation. Explicit reduction from k raw completions to one independent example statistic.
`sampling`	no	`Sampling`	-	Sampling. Synthesis sampling and filtering controls.
`system_prompt`	no	`String \| null`	-	System prompt.
`strip_system_prompt`	no	`Boolean`	-	Strip system prompt. Either true or false.
`suite_tools`	no	`array<AnyJson>`	-	Suite tools.
`force`	no	`Boolean`	-	Force. Either true or false.
`run_against`	no	`array<String> \| null`	-	Run against.
`source_split`	no	`DatasetSplit`	-	Source split. Persisted dataset partition.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "SynthesizeBody",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

SynthesizeDatasetResponsePersisted suite summary plus any auto-queued eval job IDs.3 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`suite`	yes	`EvalSuiteSummary`	-	Suite. List-view projection of a registered suite.
`stats`	yes	`SynthesisStats`	-	Stats. Synthesis input, filtering, sampling, output, scorer, and effective-seed counts.
`queued_eval_job_ids`	yes	`array<String>`	-	Queued eval job IDs.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "SynthesizeDatasetResponse"
}

TagBreakdownPer-tag pass counts and interval.4 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`num_examples`	yes	`NonNegativeInteger`	-	Num examples. A whole number greater than or equal to zero.
`num_pass`	yes	`NonNegativeInteger`	-	Num pass. A whole number greater than or equal to zero.
`pass_rate`	yes	`UnitInterval`	-	Pass rate. A score or rate from 0 through 1, inclusive.
`confidence_interval`	yes	`PassRateConfidenceInterval`	-	Confidence interval. Wilson confidence interval for a pass rate.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::TagBreakdown"
}

ToolBreakdownPer-tool pass counts and interval.4 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`num_examples`	yes	`NonNegativeInteger`	-	Num examples. A whole number greater than or equal to zero.
`num_pass`	yes	`NonNegativeInteger`	-	Num pass. A whole number greater than or equal to zero.
`pass_rate`	yes	`UnitInterval`	-	Pass rate. A score or rate from 0 through 1, inclusive.
`confidence_interval`	yes	`PassRateConfidenceInterval`	-	Confidence interval. Wilson confidence interval for a pass rate.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::ToolBreakdown"
}

ToolCallWeightsRelative tool-call name, structure, and content weights.3 fields

Type: object.

Field	Required	Type	Constraints and default	Description
`name`	yes	`FiniteNumber`	-	Name. A finite JSON number; NaN and infinities are not valid JSON.
`structure`	yes	`FiniteNumber`	-	Structure. A finite JSON number; NaN and infinities are not valid JSON.
`content`	yes	`FiniteNumber`	-	Content. A finite JSON number; NaN and infinities are not valid JSON.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "kiln_eval::scorers::ToolCallWeights",
  "x-kiln-unknown-field-policy": "accepted_and_ignored"
}

UnitIntervalA score or rate from 0 through 1, inclusive.0 fields

Type: number. Constraints: minimum 0; maximum 1.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "f32"
}

ValidateJudgmentResponseHeld-out judge-validation admission receipt.5 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`status`	yes	`any`	const "queued"	Status.
`eval_job_id`	yes	`NonEmptyString`	-	Eval job ID. Text containing at least one character.
`effective_seed`	yes	`DecimalU64`	-	Effective seed. An unsigned 64-bit value encoded as decimal text so JavaScript clients do not round it.
`validation_suite`	yes	`EvalResourceName`	-	Validation suite. A non-blank suite, dataset, or judgment name without path separators or the sequence '..'.
`warnings`	yes	`array<String>`	-	Warnings.

Kiln contract annotations

These machine-readable annotations are copied exactly from this schema node.

{
  "x-kiln-field-schema-status": "complete",
  "x-kiln-rust-type": "ValidateJudgmentResponse"
}