Sort anything with Jev.

Jev judges “which of these two is better?” for a fraction of a cent. pairsort turns its answers into one ranking with honest probabilities.

# pip install pairsort
import pairsort

ideas = ["A CSV-to-chart CLI",
         "A blockchain to-do app",
         "A PR-urgency ranker",
         "A local-first notes app",
         "A tweeting fridge magnet"]
r = pairsort.sort(ideas,
      "Which is most useful?")
r.best     # 'A local-first notes app'
r.scores   # P(best) for each idea

# no install needed
uvx pairsort sort - \
  "Which is most useful?" <<EOF
A CSV-to-chart CLI
A blockchain to-do app
A PR-urgency ranker
A local-first notes app
A tweeting fridge magnet
EOF

Jev is the default judge: set OPENROUTER_API_KEY. Ranking six ideas took 30 Jev judgments, 0.52 s and $0.0001 (watch it). Several questions? ↓

Several questions, one ranking

Name each question. pairsort ranks by each one separately, then blends them into one answer.

r = pairsort.sort(ideas,
      useful="Which is more useful?",
      easy="Which is easier to build?")
r.best     # 'A CSV-to-chart CLI'
print(r)   # this table:

pairsort sort ideas.txt \
  useful="Which is more useful?" \
  easy="Which is easier to build?"

#	idea	useful	easy	P(best)
1	A CSV-to-chart CLI	#2	#1	54%
2	A local-first notes app	#1	#4	20%
3	A PR-urgency ranker	#3	#3	19%
4	A tweeting fridge magnet	#5	#2	5%
5	A blockchain to-do app	#4	#5	2%

Asked only “most useful?”, the notes app wins. It's also hard to build, so the blend picks the CSV CLI: #2 on useful, #1 on easy. To make one question count double, write useful:2="…" in the CLI or useful=("…", 2) in Python. Real Jev run: 40 judgments, $0.00013.

Watch Jev sort

A recorded run: six side-project ideas, every pair asked in both orders.

0 / 30 judgments cost $0.000000

Starting…

Live ranking = Bradley–Terry on the pairs so far. Recorded 2026-09-23 against typesafe/jev-1.13 on OpenRouter; the numbers are the real ones.

Jev: cheap and good

The same pairs, scored against ground truth, with Jev and three general LLM judges. Jev was the cheapest judge in every eval, by 1.5–20×, and landed within 2.3 points of the best judge every time.

eval (ground truth)	Jev	best other judge	Jev cost	others
Same answer in 4 languages	96%	94% Gemma 4 31B	5.2¢	15–21¢
Next-day stock returns, Mon	τ +0.14	+0.08 Gemma 4 31B	7.3¢	11–17¢
Exact error counts	96.4%	97.2% Gemma 4 31B	2.2¢	6.8–9.7¢
Which code runs faster	97.3%	97.9% DeepSeek V4.1 Flash	1.2¢	3.2–6.7¢
Damage ladder	94.4%	96.7% Gemma 4 31B	1.6¢	6.3–32¢
Next-day stock returns, Tue	τ +0.06	+0.08 Gemma 4 31B	—	—

Percentages are pairs ordered correctly (cross-lingual: identical answer in EN/ES/DE/JA). Market rows are Kendall τ against realized returns; neither day is strong evidence yet. Costs are what each judge actually billed for its whole eval (Tuesday re-used cached answers, so its cost is not comparable). On the 100-model Summary Showdown, Jev's 4,800 judgments cost 11¢; the other judges cost $0.37–$1.55 for the same pairs.

100 AI models, one paper, one jury

Jev and three other judges ranked 100 popular models' summaries from 8% of the possible pairs.

Loading the leaderboard…

Judge the summaries yourself → Results & method

How it works

One hard question (“rank these 100 things”) becomes many easy ones (“which of these two?”), put back together with the pairwise-coupling rule of Price, Knerr, Personnaz & Dreyfus (NeurIPS 1994).

1
Pick informative pairs
An active schedule asks about the pairs the ranking is least sure of, and stops when it settles. No all-vs-all.
2
Ask both orders
A-first and B-first are averaged: P_ij = (q(i,j) + 1 − q(j,i)) / 2 which cancels position bias exactly.
3
Couple into one ranking
PKPD Eq. 7 turns pairwise probabilities into a probability per item, even when the judge contradicts itself. Bradley–Terry takes over for large or sparse sets.

P_i = 1 / ( Σ_j≠i 1/P_ij − (K − 2) )
4
Blend questions, abstain when unsure
Each question is coupled separately, then fused. When the top two are too close to call, pairsort says so.

Worked example: a judge that goes in circles

The judge says A beats B (70%), B beats C (65%) and C beats A (55%). Everyone beats D. Counting wins gives a three-way tie; coupling uses how sure each answer was.

	A	B	C	D
A	·	.70	.45	.85
B	.30	·	.65	.80
C	.55	.35	·	.75
D	.15	.20	.25	·

38%

27%

26%

P(best) from PKPD Eq. 7. A wins its fights by wider margins; B and C are too close to separate.

The full pipeline Other judges: any OpenRouter LLM, open Jev models, your own function

Go deeper

Evidence Eight evals, ground truth first

Exact counts, stock returns, sandboxed code timings, tomorrow's weather, a damage ladder, four languages, synthetic data.

See the results → Play Summary Showdown

Judge 8 pairs of summaries and find out which AI judge thinks most like you.

Play → Science Humans vs judges

Agreement rate, Cohen's κ and Kendall τ between reader ballots and each AI judge.

How it's measured → Docs Usage & API

Budgets, active and referee schedules, meta-judges, calibration, every backend, the full CLI.

Read the docs →

What's next

Weather resolves Friday. 35 cities ranked by tomorrow's heat and rain, committed before the day starts, scored against airport sensors from 2026-09-25.
More market sessions. Two days is not evidence. Each new session gets added as it resolves.
Humans vs judges. Every submitted showdown ballot grows the agreement study.
Open Jev models, locally. Running the in-process adapters (Laya, Decider, NanoJev, Verdict) against real checkpoints.

Ideas and PRs welcome on GitHub.