Training a small model on the arcade, three ways
This page shows how to train a small language model on the arcade's environments jointly, with rewards the environments themselves compute. The same recipe is walked through three times — in TRL, in prime-rl, and against the Tinker API via SkyRL.
This walkthrough uses the arcade's recorded fourteen-server gallery, indexed
in arcade/README.md. Ten are games and four are task
generators, and every one of them speaks the same JSON over the same socket. No
framework below is asked to learn a new protocol: each already speaks OpenEnv,
and these servers answer it. The separate math room adds
eight one-step, text-first environments and its own manifest; start those with
make math-up.
Each walkthrough is the same three steps. Start the servers with make arcade-up. Point the adapter this repository ships at the manifest that target
wrote. Train.
One policy, several servers
Joint training here means one model, several environments, and one update. A batch of episodes is drawn from every server in the manifest at once, each episode is scored by the server that minted it, and the model is updated from the mixed batch.
Four terms, defined once. An episode is one run of the loop: a reset, then
steps, until done. A rollout is one episode played by the model being
trained. A verifiable reward is a number a program computed from the answer,
rather than a judge's opinion of it. The four task servers compare the answer
they are handed against the one they computed at reset, and pay 1 or 0. GRPO
is group-relative policy optimisation: sample several rollouts of the same task,
score them all, and push the model toward the ones that beat the group's own
mean. All three walkthroughs below run GRPO or a close relative.
The reason a language model fits at all is that the interface is text. Every one
of the fourteen arcade servers prints the whole decision into one observation
field, input_text — the situation, the board where there is one, and how to
reply on the last line. It carries nothing the other fields do not carry, so a
prompt is a paste and never a rewrite;
arcade/CHARTER.md ↗ §7a writes out all five clauses of
that promise. Every server reads the answer back out of a one-field action:
{"answer": "..."} for the task servers, {"guess": "crane"} for wordle,
{"col": 3} for connect four. So the model needs no action head, no image
encoder and no tokenizer surgery — it needs to emit text, which is what a chat
model already does. Where the field is a number, the adapter takes the first run
of digits out of the reply; the declared type says how the environment reads an
answer, not who can write one.
Nothing to hand-write
An earlier draft of this page asked a reader to type five things before they could train anything. A table of ports and action fields. A prompt for wordle, whose observation carried no readable task text. Action-field discovery. A reset special case, because wordle took a seed and the task servers took a difficulty too. And an episode-return sum, because a framework that reads one number off a finished episode reads the last step's reward.
None of the five is protocol. Four are on the wire, and the fifth is one line of arithmetic. So all five moved out of the walkthrough:
| What a reader used to write | Where it lives now |
|---|---|
| the table of ports | build/rel/arcade.json, which make arcade-up writes |
| the wordle prompt | input_text, which all fourteen servers publish |
| action-field discovery | the one action property that is not metadata, off GET /schema |
| the reset special case | {"seed": N, "difficulty": D}, sent to every server |
| the episode-return sum | Session.total, over the rewards the server paid |
interop/openenv_text.py ↗ is that whole column. It
reads the manifest, describes each server off its own schema, coerces a model's
reply onto whichever field that schema named, and plays one episode over a
WebSocket. It imports nothing but the standard library, and no branch in it
names an environment. The three files below are the framework halves.
Verified here, and illustrative here
Verified: the environment side. The servers, the recorded episodes and the
reward channel are arcade/recordings.txt ↗, quoted.
make record-arcade writes that file by running the real binaries, and
make site refuses any transcript line the arcade page cannot find in it. The
wire has already trained a learner from the outside, too. make interop's
eighth oracle runs epsilon-greedy inside upstream's own EnvClient
(interop/train_bandit.py ↗) against
arcade/bandit.c ↗, and fails the suite unless the run
improved. An oracle here is an independent judge, usually upstream's own code.
Verified: the adapter layer, leg by leg. make adapters drives the shipped
files against a running arcade, with no GPU and no model weights. It is four
legs, and they are deliberately unequal: what a CPU can test differs per
framework, and the table says which is which. Wherever a model would be, a
script stands in — six real Wordle words, or a bare 1.
| Leg | What it asserts | What it needs |
|---|---|---|
| S | the five shipped .py files parse; train_trl.py imports trl, datasets and transformers inside main(); two of the five import nothing beyond the standard library and their sibling | nothing |
| 0 | each of the fourteen servers names exactly one action property that is not metadata, with a declared type; all fourteen publish input_text with an action field there is a coercion rule for; every server that declares the field returns a non-empty one; every server accepts {seed, difficulty, episode_id}, and the preset is read where it means something and ignored where it does not; Session.total is the sum of what the server paid; dyna ends in three steps and wordle inside six; six real words score above zero on wordle, and a constant 1 over sixty-four seeds of logic has a mean between 0.3 and 0.7; connect four is played to the end by a scripted answerer writing a sentence per column, and both forfeits score zero without raising | the arcade binaries, and a manifest |
| A | upstream's own MCPHarnessAdapter drives our TRL session over a real WebSocket; the first user turn is the server's input_text, pasted whole; verify().env_reward is the episode return, and it is the sum of what the server paid rather than its last step; connect four plays to the end on that path too, and both forfeits come back as a scored result rather than an exception | the venv sh interop/setup.sh builds, for openenv |
| D | both Tinker builders survive a pickle round trip, which is what a rollout worker does to them | chz and tinker_cookbook |
Leg A is the one that drives production code nobody here wrote. Leg D skips on
a machine without the cookbook, which is most machines. A skip names the missing
thing and the command that supplies it, and a skip is never a failure: arcade/
is outside this repository's proof boundary, and every target that touches it
keeps that promise.
make adapters is not an oracle and does not claim to be one. One leg of the
four runs somebody else's code; the other three hold what this repository wrote
against what this repository wrote. That is a smoke test rather than an
independent judge, so the target wears no oracle label and the live oracle count
does not move.
Illustrative: the training runs. No GPU ran here. No optimizer step was
taken, no model weights were loaded, and every reply in every leg was scripted.
make adapters prints its own list of what it did not test at the end of every
run, passed or skipped: the trainers, the model, the prime-rl run, the renderer.
Each config and flag below is written from the named files of the frameworks'
own sources, cloned under reference/ at pinned revisions: trl c285dc1,
prime-rl f50ec5c, verifiers 8233c84, SkyRL b5fdfd0, tinker-cookbook
4973dc0. The reference is pinned too, at 60ecc7b7 — the TRL walkthrough
builds on its openenv.core.harness. There is no learning curve to show, and no
training-performance figure appears anywhere on this page.
The three claims are not levelled up to the strongest one. TRL's session half is
driven end to end here. prime-rl's config is written by a program that reads
each server's schema at generation time, and make adapters reads that
program's source without running it. The Tinker adapter has one assertion
against it, and that assertion is about pickling.
That is the whole hedge. The walkthroughs below are written in the indicative, and nothing after this section claims a run that did not happen.
The task set
The adapters keep all fourteen servers. Nothing picks them by name.
describe_all keeps a server on two clauses, both read off GET /schema. Its
observation schema must declare input_text, and its action field must be a
declared type there is a coercion rule for. There are two such types, a string
and an integer, and every arcade server passes. Five answer with a string; the
other nine are the games, which answer with a small integer the adapter reads
out of the reply.
The five walked through below are a worked subset and not a filter. They are
here because their rewards are the simplest to read: a task server pays 1 or 0
on exact match, and there is nothing to argue about. Point an adapter at the
whole manifest and it plays the games too; --envs narrows it to these five.
The games also keep the classical learners in
arcade/train.c ↗, which is a different thing from what this
page does and remains the fastest way to check a game works.
| Server | Port | Action | The skill it pays for | Reward |
|---|---|---|---|---|
wordle | 8805 | {"guess": "crane"} | deduction under feedback | shaped: partial credit per guess, a bonus for solving and for solving early |
dyna | 8821 | {"answer": "B"} | state tracking | 1 or 0 per query, three queries per task |
inducto | 8822 | {"answer": "A B C"} | rule induction | 1 or 0, exact match |
logic | 8823 | {"answer": "1"} | circuit evaluation | 1 or 0, exact match |
filtro | 8824 | {"answer": "alpha"} | source separation | 1 or 0, exact match |
The Action column is the shape of the wire, and it is here because a reader wants it. No adapter reads it from here. Each one asks the server what its action field is called, and gets the answer in the same request that carries that field's own description.
These five earn their place: their rewards are computed by the server, and their
tasks are minted fresh from the reset seed. A training run therefore never sees
the same problem twice, and never needs a dataset file. The four task servers
take a difficulty preset as well as a seed: {"seed": N, "difficulty": 1|2|3},
and the knob moves the real generator parameters. Their episodes are one step
long, except dyna's, which asks three questions of one arrangement and grades
each on its own step.
The ten games are in reach of the same adapters, and they are worth training
against. Connect four is the one to read first. Its observation renders the
board, lists the columns with room, and ends Reply with one digit, the column to drop into. That is a whole decision in text. The answer happens to land on
a field the schema calls an integer, which is the environment's business and
not the model's. make adapters plays it end to end, and that leg is where the
integer path is proved. Bandit is the smoke test to run
first against any new harness, because oracle 8 already trains it: if a harness
cannot learn the bandit, the harness is what is broken.
A reply the environment cannot use
Two things a model can send end an episode without earning anything, and the two adapters this repository ships as code treat both the same way. A forfeited step scores zero and ends that rollout. It is never an exception a trainer has to catch.
The first is a reply there is nothing in to put on the action field: no digits
at all, where the field is a number. openenv_text.action_of refuses to invent
one, and nothing reaches the socket. The second is an action the environment ran
and would not take — a full connect four column, a snake turn outside 0..2. The
server answers an EXECUTION_ERROR frame, the episode does not advance, and the
socket survives.
Two things follow, and both are deliberate.
Nothing is substituted for the model's own reply. Not the field's declared
default. Snake's turn defaults to 1, which is a legal move that could score
well by luck. A reward the reply did not earn would teach the model that a shrug
is an answer, so zero is the only number a forfeited step can honestly be worth.
Only the model's own failure is scored. A CAPACITY_REACHED frame means the
trainer asked for more concurrent sessions than --max-sessions allows. A run
that scored that zero would train on a flat line instead of saying so. That
frame still raises, and so does every other one.
prime-rl is the exception, and it is one this page cannot fix. That route runs
no code of ours at rollout time, so a reply its parser cannot place raises
inside OpenEnvEnv.run rather than scoring zero. The generated config says so
in its own header.
The environment side, once
Build the arcade, then start the recorded gallery. The build includes the math room as well; the launcher below starts only the fourteen servers on the ports their recordings pin down and writes the manifest every adapter below reads:
$ make arcade
arcade: 25 binaries in build/rel/bin — algebra bandit blackjack cartpole chance change connect4 discrete dyna evidence filtro g2048 inducto kuhn logic mathlab maze measure pong reason snake space synthlab train wordle
plus oe_probe for poking them by hand
try: make train-bandit (or run one by hand: build/rel/bin/bandit --port 8801 &)
$ make arcade-up
arcade-up: bandit http://127.0.0.1:8801
arcade-up: cartpole http://127.0.0.1:8802
arcade-up: snake http://127.0.0.1:8803
arcade-up: g2048 http://127.0.0.1:8804
arcade-up: wordle http://127.0.0.1:8805
arcade-up: blackjack http://127.0.0.1:8806
arcade-up: maze http://127.0.0.1:8807
arcade-up: pong http://127.0.0.1:8808
arcade-up: connect4 http://127.0.0.1:8809
arcade-up: kuhn http://127.0.0.1:8810
arcade-up: dyna http://127.0.0.1:8821
arcade-up: inducto http://127.0.0.1:8822
arcade-up: logic http://127.0.0.1:8823
arcade-up: filtro http://127.0.0.1:8824
arcade-up: 14 servers on 127.0.0.1, manifest build/rel/arcade.json
make arcade-down stops them
The first command builds every downstream arcade program. The arcade-up
block is that target's own output, copied from a run. It is the stable,
recorded-gallery manifest; math-up writes a separate math.json so a training
run can mix the two manifests deliberately instead of changing an existing
experiment by surprise.
Five things about that launch decide how the rest goes.
- The manifest is an address book and nothing else.
build/rel/arcade.jsonmaps a server's name to its base URL. Not the action field, not the episode length, not the concurrency cap:GET /schemacarries the first, no conformant environment publishes the second, and the third is the trainer's. It is written last, after every server has answered, so its existence means the arcade is up.build/is gitignored andmake arcade-downremoves the file, so it cannot go stale in a checkout. - Concurrency is a flag. Each server accepts eight WebSocket sessions at
once by default, and
--max-sessions Nmoves it. Size it at or above the number of rollouts a trainer keeps in flight, or the surplus clients are refused with aCAPACITY_REACHEDframe that says the limit out loud. - The seed is the dataset.
reset {"seed": N}mints a task no file has ever held, and the same seed twice comes back byte-identical. Give each rollout a seed and the batch is the sample. - The episode is the unit. Reset, step until
done, then reset again. Afterdonea server returns the terminal observation unchanged with rewardnull, so a trainer bug shows up as a flat line rather than a haunted curve. - Drive the WebSocket.
POST /resetis stateless by design, so an episode needs/ws. Every adapter below opens it and stays on it, which is what dyna's three steps and wordle's six guesses need.
Everything above this line is the environment side, and it runs here. Below,
each walkthrough's adapter is exercised by make adapters. None of the three
trainers is.
TRL
TRL is Hugging Face's post-training library, and it ships a real OpenEnv integration rather than an example that has to be maintained by hand.
Prerequisites. One CUDA GPU upward — two is more comfortable, one for vLLM
and one for the trainer. trl with its vLLM extras, openenv-core, and a small
instruction-tuned model in the Qwen3 class. The arcade must be reachable from
the trainer host. Nothing here is CPU-viable except a dry run with a tiny model.
TRL: the run
make arcade-up
trl vllm-serve --model Qwen/Qwen3-1.7B --port 8000
python interop/train_trl.py --manifest build/rel/arcade.json \
--model Qwen/Qwen3-1.7B --vllm-url http://localhost:8000
That is the walkthrough. interop/train_trl.py ↗ is
one session factory and one main(), and it holds no table.
Flags, with their defaults: --manifest build/rel/arcade.json, --model,
--vllm-url http://localhost:8000, --envs (empty means every text-profiled
entry), --difficulty 2, --steps 500, --generations 8, --inflight 8,
--max-turns 6. --inflight is how many sessions are open at once, so it is
the number to match with --max-sessions on each server.
TRL: what the file does
TRL's OpenEnv harness lives in trl/experimental/async_grpo/openenv_harness.py.
It drives an OpenEnv ResourceSessionFactory: one isolated session per rollout,
the session exposes MCP-style tools, and the reward the trainer uses is the one
the session's verify() hands back. OpenEnv ships the bridge from a step/reset
server to that interface, openenv.core.harness.StepEnvSessionAdapter.
ManifestSessionFactory is what fills it in.
__init__ reads GET /schema once per manifest entry and keeps what it learned.
It drops the entries a text model cannot play and names them on stderr, because
a filtered-out environment is the thing a reader is most likely to think they
are training on. create then opens a client, builds one tool from the schema,
and hands both to the adapter.
The seed is the task. _HarnessRolloutLoop._run_session passes the GRPO
group id as the seed, so every generation in a group meets the same server at
the same seed. That is what makes a group's rewards comparable. Across groups
the seed rotates through the survivors, and that rotation is the joint training.
The tool is called act, not the field it fills. RESERVED_TOOL_NAMES is
{"reset", "step", "state", "close"}, so a field-named tool would break on an
environment whose action field was called step. The model puts its answer in
value instead, and the adapter moves it onto whatever the schema said the
field was. The tool's description is the schema's own description of that field,
which is the prose a hand-written table used to supply.
Summing in verify is safe on this path and not on every path, and the file
says so at the method that does it. TRL calls session.verify() itself and
reads env_reward straight off the result. OpenEnv's own collection helpers
take a different route: _resolve_env_reward raises if verify.env_reward
disagrees with the last tool result's reward. A summing verify is for the
trainer that reads it directly.
The factory is picklable, which is not decoration. AsyncRolloutWorker forwards
every constructor argument to a spawned child process, and raises a TypeError
up front if any of them is a lambda or a closure. So the factory holds a path,
a comma-joined list and two ints, and the per-session callables are built inside
create, which runs in the child.
The simpler wiring. If rollouts do not need to leave the trainer process,
trl.GRPOTrainer takes an environment_factory instead. It has two forms, and
they are worth telling apart. Pass a dict of factories and every example must
carry an environment field naming one of them; only that environment's tools
are exposed in its prompt. Pass a single factory and every example uses it,
one instance per rollout. Either way the environment class defines a reset
method and its tool methods. If it also defines get_reward, the trainer's own
docstring says that "the environment owns the reward", and reward_funcs "then
becomes optional".
examples/scripts/openenv/multi_env.py mixes two environments in one run, and
it takes the second route rather than the first. One MultiEnv class holds both
clients, the dataset carries an "env" column, and MultiEnv.reset reads that
column to choose which client to open. It also passes a reward function per
environment rather than leaning on get_reward. It is worth reading as the
worked mixture; it is not an example of the dict form. Either route would serve
the arcade. The harness above is the deeper cut, for when the sessions have to be
isolated processes.
What to expect. Nothing until vLLM is up: the trainer's child process holds the rollout loop and blocks on the first generation. Then completions in the log, and one reward column named for the harness. A GRPO group in which every rollout scored the same produces no gradient, and early on that is most of them.
prime-rl
prime-rl is Prime Intellect's RL trainer, and its environments are verifiers environments. That is the useful fact here: verifiers already ships an OpenEnv taskset, so the arcade needs a config rather than an adapter.
Prerequisites. The prime-rl uv workspace, a trainer GPU and an inference GPU
(its own basic configs assume two), verifiers with its openenv extra, and the
arcade reachable from the orchestrator host.
prime-rl: the taskset
verifiers/v1/tasksets/openenv/taskset.py defines OpenEnvTaskset,
OpenEnvConfig and OpenEnvEnv. The config takes either an OpenEnv environment
id to launch, or a base_url of a server already running. The arcade wants the
second. Its resets field is a list of keyword sets, one finite task per entry,
passed straight to the environment's reset. So a difficulty sweep over seeds
is a list literal:
# one task family, and the shape the generator below emits
[env.taskset]
id = "openenv"
base_url = "http://127.0.0.1:8823"
resets = [
{ seed = 1, difficulty = 1 },
{ seed = 2, difficulty = 2 },
{ seed = 3, difficulty = 3 },
]
OpenEnvEnv.run is the loop, and it is worth reading before trusting it. It
fetches GET /schema over HTTP, then opens the player's interaction and sends
the observation and that action schema as JSON. It parses each reply with
parse_action, steps the client, and sums the per-step rewards onto the trace
as openenv_reward. So prime-rl is the framework that needs no adapter from us:
four of the five pieces of boilerplate are already off the wire, and the fifth
is the sum it already does.
The reply that works everywhere is a JSON object. parse_action calls
json.loads on the reply first, and keeps the raw string only on a decode
error. A bare value is wrapped only when the action schema names exactly one
required field. Wordle declares required: ["guess"], so a bare crane works
there. The four task servers declare no required key, because their one field
carries a default, so the same reply raises rather than scoring zero.
And widening that shortcut to them would not help. A bare value keeps
whatever JSON type it parsed as. logic's whole answer alphabet is {"0", "1"}
and both parse as integers, and a string field answers an integer with HTTP 422
— measured against build/rel/bin/logic. So the shortcut would advertise a path
that is broken for every one of that environment's answers.
arcade/CHARTER.md ↗ §7b records the schema change that
was proposed here and the four reasons it was refused.
One JSON object naming the field works everywhere, with its value typed the way
the schema types the field. The action schema the taskset ships to the model on
every turn names that field, so the model has what it needs to write the object.
parse_action is upstream's parser and its behaviour is upstream's decision, so
it is named here rather than patched. The shipped adapters take the other
route. They read the schema's shape rather than its required list, send a
string field the raw text, and take the first run of digits for an integer
field. A wrong answer then scores zero instead of killing the episode.
And a reply this route cannot place ends the run rather than the episode.
That is the one thing prime-rl does not get from the section above. parse_action
raises where the schema names no single required field, and an action the
environment refuses comes back as an error frame the client raises on. Both are
upstream's, both are on the far side of a config file, and there is nothing a
generated TOML can do about either. The generated header says so where a reader
of that file will meet it.
One more thing that schema decides. run switches to an MCP list_tools path
when the action schema's title is Action, CallToolAction or
ListToolsAction. The arcade's titles are WordleAction, DynaAction,
Connect4Action and eleven more of the same shape, so that branch is not taken
and the ordinary step path runs. The generator below reads each title and says
so on stderr if one of those three ever turns up.
prime-rl: the run
make arcade-up
python interop/train_prime.py --manifest build/rel/arcade.json \
--model Qwen/Qwen3-1.7B --seeds 64 --difficulties 1,2,3 > arcade-rl.toml
uv run rl @ arcade-rl.toml
interop/train_prime.py ↗ is not a trainer. It is a
config writer, and it imports nothing but the standard library. Joint training
in prime-rl is several [[orchestrator.train.env]] blocks in one run config.
Each block names its own taskset, and each may carry a ratio that weights how
often the buffer draws from it. The generator emits one block per surviving
manifest entry, in a run config whose shape follows
configs/basic/wordle/rl.toml, prime-rl's own two-GPU example.
Two things a hand-typed config cannot promise.
It cannot be wrong about the environment. Every entry is described from its
own GET /schema at generation time. An entry whose observation declares no
input_text, or whose action field is a declared type there is no coercion rule
for, gets no block, and what was dropped is named on stderr.
It ships the resets list. OpenEnvConfig.resets defaults to [{}], which
is one task at the environment's own default seed, forever. A finite resets
list is what bounds the taskset, so the generator emits seeds crossed with
difficulties, per environment.
The GPU counts and seq_len are emitted as a skeleton, under a comment saying
they are the reader's. The batch size and the group size are flags with
defaults. No environment publishes any of those numbers and no manifest can
supply them. The null harness is pinned on every block: the arcade needs no
agent scaffolding, only a model that answers. Every env name must be unique, and
the orchestrator reserves agg for its cross-environment metric rollup, so the
generator refuses a manifest key by that name.
uv run rl @ arcade-rl.toml is prime_rl.entrypoints.rl:main, and it launches
the trainer, the orchestrator and the inference server together. Under the hood
prime_rl/orchestrator/envs.py builds a TrainEnv per block and spawns a
verifiers env server for each. The orchestrator owns the taskset: tasks are
loaded on its side, once, and each dispatched rollout ships its task's data on
the request. In that file's own words, "The server answers one Episode per
env-rollout". Envs.shutdown terminates them all on exit.
What to expect. One log line per environment at startup, giving each one's
task count. Per-environment reward metrics from then on, keyed by the name in
each block. That is the point of naming them: a mixture whose metrics are pooled
tells you the average moved, not which task moved it. A finite resets list is
a finite taskset, so bound the run by max_steps or supply enough seeds.
Tinker, via SkyRL
Tinker is Thinking Machines' training API: the loop runs on your machine, and
sampling, forward-backward and the optimizer step are calls. SkyRL implements the
same API on your own GPUs, so the same script runs against either — one
base_url apart.
Prerequisites. Either a Tinker API key, or SkyRL installed with its tinker
and fsdp extras on a machine with GPUs. tinker-cookbook for the RL loop and the
renderers. The arcade must be reachable from wherever the loop runs, which is
the local machine and not the service.
Tinker: the run
make arcade-up
uv run --extra tinker --extra fsdp -m skyrl.tinker.api \
--base-model "Qwen/Qwen3-1.7B" --backend fsdp
TINKER_API_KEY=tml-dummy python interop/train_tinker.py \
manifest=build/rel/arcade.json model_name=Qwen/Qwen3-1.7B \
base_url=http://localhost:8000
Against Thinking Machines' hosted service, set TINKER_API_KEY and leave
base_url unset. Against SkyRL, start its Tinker API server first and point the
script at it — SkyRL's own quickstart says no code changes are needed beyond the
URL. base_url is a real field of tinker_cookbook.rl.train.Config, and not an
invention of this page.
Tinker: what the file does
tinker-cookbook's RL types are in tinker_cookbook/rl/types.py. There are
three: an Env with async initial_observation and step, an
EnvGroupBuilder that makes the group GRPO centres rewards across, and an
RLDataset that hands the training loop a batch of those builders.
interop/train_tinker.py ↗ writes all three
generically, and adds the @chz.chz config and chz.entrypoint the cookbook's
own recipes use.
The renderer cannot be removed, only hidden. A Tinker observation is a
tinker.ModelInput and not a string, so something has to turn the environment's
text into tokens and the sampled tokens back into a reply. That is one class,
Seam, and it is the only thing in the file that knows what a tokenizer is.
ManifestGroupBuilder.make_seams is the one method to override to frame the
environment's text some other way: a system turn, a few-shot prefix, a renderer
of your own. Nothing that speaks to a server has to move.
The mixture is a lattice, not a draw. Member j of batch i is slot
i * batch_size + j, and the slot alone picks the server, the seed and the
difficulty. Two things follow. Any batch index replays exactly, with no
generator state to carry. And every server-and-difficulty pair comes up the same
number of times, so the mixture is balanced by construction.
Nothing sums. TrajectoryGroup.get_total_rewards already adds the
per-transition rewards, so step returns each step's reward as the environment
paid it. The episode's running total appears once, in a log line, and never as a
reward.
The builders pickle, and that is a requirement rather than a preference.
rl/types.py says so, because a builder is serialized out to rollout workers.
So the builder holds the manifest path, a comma-joined env list, ints, and the
Spec read off GET /schema at startup. Every socket, the renderer and the
tokenizer are built on the far side of that boundary. make adapters leg D is
the assertion, and the cookbook ships a pickle test of its own as the precedent.
SkyRL's skyrl/tinker/api.py is a FastAPI server that records each request and
returns a future. A background engine batches those requests onto the backend in
skyrl/backends/, which is where forward_backward, forward and optim_step
become real work on Ray workers. --backend-config takes a JSON dictionary of
dotted overrides for the underlying trainer config, and the GPU counts and the
inference engine settings live there.
What to expect. The rollouts run locally and the model does not, so watch the gap between them. A batch spends most of its wall clock in sampling, and the arcade servers are not the queue. If the loop looks idle, check the Tinker service, not the environments.
What none of the three removes
These are the things a shipped adapter cannot take off a reader's hands, and a reader deserves to know which they are.
- TRL needs a script either way. Its CLI registers
grpo, and that subcommand runstrl/scripts/grpo.pywith the ordinaryGRPOTrainer. There is no entry point that takes a rollout worker. We ship the script so the reader writes none; somebody still wrote one. - TRL needs a dummy dataset.
Trainerderives its step count from the dataset unlessmax_stepsis set, so the empty rows are a load-bearing wart. They are inside the shipped file, which is the most that can be done to them. - prime-rl needs a run config. GPU counts, batch size, group size. No manifest can supply those, and the generator writes them as a skeleton to edit.
- Tinker needs the renderer and the tokenizer. Both belong to the model, not to the environment.
- All three need concurrency sized by hand.
--max-sessions Non each server has to match the trainer's in-flight number, and that number is the trainer's. - All three need a turn cap. A conformant environment does not publish its episode length, so nothing can discover it. Wordle wants six; a one-step task server ends before any cap does.
What to expect
The environments answer faster than any learner will ask them to. The recorded
runs of the five servers here span 30155 steps/s for logic to 46154 steps/s for
dyna, on one core with no GPU — measured by make record-arcade and printed
into arcade/recordings.txt ↗. A model that generates
a few hundred tokens per action is several orders of magnitude slower than that.
The environment will not be the bottleneck, and if a run stalls the environment
is not the first place to look.
Four of the five rewards are an integer, 1 or 0, on exact match. That is a narrow, honest signal with no partial credit to drift on: a rollout either answered the question the server asked or it did not. It also means early batches are mostly zeros, and a GRPO group in which every member scored zero contributes no gradient at all. Expect the first visible change to be format compliance: the model learning to answer in the shape its harness asks for, and to add nothing else. Expect difficulty 1 to move before difficulty 3. Difficulty 1 is a shorter circuit, a smaller graph, more examples of the hidden rule; difficulty 3 is the same task with the slack taken out.
A run that includes the games has one more thing in its early batches: forfeits. A game names its legal moves in its own prompt, and a model that has not yet learned to read them will answer outside the range. Those rollouts score zero and end, so they look exactly like a wrong answer in the reward column, which is what they are.
One last line, repeated because it is the hedge this page runs on. The
environment side runs here. So does each adapter above, as far as a machine with
no GPU can drive one, and make adapters names in writing whatever it had to
skip. The trainers are the part that did not run. Nothing here claims a run that
did not happen.