miniopenenv

docs/TRAINING.md → html

Training a small model on the arcade, three ways

This page shows how to train a small language model on the arcade's environments jointly, with rewards the environments themselves compute. The same recipe is walked through three times — in TRL, in prime-rl, and against the Tinker API via SkyRL.

This walkthrough uses the arcade's recorded fourteen-server gallery, indexed in arcade/README.md. Ten are games and four are task generators, and every one of them speaks the same JSON over the same socket. No framework below is asked to learn a new protocol: each already speaks OpenEnv, and these servers answer it. The separate math room adds eight one-step, text-first environments and its own manifest; start those with make math-up.

Each walkthrough is the same three steps. Start the servers with make arcade-up. Point the adapter this repository ships at the manifest that target wrote. Train.

§ 1 · One policy, several servers

One policy, several servers

Joint training here means one model, several environments, and one update. A batch of episodes is drawn from every server in the manifest at once, each episode is scored by the server that minted it, and the model is updated from the mixed batch.

Four terms, defined once. An episode is one run of the loop: a reset, then steps, until done. A rollout is one episode played by the model being trained. A verifiable reward is a number a program computed from the answer, rather than a judge's opinion of it. The four task servers compare the answer they are handed against the one they computed at reset, and pay 1 or 0. GRPO is group-relative policy optimisation: sample several rollouts of the same task, score them all, and push the model toward the ones that beat the group's own mean. All three walkthroughs below run GRPO or a close relative.

The reason a language model fits at all is that the interface is text. Every one of the fourteen arcade servers prints the whole decision into one observation field, input_text — the situation, the board where there is one, and how to reply on the last line. It carries nothing the other fields do not carry, so a prompt is a paste and never a rewrite; arcade/CHARTER.md ↗ §7a writes out all five clauses of that promise. Every server reads the answer back out of a one-field action: {"answer": "..."} for the task servers, {"guess": "crane"} for wordle, {"col": 3} for connect four. So the model needs no action head, no image encoder and no tokenizer surgery — it needs to emit text, which is what a chat model already does. Where the field is a number, the adapter takes the first run of digits out of the reply; the declared type says how the environment reads an answer, not who can write one.

§ 2 · Nothing to hand-write

Nothing to hand-write

An earlier draft of this page asked a reader to type five things before they could train anything. A table of ports and action fields. A prompt for wordle, whose observation carried no readable task text. Action-field discovery. A reset special case, because wordle took a seed and the task servers took a difficulty too. And an episode-return sum, because a framework that reads one number off a finished episode reads the last step's reward.

None of the five is protocol. Four are on the wire, and the fifth is one line of arithmetic. So all five moved out of the walkthrough:

What a reader used to writeWhere it lives now
the table of portsbuild/rel/arcade.json, which make arcade-up writes
the wordle promptinput_text, which all fourteen servers publish
action-field discoverythe one action property that is not metadata, off GET /schema
the reset special case{"seed": N, "difficulty": D}, sent to every server
the episode-return sumSession.total, over the rewards the server paid

interop/openenv_text.py ↗ is that whole column. It reads the manifest, describes each server off its own schema, coerces a model's reply onto whichever field that schema named, and plays one episode over a WebSocket. It imports nothing but the standard library, and no branch in it names an environment. The three files below are the framework halves.

§ 3 · Verified here, and illustrative here

Verified here, and illustrative here

Verified: the environment side. The servers, the recorded episodes and the reward channel are arcade/recordings.txt ↗, quoted. make record-arcade writes that file by running the real binaries, and make site refuses any transcript line the arcade page cannot find in it. The wire has already trained a learner from the outside, too. make interop's eighth oracle runs epsilon-greedy inside upstream's own EnvClient (interop/train_bandit.py ↗) against arcade/bandit.c ↗, and fails the suite unless the run improved. An oracle here is an independent judge, usually upstream's own code.

Verified: the adapter layer, leg by leg. make adapters drives the shipped files against a running arcade, with no GPU and no model weights. It is four legs, and they are deliberately unequal: what a CPU can test differs per framework, and the table says which is which. Wherever a model would be, a script stands in — six real Wordle words, or a bare 1.

LegWhat it assertsWhat it needs
Sthe five shipped .py files parse; train_trl.py imports trl, datasets and transformers inside main(); two of the five import nothing beyond the standard library and their siblingnothing
0each of the fourteen servers names exactly one action property that is not metadata, with a declared type; all fourteen publish input_text with an action field there is a coercion rule for; every server that declares the field returns a non-empty one; every server accepts {seed, difficulty, episode_id}, and the preset is read where it means something and ignored where it does not; Session.total is the sum of what the server paid; dyna ends in three steps and wordle inside six; six real words score above zero on wordle, and a constant 1 over sixty-four seeds of logic has a mean between 0.3 and 0.7; connect four is played to the end by a scripted answerer writing a sentence per column, and both forfeits score zero without raisingthe arcade binaries, and a manifest
Aupstream's own MCPHarnessAdapter drives our TRL session over a real WebSocket; the first user turn is the server's input_text, pasted whole; verify().env_reward is the episode return, and it is the sum of what the server paid rather than its last step; connect four plays to the end on that path too, and both forfeits come back as a scored result rather than an exceptionthe venv sh interop/setup.sh builds, for openenv
Dboth Tinker builders survive a pickle round trip, which is what a rollout worker does to themchz and tinker_cookbook

Leg A is the one that drives production code nobody here wrote. Leg D skips on a machine without the cookbook, which is most machines. A skip names the missing thing and the command that supplies it, and a skip is never a failure: arcade/ is outside this repository's proof boundary, and every target that touches it keeps that promise.

make adapters is not an oracle and does not claim to be one. One leg of the four runs somebody else's code; the other three hold what this repository wrote against what this repository wrote. That is a smoke test rather than an independent judge, so the target wears no oracle label and the live oracle count does not move.

Illustrative: the training runs. No GPU ran here. No optimizer step was taken, no model weights were loaded, and every reply in every leg was scripted. make adapters prints its own list of what it did not test at the end of every run, passed or skipped: the trainers, the model, the prime-rl run, the renderer. Each config and flag below is written from the named files of the frameworks' own sources, cloned under reference/ at pinned revisions: trl c285dc1, prime-rl f50ec5c, verifiers 8233c84, SkyRL b5fdfd0, tinker-cookbook 4973dc0. The reference is pinned too, at 60ecc7b7 — the TRL walkthrough builds on its openenv.core.harness. There is no learning curve to show, and no training-performance figure appears anywhere on this page.

The three claims are not levelled up to the strongest one. TRL's session half is driven end to end here. prime-rl's config is written by a program that reads each server's schema at generation time, and make adapters reads that program's source without running it. The Tinker adapter has one assertion against it, and that assertion is about pickling.

That is the whole hedge. The walkthroughs below are written in the indicative, and nothing after this section claims a run that did not happen.

§ 4 · The task set

The task set

The adapters keep all fourteen servers. Nothing picks them by name. describe_all keeps a server on two clauses, both read off GET /schema. Its observation schema must declare input_text, and its action field must be a declared type there is a coercion rule for. There are two such types, a string and an integer, and every arcade server passes. Five answer with a string; the other nine are the games, which answer with a small integer the adapter reads out of the reply.

The five walked through below are a worked subset and not a filter. They are here because their rewards are the simplest to read: a task server pays 1 or 0 on exact match, and there is nothing to argue about. Point an adapter at the whole manifest and it plays the games too; --envs narrows it to these five. The games also keep the classical learners in arcade/train.c ↗, which is a different thing from what this page does and remains the fastest way to check a game works.

ServerPortActionThe skill it pays forReward
wordle8805{"guess": "crane"}deduction under feedbackshaped: partial credit per guess, a bonus for solving and for solving early
dyna8821{"answer": "B"}state tracking1 or 0 per query, three queries per task
inducto8822{"answer": "A B C"}rule induction1 or 0, exact match
logic8823{"answer": "1"}circuit evaluation1 or 0, exact match
filtro8824{"answer": "alpha"}source separation1 or 0, exact match

The Action column is the shape of the wire, and it is here because a reader wants it. No adapter reads it from here. Each one asks the server what its action field is called, and gets the answer in the same request that carries that field's own description.

These five earn their place: their rewards are computed by the server, and their tasks are minted fresh from the reset seed. A training run therefore never sees the same problem twice, and never needs a dataset file. The four task servers take a difficulty preset as well as a seed: {"seed": N, "difficulty": 1|2|3}, and the knob moves the real generator parameters. Their episodes are one step long, except dyna's, which asks three questions of one arrangement and grades each on its own step.

The ten games are in reach of the same adapters, and they are worth training against. Connect four is the one to read first. Its observation renders the board, lists the columns with room, and ends Reply with one digit, the column to drop into. That is a whole decision in text. The answer happens to land on a field the schema calls an integer, which is the environment's business and not the model's. make adapters plays it end to end, and that leg is where the integer path is proved. Bandit is the smoke test to run first against any new harness, because oracle 8 already trains it: if a harness cannot learn the bandit, the harness is what is broken.

§ 5 · A reply the environment cannot use

A reply the environment cannot use

Two things a model can send end an episode without earning anything, and the two adapters this repository ships as code treat both the same way. A forfeited step scores zero and ends that rollout. It is never an exception a trainer has to catch.

The first is a reply there is nothing in to put on the action field: no digits at all, where the field is a number. openenv_text.action_of refuses to invent one, and nothing reaches the socket. The second is an action the environment ran and would not take — a full connect four column, a snake turn outside 0..2. The server answers an EXECUTION_ERROR frame, the episode does not advance, and the socket survives.

Two things follow, and both are deliberate.

Nothing is substituted for the model's own reply. Not the field's declared default. Snake's turn defaults to 1, which is a legal move that could score well by luck. A reward the reply did not earn would teach the model that a shrug is an answer, so zero is the only number a forfeited step can honestly be worth.

Only the model's own failure is scored. A CAPACITY_REACHED frame means the trainer asked for more concurrent sessions than --max-sessions allows. A run that scored that zero would train on a flat line instead of saying so. That frame still raises, and so does every other one.

prime-rl is the exception, and it is one this page cannot fix. That route runs no code of ours at rollout time, so a reply its parser cannot place raises inside OpenEnvEnv.run rather than scoring zero. The generated config says so in its own header.

§ 6 · The environment side, once

The environment side, once

Build the arcade, then start the recorded gallery. The build includes the math room as well; the launcher below starts only the fourteen servers on the ports their recordings pin down and writes the manifest every adapter below reads:

$ make arcade
arcade: 25 binaries in build/rel/bin — algebra bandit blackjack cartpole chance change connect4 discrete dyna evidence filtro g2048 inducto kuhn logic mathlab maze measure pong reason snake space synthlab train wordle
        plus oe_probe for poking them by hand
        try: make train-bandit   (or run one by hand: build/rel/bin/bandit --port 8801 &)

$ make arcade-up
arcade-up: bandit    http://127.0.0.1:8801
arcade-up: cartpole  http://127.0.0.1:8802
arcade-up: snake     http://127.0.0.1:8803
arcade-up: g2048     http://127.0.0.1:8804
arcade-up: wordle    http://127.0.0.1:8805
arcade-up: blackjack http://127.0.0.1:8806
arcade-up: maze      http://127.0.0.1:8807
arcade-up: pong      http://127.0.0.1:8808
arcade-up: connect4  http://127.0.0.1:8809
arcade-up: kuhn      http://127.0.0.1:8810
arcade-up: dyna      http://127.0.0.1:8821
arcade-up: inducto   http://127.0.0.1:8822
arcade-up: logic     http://127.0.0.1:8823
arcade-up: filtro    http://127.0.0.1:8824
arcade-up: 14 servers on 127.0.0.1, manifest build/rel/arcade.json
           make arcade-down stops them

The first command builds every downstream arcade program. The arcade-up block is that target's own output, copied from a run. It is the stable, recorded-gallery manifest; math-up writes a separate math.json so a training run can mix the two manifests deliberately instead of changing an existing experiment by surprise.

Five things about that launch decide how the rest goes.

Everything above this line is the environment side, and it runs here. Below, each walkthrough's adapter is exercised by make adapters. None of the three trainers is.

§ 7 · TRL

TRL

TRL is Hugging Face's post-training library, and it ships a real OpenEnv integration rather than an example that has to be maintained by hand.

Prerequisites. One CUDA GPU upward — two is more comfortable, one for vLLM and one for the trainer. trl with its vLLM extras, openenv-core, and a small instruction-tuned model in the Qwen3 class. The arcade must be reachable from the trainer host. Nothing here is CPU-viable except a dry run with a tiny model.

TRL: the run

make arcade-up
trl vllm-serve --model Qwen/Qwen3-1.7B --port 8000
python interop/train_trl.py --manifest build/rel/arcade.json \
    --model Qwen/Qwen3-1.7B --vllm-url http://localhost:8000

That is the walkthrough. interop/train_trl.py ↗ is one session factory and one main(), and it holds no table.

Flags, with their defaults: --manifest build/rel/arcade.json, --model, --vllm-url http://localhost:8000, --envs (empty means every text-profiled entry), --difficulty 2, --steps 500, --generations 8, --inflight 8, --max-turns 6. --inflight is how many sessions are open at once, so it is the number to match with --max-sessions on each server.

TRL: what the file does

TRL's OpenEnv harness lives in trl/experimental/async_grpo/openenv_harness.py. It drives an OpenEnv ResourceSessionFactory: one isolated session per rollout, the session exposes MCP-style tools, and the reward the trainer uses is the one the session's verify() hands back. OpenEnv ships the bridge from a step/reset server to that interface, openenv.core.harness.StepEnvSessionAdapter. ManifestSessionFactory is what fills it in.

__init__ reads GET /schema once per manifest entry and keeps what it learned. It drops the entries a text model cannot play and names them on stderr, because a filtered-out environment is the thing a reader is most likely to think they are training on. create then opens a client, builds one tool from the schema, and hands both to the adapter.

The seed is the task. _HarnessRolloutLoop._run_session passes the GRPO group id as the seed, so every generation in a group meets the same server at the same seed. That is what makes a group's rewards comparable. Across groups the seed rotates through the survivors, and that rotation is the joint training.

The tool is called act, not the field it fills. RESERVED_TOOL_NAMES is {"reset", "step", "state", "close"}, so a field-named tool would break on an environment whose action field was called step. The model puts its answer in value instead, and the adapter moves it onto whatever the schema said the field was. The tool's description is the schema's own description of that field, which is the prose a hand-written table used to supply.

Summing in verify is safe on this path and not on every path, and the file says so at the method that does it. TRL calls session.verify() itself and reads env_reward straight off the result. OpenEnv's own collection helpers take a different route: _resolve_env_reward raises if verify.env_reward disagrees with the last tool result's reward. A summing verify is for the trainer that reads it directly.

The factory is picklable, which is not decoration. AsyncRolloutWorker forwards every constructor argument to a spawned child process, and raises a TypeError up front if any of them is a lambda or a closure. So the factory holds a path, a comma-joined list and two ints, and the per-session callables are built inside create, which runs in the child.

The simpler wiring. If rollouts do not need to leave the trainer process, trl.GRPOTrainer takes an environment_factory instead. It has two forms, and they are worth telling apart. Pass a dict of factories and every example must carry an environment field naming one of them; only that environment's tools are exposed in its prompt. Pass a single factory and every example uses it, one instance per rollout. Either way the environment class defines a reset method and its tool methods. If it also defines get_reward, the trainer's own docstring says that "the environment owns the reward", and reward_funcs "then becomes optional".

examples/scripts/openenv/multi_env.py mixes two environments in one run, and it takes the second route rather than the first. One MultiEnv class holds both clients, the dataset carries an "env" column, and MultiEnv.reset reads that column to choose which client to open. It also passes a reward function per environment rather than leaning on get_reward. It is worth reading as the worked mixture; it is not an example of the dict form. Either route would serve the arcade. The harness above is the deeper cut, for when the sessions have to be isolated processes.

What to expect. Nothing until vLLM is up: the trainer's child process holds the rollout loop and blocks on the first generation. Then completions in the log, and one reward column named for the harness. A GRPO group in which every rollout scored the same produces no gradient, and early on that is most of them.

§ 8 · prime-rl

prime-rl

prime-rl is Prime Intellect's RL trainer, and its environments are verifiers environments. That is the useful fact here: verifiers already ships an OpenEnv taskset, so the arcade needs a config rather than an adapter.

Prerequisites. The prime-rl uv workspace, a trainer GPU and an inference GPU (its own basic configs assume two), verifiers with its openenv extra, and the arcade reachable from the orchestrator host.

prime-rl: the taskset

verifiers/v1/tasksets/openenv/taskset.py defines OpenEnvTaskset, OpenEnvConfig and OpenEnvEnv. The config takes either an OpenEnv environment id to launch, or a base_url of a server already running. The arcade wants the second. Its resets field is a list of keyword sets, one finite task per entry, passed straight to the environment's reset. So a difficulty sweep over seeds is a list literal:

# one task family, and the shape the generator below emits
[env.taskset]
id = "openenv"
base_url = "http://127.0.0.1:8823"
resets = [
  { seed = 1, difficulty = 1 },
  { seed = 2, difficulty = 2 },
  { seed = 3, difficulty = 3 },
]

OpenEnvEnv.run is the loop, and it is worth reading before trusting it. It fetches GET /schema over HTTP, then opens the player's interaction and sends the observation and that action schema as JSON. It parses each reply with parse_action, steps the client, and sums the per-step rewards onto the trace as openenv_reward. So prime-rl is the framework that needs no adapter from us: four of the five pieces of boilerplate are already off the wire, and the fifth is the sum it already does.

The reply that works everywhere is a JSON object. parse_action calls json.loads on the reply first, and keeps the raw string only on a decode error. A bare value is wrapped only when the action schema names exactly one required field. Wordle declares required: ["guess"], so a bare crane works there. The four task servers declare no required key, because their one field carries a default, so the same reply raises rather than scoring zero.

And widening that shortcut to them would not help. A bare value keeps whatever JSON type it parsed as. logic's whole answer alphabet is {"0", "1"} and both parse as integers, and a string field answers an integer with HTTP 422 — measured against build/rel/bin/logic. So the shortcut would advertise a path that is broken for every one of that environment's answers. arcade/CHARTER.md ↗ §7b records the schema change that was proposed here and the four reasons it was refused.

One JSON object naming the field works everywhere, with its value typed the way the schema types the field. The action schema the taskset ships to the model on every turn names that field, so the model has what it needs to write the object. parse_action is upstream's parser and its behaviour is upstream's decision, so it is named here rather than patched. The shipped adapters take the other route. They read the schema's shape rather than its required list, send a string field the raw text, and take the first run of digits for an integer field. A wrong answer then scores zero instead of killing the episode.

And a reply this route cannot place ends the run rather than the episode. That is the one thing prime-rl does not get from the section above. parse_action raises where the schema names no single required field, and an action the environment refuses comes back as an error frame the client raises on. Both are upstream's, both are on the far side of a config file, and there is nothing a generated TOML can do about either. The generated header says so where a reader of that file will meet it.

One more thing that schema decides. run switches to an MCP list_tools path when the action schema's title is Action, CallToolAction or ListToolsAction. The arcade's titles are WordleAction, DynaAction, Connect4Action and eleven more of the same shape, so that branch is not taken and the ordinary step path runs. The generator below reads each title and says so on stderr if one of those three ever turns up.

prime-rl: the run

make arcade-up
python interop/train_prime.py --manifest build/rel/arcade.json \
    --model Qwen/Qwen3-1.7B --seeds 64 --difficulties 1,2,3 > arcade-rl.toml
uv run rl @ arcade-rl.toml

interop/train_prime.py ↗ is not a trainer. It is a config writer, and it imports nothing but the standard library. Joint training in prime-rl is several [[orchestrator.train.env]] blocks in one run config. Each block names its own taskset, and each may carry a ratio that weights how often the buffer draws from it. The generator emits one block per surviving manifest entry, in a run config whose shape follows configs/basic/wordle/rl.toml, prime-rl's own two-GPU example.

Two things a hand-typed config cannot promise.

It cannot be wrong about the environment. Every entry is described from its own GET /schema at generation time. An entry whose observation declares no input_text, or whose action field is a declared type there is no coercion rule for, gets no block, and what was dropped is named on stderr.

It ships the resets list. OpenEnvConfig.resets defaults to [{}], which is one task at the environment's own default seed, forever. A finite resets list is what bounds the taskset, so the generator emits seeds crossed with difficulties, per environment.

The GPU counts and seq_len are emitted as a skeleton, under a comment saying they are the reader's. The batch size and the group size are flags with defaults. No environment publishes any of those numbers and no manifest can supply them. The null harness is pinned on every block: the arcade needs no agent scaffolding, only a model that answers. Every env name must be unique, and the orchestrator reserves agg for its cross-environment metric rollup, so the generator refuses a manifest key by that name.

uv run rl @ arcade-rl.toml is prime_rl.entrypoints.rl:main, and it launches the trainer, the orchestrator and the inference server together. Under the hood prime_rl/orchestrator/envs.py builds a TrainEnv per block and spawns a verifiers env server for each. The orchestrator owns the taskset: tasks are loaded on its side, once, and each dispatched rollout ships its task's data on the request. In that file's own words, "The server answers one Episode per env-rollout". Envs.shutdown terminates them all on exit.

What to expect. One log line per environment at startup, giving each one's task count. Per-environment reward metrics from then on, keyed by the name in each block. That is the point of naming them: a mixture whose metrics are pooled tells you the average moved, not which task moved it. A finite resets list is a finite taskset, so bound the run by max_steps or supply enough seeds.

§ 9 · Tinker, via SkyRL

Tinker, via SkyRL

Tinker is Thinking Machines' training API: the loop runs on your machine, and sampling, forward-backward and the optimizer step are calls. SkyRL implements the same API on your own GPUs, so the same script runs against either — one base_url apart.

Prerequisites. Either a Tinker API key, or SkyRL installed with its tinker and fsdp extras on a machine with GPUs. tinker-cookbook for the RL loop and the renderers. The arcade must be reachable from wherever the loop runs, which is the local machine and not the service.

Tinker: the run

make arcade-up
uv run --extra tinker --extra fsdp -m skyrl.tinker.api \
    --base-model "Qwen/Qwen3-1.7B" --backend fsdp
TINKER_API_KEY=tml-dummy python interop/train_tinker.py \
    manifest=build/rel/arcade.json model_name=Qwen/Qwen3-1.7B \
    base_url=http://localhost:8000

Against Thinking Machines' hosted service, set TINKER_API_KEY and leave base_url unset. Against SkyRL, start its Tinker API server first and point the script at it — SkyRL's own quickstart says no code changes are needed beyond the URL. base_url is a real field of tinker_cookbook.rl.train.Config, and not an invention of this page.

Tinker: what the file does

tinker-cookbook's RL types are in tinker_cookbook/rl/types.py. There are three: an Env with async initial_observation and step, an EnvGroupBuilder that makes the group GRPO centres rewards across, and an RLDataset that hands the training loop a batch of those builders. interop/train_tinker.py ↗ writes all three generically, and adds the @chz.chz config and chz.entrypoint the cookbook's own recipes use.

The renderer cannot be removed, only hidden. A Tinker observation is a tinker.ModelInput and not a string, so something has to turn the environment's text into tokens and the sampled tokens back into a reply. That is one class, Seam, and it is the only thing in the file that knows what a tokenizer is. ManifestGroupBuilder.make_seams is the one method to override to frame the environment's text some other way: a system turn, a few-shot prefix, a renderer of your own. Nothing that speaks to a server has to move.

The mixture is a lattice, not a draw. Member j of batch i is slot i * batch_size + j, and the slot alone picks the server, the seed and the difficulty. Two things follow. Any batch index replays exactly, with no generator state to carry. And every server-and-difficulty pair comes up the same number of times, so the mixture is balanced by construction.

Nothing sums. TrajectoryGroup.get_total_rewards already adds the per-transition rewards, so step returns each step's reward as the environment paid it. The episode's running total appears once, in a log line, and never as a reward.

The builders pickle, and that is a requirement rather than a preference. rl/types.py says so, because a builder is serialized out to rollout workers. So the builder holds the manifest path, a comma-joined env list, ints, and the Spec read off GET /schema at startup. Every socket, the renderer and the tokenizer are built on the far side of that boundary. make adapters leg D is the assertion, and the cookbook ships a pickle test of its own as the precedent.

SkyRL's skyrl/tinker/api.py is a FastAPI server that records each request and returns a future. A background engine batches those requests onto the backend in skyrl/backends/, which is where forward_backward, forward and optim_step become real work on Ray workers. --backend-config takes a JSON dictionary of dotted overrides for the underlying trainer config, and the GPU counts and the inference engine settings live there.

What to expect. The rollouts run locally and the model does not, so watch the gap between them. A batch spends most of its wall clock in sampling, and the arcade servers are not the queue. If the loop looks idle, check the Tinker service, not the environments.

§ 10 · What none of the three removes

What none of the three removes

These are the things a shipped adapter cannot take off a reader's hands, and a reader deserves to know which they are.

§ 11 · What to expect

What to expect

The environments answer faster than any learner will ask them to. The recorded runs of the five servers here span 30155 steps/s for logic to 46154 steps/s for dyna, on one core with no GPU — measured by make record-arcade and printed into arcade/recordings.txt ↗. A model that generates a few hundred tokens per action is several orders of magnitude slower than that. The environment will not be the bottleneck, and if a run stalls the environment is not the first place to look.

Four of the five rewards are an integer, 1 or 0, on exact match. That is a narrow, honest signal with no partial credit to drift on: a rollout either answered the question the server asked or it did not. It also means early batches are mostly zeros, and a GRPO group in which every member scored zero contributes no gradient at all. Expect the first visible change to be format compliance: the model learning to answer in the shape its harness asks for, and to add nothing else. Expect difficulty 1 to move before difficulty 3. Difficulty 1 is a shorter circuit, a smaller graph, more examples of the hidden rule; difficulty 3 is the same task with the slack taken out.

A run that includes the games has one more thing in its early batches: forfeits. A game names its legal moves in its own prompt, and a model that has not yet learned to read them will answer outside the range. Those rollouts score zero and end, so they look exactly like a wrong answer in the reward column, which is what they are.

One last line, repeated because it is the hedge this page runs on. The environment side runs here. So does each adapter above, as far as a machine with no GPU can drive one, and make adapters names in writing whatever it had to skip. The trainers are the part that did not run. Nothing here claims a run that did not happen.