Reference notes — the measurements the source cites by number
This file holds the measurements this implementation was derived from, one
numbered section per subject. It is the document the EVIDENCE line of every
header in src/proto/ and of both installed headers points at. When a comment
says "REFERENCE-NOTES.md section 3.3", this is section 3.3, and the number
means what it means here.
Why the numbering looks like somebody else's. It is. These sections
re-derive the internal protocol notes this implementation was written from,
and they keep those notes' numbering so that a citation written during
implementation still resolves. Those notes are not part of this repository —
/reference/ is gitignored, because it holds a checkout of somebody else's
project and an analysis of it. What ships is this file: the same numbers,
holding the measurement rather than the prose, so a reader can re-run it.
Every exchange below was re-measured for this document, against
openenv at 60ecc7b7d6ecf830448aa9fbf330e3c7cecbc542 with fastapi 0.140.0,
pydantic 2.13.4, starlette 1.3.1, uvicorn 0.51.0, CPython 3.13.7. The
probe environments are in ../../interop/ ↗ and the setup that
produces the venv is ../../interop/setup.sh ↗.
Where a fact is pinned by the captured corpus instead of by a probe, the
section says so and names the case. The distinction matters. A wart
(../WARTS.md) backed by captured bytes is a fact about the
protocol; a wart backed by a probe is a fact about one implementation of it.
../PROVENANCE.md
explains why a third implementer deserves to know which they are reading.
Sections 3.4, 3.7, 3.12 and 3.13 are not here. Nothing in the tree cites them; the numbers are left unused rather than reassigned, so every citation in the source keeps meaning what it meant.
2. The protocol line — what is protocol and what is one implementation
This is the distinction the whole project is organised around, and it is not a
matter of taste. include/openenv/openenv.h tags every public declaration
[WIRE], [SDK] or [POLICY]. ../BOUNDARY.md is the
generated index of all of them, and make check fails if the index is stale.
The short form: the protocol is sixteen routes, five client message types,
four server response types, seven error strings and a four-key envelope.
Everything else — arenas, the poll loop, the field table, the log format, the
64-connection cap — is this implementation's, and is tagged [SDK] or
[POLICY] to say so. ../PROTOCOL.md states the protocol
half in full.
3. The wire, exhaustively
Sections 3.0 to 3.11 are the routes and the frames, one subject each.
3.0 The route table, and the four framework defaults
GET /openapi.json on a live reference server, paths in publication order —
the order matters, because openenv validate --url infers the server's MODE
from which keys are present:
$ curl -s http://127.0.0.1:PORT/openapi.json | python -c '...'
3.1.0 1.0.0
/list_environments ['GET']
/{env_name}/splits ['GET']
/{env_name}/tasks ['POST']
/{env_name}/num_tasks ['POST']
/{env_name}/task ['POST']
/{env_name}/task_range ['POST']
/reset ['POST']
/step ['POST']
/state ['GET']
/metadata ['GET']
/health ['GET']
/schema ['GET']
/mcp ['POST']
Thirteen keys. GET /openapi.json does not list itself, and the two WebSocket
routes — WS /ws and WS /mcp — cannot appear, because an OpenAPI 3.1 document
has no way to describe a WebSocket endpoint. 13 + 1 + 2 is the sixteen rows
of src/proto/routes.c, which grep -c 'OE_STR_INIT("/' src/proto/routes.c
counts. The reference also registers /docs, /redoc and
/docs/oauth2-redirect; they are Swagger UI pages, they are in no conformance
criterion, and this project refuses them
(../NOT-THE-PROTOCOL.md).
The four defaults that come from the framework rather than the application.
These are on the wire, so a compatible server produces them (CONF-22).
Measured, same server:
GET /health/ 307 location=http://127.0.0.1:PORT/health
GET /step 405 allow=POST
HEAD /health 405 allow=GET
GET /nope {"detail":"Not Found"} <- 404
HEAD is the one worth stopping at: it is an ordinary wrong method and not a
separate mechanism (WART-21). Honoured by src/proto/routes.c and
oe__route_allow.
3.1 info.version
"1.0.0", from the same document above — a string starting "1.", which is
what makes the profile openenv-http/1.x. It is server requirement 8.A.3 and
the official validator's criterion 1. Honoured by src/proto/openapi.c.
3.2 Request decoding: the action is wrapped on HTTP and bare on WebSocket
POST /step {"action":{"amount":1},"timeout_s":2.5} <- WRAPPED
WS /ws {"type":"step","data":{"amount":1}} <- BARE
Pinned by the corpus rather than probed: cases 12–25 are HTTP request bodies and
48–67 are WebSocket frames, in
../../tests/fixtures/reference-corpus.json ↗.
Two further facts from the same cases, both of which a rewrite gets wrong:
POST /resetdeclaresBody(default_factory=ResetRequest), so no body at all constructs an empty one and succeeds (case 14).- A non-JSON
Content-Typedoes not reach the model: the raw body text is handed to it, so the 422'sinputis the body as a STRING. Case 15 echoes"{}"— two characters — which is the clearest possible evidence that nothing parsed it (WART-18).
Honoured by src/proto/decode.c, whose four numbered steps each name their case.
3.3 The reward divergence — the two transports disagree about one value
This is the fact this repository repeats most often, so here is the
measurement rather than a citation to one. interop/reward_env.py returns a
reward of a different JSON type on each step. The same environment, the same
server process, the two transports:
$ # HTTP: POST /step, one call per reward kind
null {"observation":{"metadata":{},"kind":"null"},"reward":null,"done":false,"metadata":null}
bool_true {"observation":{"metadata":{},"kind":"bool_true"},"reward":1.0,"done":false,"metadata":null}
bool_false {"observation":{"metadata":{},"kind":"bool_false"},"reward":0.0,"done":false,"metadata":null}
int {"observation":{"metadata":{},"kind":"int"},"reward":3.0,"done":false,"metadata":null}
int_zero {"observation":{"metadata":{},"kind":"int_zero"},"reward":0.0,"done":false,"metadata":null}
float {"observation":{"metadata":{},"kind":"float"},"reward":1.5,"done":false,"metadata":null}
float_int {"observation":{"metadata":{},"kind":"float_int"},"reward":2.0,"done":false,"metadata":null}
$ # WS /ws: {"type":"step",...}, one frame per reward kind, same server
null {"type":"observation","data":{"observation":{"metadata":{},"kind":"null"},"reward":null,"done":false}}
bool_true {"type":"observation","data":{"observation":{"metadata":{},"kind":"bool_true"},"reward":true,"done":false}}
bool_false {"type":"observation","data":{"observation":{"metadata":{},"kind":"bool_false"},"reward":false,"done":false}}
int {"type":"observation","data":{"observation":{"metadata":{},"kind":"int"},"reward":3,"done":false}}
int_zero {"type":"observation","data":{"observation":{"metadata":{},"kind":"int_zero"},"reward":0,"done":false}}
float {"type":"observation","data":{"observation":{"metadata":{},"kind":"float"},"reward":1.5,"done":false}}
float_int {"type":"observation","data":{"observation":{"metadata":{},"kind":"float_int"},"reward":2.0,"done":false}}
true becomes 1.0 on HTTP and stays true on WebSocket. 3 becomes 3.0
and stays 3. null survives both, because the HTTP response model is
Optional[float] and None is not coerced.
Why: StepResponse.reward is Optional[float] while Observation.reward
is bool | int | float | None, so the HTTP response model re-types the field
the environment set. The coercion is float(), not a cast.
Why the corpus cannot show it: the corpus environment returns
float(total), and an integral float is the one value both encoders agree on.
Cases 12/18 and 48 pin the two SHAPES and neither pins the divergence. That is
why this is a probe and why envs/rewards.c exists as the same probe in C.
Re-measured on every make interop run, as oracle 2b: "upstream EnvClient
read bool/int/float rewards back as themselves". Honoured by
write_reward_http against oe__encode_obs_ws in src/proto/encode.c —
WART-03, and the two functions sit either side of the divergence table.
3.5 GET /state drops a State subclass's own fields
Probed with a State subclass carrying four extra fields
(scratchpad probe_state.py, the same shape as interop/probe_env.py):
$ curl -s http://127.0.0.1:PORT/state
{"episode_id":"ep-probe","step_count":0}
$ # the same environment, over WS /ws
>> {"type":"state"}
<< {"type":"state","data":{"episode_id":"ep-probe","step_count":2,"total":20,"label":"four fields","depth":7,"flag":true}}
Four fields declared, four fields gone on HTTP, all four present on the
WebSocket. Why: the route is registered with response_model=State, so
FastAPI re-serialises the returned object through the BASE model and a subclass
field has nowhere to go.
The corpus pins the one-field version of this: case 3 is
{"episode_id":null,"step_count":0} from an environment whose total is 0, and
case 48 frame 4 is {"episode_id":"ep-fixed","step_count":2,"total":6}. The
general case above is the probe that shows it is not a coincidence of the value
being zero. Honoured by oe__encode_state_http in src/proto/encode.c
(WART-06), which is write_base_state and a closing brace.
3.6 The error shapes
Two of them, and which one you get is decided by the code path rather than by what is wrong:
| origin | body |
|---|---|
| a validation failure | 422 {"detail":[{type,loc,msg,input[,ctx][,url]}]} |
| anything Starlette did not expect | 500 text/plain "Internal Server Error" |
Corpus cases 15, 17, 20–24 and 30 pin the HTTP forms and 54–64 the WebSocket
ones. The url key is the trap, and the rule that looks right is wrong:
| transport | loc | type | carries url? |
|---|---|---|---|
| http | ["amount"] | int_parsing | yes |
| http | ["body","action"] | missing | no |
| http | ["bogus"] | extra_forbidden | yes |
| ws | ["amount"] | int_parsing | yes |
| ws | ["data"] | missing | yes |
loc:["data"] is an envelope-level error and it does carry url. The
discriminator is not the locality of loc — it is that FastAPI's request-body
validation calls errors(include_url=False) and every other path, the WebSocket
handler included, uses plain .errors(). Honoured by the ORIGIN flag on
oe_diag, fixed when the report is initialised in src/proto/errors.c and
never inferred from loc.
3.7 GET /schema — what required looks like, and where it sorts
The corpus cannot pin required: both of interop/probe_env.py's action fields
have defaults, and pydantic then omits the keyword entirely. So a second probe,
class EchoAction(Action): message: str — one field, no default:
$ curl -s http://127.0.0.1:PORT/schema | python -m json.tool # .action only
{
"additionalProperties": false,
"properties": {
"metadata": {
"additionalProperties": true,
"description": "Additional metadata for the action",
"title": "Metadata",
"type": "object"
},
"message": {
"title": "Message",
"type": "string"
}
},
"required": [
"message"
],
"title": "EchoAction",
"type": "object"
}
Three facts, and each is a thing a hand-written generator gets wrong:
requiredsorts betweenpropertiesandtitle— the keys are in pydantic's own order, which is neither alphabetical nor declaration order.metadatais present in an ACTION schema and the author did not declare it: the framework injects it, exactly as it injectsdone/reward/metadatainto an observation.additionalProperties: falseat the top andtrueinsidemetadata.
Honoured by src/proto/schema.c, whose required array is derived from
json_default == NULL and never stored twice. tests/t_schema.c uses this probe
as its second oracle and corpus case 2 as its first.
3.8 The Task API
Six routes, all POST except GET /list_environments and GET /{env_name}/splits.
501 {"detail":"<method> is not supported for this environment"} is a
conforming answer, and it is the one this project gives. It is captured now
against the REGISTERED name counter_env (cases 91-95), which closed the old
GAP-20: every earlier case named the class CounterEnvironment and hit the
name check first. An env_name that does not match, case-insensitively, gets
a SECOND 404 body, {"detail":"unknown environment 'X'"} — on a server that
also answers {"detail":"Not Found"} for an unmatched path (WART-15).
Honoured by src/server/http_const.c.
What a reference environment WITH a provider returns — captured from
interop/task_env.py (cases 96-102), the shapes no shipped env can produce:
GET /list_environments ["task_env"] (no provider needed)
GET /task_env/splits [{"name":"train","type":"train"},
{"name":"test","type":"test"},
{"name":"holdout","type":"validation"}]
POST /task_env/tasks {"tasks":[…],"env_name":"task_env"}
POST /task_env/num_tasks {"num_tasks":3}
POST /task_env/task {"task":{…}}
POST /task_env/task_range {"tasks":[…]} (Python slice; no env_name)
POST /task_env/task idx=99 -> 400 {"detail":"Invalid task index"}
Two facts the reference states and no shipped env emits. _normalize_split
(http_server.py:1070-1080) wraps each bare split string as {"name","type"}
and types a name outside {train,validation,test} as validation — so holdout
above is type:"validation". And the 400 (http_server.py:1094-1098) is an
IndexError turned into {"detail":"Invalid task index"}, reachable only with a
provider present (the 501 wins first) and named in no OpenAPI document. This
project DECLINES providers on evidence (GAP-24), so the corpus buckets cases
97-102 REFUSED.
3.9 POST /mcp — the dispatch order, and the three tool-table shapes
The session id is resolved before the method is dispatched. Probed:
$ # unknown session, unknown method
{"jsonrpc":"2.0","id":7,"error":{"code":-32602,"message":"Unknown session_id: zzz","data":null}}
$ # no session, unknown method
{"jsonrpc":"2.0","id":8,"error":{"code":-32601,"message":"Method not found: foo/bar","data":null}}
$ # the empty request the official validator sends — 200, and an error INSIDE it
200 {"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"Invalid request: 2 validation errors for JsonRpcRequest\njsonrpc\n Field required ...","data":null}}
So an id that names nothing returns -32602 and the method is never looked at —
the opposite order from the one a reader assumes. Honoured by
src/proto/jsonrpc.c, which fills session_id whatever the method is and
leaves the ORDER to the caller, because the session table cannot live in a pure
function.
The three tools/list shapes. tools/list is implemented in at least three
places in the reference tree with mutually incompatible result shapes
(core/mcp_client.py, core/env_server/mcp_types.py,
envs/calendar_env/.../mcp_handler.py), and no single OpenEnv-defined shape
covers all of them. That observation is why a door rule is needed at all. The
rule this project implements: one shape per door, each one the reference's
own FastMCP-free spelling. The trainer door gets mcp_types.py:179-195 — a
pydantic model with extra="forbid" and exactly name, description,
input_schema; the agent door gets http_server.py:945-958, the same server's
hand-written inputSchema camelCase three. The third shape,
mcp.types.Tool.model_dump()'s nine keys, is the one we decline. Its
provenance is a Python library's model rather than an OpenEnv decision. And
that call site omits by_alias=True, so its _meta key comes out spelled
meta — it is not even the MCP specification's shape
(../NOT-THE-PROTOCOL.md § out of scope).
An earlier revision of this paragraph said this project answers an empty tool
table on every door and refuses the rest outright. That was true when it was
written and is now false in every clause: build/rel/bin/echo_tools publishes
two tools on both doors. The refusal narrowed rather than vanished; the page
linked above opens by recanting the wide version.
3.10 WS /ws — the envelope, and metadata when it is empty
The four keys, in this order, and nothing else:
{"observation": {...}, "reward": ..., "done": false, "metadata": {...}}
reward and done are HOISTED to the top level of data and are not left
behind: the nested dump is taken with exclude={"reward","done"}
(serialization.py:155-160), so neither key appears inside observation on
either transport. Zero of the observation payloads in the captured corpus carry
one, on either transport, and
../../tests/t_client.c ↗ walks every one of them and
asserts that zero. GET /schema nevertheless advertises both as properties of
the observation, with defaults false and null, because it is generated from
the model that declares them. That document is where a client learns to look in
the wrong place. What it reads there is its own lookup default rather than
anything the server sent — done stays false forever, and the episode never
ends. Two shipped upstream environments read the keys from exactly that wrong
place.
MUST 8.B.12 in ../PROTOCOL.md
is the client's half of it.
metadata differs by transport and it is not the same difference as
reward: HTTP emits the key always, with null when it is empty; the
WebSocket omits the key entirely (if metadata: in the reference's serialiser).
Visible in the two transcripts in 3.3 above — every HTTP line ends
"metadata":null and no WebSocket line has the key at all. WART-02 and
WART-04; honoured by oe__encode_obs_http against oe__encode_obs_ws.
There is no episode state machine: a step before any reset is accepted,
a step after done is accepted repeatedly, and a second reset
re-initialises. That probe is in
OBSERVED-WIRE.md with
both servers' bytes side by side, and it is why oe_session has no phase field.
3.11 The three MCP framings
The same JSON-RPC dialect arrives three ways, and the envelope differs each time:
| framing | request | response |
|---|---|---|
POST /mcp | the JSON-RPC object as the whole body | the JSON-RPC object as the whole body |
WS /mcp | the JSON-RPC object as the whole frame | ditto, and spaced JSON (WART-13) |
{"type":"mcp"} over WS /ws | the object under data | the object under data |
Corpus cases 31–38 (POST /mcp), 67 ({"type":"mcp"} over /ws) and 68–69
(WS /mcp). Honoured by src/proto/jsonrpc.c — one dialect, three framings, and
the framing is a parameter rather than three code paths.
4. The data model
Three base models — Action, Observation, State — and the framework injects
its own fields into every subclass: metadata into an action, and
done/reward/metadata into an observation. Those belong to OpenEnv rather
than to the environment author, which is why include/openenv/openenv.h refuses
to let a field table declare them. Corpus case 2 (GET /schema, 1468 bytes) is
the oracle for what that looks like as JSON Schema.
4.5 Deserialisation: pydantic's LAX mode, and it is load-bearing
The reference builds actions with model_validate and never asks for strict
mode, so "42" is an acceptable integer. Probed directly against
pydantic 2.13.4:
int <- "42" -> ok i=42 s='x'
int <- true -> ok i=1 s='x'
int <- 1.0 -> ok i=1 s='x'
int <- 1.5 -> int_from_float
int <- "x" -> int_parsing
str <- 5 -> string_type
str <- true -> string_type
An int field coerces from a string, a boolean and an integral float. A str
field coerces from nothing. grep -rn strict reference/OpenEnv/src/openenv
returns four hits, none of them a validation mode: two are a CLI helper's
parameter and two are a comment about network egress.
This cannot be skipped. int_parsing is what a conversion reports when it
was ATTEMPTED and failed, so its presence in the corpus (case 20) is proof that
the reference attempts one. Case 25 is the same conversion succeeding:
{"action":{"amount":"42"}} returns 200 and drives the counter to 42. A
validator that never attempts a conversion emits int_type there and fails the
replay. Honoured by the six functions in part 3 of src/proto/validate.c, one
per field kind, each listing its accepted inputs in the order the reference
tries them (WART-20).
5. Lifecycle
5.4 The handshake, and ws_max_size
Uvicorn's default inbound frame limit is 16 MiB
(uvicorn/config.py:201, ws_max_size: int = 16 * 1024 * 1024) and the
reference closes above it. OE_WS_MAX_FRAME is deliberately lower — see the
DIVERGENCE note at that #define in src/wire/limits.h, and divergence 3 in
../NOT-THE-PROTOCOL.md.
The asymmetry is deliberate on the client side too. The reference CLIENT
raises its own inbound limit a hundredfold above its library's default
(max_message_size_mb: float = 100.0, core/env_client.py:284). That is the
same reasoning that makes OE_WS_MAX_FRAME_CLIENT larger than the server's
number. A server chooses what it will accept; a client must accept what the
peer sends.
The reference can afford 100 MB there and this implementation cannot afford
anything like it. That is a fact about the two memory models rather than about
nerve. CPython's parser produces objects the allocator can free individually;
ours produces an arena that is released in one piece, and that arena is the
binding resource. So OE_WS_MAX_FRAME_CLIENT is 512 KiB: the arena divided by
the measured cost of parsing a byte. See the #define for the measurements.
5.5 Close-frame reason strings — 1002, 1009, and the 1011 that never comes
Captured against the live reference on /ws (8901), one malformed frame per
fresh connection, websockets 16.1.1 under uvicorn 0.51.0. These confirm the
reason column of the table in src/wire/ws.c — measured there, re-measured here
for NEXT item 24:
trigger close reason
RSV1 bit set 1002 "reserved bits must be 0"
reserved opcode 0x3 1002 "invalid opcode"
unmasked client frame 1002 "incorrect masking"
fragmented control (FIN=0) 1002 "fragmented control frame"
close payload of 1 byte 1002 "close frame too short"
close code 999 or 1005 1002 "invalid status code"
frame over ws_max_size 1009 "frame with N bytes exceeds limit of M bytes"
The reason strings are the websockets library's, not ours: ws.c keeps its own
clearer prose and records both columns so a byte diff is never a surprise. The one
code difference is rule 5 — a 64-bit length with the sign bit set is 1009 there
and 1002 here, because a malformed header should not invite a retry with a smaller
number.
The 1011 class has no reference counterpart on this transport. 1011 is our
close for an internal failure — an exhausted per-message arena. The reference
never reaches it through the OpenEnv handler. A non-object payload, the sharpest
thing a well-framed peer can send, is answered with a SESSION_ERROR frame
({"type":"error","data":{"message":"'int' object has no attribute 'get'", "code":"SESSION_ERROR"}}) and then a NORMAL close: code 1000, empty reason.
The handler's blanket except Exception turns every application failure into
a frame before the transport could 1011 it. CPython's heap is unbounded, so it
never hits the budget that produces ours. Recorded so a reader who greps for
our 1011 does not go hunting for the reference's.
8. The conformance checklist
8.A is the twenty-three server requirements and 8.B is the twenty
client requirements. Both are published in full, with their numbers, in
../PROTOCOL.md — that is
where MUST 8.A.n and MUST 8.B.n in the source resolve to, and make check
fails on a citation whose number has no row there.
9. Implications for a C99 implementation
9.1 The three that are genuinely hard, and one piece of advice that was wrong
- Dynamic typing without reflection. The reference derives
/schema, the action validator and the observation serialiser from one pydantic class each. C has no reflection, so the same three come from an explicit field-descriptor table:oe_fieldininclude/openenv/openenv.h, and one table per environment. One fact, three consumers, no way for them to disagree. - A JSON parser that keeps
truedistinct from1. Section 3.3 is why: a parser that widens everything to double cannot reproduce the WebSocket encoder at all.src/wire/json_parse.ckeeps the distinction, andoe_rewardis a tagged union with four arms rather than adouble. - Messages that are genuinely large. Observations can be megabytes.
envs/atari_envpublishes its screen as a flattenedList[int]of[210,160,3]— 100,800 integers per frame (envs/atari_env/models.py:47). When a uvicorn server sends that shape it is 403,316 bytes on the wire, measured, and comfortably over 256 KiB. That number is whyOE_WS_MAX_FRAME_CLIENTis not the server's cap, and the argument is beside both constants insrc/wire/limits.h. It is also the number that sets the ceiling's lower bound: whatever the client's cap is derived from, it has to clear 403,316, and at 512 KiB it clears it by 23%.
The advice that was wrong, recorded because getting it wrong is instructive.
The original note said to skip pydantic's lax coercion: "a compatibility
nicety, not a requirement, and getting it half-right is worse than not doing
it." Corpus cases 20 and 25 make both of those cases unpassable without it,
and they are inside the project's primary acceptance test. So the coercion is
implemented and the advice is superseded. src/proto/validate.c records the
supersession at the line that decides it.
The same note also said to emit url on action-relative errors and omit it on
envelope errors, "that distinction is observable". It is not the
distinction — see the table in 3.6, where loc:["data"] is envelope-level and
carries url anyway. Both corrections were found by adversarial review of a
claim that was plausible, was written after genuinely probing a server, and was
still false. That is the best argument in this repository for grading every
claim against bytes.
See also
OBSERVED-WIRE.md— the probed exchanges, with both servers' bytes../PROTOCOL.md— the protocol itself, and the two numbered requirement lists../PROVENANCE.md— the corpus, the fifteen oracles, and the four kinds of claim../WARTS.md,../GAPS.md— the generated ledgers, where each of the facts above lands in the source