The /v1 endpoint: what it is, and what it is not

POST /v1/chat/completions is OpenAI's path, and a path sets an expectation. This page is the expectation stated in advance, so that the gaps are documented rather than discovered on the wire. Since #809 there is a second OpenAI path, POST /v1/responses, adapted onto the first; it has its own section below and everything else on this page applies to it unchanged.

Roteiro serves your installed GGUF models over an OpenAI-compatible /v1 endpoint, bound to loopback (ADR-0006). It is enabled by building with the serve feature and started with roteiro serve.

It has two modes, and the request chooses

This is the single most useful thing to know about it. The mode is decided by whether you send a tools array — nothing else.

The question is did you bring tools, so an empty array answers no and is read exactly as an absent one. Both wires agree on that, and a test pins them together so they cannot drift apart on it.

In Ask mode the endpoint is the Ask panel over HTTP: the server runs an agent loop, calls the graph's own tools, and answers from what it finds. That is the mode the Ask page describes.

In general mode it is a local model backend. Your tools replace the graph tools rather than joining them — a client sending its own tools is using this as a backend, and adding roughly 3,100 tokens of graph schemas nobody asked for is the surprise this behaviour exists to remove.

If you want both — an agent with its own tools and the graph — that is what MCP is for. Run roteiro mcp and your agent gets search, context, path, debt and the rest natively, alongside whatever else it already has.

Roteiro never executes a tool you supplied

When the model calls one of your tools, Roteiro returns the call with finish_reason: "tool_calls" and stops. You run it, and send the result back as a role: "tool" turn.

This is why client tools need no consent gate and no authorisation story: there is no code path that reaches one.

If a single turn calls both a client tool and a graph tool, all the calls are returned and none are executed — never "run the graph half, return the client half", which would hand you a tool_calls response for an assistant turn you could not reconstruct, having never seen the graph call or its result.

Scoped routes

Serving several repositories from one process (ADR-0008) scopes the graph tools by route. The unscoped route answers about the single repository the server was started in.

Declared divergences from OpenAI

Stated rather than half-implemented, so that a gap is a documented decision rather than a surprise.

A model's reasoning never reaches you, and that is a decision. A reasoning-capable GGUF (Qwen3, DeepSeek-R1, …) writes a <think>…</think> block before its answer. Roteiro drops it here exactly as it drops it for spec draft and review --llm, and the fact that it did not used to was an omission rather than a policy (#582): the same model producing the same block was cleaned for a CLI consumer and passed through raw over HTTP. Measured live, that was ~95 of 105 completion tokens of deliberation for a one-word answer.

Three reasons this endpoint strips rather than forwarding:

• It is Roteiro's Ask over HTTP, not a general model backend — ADR-0006 and #487 settled that framing, and Ask's consumer wants the answer.
• Multi-turn callers echo assistant turns back as history, so a block passed through is re-sent verbatim on every subsequent turn, against a prompt budget with nothing reused unless [serve] prefix_cache_mb is set (#578). The same compounding happens inside Roteiro's own tool loop, which is why the block is dropped before a turn is fed back into the next round.
• The rule now lives in one place. rto_llama::thinking is shared by this endpoint and by every CLI path, so the two cannot drift apart again.

Stripping keys on a block being opened, never on a tag appearing. The rule asks whether the content starts with <think> — after leading whitespace — and only then looks for the close tag. This is the whole reason you can ask this endpoint about its own reasoning handling and get the answer back intact: a reply quoting </think> is prose, and cutting everything before a quoted tag would be the silent truncation docs/REVIEW_CHECKLIST.md §Refusals forbids. The streaming filter has always read it this way; since #589 the non-streaming half does too, so the two surfaces agree on what counts as a block as well as on what to do with one.

reasoning_content is deliberately not implemented. Returning the block under a field name would be adopting a convention Roteiro does not otherwise speak, and the block would still cost a multi-turn caller its budget the moment that caller echoed the turn back. If you want a model's deliberation, run the model directly — this endpoint is not the place to get it.

An unterminated block is a refusal, not a short answer. If generation stops inside <think>, there is no answer in the completion at all, so content carries a Roteiro: -prefixed sentence saying which budget ran out rather than the raw deliberation (#583). This is not hypothetical for the models this registry serves: qwen3.8-27b was measured spending an entire 1,200-token budget inside <think> and emitting no answer. finish_reason and the token counts are left exactly as the model produced them, so a machine client still sees length and still bills for what was really spent.

role: "tool" becomes a user turn carrying <tool_response>, deliberately. llama_chat_apply_template does not run a Jinja parser — it renders a fixed set of templates and emits unknown roles literally, so passing tool through would put a role token in the prompt these models were never trained on. A <tool_response> user turn is what every Qwen template emits natively for a tool result, so this is the native form rather than a workaround.

Every request parameter, and what happens to it

This is the whole boundary. All 37 parameters of OpenAI's chat-completions request body are below, and there is nothing else to send — a key in no row here is either a typo or newer than this table, and is ignored.

Four statuses, the same four the divergence table above uses:

• supported — read and acted on.
• accepted, not enforced — parsed and carried, then deliberately not acted on. Declared, which is the entire difference between this and the silent drop the endpoint used to do.
• dropped — ignored, and ignoring it leaves you believing nothing false about the response. These are the parameters that drive OpenAI's own bookkeeping — storage, billing, cache bucketing, abuse attribution — which a loopback server has no analogue for and whose absence changes nothing you can observe.
• 400 — refused. Ignoring it would hand you a response that silently contradicts your request, and a wrong answer is worse than a refusal. The cell quotes the message you will receive, verbatim.

Sending a default is not making a decision

A refusal that fires at a caller who chose nothing would be a worse defect than the silence it replaces, and client libraries serialise defaults constantly. So the check reads the value, never the presence of the key:

• null is never a decision, for any parameter.
• Neither is OpenAI's own documented default — n: 1, top_p: 1, frequency_penalty: 0, logit_bias: {}, response_format: {"type":"text"}, modalities: ["text"], and the rest.

n: 1 and n: 3 are therefore answered differently, and both correctly: the first asks for what this endpoint does, the second asks for something it cannot do. If your client sends n: 1 and top_p: 1 on every request — most do — none of this is aimed at you.

Why not deny_unknown_fields

It was the obvious instrument and it is the wrong one. It would refuse user, store, metadata, service_tier and prompt_cache_key — keys clients send out of habit and whose absence harms nobody — so it would break working callers in order to fix a correctness problem those callers do not have. The list above is the narrower answer: refuse what would mislead you, ignore what would not, and publish both.

The declaration is enforced from the same table that generates this page (crates/rto-serve/src/openai_params.rs), and the tests beside it assert the table against the request type and against this document in both directions — so a parameter added to the code without a row here, or a row here without the code behind it, fails the build rather than the reader.

Tool arguments are the other way round

Everything above is about the request envelope, and none of it extends to the arguments of a tool call. There, an unrecognised key is refused.

The two cases look alike and are not. A stray user on the request is a field Roteiro has no use for, and dropping it costs nobody anything. A stray key inside a tool call's arguments is a filter the model asked for and did not get. Roteiro's two tool surfaces used to spell one of debt's arguments differently — kind on the MCP surface, categories here — and while unknown keys were dropped, {"categories":["todo"]} sent to the MCP debt deserialised to an empty filter, which means every category. The model asked for one kind of marker, received the whole repository's debt presented as the filtered set, and had nothing in the result to tell the two apart.

Both surfaces now spell it categories, and a test drives that comparison from the two sets of definitions rather than from a hard-coded pair, so a future divergence in any shared tool's argument names fails the build. Making unknown keys an error is what allowed that rename: while they were dropped, renaming the MCP argument would have left every existing caller parsing fine and silently receiving unfiltered results. Refused, the same caller is told the new name.

So a tool that declares "additionalProperties": false has it enforced: a call carrying a key the schema does not list is refused by name, with the keys that would have worked, and the tool is not run.

tool `debt` error: unknown argument `kind` — `debt` takes only `categories`,
`project`. Nothing was run: an argument this tool does not declare is refused
rather than ignored, because ignoring it would answer a narrower question than
the one you asked and give you no way to tell.

This is about Roteiro's own graph tools. A tool you supplied is unaffected in either direction — Roteiro never executes one (see above), so it never inspects its arguments, and the call comes back to you exactly as the model wrote it.

The Responses API (POST /v1/responses)

OpenAI has a second wire, and some clients now speak only that one: codex-cli 0.147 removed Chat Completions support — wire_api = "chat" is no longer accepted — so for that family this path is not a convenience, it is the only one.

It is served here as an adapter, not a second implementation. A Responses request is translated into the request the chat path builds, run through the same tool loop, and rendered back out as Responses events. Everything above about modes, about tools you supplied, about <think> blocks and about refusals is therefore true of this path too, because it is the same code deciding it.

The surface is deliberately small, and what is missing is refused by name rather than ignored.

Nothing is stored, and the response says so rather than leaving you to infer it. Every event this endpoint emits carries "store": false inside its response object — response.created, response.completed and response.incomplete alike — whatever you sent or omitted. That is the declaration, and it is in the field a client would read to find out.

store: true is refused, because it asks for something that will not happen. A request that omits store is not refused, and that is a deliberate decision rather than an oversight:

• An absent key is no decision, for every parameter on both wires. Refusing everyone who left a default unstated would be the refusal-fired-at-nobody this page's chat half is at pains to avoid.
• The only thing storage buys is retrieval, and every retrieval path here is already a 400 that names the cause. previous_response_id and conversation both refuse, saying that nothing is stored and that the whole conversation must be sent in input. So there is no sequence in which a client believes a turn was kept and later gets a wrong answer instead of a refusal — the worst case is a refusal one turn later than it could have come, with the same sentence.
• The default OpenAI applies when store is absent could not be verified from anything available offline: openai 6.40.0's own type documentation says only "Whether to store the generated model response for later retrieval via API" and states no default, while documenting one three fields earlier for service_tier ("When not set, the default behavior is 'auto'"). Building a refusal on an unverified default would be worse than declaring the behaviour, which is true whatever that default turns out to be.

A tool round trip, and why it is byte-identical to the chat one

Responses models a tool call as typed items where Chat Completions puts tool_calls on a message. Both are translated into the in-band protocol the served models actually speak, by the same function:

This is not a parallel rendering that happens to agree. The adapter builds the chat structures and hands them to the same normalisation the chat wire uses, so there is one implementation of the byte format and a test asserts the two wires produce identical prompt turns for the same conversation. A second copy would have drifted, and the symptom would have been a multi-turn tool conversation degrading at turn three with nothing on the wire to say so.

One inherited detail worth knowing: the keys inside arguments come back sorted, because the rendering re-parses the arguments string into a JSON object. That is true of the chat wire already; the adapter inherits it rather than introducing it.

The event sequence, and which parts a client actually needs

response.created → response.output_item.added → (response.output_text.delta | response.function_call_arguments.delta) → response.output_item.done → response.completed, then data: [DONE]. A generation the token budget cut short ends on response.incomplete instead; codex-cli 0.147.0 was measured reading that event and reporting Incomplete response returned, reason: max_output_tokens.

Which of those are load-bearing was measured, by serving each event set from a mock and running a real tool-using turn of codex-cli 0.147.0 against it, rather than by reading a schema or grepping a binary for string literals — a literal that is absent from a packed binary is not a literal the client does not need.

So the load-bearing pair is response.output_item.done and the terminal event. Wait for either terminal event, not for response.completed alone — a turn that reaches max_output_tokens ends on response.incomplete, and a client waiting only for completed would read the closed stream as a disconnect and retry a turn that was in fact delivered.

The full sequence is emitted anyway: it is what OpenAI's own wire carries, and a client that reads the optional events gets a live stream instead of a silent wait.

Every Responses request parameter

Read the four status words exactly as the chat table above defines them. This table is shorter than that one on purpose: the chat table is exhaustive over a frozen wire, and this one carries the parameters a Responses client sends plus every one whose absence would mislead you. A key in no row here is passed through untouched, exactly as an unknown chat key is — and for the same reason (deny_unknown_fields would refuse the habitual keys clients send and harm nobody).

That rule puts a duty on this table rather than on you. A parameter that moves state or output off the request — previous_response_id, conversation — is a silently wrong answer if it is missing here, not a harmless unknown, so those are listed and refused. The rows below were read from the Responses request schema on 2026-09-13 and cross-checked against a captured codex-cli 0.147.0 request.

How a failure is reported

Every failure carries an OpenAI-shaped {"error": {"message", "type"}} body. The type is the part worth reading: it says whose problem this is, and therefore whether retrying can help.

A streaming request has already sent 200 OK and its first event before generation can fail, so there is no status code left to carry that. Both streaming wires put the same word in their terminal event instead — response.failed's error.code on /v1/responses, and the final error frame on a streaming /v1/chat/completions. So the table below reads the same on all three surfaces.

The message is never a summary. When a model's chat template refuses a conversation, the message carries the template's own sentence, verbatim — System message must be at the beginning. — because that sentence is the answer and anything of Roteiro's in its place would be a paraphrase of a fact only the model's own metadata knows.

One asymmetry is deliberate. Roteiro prepends a system turn of its own to every tooled conversation, carrying the grounding rules for the tools it advertises. Once it has done that, the conversation a template refuses is no longer the one you sent, and Roteiro can no longer tell whether the refusal is about your turns or its own. It does not guess: it reports those as 500, names the turn it added, and leaves the template's sentence intact so you can still recognise your own case if it is yours. An untooled request is passed through untouched, so a refusal there is reported as 400 and can be trusted to be about what you sent.

Why the tools array is bounded

The limits are a security bound, not a tidiness rule. Roteiro sizes the model's context window to the request — prompt_tokens + max_tokens + headroom, capped at the model's trained window — so with an unbounded tools array a caller would choose Roteiro's memory allocation. On qwen3.8-27b the trained window is 262,144 tokens and KV runs about 64 KiB per token: roughly 16.4 GiB reserved for a single request.

32 KiB of names, descriptions and schemas is about 8k tokens, keeping the tool surface's contribution around 32× below that ceiling. Raising the bound re-opens the hole in proportion.

The bound is measured on what you send, not on what the model reads. Roteiro renders a tool's arguments as a signature — search(query: str, limit?: int 1..25) — rather than as raw JSON Schema, so the same 32 KiB of wire bytes becomes fewer prompt tokens than it used to. That makes the bound more conservative, never less: it is applied to the compact JSON before any rendering happens. Measured on Roteiro's own MCP surface driven back in as a client payload, 20,045 wire bytes rendered to 21,180 prompt bytes before and 17,454 after, with tool-call accuracy unchanged on a fixed question set.

A schema the renderer cannot state without losing something — a nested object, a $ref, a oneOf, additionalProperties: true — is sent verbatim instead. Your argument shape is the contract Roteiro hands back for you to execute, and a lossy summary of it would leave the model calling a tool whose arguments no longer match what you will run. That is the same failure the size bound refuses to truncate for.

Reusing a preamble instead of re-prefilling it

A context is built per generation and the prompt is prefilled in full, so a multi-turn client re-pays for the part of its prompt that never changes — its system message and its tools array. Measured on qwen3.8-27b at 3.13 ms per prompt token, a 32 KiB tool surface is about 21 s per turn, every turn.

[serve] prefix_cache_mb stops that. Set it in ~/.roteiro/config.toml:

[serve]
prefix_cache_mb = 512

Unset — the default — nothing is cached and behaviour is exactly as before.

Size it in whole preambles. An entry costs a fixed ~150 MiB plus 64 KiB per token, because qwen3.8-27b is a hybrid attention/SSM model whose recurrent state serialises whole regardless of prompt length. There is no such thing as a small entry: 256 holds roughly one preamble, 1024 roughly three, and anything below ~200 stores nothing at all.

Several preambles for one model coexist, so two clients with different system prompts each get their own — an entry is only displaced when a new one that it is a prefix of arrives, or when the budget forces a least-recently-used eviction.

What to expect. The boundary is learned by comparing consecutive prompts, so the first two turns pay full price and reuse begins at the third. Measured end to end on a 1,462-token prompt: 5.80 s → 2.11 s, with byte-identical output.

Three things it deliberately does not do:

• No partial credit. A prompt that shares only part of a cached preamble is a miss and prefills in full. Recurrent state has no per-position structure and cannot be rewound, so a trimmed restore would be silently wrong on 48 of this model's 64 layers rather than an error. A prompt equal to a cached preamble misses too, for a different reason: with everything restored there would be no token left to batch, and nothing to carry logits.
• No client key. prompt_cache_key stays dropped. The preamble is found by comparing prompts, so a client gets this without asking and cannot mistakenly ask for another client's state.
• Nothing under speculative decoding. A speculative generation runs target and draft contexts whose states are not independent, so the cache stays inert there rather than restoring half a pair.

Everything it holds is recomputable, so a failed restore or a failed snapshot costs a request nothing: it falls back to prefilling in full.

Not a hosted API

Two things follow from this being a local server rather than a service, and both are deliberate (ADR-0006):

• It binds loopback by default. Front a public bind with a reverse proxy; TLS and authentication terminate there.
• It speaks HTTP/1.1, not HTTP/2. HTTP/2 in practice requires TLS, which is the proxy's job, and the multiplexing it buys is worth nothing over loopback against a bottleneck that is model inference. See ADR-0006 for the full reasoning.

← Back to roteiro.dev