On a reasoning model the reasoning is spent from the same completion budget
and it goes first, so a small ceiling truncates the actual reply away. It
fails silently and looks exactly like a model that cannot follow the system
prompt, which is the expensive way to debug it.
Measured on gpt-oss-20b against the real ~3.5k-token system prompt, same
prompt and same model, only the ceiling changing:
1400 an empty string, 0 bytes, after 27s
2048 a truncated half-Korean fragment, none of the required blocks
8000 99% English prose, no romanization, 0 batchim violations in the
task lines, all three blocks, 11s
The old comment justified 2048 by worrying about 4k-context local models.
That reasoning was wrong: the system prompt alone is ~3.5k tokens, so such
a model cannot run this app at all and there was nothing to protect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
HANKAN_TUTOR_BACKEND=openai talks to anything serving
/chat/completions -- LM Studio, Ollama, llama.cpp, vLLM, LiteLLM,
OpenRouter, OpenAI. The TutorBackend seam already existed for this, so the
model becomes a config line rather than a code change.
Written against fetch rather than the openai package. The Anthropic SDK
alone is 14MB in the image, this backend uses one endpoint with no tools
and no retries, and local servers are the ones most likely to deviate from
an SDK's expectations. The real risk in hand-rolling it is SSE reassembly,
so that is where the tests are: a JSON payload split across two TCP reads,
an event whose blank-line terminator lands in the next read, heartbeat
comments, CRLF framing, and a stream that ends without [DONE]. The two
split cases both fail against a naive per-read parser, which is what makes
them worth having.
<think> blocks are stripped from the stream, tags split across chunks
included. Reasoning models served locally often emit chain-of-thought
inline in `content` rather than in a separate field, and left in it lands
in the lesson transcript where the block parser reads it as prose.
WHAT THIS COSTS: prompt caching. The Anthropic backend marks the ~12k
character gate as a cached prefix, so every turn after the first reads it
at a fraction of the input price. There is no portable equivalent, so
against a paid hosted endpoint the system prompt is re-billed every turn --
the biggest cost lever in the design, gone.
Against a local model it costs nothing, and the shape still pays: llama.cpp
and LM Studio reuse their KV cache for an unchanged prefix. Measured on a
6,948-character prompt against gpt-oss-20b, first token 1,563ms cold and
324ms warm, so the system prompt goes first and stays put here too.
Verified against LM Studio running openai/gpt-oss-20b, not only a fake: a
turn streams from the browser through this server to the model and back,
rendered in the chat, no page errors.
Also makes test/server/http.test.ts backend-agnostic. It asserted the echo
backend's wording and so failed the moment the server was pointed at a real
model -- precisely the case a transport test should survive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compose.yaml joins the Pi's existing network as external and reaches the
Postgres and Caddy already there by name; nothing is published to the
host. The schema applies itself on boot, so there is no migration step to
run by hand.
The Caddy snippet in the README is the part worth reading. Excluding
/api/tutor from `encode` matters more than flush_interval: compression
delays the header flush until body bytes arrive and holds already-flushed
events inside an unfinished frame, which presents as a stream that hangs
rather than as an error.
Untested against the actual Pi — written from PORT.md and verified only
as far as building the image and running it against a local Postgres.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
