feat(tutor): the real endpoint, streamed, with the system prompt cached

POST /api/tutor takes the assembled system prompt, the transcript the
client owns, and the new message, and streams tokens back. It holds
nothing between requests, so a dropped connection costs one turn rather
than the conversation.

The seam moved first: Sample took the assembled prompt as messages[0] with
role user. It now has an explicit system field, which is what lets the
backend put it in the API's system parameter as a cached block. The gate
is ~12k characters and is byte-identical for as long as the learner stays
in one unit, so every turn after the first reads the prefix at a fraction
of the input price. That is the single biggest cost lever in the design,
and it was unreachable through the old shape.

prompt/tutor-system.md still ships unchanged; only where the string is
placed changed.

SSE has three rules that are silent when broken, and all three are
handled: every event ends with a blank line, payloads are JSON-encoded
because a raw newline in Korean text would break the framing, and a `:`
heartbeat every 15s keeps intermediaries from timing the stream out.
Cache-Control is set on the returned Response rather than inside
streamSSE, which writes its own and would overwrite it; `no-transform` is
there because compression, not buffering, is what usually makes SSE look
like it hangs behind a proxy.

The client uses fetch + getReader, not EventSource — EventSource cannot
POST, and the body is {system, history, message}. Aborting closes the
connection, the server aborts upstream, and a cancelled turn stops
billing. With no server configured the app falls back to the stub, so the
offline build is untouched.

backends/anthropic.ts is the default. backends/agent-sdk.ts is deliberately
unimplemented and documents why the plain API was chosen over PORT.md's
Agent SDK — chiefly that its prompt accepts only user-role messages, so
the transcript would have to be flattened into a single turn.

The endpoint's own tests use a mock backend and need no API key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-09-08 19:58:01 +02:00
parent c2e1fc23fe
commit 66f92247d2
8 changed files with 582 additions and 6 deletions

103
server/src/tutor.ts Normal file
View File

@@ -0,0 +1,103 @@
/* POST /api/tutor — one turn in, a token stream out.
No session state. The client posts the assembled system prompt, the
transcript it owns, and the new message; the server holds nothing between
requests, so a dropped connection costs one turn rather than the
conversation.
SSE framing has three rules that are easy to get wrong and silent when
broken: every event ends with a blank line, a `data:` field may not
contain a raw newline (Korean tutor replies are full of them, so payloads
are JSON-encoded), and a comment line starting with `:` is a heartbeat
that keeps intermediaries from timing the stream out. */
import { Hono } from "hono";
import { streamSSE } from "hono/streaming";
import { anthropicBackend } from "./backends/anthropic.ts";
import type { TutorBackend, TutorTurn } from "./backends/types.ts";
const HEARTBEAT_MS = 15_000;
/** Bound the transcript the client may send, so one request cannot be huge. */
const MAX_HISTORY = 40;
function pickBackend(): TutorBackend {
const name = process.env.HANKAN_TUTOR_BACKEND ?? "anthropic";
if (name === "anthropic") return anthropicBackend();
throw new Error(`unknown HANKAN_TUTOR_BACKEND: ${name}`);
}
export function tutorRoute(backend?: TutorBackend) {
const app = new Hono();
// Constructed lazily so the process can start — and serve sync — without
// an API key present.
let resolved: TutorBackend | null = backend ?? null;
app.post("/", async (c) => {
const body = (await c.req.json()) as {
system?: string;
history?: TutorTurn[];
message?: string;
};
if (!body.system || !body.message) {
return c.json({ error: "system and message are required" }, 400);
}
const history = (Array.isArray(body.history) ? body.history : [])
.filter((t) => t && (t.role === "user" || t.role === "assistant") && typeof t.content === "string")
.slice(-MAX_HISTORY);
try {
resolved ??= pickBackend();
} catch (err) {
return c.json({ error: err instanceof Error ? err.message : "no backend" }, 503);
}
const res = streamSSE(c, async (stream) => {
const abort = new AbortController();
// The client going away is the cancel signal: abort upstream so a
// cancelled turn stops billing.
stream.onAbort(() => abort.abort());
const beat = setInterval(() => {
void stream.writeSSE({ data: "", event: "ping" });
}, HEARTBEAT_MS);
try {
for await (const event of resolved!.stream({
system: body.system!,
history,
message: body.message!,
signal: abort.signal,
})) {
await stream.writeSSE({ event: event.type, data: JSON.stringify(event) });
if (event.type === "done" || event.type === "error") break;
}
} catch (err) {
await stream.writeSSE({
event: "error",
data: JSON.stringify({
type: "error",
code: "server",
message: err instanceof Error ? err.message : String(err),
}),
});
} finally {
clearInterval(beat);
}
});
/* Set after streamSSE, which writes its own Cache-Control and would
overwrite these. `no-transform` tells intermediaries not to re-encode
the body — compression is the usual reason SSE appears to hang behind
a proxy, because an unfinished compression frame holds the events.
X-Accel-Buffering is nginx's opt-out and harmless elsewhere. */
res.headers.set("Cache-Control", "no-cache, no-transform");
res.headers.set("X-Accel-Buffering", "no");
return res;
});
return app;
}