feat(tutor): the real endpoint, streamed, with the system prompt cached
POST /api/tutor takes the assembled system prompt, the transcript the
client owns, and the new message, and streams tokens back. It holds
nothing between requests, so a dropped connection costs one turn rather
than the conversation.
The seam moved first: Sample took the assembled prompt as messages[0] with
role user. It now has an explicit system field, which is what lets the
backend put it in the API's system parameter as a cached block. The gate
is ~12k characters and is byte-identical for as long as the learner stays
in one unit, so every turn after the first reads the prefix at a fraction
of the input price. That is the single biggest cost lever in the design,
and it was unreachable through the old shape.
prompt/tutor-system.md still ships unchanged; only where the string is
placed changed.
SSE has three rules that are silent when broken, and all three are
handled: every event ends with a blank line, payloads are JSON-encoded
because a raw newline in Korean text would break the framing, and a `:`
heartbeat every 15s keeps intermediaries from timing the stream out.
Cache-Control is set on the returned Response rather than inside
streamSSE, which writes its own and would overwrite it; `no-transform` is
there because compression, not buffering, is what usually makes SSE look
like it hangs behind a proxy.
The client uses fetch + getReader, not EventSource — EventSource cannot
POST, and the body is {system, history, message}. Aborting closes the
connection, the server aborts upstream, and a cancelled turn stops
billing. With no server configured the app falls back to the stub, so the
offline build is untouched.
backends/anthropic.ts is the default. backends/agent-sdk.ts is deliberately
unimplemented and documents why the plain API was chosen over PORT.md's
Agent SDK — chiefly that its prompt accepts only user-role messages, so
the transcript would have to be flattened into a single turn.
The endpoint's own tests use a mock backend and need no API key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
103
server/src/tutor.ts
Normal file
103
server/src/tutor.ts
Normal file
@@ -0,0 +1,103 @@
|
||||
/* POST /api/tutor — one turn in, a token stream out.
|
||||
|
||||
No session state. The client posts the assembled system prompt, the
|
||||
transcript it owns, and the new message; the server holds nothing between
|
||||
requests, so a dropped connection costs one turn rather than the
|
||||
conversation.
|
||||
|
||||
SSE framing has three rules that are easy to get wrong and silent when
|
||||
broken: every event ends with a blank line, a `data:` field may not
|
||||
contain a raw newline (Korean tutor replies are full of them, so payloads
|
||||
are JSON-encoded), and a comment line starting with `:` is a heartbeat
|
||||
that keeps intermediaries from timing the stream out. */
|
||||
|
||||
import { Hono } from "hono";
|
||||
import { streamSSE } from "hono/streaming";
|
||||
|
||||
import { anthropicBackend } from "./backends/anthropic.ts";
|
||||
import type { TutorBackend, TutorTurn } from "./backends/types.ts";
|
||||
|
||||
const HEARTBEAT_MS = 15_000;
|
||||
|
||||
/** Bound the transcript the client may send, so one request cannot be huge. */
|
||||
const MAX_HISTORY = 40;
|
||||
|
||||
function pickBackend(): TutorBackend {
|
||||
const name = process.env.HANKAN_TUTOR_BACKEND ?? "anthropic";
|
||||
if (name === "anthropic") return anthropicBackend();
|
||||
throw new Error(`unknown HANKAN_TUTOR_BACKEND: ${name}`);
|
||||
}
|
||||
|
||||
export function tutorRoute(backend?: TutorBackend) {
|
||||
const app = new Hono();
|
||||
// Constructed lazily so the process can start — and serve sync — without
|
||||
// an API key present.
|
||||
let resolved: TutorBackend | null = backend ?? null;
|
||||
|
||||
app.post("/", async (c) => {
|
||||
const body = (await c.req.json()) as {
|
||||
system?: string;
|
||||
history?: TutorTurn[];
|
||||
message?: string;
|
||||
};
|
||||
|
||||
if (!body.system || !body.message) {
|
||||
return c.json({ error: "system and message are required" }, 400);
|
||||
}
|
||||
|
||||
const history = (Array.isArray(body.history) ? body.history : [])
|
||||
.filter((t) => t && (t.role === "user" || t.role === "assistant") && typeof t.content === "string")
|
||||
.slice(-MAX_HISTORY);
|
||||
|
||||
try {
|
||||
resolved ??= pickBackend();
|
||||
} catch (err) {
|
||||
return c.json({ error: err instanceof Error ? err.message : "no backend" }, 503);
|
||||
}
|
||||
|
||||
const res = streamSSE(c, async (stream) => {
|
||||
const abort = new AbortController();
|
||||
// The client going away is the cancel signal: abort upstream so a
|
||||
// cancelled turn stops billing.
|
||||
stream.onAbort(() => abort.abort());
|
||||
|
||||
const beat = setInterval(() => {
|
||||
void stream.writeSSE({ data: "", event: "ping" });
|
||||
}, HEARTBEAT_MS);
|
||||
|
||||
try {
|
||||
for await (const event of resolved!.stream({
|
||||
system: body.system!,
|
||||
history,
|
||||
message: body.message!,
|
||||
signal: abort.signal,
|
||||
})) {
|
||||
await stream.writeSSE({ event: event.type, data: JSON.stringify(event) });
|
||||
if (event.type === "done" || event.type === "error") break;
|
||||
}
|
||||
} catch (err) {
|
||||
await stream.writeSSE({
|
||||
event: "error",
|
||||
data: JSON.stringify({
|
||||
type: "error",
|
||||
code: "server",
|
||||
message: err instanceof Error ? err.message : String(err),
|
||||
}),
|
||||
});
|
||||
} finally {
|
||||
clearInterval(beat);
|
||||
}
|
||||
});
|
||||
|
||||
/* Set after streamSSE, which writes its own Cache-Control and would
|
||||
overwrite these. `no-transform` tells intermediaries not to re-encode
|
||||
the body — compression is the usual reason SSE appears to hang behind
|
||||
a proxy, because an unfinished compression frame holds the events.
|
||||
X-Accel-Buffering is nginx's opt-out and harmless elsewhere. */
|
||||
res.headers.set("Cache-Control", "no-cache, no-transform");
|
||||
res.headers.set("X-Accel-Buffering", "no");
|
||||
return res;
|
||||
});
|
||||
|
||||
return app;
|
||||
}
|
||||
Reference in New Issue
Block a user