fix(gate): the gate was being spliced into the prompt's header sentence

assemblePrompt used template.replace("{{GATE}}", ...), and String.replace
with a string argument substitutes only the FIRST occurrence. All three
placeholders appear twice in tutor-system.md, because the document names
them in its own header paragraph before using them:

    Assembled per turn. `{{GATE}}` is `renderGate()` from `lib/gate.js`;
    `{{VARIETY}}` and `{{FOCUS}}` are one-liners built from recent state.

So the rendered gate replaced the backticked mention mid-sentence, and the
real slot further down was sent to the model as the literal text "{{GATE}}".
The mechanism that decides what the tutor is allowed to teach was delivered
in the wrong place, with a template token standing where it belonged, and
the same for VARIETY and FOCUS. Confirmed by capturing what the app
actually put on the wire: three unfilled placeholders at lines 60, 86 and
141.

Substitution is now anchored to a whole line, which is what distinguishes a
slot from a mention -- the header's are inline and backticked. The
replacement is a function because renderGate() output contains "$"
sequences that String.replace would otherwise interpret.

The existing tests could not have caught this. They ran against a synthetic
template naming each placeholder exactly once, which is precisely the
property the shipped file lacks. The new ones run against
prompt/tutor-system.md itself: no slot may survive unfilled, the header
must come through intact, and the gate must land between the profile and
the pre-flight check. Both fail against the old code.

Also appends a HOUSE STYLE section after the shipped prompt -- an addition
by the app, not an edit to the file, which still ships byte-identical.
It covers two things the prompt leaves to inference. The language of
explanation is never actually stated: "한글 and English only" is a rule
about retiring romanization, and everything else only implies English. A
strong model infers it; gpt-oss-20b delivered a full grammar lesson in
Korean to a student on unit 1.1 who cannot yet read it. And no-markdown is
stated outright at line 96 and was ignored anyway, so it is restated where
the consequence is visible: the app renders **bold** and nothing else, so a
table arrives as rows of literal pipes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-09-10 07:09:59 +02:00
parent 7b9c92eb98
commit 9a925a5e3f
2 changed files with 105 additions and 4 deletions

View File

@@ -154,11 +154,64 @@ export interface PromptInputs {
* Assemble the system prompt. The template is prompt/tutor-system.md,
* shipped unchanged — this fills its three placeholders and nothing else.
*/
/* Appended by the app, after the shipped prompt — not an edit to it.
tutor-system.md ships unchanged. Two things it leaves to inference, which
a strong model supplies on its own and a small local one does not:
THE LANGUAGE OF EXPLANATION is never actually stated. The prompt says
"한글 and English only", but that rule is about retiring romanization, not
about which language to teach in. Everything else only implies it —
English translations, English glosses. gpt-oss-20b read the room
differently and delivered a whole grammar lesson in Korean, to a student
on unit 1.1 who cannot yet read it.
NO MARKDOWN is stated outright at line 96, and was ignored anyway. It is
restated here because the consequence is invisible to the model: the app
renders **bold** and nothing else, so a table arrives as rows of literal
pipes and a heading as literal hashes.
Both are reinforcement, never contradiction. If the shipped prompt and
this ever disagree, the shipped prompt is right and this should go. */
const HOUSE_STYLE = `
════ HOUSE STYLE ════
Explain in English. Korean is the material — words, examples, exercise lines, anything you are asking him to read. It is not the language you teach in. He is a beginner working through the writing system; an explanation he cannot read teaches him nothing.
Plain text only. No tables, headings, bullet characters or code fences — the app renders **bold** and nothing else, so anything else reaches him as literal pipes and hashes.`;
/**
* Substitute one placeholder — the one standing alone on its own line.
*
* This was `template.replace("{{GATE}}", …)`, which is wrong twice over.
* String.replace with a string argument replaces only the FIRST occurrence,
* and tutor-system.md names all three placeholders in its own header
* paragraph before using them:
*
* Assembled per turn. `{{GATE}}` is `renderGate()` from `lib/gate.js`;
* `{{VARIETY}}` and `{{FOCUS}}` are one-liners built from recent state.
*
* So the gate was spliced into the middle of that sentence and the real slot
* further down was shipped to the model as the literal text "{{GATE}}" —
* the curriculum gate, the whole mechanism that decides what may be taught,
* delivered in the wrong place with a template token where it belonged.
*
* Anchoring to a whole line fixes it: the header mentions are inline and
* backticked, the real slots are alone on a line. A function replacement is
* used because the rendered gate contains "$" sequences that String.replace
* would otherwise interpret.
*/
function fill(template: string, name: string, value: string): string {
const token = `{{${name}}}`;
const line = new RegExp(`^${token.replace(/[{}]/g, "\\$&")}[ \\t]*$`, "m");
if (line.test(template)) return template.replace(line, () => value);
// A template that inlines the placeholder still works.
return template.replace(token, () => value);
}
export function assemblePrompt({ template, gate, recent, focus }: PromptInputs): string {
return template
.replace("{{GATE}}", renderGate(gate))
.replace("{{VARIETY}}", varietyLine(recent))
.replace("{{FOCUS}}", focusLine(focus));
let out = fill(template, "GATE", renderGate(gate));
out = fill(out, "VARIETY", varietyLine(recent));
out = fill(out, "FOCUS", focusLine(focus));
return out + HOUSE_STYLE;
}
export { renderGate, REFERENCE_BAND };

View File

@@ -4,6 +4,7 @@
the prompt assembly and the confidence clamp. */
import { describe, it, expect } from "vitest";
import { readFileSync } from "node:fs";
import {
UNITS,
VOCAB_CAP,
@@ -158,6 +159,13 @@ describe("vocabQuery is never wider than the band", () => {
describe("the prompt", () => {
const TEMPLATE = "BEFORE\n{{GATE}}\nMIDDLE\n{{VARIETY}}\nAFTER\n{{FOCUS}}";
/* The shipped prompt, byte for byte. The synthetic TEMPLATE above names
each placeholder exactly once, and that is precisely the property the
real file does not have — it mentions all three in its own header
paragraph before using them, which is how a first-occurrence replace
went unnoticed. Anything about substitution has to be asserted here. */
const SHIPPED = readFileSync(new URL("../../prompt/tutor-system.md", import.meta.url), "utf8");
it("fills all three placeholders and leaves the rest alone", () => {
const gate = gateFor({ progress: at("2.1"), bandQuery: bandOffering("학교") });
const out = assemblePrompt({
@@ -172,6 +180,46 @@ describe("the prompt", () => {
expect(out).toContain("AFTER");
});
it("leaves no placeholder in the SHIPPED prompt, which names each one twice", () => {
const gate = gateFor({ progress: at("2.1"), bandQuery: bandOffering("학교") });
const out = assemblePrompt({ template: SHIPPED, gate, recent: ["translate"], focus: "sentence" });
// The header paragraph mentions all three inline, in backticks.
expect(SHIPPED.match(/\{\{GATE\}\}/g)).toHaveLength(2);
/* No placeholder may survive as a line of its own — that is a slot that
did not get filled. The backticked mentions in the header are prose
about how assembly works and are left exactly as written. */
for (const name of ["GATE", "VARIETY", "FOCUS"]) {
const unfilled = new RegExp(`^\\{\\{${name}\\}\\}[ \\t]*$`, "m");
expect(unfilled.test(out), `{{${name}}} slot was never filled`).toBe(false);
}
});
it("puts the gate in its own slot, not into the header sentence", () => {
const gate = gateFor({ progress: at("2.1"), bandQuery: bandOffering("학교") });
const out = assemblePrompt({ template: SHIPPED, gate, recent: [], focus: "auto" });
// The header must survive intact, with its backticked mentions.
expect(out).toContain("`{{GATE}}` is `renderGate()`");
// And the gate must land after the profile and before the pre-flight
// check, which is where the shipped file puts the slot.
const knows = out.indexOf("WHAT HE KNOWS");
expect(knows).toBeGreaterThan(out.indexOf("Register: manhwa is written"));
expect(knows).toBeLessThan(out.indexOf("PRE-FLIGHT CHECK"));
});
it("survives a gate containing $ sequences, which replace() would eat", () => {
const out = assemblePrompt({
template: "A\n{{GATE}}\nB\n{{VARIETY}}\n{{FOCUS}}",
gate: { ...gateFor({ progress: at("1.1"), bandQuery: bandOffering() }) },
recent: [],
focus: "auto",
});
expect(out).not.toContain("$&");
});
it("renders the gate's own headings — the model keys off them", () => {
const gate = gateFor({ progress: at("2.1"), bandQuery: bandOffering("학교") });
const out = assemblePrompt({ template: TEMPLATE, gate, recent: [], focus: "auto" });