fix(gate): the gate was being spliced into the prompt's header sentence
assemblePrompt used template.replace("{{GATE}}", ...), and String.replace
with a string argument substitutes only the FIRST occurrence. All three
placeholders appear twice in tutor-system.md, because the document names
them in its own header paragraph before using them:
Assembled per turn. `{{GATE}}` is `renderGate()` from `lib/gate.js`;
`{{VARIETY}}` and `{{FOCUS}}` are one-liners built from recent state.
So the rendered gate replaced the backticked mention mid-sentence, and the
real slot further down was sent to the model as the literal text "{{GATE}}".
The mechanism that decides what the tutor is allowed to teach was delivered
in the wrong place, with a template token standing where it belonged, and
the same for VARIETY and FOCUS. Confirmed by capturing what the app
actually put on the wire: three unfilled placeholders at lines 60, 86 and
141.
Substitution is now anchored to a whole line, which is what distinguishes a
slot from a mention -- the header's are inline and backticked. The
replacement is a function because renderGate() output contains "$"
sequences that String.replace would otherwise interpret.
The existing tests could not have caught this. They ran against a synthetic
template naming each placeholder exactly once, which is precisely the
property the shipped file lacks. The new ones run against
prompt/tutor-system.md itself: no slot may survive unfilled, the header
must come through intact, and the gate must land between the profile and
the pre-flight check. Both fail against the old code.
Also appends a HOUSE STYLE section after the shipped prompt -- an addition
by the app, not an edit to the file, which still ships byte-identical.
It covers two things the prompt leaves to inference. The language of
explanation is never actually stated: "한글 and English only" is a rule
about retiring romanization, and everything else only implies English. A
strong model infers it; gpt-oss-20b delivered a full grammar lesson in
Korean to a student on unit 1.1 who cannot yet read it. And no-markdown is
stated outright at line 96 and was ignored anyway, so it is restated where
the consequence is visible: the app renders **bold** and nothing else, so a
table arrives as rows of literal pipes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -154,11 +154,64 @@ export interface PromptInputs {
|
||||
* Assemble the system prompt. The template is prompt/tutor-system.md,
|
||||
* shipped unchanged — this fills its three placeholders and nothing else.
|
||||
*/
|
||||
/* Appended by the app, after the shipped prompt — not an edit to it.
|
||||
|
||||
tutor-system.md ships unchanged. Two things it leaves to inference, which
|
||||
a strong model supplies on its own and a small local one does not:
|
||||
|
||||
THE LANGUAGE OF EXPLANATION is never actually stated. The prompt says
|
||||
"한글 and English only", but that rule is about retiring romanization, not
|
||||
about which language to teach in. Everything else only implies it —
|
||||
English translations, English glosses. gpt-oss-20b read the room
|
||||
differently and delivered a whole grammar lesson in Korean, to a student
|
||||
on unit 1.1 who cannot yet read it.
|
||||
|
||||
NO MARKDOWN is stated outright at line 96, and was ignored anyway. It is
|
||||
restated here because the consequence is invisible to the model: the app
|
||||
renders **bold** and nothing else, so a table arrives as rows of literal
|
||||
pipes and a heading as literal hashes.
|
||||
|
||||
Both are reinforcement, never contradiction. If the shipped prompt and
|
||||
this ever disagree, the shipped prompt is right and this should go. */
|
||||
const HOUSE_STYLE = `
|
||||
════ HOUSE STYLE ════
|
||||
Explain in English. Korean is the material — words, examples, exercise lines, anything you are asking him to read. It is not the language you teach in. He is a beginner working through the writing system; an explanation he cannot read teaches him nothing.
|
||||
Plain text only. No tables, headings, bullet characters or code fences — the app renders **bold** and nothing else, so anything else reaches him as literal pipes and hashes.`;
|
||||
|
||||
/**
|
||||
* Substitute one placeholder — the one standing alone on its own line.
|
||||
*
|
||||
* This was `template.replace("{{GATE}}", …)`, which is wrong twice over.
|
||||
* String.replace with a string argument replaces only the FIRST occurrence,
|
||||
* and tutor-system.md names all three placeholders in its own header
|
||||
* paragraph before using them:
|
||||
*
|
||||
* Assembled per turn. `{{GATE}}` is `renderGate()` from `lib/gate.js`;
|
||||
* `{{VARIETY}}` and `{{FOCUS}}` are one-liners built from recent state.
|
||||
*
|
||||
* So the gate was spliced into the middle of that sentence and the real slot
|
||||
* further down was shipped to the model as the literal text "{{GATE}}" —
|
||||
* the curriculum gate, the whole mechanism that decides what may be taught,
|
||||
* delivered in the wrong place with a template token where it belonged.
|
||||
*
|
||||
* Anchoring to a whole line fixes it: the header mentions are inline and
|
||||
* backticked, the real slots are alone on a line. A function replacement is
|
||||
* used because the rendered gate contains "$" sequences that String.replace
|
||||
* would otherwise interpret.
|
||||
*/
|
||||
function fill(template: string, name: string, value: string): string {
|
||||
const token = `{{${name}}}`;
|
||||
const line = new RegExp(`^${token.replace(/[{}]/g, "\\$&")}[ \\t]*$`, "m");
|
||||
if (line.test(template)) return template.replace(line, () => value);
|
||||
// A template that inlines the placeholder still works.
|
||||
return template.replace(token, () => value);
|
||||
}
|
||||
|
||||
export function assemblePrompt({ template, gate, recent, focus }: PromptInputs): string {
|
||||
return template
|
||||
.replace("{{GATE}}", renderGate(gate))
|
||||
.replace("{{VARIETY}}", varietyLine(recent))
|
||||
.replace("{{FOCUS}}", focusLine(focus));
|
||||
let out = fill(template, "GATE", renderGate(gate));
|
||||
out = fill(out, "VARIETY", varietyLine(recent));
|
||||
out = fill(out, "FOCUS", focusLine(focus));
|
||||
return out + HOUSE_STYLE;
|
||||
}
|
||||
|
||||
export { renderGate, REFERENCE_BAND };
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
the prompt assembly and the confidence clamp. */
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { readFileSync } from "node:fs";
|
||||
import {
|
||||
UNITS,
|
||||
VOCAB_CAP,
|
||||
@@ -158,6 +159,13 @@ describe("vocabQuery is never wider than the band", () => {
|
||||
describe("the prompt", () => {
|
||||
const TEMPLATE = "BEFORE\n{{GATE}}\nMIDDLE\n{{VARIETY}}\nAFTER\n{{FOCUS}}";
|
||||
|
||||
/* The shipped prompt, byte for byte. The synthetic TEMPLATE above names
|
||||
each placeholder exactly once, and that is precisely the property the
|
||||
real file does not have — it mentions all three in its own header
|
||||
paragraph before using them, which is how a first-occurrence replace
|
||||
went unnoticed. Anything about substitution has to be asserted here. */
|
||||
const SHIPPED = readFileSync(new URL("../../prompt/tutor-system.md", import.meta.url), "utf8");
|
||||
|
||||
it("fills all three placeholders and leaves the rest alone", () => {
|
||||
const gate = gateFor({ progress: at("2.1"), bandQuery: bandOffering("학교") });
|
||||
const out = assemblePrompt({
|
||||
@@ -172,6 +180,46 @@ describe("the prompt", () => {
|
||||
expect(out).toContain("AFTER");
|
||||
});
|
||||
|
||||
it("leaves no placeholder in the SHIPPED prompt, which names each one twice", () => {
|
||||
const gate = gateFor({ progress: at("2.1"), bandQuery: bandOffering("학교") });
|
||||
const out = assemblePrompt({ template: SHIPPED, gate, recent: ["translate"], focus: "sentence" });
|
||||
|
||||
// The header paragraph mentions all three inline, in backticks.
|
||||
expect(SHIPPED.match(/\{\{GATE\}\}/g)).toHaveLength(2);
|
||||
|
||||
/* No placeholder may survive as a line of its own — that is a slot that
|
||||
did not get filled. The backticked mentions in the header are prose
|
||||
about how assembly works and are left exactly as written. */
|
||||
for (const name of ["GATE", "VARIETY", "FOCUS"]) {
|
||||
const unfilled = new RegExp(`^\\{\\{${name}\\}\\}[ \\t]*$`, "m");
|
||||
expect(unfilled.test(out), `{{${name}}} slot was never filled`).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("puts the gate in its own slot, not into the header sentence", () => {
|
||||
const gate = gateFor({ progress: at("2.1"), bandQuery: bandOffering("학교") });
|
||||
const out = assemblePrompt({ template: SHIPPED, gate, recent: [], focus: "auto" });
|
||||
|
||||
// The header must survive intact, with its backticked mentions.
|
||||
expect(out).toContain("`{{GATE}}` is `renderGate()`");
|
||||
|
||||
// And the gate must land after the profile and before the pre-flight
|
||||
// check, which is where the shipped file puts the slot.
|
||||
const knows = out.indexOf("WHAT HE KNOWS");
|
||||
expect(knows).toBeGreaterThan(out.indexOf("Register: manhwa is written"));
|
||||
expect(knows).toBeLessThan(out.indexOf("PRE-FLIGHT CHECK"));
|
||||
});
|
||||
|
||||
it("survives a gate containing $ sequences, which replace() would eat", () => {
|
||||
const out = assemblePrompt({
|
||||
template: "A\n{{GATE}}\nB\n{{VARIETY}}\n{{FOCUS}}",
|
||||
gate: { ...gateFor({ progress: at("1.1"), bandQuery: bandOffering() }) },
|
||||
recent: [],
|
||||
focus: "auto",
|
||||
});
|
||||
expect(out).not.toContain("$&");
|
||||
});
|
||||
|
||||
it("renders the gate's own headings — the model keys off them", () => {
|
||||
const gate = gateFor({ progress: at("2.1"), bandQuery: bandOffering("학교") });
|
||||
const out = assemblePrompt({ template: TEMPLATE, gate, recent: [], focus: "auto" });
|
||||
|
||||
Reference in New Issue
Block a user