fabi/Hankan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(server): default max_tokens 2048 -> 8192, which was starving the lesson

Browse Source 
On a reasoning model the reasoning is spent from the same completion budget
and it goes first, so a small ceiling truncates the actual reply away. It
fails silently and looks exactly like a model that cannot follow the system
prompt, which is the expensive way to debug it.

Measured on gpt-oss-20b against the real ~3.5k-token system prompt, same
prompt and same model, only the ceiling changing:

  1400  an empty string, 0 bytes, after 27s
  2048  a truncated half-Korean fragment, none of the required blocks
  8000  99% English prose, no romanization, 0 batchim violations in the
        task lines, all three blocks, 11s

The old comment justified 2048 by worrying about 4k-context local models.
That reasoning was wrong: the system prompt alone is ~3.5k tokens, so such
a model cannot run this app at all and there was nothing to protect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-09-10 07:17:16 +02:00
parent 9a925a5e3f
commit 988d33bd5f
3 changed files with 28 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
server/src/backends/openai.ts
View File

@@ -38,10 +38,20 @@ const DEFAULT_BASE_URL = "http://localhost:1234/v1";
const DEFAULT_MODEL = "local-model";
/* A tutor turn is a lesson intro at most — the prompt caps it at 250-450
words. Deliberately far below the Anthropic backend's 16k: this one may
be pointed at a 4k-context local model, where asking for more completion
tokens than the context holds is an outright error rather than a ceiling. */
const DEFAULT_MAX_TOKENS = 2048;
words, so roughly 1,200 tokens of visible text.
This is 8k anyway, because on a reasoning model the reasoning is spent
from the SAME budget and it goes first. Measured on gpt-oss-20b with the
real ~3.5k-token system prompt: at 1,400 it returned an empty string, at
2,048 a truncated fragment of Korean with none of the required blocks,
and at 8,000 a correct English lesson with all three. The failure is
silent and looks exactly like a model that cannot follow instructions,
which is the expensive way to debug it.
An earlier note here worried about 4k-context models. That was wrong:
the system prompt alone is ~3.5k tokens, so such a model cannot run this
app at all and there is nothing to protect. */
const DEFAULT_MAX_TOKENS = 8192;
/**
* Strip <think> blocks from a token stream.