Add Postgres store, path safety, and the crawl indexer

Store: crawl generations as the sync cursor. Diffs compare generations
on entity identity plus a content digest, never on upstream timestamps —
GET /course-rooms/{id}/board returns request time as updatedAt for most
elements, so a timestamp cursor would report every board as changed on
every crawl. Identity diffing also yields deletions, which no timestamp
scheme can. A per-course crawl carries the other courses' rows forward
so every completed generation is a complete picture and any two diff
directly; without that a partial crawl reads as a mass deletion.

FTS uses the german dictionary with weighted title/body, plus a pg_trgm
arm because stemming will not match "Datenschutz" inside
"Datenschutzgrundverordnung" and German compounds make that the common
case. file_texts is keyed by file record id and deliberately outlives
generations: records are immutable upstream, so text extracted once is
valid forever and a re-crawl of unchanged content costs nothing.

Store.open returns undefined instead of throwing when Postgres is
unreachable — the index is an accelerator, and a Pi that loses its
database should get slower, not broken.

core/paths.ts is the security boundary for the mirror. Course titles,
card titles and filenames are all user-supplied upstream, so this is
where a hostile name stops being text and becomes a path. Two bugs found
by its own tests: "///" produced "---" instead of falling back, and dot
runs survived mid-component. Now no ".." can survive anywhere, which
makes the invariant checkable rather than a claim about ordering.

Indexer coalesces concurrent refreshes onto one run and enforces a
minimum interval, since a full crawl is ~270 requests from an account
that looks like a student.

9 store tests against a real Postgres (mocks would test nothing here)
and 13 path tests; 47 total.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-12 21:10:53 +02:00
parent 81dd633863
commit a18b526267
11 changed files with 1547 additions and 1 deletions

View File

@@ -23,9 +23,18 @@ export interface Config {
/**
* How often to ping the instance to hold the session open. Must stay well
* under the instance's `JWT_TIMEOUT_SECONDS` (7200s here) — see
* src/keepalive.ts. Zero disables the keepalive.
* src/core/keepalive.ts. Zero disables the keepalive.
*/
keepaliveIntervalMs: number;
/** Postgres for the search index and file mirror. Unset = live-only mode. */
databaseUrl: string | undefined;
/** Where mirrored file bytes live on disk. */
mirrorDir: string;
/** Files larger than this are indexed as metadata but not mirrored. */
mirrorMaxBytes: number;
/** How often to re-crawl on a timer. Zero = only on demand. */
crawlIntervalMs: number;
}
function required(name: string): string {
@@ -66,5 +75,9 @@ export function loadConfig(): Config {
maxExtractedChars: int('MAX_EXTRACTED_CHARS', 120_000),
requestTimeoutMs: int('REQUEST_TIMEOUT_MS', 30_000),
keepaliveIntervalMs: intAllowingZero('KEEPALIVE_INTERVAL_MS', 30 * 60_000),
databaseUrl: process.env.DATABASE_URL?.trim() || undefined,
mirrorDir: process.env.MIRROR_DIR?.trim() || '/var/lib/schulcloud-mcp/mirror',
mirrorMaxBytes: int('MIRROR_MAX_BYTES', 64 * 1024 * 1024),
crawlIntervalMs: intAllowingZero('CRAWL_INTERVAL_MS', 6 * 60 * 60_000),
};
}