Add Postgres store, path safety, and the crawl indexer
Store: crawl generations as the sync cursor. Diffs compare generations
on entity identity plus a content digest, never on upstream timestamps —
GET /course-rooms/{id}/board returns request time as updatedAt for most
elements, so a timestamp cursor would report every board as changed on
every crawl. Identity diffing also yields deletions, which no timestamp
scheme can. A per-course crawl carries the other courses' rows forward
so every completed generation is a complete picture and any two diff
directly; without that a partial crawl reads as a mass deletion.
FTS uses the german dictionary with weighted title/body, plus a pg_trgm
arm because stemming will not match "Datenschutz" inside
"Datenschutzgrundverordnung" and German compounds make that the common
case. file_texts is keyed by file record id and deliberately outlives
generations: records are immutable upstream, so text extracted once is
valid forever and a re-crawl of unchanged content costs nothing.
Store.open returns undefined instead of throwing when Postgres is
unreachable — the index is an accelerator, and a Pi that loses its
database should get slower, not broken.
core/paths.ts is the security boundary for the mirror. Course titles,
card titles and filenames are all user-supplied upstream, so this is
where a hostile name stops being text and becomes a path. Two bugs found
by its own tests: "///" produced "---" instead of falling back, and dot
runs survived mid-component. Now no ".." can survive anywhere, which
makes the invariant checkable rather than a claim about ordering.
Indexer coalesces concurrent refreshes onto one run and enforces a
minimum interval, since a full crawl is ~270 requests from an account
that looks like a student.
9 store tests against a real Postgres (mocks would test nothing here)
and 13 path tests; 47 total.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -23,9 +23,18 @@ export interface Config {
|
||||
/**
|
||||
* How often to ping the instance to hold the session open. Must stay well
|
||||
* under the instance's `JWT_TIMEOUT_SECONDS` (7200s here) — see
|
||||
* src/keepalive.ts. Zero disables the keepalive.
|
||||
* src/core/keepalive.ts. Zero disables the keepalive.
|
||||
*/
|
||||
keepaliveIntervalMs: number;
|
||||
|
||||
/** Postgres for the search index and file mirror. Unset = live-only mode. */
|
||||
databaseUrl: string | undefined;
|
||||
/** Where mirrored file bytes live on disk. */
|
||||
mirrorDir: string;
|
||||
/** Files larger than this are indexed as metadata but not mirrored. */
|
||||
mirrorMaxBytes: number;
|
||||
/** How often to re-crawl on a timer. Zero = only on demand. */
|
||||
crawlIntervalMs: number;
|
||||
}
|
||||
|
||||
function required(name: string): string {
|
||||
@@ -66,5 +75,9 @@ export function loadConfig(): Config {
|
||||
maxExtractedChars: int('MAX_EXTRACTED_CHARS', 120_000),
|
||||
requestTimeoutMs: int('REQUEST_TIMEOUT_MS', 30_000),
|
||||
keepaliveIntervalMs: intAllowingZero('KEEPALIVE_INTERVAL_MS', 30 * 60_000),
|
||||
databaseUrl: process.env.DATABASE_URL?.trim() || undefined,
|
||||
mirrorDir: process.env.MIRROR_DIR?.trim() || '/var/lib/schulcloud-mcp/mirror',
|
||||
mirrorMaxBytes: int('MIRROR_MAX_BYTES', 64 * 1024 * 1024),
|
||||
crawlIntervalMs: intAllowingZero('CRAWL_INTERVAL_MS', 6 * 60 * 60_000),
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user