Extract core/, lift the crawler out of the search tool

Moves the reusable half into src/core/ (client, types, board, extract,
text, keepalive) and the MCP half into src/mcp/. The layering was
already clean — nothing in core imported app code or read process.env —
so this is a move, not a redesign, and the smoke suite stayed the oracle
throughout.

The substantive part is core/crawl.ts. The course->board->card->element
->file traversal previously existed only inside tools/search.ts, and the
indexer, what's-new diff and file mirror all need it. It now returns a
typed Snapshot with breadcrumbs, sorted so two crawls of unchanged
content compare equal. Metadata only: downloading and extracting bytes
is an order of magnitude more expensive and only the indexer wants it.

core/match.ts holds the keyword matching, which makes it testable
without a network, and core/text.ts gains the fold/tokenize/snippet
helpers (accent folding is not optional for German).

search now finds strictly more than before — 5 hits vs 3 for
"Datenschutz" — because the snapshot surfaces file-name matches the old
streaming walk skipped. 34 unit tests and 30/30 smoke checks pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-12 21:04:52 +02:00
parent c9bcd3de31
commit 81dd633863
23 changed files with 514 additions and 264 deletions

View File

@@ -1,6 +1,6 @@
import assert from 'node:assert/strict';
import { describe, it } from 'node:test';
import { daysUntil, formatDate, htmlToText, joinSections, normalizeObjectId } from '../src/render.ts';
import { daysUntil, formatDate, htmlToText, joinSections, normalizeObjectId } from '../src/core/text.ts';
describe('htmlToText', () => {
it('unwraps the CKEditor markup Schulcloud stores', () => {