Survive a first full crawl: poll, time out on silence, retry downloads

The first full crawl with the file manager ran 14 minutes, downloading every
file once, and broke in three ways:

- `schulcloud refresh` reported "fetch failed" for a crawl that was
  succeeding: Node's fetch abandons a response without headers after five
  minutes. POST /api/refresh takes wait:false and the CLI polls /api/status;
  refresh_index answers after 50 s and leaves the crawl running, and
  index_status says when a first crawl is under way.
- Downloads were bounded by the 30 s request timeout, which cut 11 MB scans
  off mid-transfer. They now time out on 30 s of silence instead.
- Failures were recorded once and never retried. A download failure is now
  retried on the next crawl while an extraction failure stays final, and PDF
  text containing NUL, which Postgres refuses, is stripped.

On the re-crawl all six failed files succeeded; only two videos above the
mirror cap stay metadata-only, by design. 137 tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/http/api.ts

@@ -62,9 +62,22 @@ export function createApiRouter(services: Services): Router {
 
 	router.post('/refresh', express.json({ limit: '16kb' }), async (req: Request, res: Response) => {
 		if (!services.indexer) return res.status(503).json({ error: 'no_index' });
-		const body = (req.body ?? {}) as { courseId?: string; force?: boolean };
+		const body = (req.body ?? {}) as { courseId?: string; force?: boolean; wait?: boolean };
 		try {
-			const result = await services.indexer.refresh(body.courseId ?? 'full', { force: body.force === true });
+			const scope = body.courseId ?? 'full';
+			// `wait: false` answers at once and leaves the caller to poll /status.
+			// Waiting for the result in this request is kept for older clients, but
+			// it cannot outlast a long crawl: Node's fetch abandons a response whose
+			// headers have not arrived within five minutes.
+			if (body.wait === false) {
+				const { run, joined } = services.indexer.start(scope, { force: body.force === true });
+				// The outcome is recorded by the indexer (status().lastResult/lastError);
+				// this only keeps an unawaited failure from becoming an unhandled one.
+				run.catch(() => {});
+				const status = services.indexer.status();
+				return res.status(202).json({ started: !joined, joined, scope, startedAt: status.startedAt ?? null });
+			}
+			const result = await services.indexer.refresh(scope, { force: body.force === true });
 			return res.json(result);
 		} catch (error) {
 			// A rate-limit refusal is the caller's problem to act on, not a fault.