Survive a first full crawl: poll, time out on silence, retry downloads

The first full crawl with the file manager ran 14 minutes, downloading every
file once, and broke in three ways:

- `schulcloud refresh` reported "fetch failed" for a crawl that was
  succeeding: Node's fetch abandons a response without headers after five
  minutes. POST /api/refresh takes wait:false and the CLI polls /api/status;
  refresh_index answers after 50 s and leaves the crawl running, and
  index_status says when a first crawl is under way.
- Downloads were bounded by the 30 s request timeout, which cut 11 MB scans
  off mid-transfer. They now time out on 30 s of silence instead.
- Failures were recorded once and never retried. A download failure is now
  retried on the next crawl while an extraction failure stays final, and PDF
  text containing NUL, which Postgres refuses, is stripped.

On the re-crawl all six failed files succeeded; only two videos above the
mirror cap stay metadata-only, by design. 137 tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/cli/client.ts

@@ -103,13 +103,41 @@ export class ApiClient {
 		return (await (await this.request(`/api/manifest${query}`)).json()) as Manifest;
 	}
 
-	async refresh(courseId?: string, force = false): Promise<Record<string, unknown>> {
+	/**
+	 * Starts a re-crawl and waits for it by polling the server's status.
+	 *
+	 * Not one long request: a crawl that downloads every course file can run for
+	 * many minutes, and fetch gives up after five without response headers —
+	 * which reported "fetch failed" for a crawl that was succeeding.
+	 */
+	async refresh(
+		courseId?: string,
+		force = false,
+		onWaiting?: (seconds: number) => void,
+	): Promise<Record<string, unknown>> {
 		const response = await this.request('/api/refresh', {
 			method: 'POST',
 			headers: { 'content-type': 'application/json' },
-			body: JSON.stringify({ courseId, force }),
+			body: JSON.stringify({ courseId, force, wait: false }),
 		});
-		return (await response.json()) as Record<string, unknown>;
+		const started = (await response.json()) as { joined?: boolean; startedAt?: string | null };
+		const began = Date.now();
+
+		for (;;) {
+			await new Promise((resolve) => setTimeout(resolve, 3000));
+			const status = (await this.status()) as {
+				indexer?: { running?: boolean; lastResult?: Record<string, unknown>; lastError?: string } | null;
+			};
+			const indexer = status.indexer;
+			if (!indexer) throw new ApiError(503, 'The server has no indexer.');
+			if (indexer.running) {
+				onWaiting?.(Math.round((Date.now() - began) / 1000));
+				continue;
+			}
+			if (indexer.lastError) throw new ApiError(502, `The re-crawl failed on the server: ${indexer.lastError}`);
+			if (!indexer.lastResult) throw new ApiError(502, 'The re-crawl finished without a result.');
+			return { ...indexer.lastResult, joined: started.joined === true };
+		}
 	}
 
 	/** Streams one file's bytes. */