Survive a first full crawl: poll, time out on silence, retry downloads

The first full crawl with the file manager ran 14 minutes, downloading every
file once, and broke in three ways:

- `schulcloud refresh` reported "fetch failed" for a crawl that was
  succeeding: Node's fetch abandons a response without headers after five
  minutes. POST /api/refresh takes wait:false and the CLI polls /api/status;
  refresh_index answers after 50 s and leaves the crawl running, and
  index_status says when a first crawl is under way.
- Downloads were bounded by the 30 s request timeout, which cut 11 MB scans
  off mid-transfer. They now time out on 30 s of silence instead.
- Failures were recorded once and never retried. A download failure is now
  retried on the next crawl while an extraction failure stays final, and PDF
  text containing NUL, which Postgres refuses, is stripped.

On the re-crawl all six failed files succeeded; only two videos above the
mirror cap stay metadata-only, by design. 137 tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -147,6 +147,18 @@ These cost real time to discover; `docs/API.md` has the full list with evidence.
 - **Never swallow a per-item crawl error.** Board failures used to be caught and
   dropped, so the index lost whole boards while the crawl reported success —
   which is how the 20-id limit went unnoticed. They go into `Snapshot.failures`.
+- **Record failures by kind, or transient ones become permanent.** Every failed
+  file used to be marked done, so a timeout was never retried. A *download*
+  failure is now recorded with `retry: true` and the next crawl tries again; an
+  *extraction* failure is final. Two causes found on the first full crawl of a
+  real account: downloads bounded by the 30 s request timeout (11 MB scans cut
+  off mid-transfer — downloads now time out on 30 s of *silence*), and PDF text
+  containing NUL, which Postgres `text` refuses — stripped in `recordFileText`.
+- **A full crawl can outlast one HTTP request.** The first one with the file
+  manager took 14 minutes (every file downloaded once); Node's fetch abandons a
+  response without headers after 5. `POST /api/refresh` takes `wait: false` and
+  the CLI polls `/api/status`; `refresh_index` returns after 50 s and leaves the
+  crawl running. Don't reintroduce a caller that waits on a full crawl inline.
 - **`GET /lessons/{id}/tasks` is a bare array whose items carry no id.** Not the
   `{data,total}` envelope, and `LessonLinkedTaskResponse` has no id field at
   all. A topic-attached task is thus unidentifiable from the API and invisible