The first full crawl with the file manager ran 14 minutes, downloading every
file once, and broke in three ways:
- `schulcloud refresh` reported "fetch failed" for a crawl that was
succeeding: Node's fetch abandons a response without headers after five
minutes. POST /api/refresh takes wait:false and the CLI polls /api/status;
refresh_index answers after 50 s and leaves the crawl running, and
index_status says when a first crawl is under way.
- Downloads were bounded by the 30 s request timeout, which cut 11 MB scans
off mid-transfer. They now time out on 30 s of silence instead.
- Failures were recorded once and never retried. A download failure is now
retried on the next crawl while an extraction failure stays final, and PDF
text containing NUL, which Postgres refuses, is stripped.
On the re-crawl all six failed files succeeded; only two videos above the
mirror cap stay metadata-only, by design. 137 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The account this was built against is in no rooms, so the whole space was
invisible and easy to dismiss as an empty endpoint. It is not empty in
general — the user had rooms until a teacher removed access — and the
UI's naming actively hides the distinction: the sidebar's *Kurse* entry
links to `/rooms/courses-overview` and lists courses, while *Räume* links
to `/rooms` and lists rooms. A url containing `/rooms` identifies neither.
list_rooms and get_room cover the latter. A room holds boards and nothing
else, so get_room lists boards for get_board (which already reports "in
room" from the board context) plus who else is in it. Room boards report
`isVisible`, which the course-page projection does not, so a draft is
named as a draft instead of being offered and then answering 403.
Rooms also go through the crawl, or they would have become the next
blind spot: their boards are indexed, searchable by both the index and
the live-crawl path, diffed by what_changed, and mirrored by the CLI
under the room's name. The board traversal and the snapshot matcher are
now shared between courses and rooms rather than duplicated, which also
fixed the live-crawl path silently not searching pad contents.
The CLI needed no new command — it is file-centric and inherits rooms
through the manifest — but `--course` now accepts a room id, and says so.
`kind` gains 'room'; the column is plain TEXT, so no migration. 112 tests.
Smoke: 42/42 and 44/44 local, 41/41 and 43/43 live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Testing against a local instance turned up four things the server was
getting wrong, all of them invisible against the live account because the
data that exposes them had never been produced there.
`GET /lessons/{id}/tasks` returns a bare array, not the `{data,total}`
envelope every sibling endpoint uses, so `.data` was undefined and a
topic's tasks silently vanished. Its items also carry no id at all —
`LessonLinkedTaskResponse` has no id property — which leaves a
topic-attached task unidentifiable: it is not a task element on the
course page, and once past due it is in neither task list. So its
submission, and its grade, could not be reached by any route. That is 18
of 60 tasks on the real account, now reachable: the ids come off the
legacy topic page, where each task is linked as `/homework/{id}`.
The types said `id: string` and `status: TaskStatus` on something that
has neither, which is what let this stay quiet; `LessonLinkedTask` and
`ResolvedTask` now say what is actually there.
Collaborative text editor elements come back with `content: {}`, and the
tool said their contents were unavailable. They are available: the
content-element endpoint returns the pad url *and* an Etherpad session
cookie, and the pad exports itself as text to whoever holds it. No API
key needed. Pads are now shown by get_board and indexed for search.
The store's file digest covered id and size on the grounds that file
records are immutable. `PATCH /file/rename/{id}` renames one in place,
so a rename was reported as nothing at all.
Finally, get_board reported an unpublished board as "no permission",
which sends the reader hunting for an access problem that is not there.
smoke gains checks for topic tasks and for pads, and no longer assumes a
populated index or a search term that happens to match. 39/39 live-only
and 41/41 index-backed, against both the live instance and a local one.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds docker-compose.override.yml (local-only: publishes the server on
127.0.0.1:8080 and Postgres on 127.0.0.1:55432, crawls on demand) and
docs/LOCAL.md covering the stack, Claude Code registration over both
transports, the CLI, and the test suites.
Two bugs that only running the real container could find:
The mirror volume was root-owned while the container runs as node, so
every file write failed with EACCES. Docker initialises a named volume
from the image directory including its ownership, so the fix is to
create /data/mirror owned by node in the image. This was easy to miss
because the indexer records a per-file failure rather than crashing —
the crawl "succeeded" with 4 skipped. Earlier direct-node testing missed
it entirely by writing to a scratch dir owned by the developer.
The store tests TRUNCATE, and pointing TEST_DATABASE_URL at the dev
database put their fixtures into real data. They now refuse any database
whose name does not contain "test".
Verified against the rebuilt container: 4 files mirrored, image-only PDF
detection firing in the real pipeline, both transports showing
✔ Connected in `claude mcp list`, and a whoami tool call driven end to
end through `claude -p`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Store: crawl generations as the sync cursor. Diffs compare generations
on entity identity plus a content digest, never on upstream timestamps —
GET /course-rooms/{id}/board returns request time as updatedAt for most
elements, so a timestamp cursor would report every board as changed on
every crawl. Identity diffing also yields deletions, which no timestamp
scheme can. A per-course crawl carries the other courses' rows forward
so every completed generation is a complete picture and any two diff
directly; without that a partial crawl reads as a mass deletion.
FTS uses the german dictionary with weighted title/body, plus a pg_trgm
arm because stemming will not match "Datenschutz" inside
"Datenschutzgrundverordnung" and German compounds make that the common
case. file_texts is keyed by file record id and deliberately outlives
generations: records are immutable upstream, so text extracted once is
valid forever and a re-crawl of unchanged content costs nothing.
Store.open returns undefined instead of throwing when Postgres is
unreachable — the index is an accelerator, and a Pi that loses its
database should get slower, not broken.
core/paths.ts is the security boundary for the mirror. Course titles,
card titles and filenames are all user-supplied upstream, so this is
where a hostile name stops being text and becomes a path. Two bugs found
by its own tests: "///" produced "---" instead of falling back, and dot
runs survived mid-component. Now no ".." can survive anywhere, which
makes the invariant checkable rather than a claim about ordering.
Indexer coalesces concurrent refreshes onto one run and enforces a
minimum interval, since a full crawl is ~270 requests from an account
that looks like a student.
9 store tests against a real Postgres (mocks would test nothing here)
and 13 path tests; 47 total.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
