af4464decb
Schulcloud says what was uploaded and WebUntis says what was scheduled. Neither says what was *taught* — which point the teacher laboured, which example landed, what "will definitely come up". That lives in two places this server could not reach: the notes the user takes in the lesson, and WebUntis' class register. Notes are a directory of Markdown files (NOTES_DIR), not a table. They have to be writable from a phone in a classroom, readable when Postgres is down, and outlive this project, and files are the only shape that is all three — so the files are the truth and the index is a view of them, the same split as file_texts and the mirror. list_notes and get_note read disk, so they answer before the first crawl; search, what_changed and all three German prompts read them alongside the Schulcloud material. add_note writes one, and is the only thing in this server that writes anything. That is not a hole in the read-only invariant but a different store: it is bounded to NOTES_DIR by the same safeComponent/resolveWithin pair that stops a hostile Schulcloud filename escaping the mirror, so a note titled ../../.ssh/authorized_keys becomes a filename. Schulcloud and WebUntis stay GET-only and allowlisted respectively. NOTES_READONLY refuses writes outright. Appending targets the *lesson*, not the title: "halt das auch noch fest" mid-lesson carries a new title, and deriving the path from it would start a second note every time, which is the one thing append exists to prevent. Notes.app has no export — its bodies are compressed protobuf and the iCloud copy is encrypted — so scripting the app is not the clumsy route to the notes but the only one. scripts/export-apple-notes.js reads them through AppleScript into one JSON object per line, and `schulcloud note import` converts the HTML to Markdown, takes the Notes folder as the subject and the *creation* date as the lesson's date. Attachments cannot come across; a note that was a photo of the board imports as a line saying so, because importing it empty would hide the loss. The class register needed one API property to become cheap: getLessonTopic2017 answers per *series*, not per period, so a term is reconstructed by asking about the latest period of each lesson series and merging back by id — a few dozen calls for a school year rather than one per lesson. untis_lesson_topics now takes a subject as well as a period id, and UNTIS_HISTORY_DAYS of register goes into the index under a kind of its own, so "what did we actually do before the test" is searchable. Sharing the snapshot rather than duplicating it caught one thing on the way: the search tool's live path had to learn notes too, or fresh=true would have quietly disagreed with the index. 305 tests; 88/89 smoke against the local instance, the one failure being the H5P service that instance does not run. The live smoke could not be retaken: that session has lapsed and needs a fresh jwt cookie. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
159 lines
8.1 KiB
Markdown
159 lines
8.1 KiB
Markdown
# Extensions: built and possible
|
||
|
||
Items 1–3 are **now implemented** — see `docs/CLI.md` and the `store/`,
|
||
`indexer/` and `cli/` modules. What remains below is the rationale (worth
|
||
keeping) and the items still open.
|
||
|
||
## Decisions already taken
|
||
|
||
| Decision | Choice | Rejected |
|
||
|---|---|---|
|
||
| Endpoint auth | Static bearer token (`MCP_AUTH_TOKEN`, constant-time check) | OAuth 2.1; secret URL path |
|
||
| State | Stateless — every tool call hits the API live | Postgres cache; cache + full-text search |
|
||
| File delivery | Inline extracted text; images inline | Raw base64 only |
|
||
| **If** persistence is added | **The Pi's existing PostgreSQL, with its own database and user** | A dedicated container; SQLite |
|
||
|
||
Two of the rejected options need no revisiting. OAuth 2.1 is disproportionate
|
||
machinery for a single-user endpoint a bearer token already protects. "Raw bytes
|
||
as well as extracted text" got built anyway — `download_file` takes `raw: true`.
|
||
|
||
## 1. Full-text search over file contents — BUILT
|
||
|
||
**The gap:** `search` covers course/board/card/lesson/task titles and text, and
|
||
file *names* — never file *contents*. The 93 PDFs in this account are opaque to
|
||
it. "Where does it explain the Caesar cipher?" only hits if those words are in a
|
||
filename.
|
||
|
||
**The cost:** one `search` walks ~26 course pages, ~30 board skeletons plus card
|
||
fetches, and ~150 per-element file listings — roughly **270 upstream requests**,
|
||
several seconds, every time.
|
||
|
||
**Design sketch**
|
||
|
||
- Postgres `tsvector` + GIN. Use the **`german`** dictionary, so *Verschlüsselung*
|
||
matches *Verschlüsselungsverfahren*; the default `english` config stems these
|
||
wrongly.
|
||
- Add **`pg_trgm`** alongside. German compounds defeat stemming —
|
||
*Datenschutzgrundverordnung* will not match *Datenschutz* — and trigram
|
||
similarity also absorbs typos.
|
||
- Index: titles and text of courses, boards, cards, lessons, tasks; file names;
|
||
and **extracted file text**.
|
||
- Later, optionally: embeddings in `pgvector`, so "the thing about encryption"
|
||
finds *Verschlüsselung* without sharing a word. Needs an embedding model in the
|
||
loop, so it is a separate step, not part of this.
|
||
|
||
## 2. Cache, with bypass — BUILT (`search fresh=true`, `refresh_index`)
|
||
|
||
**The split that makes staleness tolerable** — index is *discovery*, live API is
|
||
*detail*. Search the index to find where something is; always re-fetch it to read
|
||
it. A stale index then costs a missed or spurious hit, never wrong content. That
|
||
is a far better failure mode than a cache that can serve last week's homework
|
||
list as current.
|
||
|
||
- `fresh: true` on `search` (re-crawl the relevant course first) and on the
|
||
detail tools.
|
||
- **State freshness in every cached result** ("indexed 2h ago"), regardless of
|
||
the flag — a bypass only helps if the agent knows it needs one.
|
||
- **Extracted file text is cacheable indefinitely.** A `fileRecord` id maps to
|
||
immutable bytes; an edit produces a new record. PDF parsing happens once, ever.
|
||
Cheapest large win here, and free of staleness risk.
|
||
- Sync: nightly full crawl (~270 requests, trivial) plus on-demand per course.
|
||
|
||
**Consequence to weigh:** coursework text would then live on the Pi at rest,
|
||
outside Schulcloud. It does not weaken the read-only property, but it is new —
|
||
consider disk encryption and whether it lands in a backup.
|
||
|
||
## 3. "What's new since …" — BUILT (`what_changed`)
|
||
|
||
Falls out of having an index with history: diff successive crawls to surface new
|
||
boards, cards, files and tasks. **Impossible today at any speed** — the API has no
|
||
changed-since filter anywhere. Arguably the most useful item on this list for a
|
||
student, and nearly free once the sync job exists.
|
||
|
||
## 4. Remaining gaps, measured
|
||
|
||
Scanned the instance's 98 GET routes against what the tools cover. Almost
|
||
everything student-facing is now covered; what is left, with live checks:
|
||
|
||
| Area | State on this account |
|
||
|---|---|
|
||
| `GET /groups/class` | **3 real classes** — the only uncovered endpoint with data |
|
||
| `GET /rooms` | empty — the feature is unused here |
|
||
| `GET /media-boards/me` | exists, no content |
|
||
| `GET /course-info` | 403 — teacher/admin only |
|
||
| `GET /alert` | empty |
|
||
| `tools`, `oauth2`, `school`, `registrations`, `systems`, `user-login-migrations` | admin/auth plumbing, not student-facing |
|
||
|
||
Genuinely unavailable, not merely uncovered:
|
||
|
||
- **Collaborative text editor contents** — the endpoint returns an editor URL,
|
||
never the document text.
|
||
- **Calendar** — a separate service, absent from the v3 document.
|
||
- **Image-only PDFs** — 22 of 255 indexed files are scans with no text layer, so
|
||
their contents cannot be searched without OCR.
|
||
- **Numeric grades** — the API's `grade` was null on every graded submission
|
||
here, so that path stays unverified against real data.
|
||
|
||
## 4b. A third source: the user's own notes — BUILT
|
||
|
||
Measured after a term of use: Schulcloud says what was *uploaded* and WebUntis
|
||
says what was *scheduled*. Neither says what was *taught* — which point the
|
||
teacher laboured, which example landed, what "will definitely come up". That is
|
||
only ever in what the student wrote down, and it was sitting in Apple Notes
|
||
where nothing could read it.
|
||
|
||
Built as a directory of Markdown files (`NOTES_DIR`), read by `list_notes` /
|
||
`get_note`, indexed as `kind: 'note'`, searched by both the index and the live
|
||
path, diffed by `what_changed`, and consulted by all three German prompts.
|
||
`add_note` writes one — the only write in this server, and bounded to that
|
||
directory by the same `safeComponent`/`resolveWithin` pair that guards the file
|
||
mirror. `scripts/export-apple-notes.js` plus `schulcloud note import` is the
|
||
migration path out of Notes.app, which has no export of its own. See
|
||
`docs/NOTES.md`.
|
||
|
||
**Files rather than a table**, deliberately: they have to be writable from a
|
||
classroom, readable when Postgres is down, and outlive this project.
|
||
|
||
## 4c. The class register into the index — BUILT
|
||
|
||
`untis_lesson_topics` could already answer "where did we get to" for one series
|
||
from one period id. What it could not do was answer "what have we done in
|
||
Deutsch this term", and none of it was searchable.
|
||
|
||
Both fall out of one API property: `getLessonTopic2017` answers per *series*, so
|
||
a term costs one call per lesson series (`core/untis-history.ts`). The tool now
|
||
takes a subject as well as a period id, and `UNTIS_HISTORY_DAYS` of register —
|
||
topics, the notes teachers leave on a period, announced tests, homework — go
|
||
into the index as `kind: 'untis'`.
|
||
|
||
## 5. Still open
|
||
|
||
- **Video/audio transcription** — this account has 5 MP4s and a WebM that are
|
||
currently just "here is a file you cannot read".
|
||
- **MCP resources and prompts — partly built.** Courses and rooms are resources,
|
||
and there are two German prompts (`zusammenfassung`, `pruefungsvorbereitung`).
|
||
Still open: files as resources — about a thousand of them do not fit a picker
|
||
that lists every entry, so they need a browsing UI (an MCP App) rather than a
|
||
longer list — and claude.ai, which cannot reach a server on localhost and does
|
||
not send a static bearer token.
|
||
- **Calendar** — a separate `schulcloud-calendar` service, not part of the v3 API
|
||
mapped in `API.md`.
|
||
|
||
## 5. Ruled out
|
||
|
||
- **Collaborative text editor contents.** Confirmed unavailable:
|
||
`GET /api/v3/collaborative-text-editor/{parentType}/{parentId}` returns a URL
|
||
to the editor, never the document text.
|
||
- **OCR — partially wrong, revised.** For *reading*, it remains unnecessary:
|
||
images go to Claude inline and it reads them. For *indexing* it is a real gap.
|
||
Measured after building the indexer: **3 of 4 sampled course PDFs have no
|
||
embedded fonts at all** — they are scans, so extraction legitimately yields
|
||
nothing and they are unsearchable by content. `extract.ts` now reports these
|
||
as image-only rather than as an empty result. Making them searchable would
|
||
need OCR (or page rasterisation plus a vision pass), which is the largest
|
||
remaining gap in search coverage.
|
||
- **Write tools** (submitting homework, marking tasks done). Technically easy,
|
||
but this forfeits the property that makes an internet-facing endpoint
|
||
acceptable: that a leaked token cannot act as the user. A deliberate, separate
|
||
decision — see the invariant in `CLAUDE.md`.
|