diff --git a/.env.example b/.env.example index f5ddc4e..20396fa 100644 --- a/.env.example +++ b/.env.example @@ -52,6 +52,12 @@ DATABASE_URL=postgresql://schulcloud:schulcloud@postgres:5432/schulcloud # three extra requests per task on a full crawl, so it is off by default. # INDEX_PERSONAL_FILES=false +# Also walk the file manager ("Dateien") when crawling: Kurs-Dateien for every +# course, plus Persönliche, Team- and Geteilte Dateien on a full crawl. Many +# teachers keep their material only there, so this is on by default. One page +# load per folder — about 160 on a 26-course account. +# INDEX_FILE_MANAGER=true + # How often to re-crawl on a timer, in ms. Default 21600000 (6h). 0 = on demand # only. A re-crawl of unchanged content downloads nothing, because Schulcloud # file records are immutable. diff --git a/CLAUDE.md b/CLAUDE.md index 00d554a..2455674 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,8 +39,8 @@ index. read-only with respect to Schulcloud. Run `smoke` after touching `src/core/`, `src/mcp/` or `src/http/` — the unit tests cover only pure functions. -Run smoke **both ways**: with `DATABASE_URL` set (34 checks, index-backed) and -without (32 checks, live-only). The degradation path is a supported mode, not a +Run smoke **both ways**: with `DATABASE_URL` set (57 checks, index-backed) and +without (55 checks, live-only). The degradation path is a supported mode, not a fallback nobody exercises. Store tests need a database and skip without one: @@ -71,6 +71,9 @@ bin/cli.ts ──HTTP──────────┘ cli/{config,client,sync - **`crawl.ts`** — the one traversal. Search, the indexer, the what-changed diff and the file mirror all need it; keep it here, not in a tool. - `paths.ts` — the security boundary for mirrored filenames. See Invariants. + - `legacy-files.ts` — the file manager ("Dateien": Persönliche, Kurs-, Team-, + Geteilte Dateien) as one path tree, parsed from the legacy client's pages. + A separate store from files-storage; the `fs_*` tools and `/api/fs` sit on it. - **`store/`** — crawl generations, identity diffs, `german` + `pg_trgm` FTS. `Store.open` returns `undefined` when Postgres is down; callers degrade. - **`indexer/`** — crawl → persist → mirror bytes → extract text → index. @@ -92,7 +95,12 @@ bin/cli.ts ──HTTP──────────┘ cli/{config,client,sync `GET` except `extendSession` (the keepalive's `refresh-session` call, which touches only our own session and is not exposed as a tool, so no model-driven call can be a POST). `api_get` rejects non-`/api/` paths and anything carrying a -scheme or host. `refresh_index` and `POST /api/refresh` write only to the Pi's +scheme or host. **Against the legacy client, GET-only is not enough:** +`GET /files/share/` mints a share token and `GET /files/file?share=` grants a +permission, so `getFileManagerPage` allows only the listing routes, by pattern — +widen that pattern only with a route you have read the handler of. A pre-signed +download URL is fetched with **no** credentials: it names another host, and +neither the bearer nor the `jwt` cookie may go with it. `refresh_index` and `POST /api/refresh` write only to the Pi's own index and mirror — every upstream call they make is still a GET. **Filenames from Schulcloud are untrusted paths.** Course titles, card titles @@ -217,10 +225,22 @@ These cost real time to discover; `docs/API.md` has the full list with evidence. upstream: interactive exercises are H5P elements, whose `contentId` is the only handle onto the content, or external (LTI) tools behind `contextExternalToolId`. Say that rather than looking for a quiz API. +- **The file manager is a third store, reachable only as HTML.** Persönliche, + Kurs-, Team- and Geteilte Dateien live in the legacy `files` collection, not in + files-storage: `list_files` answers 0 for a course holding dozens of + worksheets, and 21 of 26 live courses keep material there. Its Feathers service + is not in the ingress, so listings are parsed from `/files/{my,courses,teams,shared}` + pages and downloads go through `GET /files/signedurl` (JSON). Folders are + addressed by id alone — `/files/courses/{course}/{folder}` at any depth. + `permittedDirectories` returns every course with **no** folders (it queries + `refOwnerModel: 'courses'`, records say `'course'`), and `/files/search/` 504s, + so walk listings. Course names contain `/`; resolve by joining segments. A + listing page that does not parse must throw, never read as an empty folder — + "0 files" is the bug this exists to fix. `docs/API.md` has the evidence. - **Teams cannot be read at any version.** v3 exposes only `GET /team/{teamId}/news`; upstream `main`'s teams controller is write-only (`POST :teamId/create-room`). `/teams` is the legacy client's HTML page, not - an API. This one is genuinely unavailable, not merely uncovered. + an API. A team's *files* are reachable, through the file manager (`/teams/`). - **`exp` (30 days) is not the session lifetime.** The binding limit is a Valkey whitelist entry with a `JWT_TIMEOUT_SECONDS` TTL (7200s; live value at `GET /api/v3/config/public`) that every authenticated request re-sets. diff --git a/README.md b/README.md index f7c480d..1f56122 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ instance, not inferred from the upstream source. > *"Find the material about Verschlüsselung and explain the Caesar cipher worksheet."* > *"Summarise the routing lesson from the LF10 course."* -Twenty tools, all read-only: +Twenty-four tools, all read-only: | | | |---|---| @@ -30,8 +30,12 @@ Twenty tools, all read-only: | `list_tasks` | homework across all courses, by due date | | `get_task` | one task: description, due date, attachments, **and your submission** | | `list_submissions` | what you handed in, and what is still ungraded | -| `list_files` | files attached to any entity | -| `download_file` | fetch a file and extract its text, or view an image | +| `list_files` | files attached to a board element, topic, task or submission | +| `download_file` | fetch such an attachment and extract its text, or view an image | +| `fs_list` | list a folder of the file manager — Persönliche, Kurs-, Team-, Geteilte Dateien | +| `fs_tree` | everything below a file-manager folder, as a tree | +| `fs_find` | find file-manager files and folders by name | +| `fs_read` | read a file-manager file, extracted like `download_file` | | `search` | keyword search across everything — **including the text inside PDFs and Office files** | | `refresh_index` | re-read Schulcloud now, per course or in full | | `what_changed` | what appeared, changed or vanished since a date | @@ -54,6 +58,8 @@ schulcloud login --server https://mcp.example.org --token schulcloud sync --dry-run # see what would be mirrored schulcloud sync # mirror coursework to ~/Schulcloud schulcloud refresh --course +schulcloud fs tree /courses # browse the file manager ("Dateien") +schulcloud fs get "/courses//" ``` It talks only to the Pi and holds no Schulcloud credential — see @@ -151,7 +157,7 @@ npm run typecheck ``` `npm run smoke` starts the HTTP server, connects a real MCP client over -Streamable HTTP and exercises every tool against the live account — 30 checks +Streamable HTTP and exercises every tool against the live account — 55 checks (57 with the index) covering the auth gate, the protocol handshake, every content chain, file extraction, `api_get`'s guard rails and error handling. diff --git a/docs/API.md b/docs/API.md index 0778e7f..8bdccbd 100644 --- a/docs/API.md +++ b/docs/API.md @@ -225,6 +225,69 @@ hops and without Etherpad's API key; `core/etherpad.ts` does this. The url is built from the server's `ETHERPAD__PAD_URI`, so it must be checked against the instance host before the session cookie is sent to it. +## The file manager ("Dateien") is a third store + +Persönliche Dateien, Kurs-Dateien, Team-Dateien and Geteilte Dateien are the +**legacy file system**: a `files` collection with real folders (`isDirectory`, +`parent`, `owner`, `refOwnerModel`), served by the legacy Feathers +`fileStorage` service. It shares nothing with files-storage. Asking +`/api/v3/file/list/school/{school}/courses/{courseId}` answers **0** for a +course whose file manager holds dozens of worksheets — measured: 21 of 26 +courses on the live account keep files here, some nothing else. + +**Its service is not in the public ingress**, so the only way in is the legacy +client. Verified live: + +| Route (legacy client, `jwt` cookie) | Returns | Notes | +|---|---|---| +| `GET /files/my/` , `/files/my/{folder}` | HTML listing | personal root / one folder | +| `GET /files/courses/` | HTML listing | the courses, *as folders* (ids = course ids) | +| `GET /files/courses/{course}` , `/files/courses/{course}/{folder}` | HTML listing | **one** folder segment at any depth | +| `GET /files/teams/…` | HTML listing | same shape as courses | +| `GET /files/shared/` | HTML listing | flat; shared *folders* cannot be opened (no route; the UI's link 404s) | +| `GET /files/signedurl?file={id}&name={name}` | `{"url": …}` | pre-signed S3 url, **another host** (live: `s3.hidrive.strato.com`) | +| `GET /files/permittedDirectories/` | JSON tree | **lists every course with no folders in any** — see below | +| `GET /files/search/?q=` | HTML | **504** on live: an unindexed regex over every file record | + +Listings are parsed on the attributes the page's own scripts use: +`data-folder-id` with the name inside `.card-title-directory` (emitted +**unescaped**, `{{{stripOnlyScript name}}}`), and `data-file-id`, +`data-file-name`, `data-file-size`, `data-file-viewer-type` on each +`.card.file`. A blocked file carries `btn-file-danger` and no viewer type. No +dates are rendered. + +**`permittedDirectories` is broken for courses.** The directory service's +query matches course folders on `refOwnerModel: 'courses'`; the records say +`'course'`. Live result: 26 courses, 0 folders; personal folders come through. +Listings are the only complete view — which is also what the UI shows. + +**Some GET routes write.** `GET /files/share/?file=` mints a share token when +the file has none (`PATCH /fileStorage/shared/{id}`), and +`GET /files/file?…&share=…` grants the caller a permission on the file. +`GET /files/fileModel/{id}/proxy` forwards to the latter. A GET-only client is +therefore *not* read-only against this surface by itself: `core/client.ts` +allows only the listing routes and `/files/signedurl`, by pattern. + +**The signed-url service returns its error instead of throwing it** +(`.catch((err) => new Forbidden(err))`), so a refused file is a 200 whose body +has no `url`. + +**Course names contain `/`** in real data ("LF07 - FIA24A/B - Sb/Ha", +"FIA24/FIP24 IT LF12"), and a real file is literally called `..docx`. Paths +built from names cannot be split naively; `core/legacy-files.ts` resolves by +trying joined segments, and accepts ids as segments. + +Two more, seen while building the local fixture: + +- **`getRefOwnerModel(owner)` answers "a course, or else `teams`".** Any owner + id that is not a course — a *user* included — is recorded as a team's. The + upload page never sends an owner for personal files (`data-owner=""`), so the + server defaults to the creator and records `user`; send the user id and every + later permission check dereferences a team that does not exist. +- **The file permission service writes `refOwnerModel`** where the "shared with + me" query reads `refPermModel`, so a share made through it never appears + under Geteilte Dateien. The share-link flow patches `/files/{id}` directly. + ## Re-verifying after an upstream release `npm run probe` re-checks every assumption above against the live instance and diff --git a/docs/CLI.md b/docs/CLI.md index b598b2f..80052ae 100644 --- a/docs/CLI.md +++ b/docs/CLI.md @@ -50,6 +50,39 @@ alongside courses, with their files under the room's name rather than a course's what changed: that is a handful of requests, where a full re-crawl reads every course. The server refuses a repeat within a minute unless you pass `--force`. +### The file manager (`fs`) + +The Schulcloud file manager ("Dateien") — Persönliche, Kurs-, Team- and +Geteilte Dateien — browsed like a filesystem, live: + +``` +schulcloud fs ls [path] [--long] +schulcloud fs tree [path] [--depth ] [--max-folders ] +schulcloud fs find [--path ] [--type file|folder] [--long] +schulcloud fs get [--out ] [--force] [--jobs ] +``` + +```console +$ schulcloud fs ls /courses +$ schulcloud fs tree "/courses/FIA24B - SK (Rh)" +$ schulcloud fs find "*Erben*" --path /courses +$ schulcloud fs get "/courses/FIA24B - SK (Rh)/02_Erbrecht" --out ~/Erbrecht +``` + +The tree is `/my`, `/courses/`, `/teams/` and `/shared`; the +German names ("/Kurs-Dateien") work too. Names may contain `/` — course names +often do — and still resolve; any segment may also be an id from `--long`. + +`fs find` matches any part of a name, or, given `*` or `?`, the whole name as +`find -name` does. `fs get` on a folder downloads everything below it, keeps +the structure, and skips files already present at the same size — so re-running +it resumes. Each folder is one page load on the server, so large trees take a +while. + +`sync` mirrors these files too, under `/Kurs-Dateien/…`, +`Persönliche Dateien/…`, `Team-Dateien//…` and `Geteilte Dateien/`, once +the server's index includes them (`INDEX_FILE_MANAGER`, on by default). + ## How sync works It is a **one-way mirror, not a two-way sync**, and that follows from the data diff --git a/docs/LOCAL.md b/docs/LOCAL.md index 03422fe..4acf3fc 100644 --- a/docs/LOCAL.md +++ b/docs/LOCAL.md @@ -103,9 +103,9 @@ node dist/bin/cli.js sync ## Run the test suites ```bash -npm test # 58 offline tests +npm test # 135 offline tests npm run smoke # end-to-end against the live instance, live-only mode -DATABASE_URL=… npm run smoke # end-to-end with the index (34 checks) +DATABASE_URL=… npm run smoke # end-to-end with the index (57 checks) ``` Store tests need a database and skip without one: diff --git a/local-instance/README.md b/local-instance/README.md index e0ce209..6355f51 100644 --- a/local-instance/README.md +++ b/local-instance/README.md @@ -89,6 +89,7 @@ Every one of these is marked in the env files at the line it affects. ``` docker-compose.yml the stack; profiles: (default) | tools | av | preview + (minio-loopback: see "Simulating a teacher") env/ one file per service, all values local-only proxy/nginx.conf GENERATED — the single origin, see scripts/gen-proxy-conf.py scripts/seed.sh loads the demo school via the management app @@ -183,14 +184,40 @@ node scripts/simulate-teacher.mjs create # course, room, topic, task, board, # Etherpad pad, folder, files node scripts/simulate-teacher.mjs update # rename and rewrite all of it node scripts/simulate-teacher.mjs delete # remove it again +node scripts/simulate-teacher.mjs files # only the file-manager part, onto an existing fixture ``` +It also fills the **file manager** ("Dateien") — the legacy file system behind +Persönliche, Kurs-, Team- and Geteilte Dateien, a separate store from the board +files above — with a course folder tree ("Arbeitsblätter/Woche 1"), a team +folder, a folder in the demo student's own files, and a teacher's file shared +read-only with the student. The student is added to the adopted team so the +team files are visible from the account under test. Every file carries a unique +search term, so search over the index can be checked per area. + +Legacy file uploads only work because of two changes to the stack: + +- **`minio-loopback`.** The legacy service signs upload and download URLs for + its storage provider's single endpoint, and uses that endpoint for its own S3 + calls. `minio:9000` works inside the compose network and nowhere else, so the + browser and the MCP server on the host were handed URLs they could not open. + `seed.sh` now registers `http://localhost:9900`, and this socat sidecar, + sharing the api container's network namespace, forwards that address to + MinIO — the same url then works from the api, the browser and the host. + After recreating `api`, recreate `minio-loopback` too: it lives in api's + network namespace. +- **The school bucket is created up front.** The legacy service makes + `bucket-` on first upload and then calls `PutBucketCors`, which + MinIO does not implement, so the first upload failed with *"A header you + provided implies functionality that is not implemented"*. `minio-init.sh` + creates the demo school's bucket, and an existing bucket skips both calls. + Ids are kept in `.simulate-teacher.json` between phases, so the MCP server can be pointed at the instance in between: ```bash eval "$(./scripts/mcp-env.sh)" # as the demo student -cd .. && npm run smoke # 39 checks against the local instance +cd .. && npm run smoke # 57 checks against the local instance ``` `mcp-env.sh` points the index at its own database, `schulcloud_local`, and the diff --git a/local-instance/docker-compose.yml b/local-instance/docker-compose.yml index df87664..e3c9128 100644 --- a/local-instance/docker-compose.yml +++ b/local-instance/docker-compose.yml @@ -154,6 +154,24 @@ services: minio: {condition: service_healthy} restart: unless-stopped + minio-loopback: + # The legacy file service (the file manager: Persönliche/Kurs-/Team-Dateien) + # signs upload and download URLs for its storage provider's one endpoint, and + # the same endpoint serves its own S3 calls. "minio:9000" works inside the + # compose network and nowhere else, so a browser or the MCP server on this + # machine was handed URLs it could not open: uploads through the UI failed + # and downloads could not be tested. seed.sh registers the provider as + # http://localhost:9900 instead, and this forwards that address to MinIO from + # inside the api container's own network namespace — so the one URL now + # works from the api, from the browser and from the host alike. + image: alpine/socat:1.8.0.1 + network_mode: "service:api" + command: ["TCP-LISTEN:9900,fork,reuseaddr,bind=127.0.0.1", "TCP:minio:9000"] + depends_on: + api: {condition: service_started} + minio: {condition: service_healthy} + restart: unless-stopped + file-preview: # Generates thumbnails via ImageMagick, driven off RabbitMQ. Optional: with # it absent, files still upload and download, they just have no preview. diff --git a/local-instance/scripts/minio-init.sh b/local-instance/scripts/minio-init.sh index a3b74d6..ef8f4dd 100755 --- a/local-instance/scripts/minio-init.sh +++ b/local-instance/scripts/minio-init.sh @@ -11,9 +11,16 @@ for bucket in \ h5p-content-bucket ` # h5p-editor content` \ h5p-library-bucket ` # h5p content types` \ ydocs ` # tldraw whiteboard documents` \ - fwu-content # FWU media, unused but cheap to create + fwu-content ` # FWU media, unused but cheap to create` \ + bucket-5f2987e020834114b8efd6f6 # legacy file manager, demo school (see below) do mc mb --ignore-existing "local/$bucket" done +# The legacy file manager keeps one bucket per school, "bucket-", and +# creates it on first upload — then calls PutBucketCors, which MinIO does not +# implement, so the first upload fails with "A header you provided implies +# functionality that is not implemented". A bucket that already exists skips +# both calls. 5f2987e020834114b8efd6f6 is the demo school's fixed seed id. + mc ls local diff --git a/local-instance/scripts/seed.sh b/local-instance/scripts/seed.sh index 96ea5ce..325d9ba 100755 --- a/local-instance/scripts/seed.sh +++ b/local-instance/scripts/seed.sh @@ -39,7 +39,10 @@ SECRET=$(curl -fsS -X POST "$MGMT/encrypt-plain-text" \ const id = ObjectId('62949a4003839b6162aa566b'); db.storageproviders.replaceOne({ _id: id }, { _id: id, isShared: true, region: 'eu-central-1', type: 'S3', - endpointUrl: 'http://minio:9000', + // Not minio:9000: this one endpoint also goes into every signed URL, and + // those are opened by the browser and the MCP server on the host. The + // minio-loopback service makes localhost:9900 reach MinIO from the api too. + endpointUrl: 'http://localhost:9900', accessKeyId: 'miniouser', secretAccessKey: '$SECRET', maxBuckets: 150, freeBuckets: 138, diff --git a/local-instance/scripts/simulate-teacher.mjs b/local-instance/scripts/simulate-teacher.mjs index aac2b7f..562e8fe 100755 --- a/local-instance/scripts/simulate-teacher.mjs +++ b/local-instance/scripts/simulate-teacher.mjs @@ -11,6 +11,7 @@ * node scripts/simulate-teacher.mjs create # build the fixture, print ids * node scripts/simulate-teacher.mjs update # rename/edit everything it made * node scripts/simulate-teacher.mjs delete # remove it again + * node scripts/simulate-teacher.mjs files # (re)build only the file-manager part * * State lives in .simulate-teacher.json so the phases can be run one at a time * with MCP checks in between. @@ -295,8 +296,9 @@ async function create() { log(`second board left unpublished on purpose: ${draft.id}`); saveState(s); + await fileManager(); console.log(`\nstate written to ${STATE}`); - summary(s); + summary(loadState()); } /** files-storage attaches bytes to a board node (element) id, not to the card. */ @@ -307,6 +309,146 @@ async function upload(parentId, name, type, body) { return record.id; } +// ---------------------------------------------------------- file manager --- +// +// The file manager ("Dateien": Persönliche, Kurs-, Team- and Geteilte Dateien) +// is the legacy file system, a different store from files-storage above, with a +// real folder tree. Many teachers use nothing else, so the MCP server's fs_* +// tools need content there. Its services are only reachable on the server's own +// port, like the other /api/v1 writes in this script. + +const STUDENT_PASSWORD = process.env.SIM_STUDENT_PASSWORD ?? 'schulcloud'; +const TEAM_MEMBER_ROLE = '5bb5c190fb457b1c3c0c7e0f'; // "teammember" in the seed + +/** Runs `fn` signed in as another account, then restores the teacher. */ +async function asUser(email, password, fn) { + const saved = { jwt, me }; + const res = await fetch(`${API}/api/v3/authentication/local`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ username: email, password }), + }); + if (!res.ok) throw new Error(`login as ${email} failed: ${res.status}`); + jwt = (await res.json()).accessToken; + me = await v3('GET', '/me'); + try { + return await fn(); + } finally { + ({ jwt, me } = saved); + } +} + +async function legacyDir(name, owner, parent) { + return (await v1('POST', '/fileStorage/directories', { name, owner, parent }))._id; +} + +/** + * The browser's own upload sequence: a signed PUT url, the bytes, then the + * record. The url comes from the server, so it is held to the same localhost + * rule as everything else this script writes to. + */ +async function legacyUpload({ owner, parent, name, type, body }) { + const bytes = Buffer.from(body); + const signed = await v1('POST', '/fileStorage/signedUrl', { parent, filename: name, fileType: type }); + const { hostname } = new URL(signed.url); + if (hostname !== '127.0.0.1' && hostname !== 'localhost') { + throw new Error(`refusing to upload to ${hostname}: not a localhost address (see minio-loopback in docker-compose.yml)`); + } + const put = await fetch(signed.url, { method: 'PUT', headers: signed.header, body: bytes }); + if (!put.ok) throw new Error(`PUT ${name} to storage -> ${put.status} ${(await put.text()).slice(0, 200)}`); + const record = await v1('POST', '/fileStorage', { + name, + owner, + parent, + type, + size: bytes.length, + storageFileName: signed.header['x-amz-meta-flat-name'], + }); + return record._id; +} + +async function fileManager() { + const s = loadState(); + if (!s.courseId) throw new Error('run `create` first: the file-manager fixture lives in its course'); + const keep = (key, value) => { + s[key] = value; + saveState(s); + }; + const student = await studentId(); + + step('file manager: Kurs-Dateien'); + keep('fmCourseRootFileId', await legacyUpload({ + owner: s.courseId, name: 'Kursplan.txt', type: 'text/plain', + body: 'Kursplan Biologie\n\nThemen: Zelle, Gewebe, Organe. Stichwort: Photosynthese-Lichtreaktion.\n', + })); + keep('fmCourseDirId', await legacyDir('Arbeitsblätter', s.courseId)); + keep('fmCourseFileId', await legacyUpload({ + owner: s.courseId, parent: s.fmCourseDirId, name: 'Blatt 1 - Zellorganellen.txt', type: 'text/plain', + body: 'Blatt 1: Zellorganellen\n\nBeschrifte die Mitochondrienmembran und das endoplasmatische Retikulum.\n', + })); + keep('fmCourseSubDirId', await legacyDir('Woche 1', s.courseId, s.fmCourseDirId)); + keep('fmCourseDeepFileId', await legacyUpload({ + owner: s.courseId, parent: s.fmCourseSubDirId, name: 'Blatt 2 - Gewebe.txt', type: 'text/plain', + body: 'Blatt 2: Gewebe\n\nVergleiche Epithelgewebe und Bindegewebe.\n', + })); + log(`course root file, folder "Arbeitsblätter" with a file, and "Woche 1" nested inside it`); + + if (s.teamId) { + step('file manager: Team-Dateien'); + // Teams cannot be created (see the README), and the adopted one does not + // include the demo student, whose view is the one under test. + const team = await v1('GET', `/teams/${s.teamId}`); + if (!team.userIds.some((entry) => String(entry.userId?._id ?? entry.userId) === student)) { + const userIds = team.userIds.map((entry) => ({ + userId: String(entry.userId?._id ?? entry.userId), + role: String(entry.role?._id ?? entry.role), + schoolId: String(entry.schoolId?._id ?? entry.schoolId), + })); + userIds.push({ userId: student, role: TEAM_MEMBER_ROLE, schoolId: me.school.id }); + await v1('PATCH', `/teams/${s.teamId}`, { userIds }); + keep('fmStudentAddedToTeam', true); + log('demo student added to the team'); + } + keep('fmTeamDirId', await legacyDir('Projekt', s.teamId)); + keep('fmTeamFileId', await legacyUpload({ + owner: s.teamId, parent: s.fmTeamDirId, name: 'Projektplan.txt', type: 'text/plain', + body: 'Projektplan\n\nMeilenstein Chlorophyll bis Freitag.\n', + })); + log('team folder "Projekt" with a file'); + } + + step('file manager: Persönliche Dateien (the student\'s own)'); + // No `owner` for personal files, exactly as the upload page sends none: the + // server decides the owner model as "a course, or else a team", so passing a + // user id records the folder as a team's, and every later permission check on + // it then dereferences a team that does not exist. + await asUser(STUDENT_EMAIL, STUDENT_PASSWORD, async () => { + keep('fmStudentDirId', await legacyDir('Notizen')); + keep('fmStudentFileId', await legacyUpload({ + parent: s.fmStudentDirId, name: 'Lernzettel.txt', type: 'text/plain', + body: 'Lernzettel\n\nRibosomenfabrik: Proteinbiosynthese am rauen ER.\n', + })); + }); + log('student folder "Notizen" with a file'); + + step('file manager: Geteilte Dateien'); + keep('fmSharedFileId', await legacyUpload({ + name: 'Geteilt vom Lehrer.txt', type: 'text/plain', + body: 'Zusatzmaterial\n\nDie Zellkernhuelle trennt Kernplasma und Zytoplasma.\n', + })); + // What accepting a share link does in the legacy client: a read-only user + // permission. Not the permission service, which writes `refOwnerModel` where + // the "shared with me" query reads `refPermModel`, so its shares never show. + const shared = await v1('GET', `/files/${s.fmSharedFileId}`); + await v1('PATCH', `/files/${s.fmSharedFileId}`, { + permissions: [ + ...shared.permissions, + { refId: student, refPermModel: 'user', read: true, write: false, delete: false, create: false }, + ], + }); + log('teacher file shared read-only with the student'); +} + // ---------------------------------------------------------------- update --- async function update() { @@ -348,6 +490,17 @@ async function update() { await files('PATCH', `/rename/${s.fileId}`, { fileName: 'nervensystem-notiz-v2.txt' }); log('file renamed'); + if (s.fmCourseFileId) { + step('file manager edits'); + await v1('POST', '/fileStorage/rename', { id: s.fmCourseFileId, newName: 'Blatt 1 - Zellorganellen (korrigiert).txt' }); + log('course file renamed'); + s.fmCourseAddedFileId = await legacyUpload({ + owner: s.courseId, parent: s.fmCourseSubDirId, name: 'Blatt 3 - Organe.txt', type: 'text/plain', + body: 'Blatt 3: Organe\n\nNeuer Suchbegriff: Nephronschleife.\n', + }); + log('course file added in "Woche 1" (new search term: Nephronschleife)'); + } + saveState({ ...s, updated: true }); } @@ -376,6 +529,46 @@ async function remove() { ['second room', () => v3('DELETE', `/rooms/${s.roomWithoutStudentId}`)], ['course', () => v1('DELETE', `/courses/${s.courseId}`)], ]; + // File-manager content goes first, while its course and team still exist. + const fileManagerTries = [ + ['file-manager course files', async () => { + for (const id of [s.fmCourseAddedFileId, s.fmCourseDeepFileId, s.fmCourseFileId, s.fmCourseRootFileId]) { + if (id) await v1('DELETE', `/fileStorage?_id=${id}`); + } + }], + ['file-manager course folders', async () => { + for (const id of [s.fmCourseSubDirId, s.fmCourseDirId]) if (id) await v1('DELETE', `/fileStorage/directories?_id=${id}`); + }], + ['file-manager team content', async () => { + if (s.fmTeamFileId) await v1('DELETE', `/fileStorage?_id=${s.fmTeamFileId}`); + if (s.fmTeamDirId) await v1('DELETE', `/fileStorage/directories?_id=${s.fmTeamDirId}`); + }], + ['shared file', async () => { + if (s.fmSharedFileId) await v1('DELETE', `/fileStorage?_id=${s.fmSharedFileId}`); + }], + ['student personal files', async () => { + if (!s.fmStudentFileId && !s.fmStudentDirId) return; + await asUser(STUDENT_EMAIL, STUDENT_PASSWORD, async () => { + if (s.fmStudentFileId) await v1('DELETE', `/fileStorage?_id=${s.fmStudentFileId}`); + if (s.fmStudentDirId) await v1('DELETE', `/fileStorage/directories?_id=${s.fmStudentDirId}`); + }); + }], + ['student team membership', async () => { + if (!s.fmStudentAddedToTeam || !s.teamId) return; + const student = await studentId(); + const team = await v1('GET', `/teams/${s.teamId}`); + const userIds = team.userIds + .map((entry) => ({ + userId: String(entry.userId?._id ?? entry.userId), + role: String(entry.role?._id ?? entry.role), + schoolId: String(entry.schoolId?._id ?? entry.schoolId), + })) + .filter((entry) => entry.userId !== student); + await v1('PATCH', `/teams/${s.teamId}`, { userIds }); + }], + ]; + tries.unshift(...fileManagerTries); + for (const [what, fn] of tries) { try { await fn(); @@ -398,8 +591,11 @@ await login(); if (phase === 'create') await create(); else if (phase === 'update') await update(); else if (phase === 'delete') await remove(); -else if (phase === 'show') summary(loadState()); +else if (phase === 'files') { + await fileManager(); + summary(loadState()); +} else if (phase === 'show') summary(loadState()); else { - console.error(`unknown phase ${phase}; expected create | update | delete | show`); + console.error(`unknown phase ${phase}; expected create | update | delete | files | show`); process.exit(2); } diff --git a/scripts/smoke.mjs b/scripts/smoke.mjs index c45fb36..7550313 100644 --- a/scripts/smoke.mjs +++ b/scripts/smoke.mjs @@ -223,6 +223,63 @@ console.log('\n== classes and groups =='); ); } +console.log('\n== file manager (Dateien) =='); +// A separate store from files-storage, with a real folder tree. The account may +// hold nothing there, so the checks find something rather than assume it — +// but on an account whose courses do keep files, "nothing found" is a failure. +{ + const root = await call('fs_list', { path: '/' }); + check('fs_list / names the four areas', !root.isError && /\/courses\//.test(root.text) && /\/shared\//.test(root.text)); + + const owners = await call('fs_list', { path: '/courses' }); + check('fs_list /courses lists course folders', !owners.isError, owners.text.split('\n').find((line) => /course\(s\)/.test(line))); + const ownerIds = [...owners.text.matchAll(/\*\*.*?\/\*\* \(`([0-9a-f]{24})`\)/g)].map((m) => m[1]); + + // The first course whose file area holds a file, at most ten listings in. + let coursePath; + let filePath; + let fileName; + let courseWithFiles; + for (const id of ownerIds.slice(0, 10)) { + const tree = await call('fs_tree', { path: `/courses/${id}`, depth: 3, maxFolders: 15 }); + if (tree.isError) continue; + const heading = tree.text.match(/^## (\/courses\/.+?) — /m)?.[1]; + const line = tree.text.match(/^\s*([^\n]+?\.(?:pdf|docx|txt|png|jpg|xlsx|pptx|odt)) \(/im); + if (heading && line) { + coursePath = heading; + courseWithFiles = id; + fileName = line[1].trim(); + // The tree gives names; fs_find recovers the full path to read. + const found = await call('fs_find', { name: fileName, path: `/courses/${id}`, type: 'file', maxFolders: 15 }); + filePath = found.text.match(/^- (\/courses\/.+?) — /m)?.[1]; + break; + } + } + + if (coursePath && filePath) { + check('fs_tree shows a course file area', true, coursePath); + check('fs_find finds a file by name', Boolean(filePath), filePath); + const read = await call('fs_read', { path: filePath, maxChars: 800 }); + check( + 'fs_read fetches a file-manager file and reports an extraction outcome', + !read.isError && /File id:/.test(read.text), + read.text.split('\n').find((l) => /extracted|image|no extractable|image-only|Word|PDF/.test(l)) ?? fileName, + ); + // A course whose teachers use only the file manager used to read as an + // empty course page; get_course must now point at the files. + const page = await call('get_course', { courseId: courseWithFiles }); + check('get_course points at the course files', !page.isError && /Course files \(Kurs-Dateien\)/.test(page.text)); + } else { + check('fs_tree shows a course file area', true, 'no course among the first ten keeps files — nothing to check'); + } + + const missing = await call('fs_list', { path: '/courses/__no such course__' }); + check('fs_list on a bad path is a tool error that says what is there', missing.isError && /No "|not a file area|Did you mean/.test(missing.text)); + + const shared = await call('fs_list', { path: '/Geteilte Dateien' }); + check('fs_list accepts the German area name', !shared.isError, shared.text.split('\n')[0]); +} + console.log('\n== rooms =='); // Rooms ("Räume") are a separate space from courses. An account in none is // normal — and is exactly the state that hid this whole feature — so the check diff --git a/src/bin/cli.ts b/src/bin/cli.ts index f6e70eb..33748dd 100644 --- a/src/bin/cli.ts +++ b/src/bin/cli.ts @@ -7,6 +7,7 @@ import { pipeline } from 'node:stream/promises'; import { ApiClient, ApiError } from '../cli/client.ts'; import { defaultSyncDir, loadCliConfig, saveCliConfig, configPath } from '../cli/config.ts'; import { formatBytes } from '../core/extract.ts'; +import { fsFind, fsGet, fsList, fsTree } from '../cli/fs.ts'; import { sync, type SyncEvent } from '../cli/sync.ts'; /** @@ -26,6 +27,16 @@ const USAGE = `schulcloud — browse and mirror your Schulcloud files schulcloud sync [--dry-run] [--full] [--prune] [--dir ] [--jobs ] schulcloud refresh [--course ] [--force] +The file manager ("Dateien") — /my, /courses/, /teams/, /shared: + + schulcloud fs ls [path] [--long] + schulcloud fs tree [path] [--depth ] [--max-folders ] + schulcloud fs find [--path ] [--type file|folder] [--long] + schulcloud fs get [--out ] [--force] [--jobs ] + +fs get downloads a file, or a folder with everything below it. Names may contain +"/" and still resolve; any path segment can also be an id from "fs ls --long". + --course takes a course or a room id: rooms ("Räume") mirror alongside courses and their files sit under the room's name. @@ -58,6 +69,8 @@ async function main(argv: string[]): Promise { return runSync(flags); case 'refresh': return refresh(flags); + case 'fs': + return fileManager(flags); default: process.stderr.write(`Unknown command "${command}".\n\n${USAGE}`); return 2; @@ -154,6 +167,41 @@ async function get(flags: Flags): Promise { return 0; } +async function fileManager(flags: Flags): Promise { + const [sub, ...args] = flags._ as string[]; + const api = new ApiClient(await loadCliConfig()); + const out = (line: string) => process.stdout.write(`${line}\n`); + const long = Boolean(flags.long); + switch (sub) { + case 'ls': + return fsList(api, args[0] ?? '/', long, out); + case 'tree': + return fsTree(api, args[0] ?? '/', Number(flags.depth ?? 3), Number(flags['max-folders'] ?? 200), out); + case 'find': { + if (!args[0]) { + process.stderr.write('fs find needs a name, e.g.: schulcloud fs find Erbrecht --path /courses\n'); + return 2; + } + const type = flags.type === 'folder' || flags.type === 'file' ? String(flags.type) : 'any'; + return fsFind(api, args[0], String(flags.path ?? '/'), type, Number(flags['max-folders'] ?? 400), long, out); + } + case 'get': + if (!args[0]) { + process.stderr.write('fs get needs a path, e.g.: schulcloud fs get "/courses//"\n'); + return 2; + } + return fsGet( + api, + args[0], + { out: flags.out ? String(flags.out) : undefined, force: Boolean(flags.force), jobs: Number(flags.jobs ?? 3) }, + out, + ); + default: + process.stderr.write(`Unknown fs command "${sub ?? ''}". Use ls, tree, find or get.\n\n${USAGE}`); + return 2; + } +} + async function runSync(flags: Flags): Promise { const config = await loadCliConfig(); const root = flags.dir ? resolve(String(flags.dir)) : config.syncDir; diff --git a/src/cli/client.ts b/src/cli/client.ts index a3c70e2..263701b 100644 --- a/src/cli/client.ts +++ b/src/cli/client.ts @@ -26,6 +26,39 @@ export interface Manifest { entries: ManifestEntry[]; } +/** One entry of a file-manager tree or search, as /api/fs returns it. */ +export interface FsEntry { + type: 'directory' | 'file'; + path: string; + parentPath: string; + depth: number; + id: string; + name: string; + size?: number; + mimeType?: string | null; + blocked?: boolean; +} + +export interface FsListing { + path: string; + kind: 'directory' | 'file'; + area?: string | null; + directories?: { id: string; name: string; path: string }[]; + files?: { id: string; name: string; path: string; size: number; mimeType?: string; blocked: boolean }[]; + file?: { id: string; name: string; size: number; mimeType?: string; blocked: boolean }; +} + +export interface FsWalk { + path: string; + kind: 'directory' | 'file'; + entries?: FsEntry[]; + matches?: FsEntry[]; + file?: FsListing['file']; + visited?: number; + truncated?: boolean; + failures?: { path: string; reason: string }[]; +} + export class ApiError extends Error { readonly status: number; @@ -83,6 +116,28 @@ export class ApiClient { async file(fileId: string): Promise { return this.request(`/api/files/${encodeURIComponent(fileId)}`); } + + // --- the file manager ------------------------------------------------------ + + async fsList(path: string): Promise { + return (await (await this.request(`/api/fs/list?${new URLSearchParams({ path })}`)).json()) as FsListing; + } + + async fsTree(path: string, depth: number, maxFolders: number): Promise { + const query = new URLSearchParams({ path, depth: String(depth), maxFolders: String(maxFolders) }); + return (await (await this.request(`/api/fs/tree?${query}`)).json()) as FsWalk; + } + + async fsFind(name: string, path: string, type: string, maxFolders: number): Promise { + const query = new URLSearchParams({ name, path, type, maxFolders: String(maxFolders) }); + return (await (await this.request(`/api/fs/find?${query}`)).json()) as FsWalk; + } + + /** Streams one file-manager file's bytes, by path or by id. */ + async fsFile(target: { path: string } | { id: string; name: string }): Promise { + const query = 'path' in target ? new URLSearchParams({ path: target.path }) : new URLSearchParams(target); + return this.request(`/api/fs/file?${query}`); + } } function describe(status: number, detail: string, server: string): string { @@ -90,5 +145,7 @@ function describe(status: number, detail: string, server: string): string { if (status === 503) return 'The server is running without an index, so this command is unavailable. Set DATABASE_URL on the server.'; if (status === 409) return detail || 'The sync cursor is unknown to the server. Run a full sync with --full.'; if (status === 429) return detail || 'Refreshed too recently — wait a moment, or pass --force.'; + // The file manager's own errors already say what was not found and what is there. + if ((status === 400 || status === 404 || status === 422) && detail) return detail; return detail ? `HTTP ${status}: ${detail}` : `HTTP ${status}`; } diff --git a/src/cli/fs.ts b/src/cli/fs.ts new file mode 100644 index 0000000..d06de75 --- /dev/null +++ b/src/cli/fs.ts @@ -0,0 +1,218 @@ +import { createWriteStream } from 'node:fs'; +import { mkdir, rename, stat, unlink } from 'node:fs/promises'; +import { dirname, resolve } from 'node:path'; +import { Readable } from 'node:stream'; +import { pipeline } from 'node:stream/promises'; +import { formatBytes } from '../core/extract.ts'; +import { resolveWithin, safeComponent } from '../core/paths.ts'; +import type { ApiClient, FsEntry } from './client.ts'; + +/** + * `schulcloud fs` — the file manager ("Dateien") from the command line. + * + * The same tree the MCP fs_* tools show: /my, /courses/, /teams/, + * /shared. Everything goes through the server's /api/fs routes; nothing here + * talks to Schulcloud. + */ + +type Out = (line: string) => void; + +export async function fsList(api: ApiClient, path: string, long: boolean, out: Out): Promise { + const listing = await api.fsList(path); + if (listing.kind === 'file' && listing.file) { + out(`${listing.path}`); + out(` ${formatBytes(listing.file.size)} ${listing.file.mimeType ?? 'unknown type'} ${listing.file.id}${listing.file.blocked ? ' [blocked]' : ''}`); + return 0; + } + + const directories = listing.directories ?? []; + const files = listing.files ?? []; + out(listing.path); + for (const directory of directories) out(` ${directory.name}/${long ? ` ${directory.id}` : ''}`); + for (const file of files) { + const detail = long ? ` ${formatBytes(file.size).padStart(9)} ${file.id}${file.mimeType ? ` ${file.mimeType}` : ''}` : ''; + out(` ${file.name}${detail}${file.blocked ? ' [blocked]' : ''}`); + } + if (directories.length + files.length === 0) out(' (empty)'); + out(`${directories.length} folder(s), ${files.length} file(s)`); + return 0; +} + +export async function fsTree(api: ApiClient, path: string, depth: number, maxFolders: number, out: Out): Promise { + const walk = await api.fsTree(path, depth, maxFolders); + if (walk.kind === 'file') { + out(`${walk.path} is a file.`); + return 0; + } + const entries = walk.entries ?? []; + const children = groupByParent(entries); + + out(walk.path); + let files = 0; + let bytes = 0; + const visit = (parent: string, prefix: string) => { + const kids = children.get(parent) ?? []; + kids.forEach((kid, index) => { + const last = index === kids.length - 1; + const branch = last ? '└── ' : '├── '; + if (kid.type === 'directory') { + out(`${prefix}${branch}${kid.name}/`); + visit(kid.path, `${prefix}${last ? ' ' : '│ '}`); + } else { + files++; + bytes += kid.size ?? 0; + out(`${prefix}${branch}${kid.name} (${formatBytes(kid.size ?? 0)})${kid.blocked ? ' [blocked]' : ''}`); + } + }); + }; + visit(walk.path, ''); + + out(`\n${files} file(s), ${formatBytes(bytes)} — ${walk.visited ?? 0} folder(s) listed`); + if (walk.truncated) out(`Stopped after ${maxFolders} folders; pass --max-folders or start deeper.`); + for (const failure of walk.failures ?? []) out(`could not list ${failure.path}: ${failure.reason}`); + return walk.failures?.length ? 1 : 0; +} + +export async function fsFind( + api: ApiClient, + name: string, + path: string, + type: string, + maxFolders: number, + long: boolean, + out: Out, +): Promise { + const result = await api.fsFind(name, path, type, maxFolders); + for (const match of result.matches ?? []) { + const suffix = match.type === 'directory' ? '/' : ''; + const detail = long && match.type === 'file' ? ` ${formatBytes(match.size ?? 0)} ${match.id}` : long ? ` ${match.id}` : ''; + out(`${match.path}${suffix}${detail}`); + } + const count = result.matches?.length ?? 0; + process.stderr.write(`${count} match(es), ${result.visited ?? 0} folder(s) searched\n`); + if (result.truncated) process.stderr.write(`Stopped after ${maxFolders} folders; there may be more. Narrow --path.\n`); + return count > 0 ? 0 : 1; +} + +/** + * Downloads a file, or a whole folder recursively. + * + * A folder lands under `--out` (default: a directory named after it) with the + * file manager's structure. Every component is a name from Schulcloud and so + * untrusted: each goes through `safeComponent`, and the joined path through + * `resolveWithin`, exactly as sync does. + */ +export async function fsGet( + api: ApiClient, + path: string, + options: { out?: string; force: boolean; jobs: number }, + out: Out, +): Promise { + const target = await api.fsList(path); + + if (target.kind === 'file' && target.file) { + const destination = options.out ? resolve(options.out) : resolve(safeComponent(target.file.name, target.file.id)); + if (target.file.blocked) { + process.stderr.write(`${target.path}: blocked by the instance virus scanner; not downloaded.\n`); + return 1; + } + await downloadTo(api, { path: target.path }, destination); + out(destination); + return 0; + } + + const rootName = target.path === '/' ? 'Dateien' : (target.path.split('/').pop() ?? 'Dateien'); + const root = options.out ? resolve(options.out) : resolve(safeComponent(rootName, 'Dateien')); + process.stderr.write(`Listing ${target.path} …\n`); + const walk = await api.fsTree(target.path, 12, 1000); + const entries = walk.entries ?? []; + + // Rebuild each file's name segments from its parents rather than splitting + // its path: names may contain "/", which would otherwise invent folders. + const segments = new Map([[walk.path, []]]); + for (const entry of [...entries].sort((a, b) => a.depth - b.depth)) { + const parent = segments.get(entry.parentPath); + if (parent) segments.set(entry.path, [...parent, entry.name]); + } + + const files = entries.filter((entry) => entry.type === 'file'); + let downloaded = 0; + let skipped = 0; + let failed = 0; + let bytes = 0; + + const queue = [...files]; + const worker = async () => { + for (let entry = queue.shift(); entry; entry = queue.shift()) { + const parts = segments.get(entry.path); + if (!parts) continue; + const relative = parts.map((part) => safeComponent(part)).join('/'); + const destination = resolveWithin(root, relative); + if (entry.blocked) { + process.stderr.write(` blocked ${relative}\n`); + skipped++; + continue; + } + // Unchanged by size: re-running a folder download resumes rather than repeats. + const existing = await stat(destination).catch(() => undefined); + if (!options.force && existing?.isFile() && existing.size === entry.size) { + skipped++; + continue; + } + try { + await downloadTo(api, { id: entry.id, name: entry.name }, destination); + downloaded++; + bytes += entry.size ?? 0; + process.stderr.write(` get ${relative}\n`); + } catch (error) { + failed++; + process.stderr.write(` FAILED ${relative}: ${(error as Error).message}\n`); + } + } + }; + await Promise.all(Array.from({ length: Math.max(1, options.jobs) }, worker)); + + out(root); + process.stderr.write( + `Downloaded ${downloaded} file(s) (${formatBytes(bytes)}), ${skipped} skipped` + + `${failed ? `, FAILED ${failed}` : ''} — ${walk.visited ?? 0} folder(s) listed\n`, + ); + if (walk.truncated) process.stderr.write('The folder is larger than one listing pass; some files were not reached.\n'); + for (const failure of walk.failures ?? []) process.stderr.write(`could not list ${failure.path}: ${failure.reason}\n`); + return failed > 0 || (walk.failures?.length ?? 0) > 0 ? 1 : 0; +} + +async function downloadTo(api: ApiClient, target: { path: string } | { id: string; name: string }, destination: string) { + const response = await api.fsFile(target); + if (!response.body) throw new Error('empty response body'); + await mkdir(dirname(destination), { recursive: true }); + // A temporary neighbour, renamed into place, so an interrupted download never + // leaves a half-file that the size check would later accept as complete. + const temp = `${destination}.part`; + try { + await pipeline(Readable.fromWeb(response.body as never), createWriteStream(temp)); + await rename(temp, destination); + } catch (error) { + await unlink(temp).catch(() => {}); + throw error; + } +} + +function groupByParent(entries: FsEntry[]): Map { + const children = new Map(); + for (const entry of entries) { + const list = children.get(entry.parentPath) ?? []; + list.push(entry); + children.set(entry.parentPath, list); + } + for (const list of children.values()) { + list.sort((a, b) => + a.type !== b.type + ? a.type === 'directory' + ? -1 + : 1 + : a.name.localeCompare(b.name, 'de', { numeric: true, sensitivity: 'base' }), + ); + } + return children; +} diff --git a/src/config.ts b/src/config.ts index 06dd07f..b8b1dd2 100644 --- a/src/config.ts +++ b/src/config.ts @@ -37,6 +37,8 @@ export interface Config { mirrorMaxBytes: number; /** Index personal files and submitted/returned work as well as course content. */ indexPersonalFiles: boolean; + /** Walk the file manager (Kurs-, Persönliche, Team- and Geteilte Dateien) when crawling. */ + indexFileManager: boolean; /** How often to re-crawl on a timer. Zero = only on demand. */ crawlIntervalMs: number; } @@ -95,6 +97,9 @@ export function loadConfig(): Config { // cost of a full crawl. Worth turning on to make your own handed-in work // searchable, which no other route offers. indexPersonalFiles: bool('INDEX_PERSONAL_FILES', false), + // On by default: many teachers keep their material only in Kurs-Dateien, + // so an index without it misses whole courses. One page load per folder. + indexFileManager: bool('INDEX_FILE_MANAGER', true), crawlIntervalMs: intAllowingZero('CRAWL_INTERVAL_MS', 6 * 60 * 60_000), }; } diff --git a/src/context.ts b/src/context.ts index 094803f..43454d6 100644 --- a/src/context.ts +++ b/src/context.ts @@ -1,5 +1,6 @@ import type { Config } from './config.ts'; import { SchulcloudClient } from './core/client.ts'; +import { FileManager } from './core/legacy-files.ts'; import type { LegacyUser, MeResponse } from './core/types.ts'; import type { Indexer } from './indexer/indexer.ts'; import type { Store } from './store/store.ts'; @@ -14,6 +15,8 @@ import type { Store } from './store/store.ts'; export class ServerContext { readonly config: Config; readonly client: SchulcloudClient; + /** The "Dateien" file manager; shared across sessions when the process provides one. */ + readonly files: FileManager; /** Shared across sessions; undefined when running without an index. */ readonly store: Store | undefined; readonly indexer: Indexer | undefined; @@ -30,9 +33,13 @@ export class ServerContext { */ private readonly userNames = new Map>(); - constructor(config: Config, shared?: { client?: SchulcloudClient; store?: Store; indexer?: Indexer }) { + constructor( + config: Config, + shared?: { client?: SchulcloudClient; files?: FileManager; store?: Store; indexer?: Indexer }, + ) { this.config = config; this.client = shared?.client ?? new SchulcloudClient(config); + this.files = shared?.files ?? new FileManager(this.client); this.store = shared?.store; this.indexer = shared?.indexer; } diff --git a/src/core/client.ts b/src/core/client.ts index b6dc7cc..f2fe111 100644 --- a/src/core/client.ts +++ b/src/core/client.ts @@ -119,8 +119,21 @@ export class SchulcloudClient { return url; } - private async request(url: URL, accept: string): Promise { + /** + * One upstream GET, with retries for the transient failures. + * + * `auth` picks how the session travels. The v3 API takes it as a bearer + * token; the legacy client's pages take it only as the `jwt` cookie; and a + * pre-signed storage URL must get **nothing** — it lives on another host, and + * the session token has no business leaving this instance. Anything but the + * bearer form is fetched with redirects off, so a login bounce or a hop to a + * third host is seen rather than silently followed with credentials attached. + */ + private async request(url: URL, accept: string, auth: 'bearer' | 'cookie' | 'none' = 'bearer'): Promise { let lastError: unknown; + const headers: Record = { Accept: accept }; + if (auth === 'bearer') headers.Authorization = `Bearer ${this.config.jwt}`; + if (auth === 'cookie') headers.Cookie = `jwt=${this.config.jwt}`; for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) { if (attempt > 0) await delay(backoffMs(attempt)); @@ -128,9 +141,9 @@ export class SchulcloudClient { let response: Response; try { response = await fetch(url, { - headers: { Authorization: `Bearer ${this.config.jwt}`, Accept: accept }, + headers, signal: AbortSignal.timeout(this.config.requestTimeoutMs), - redirect: 'follow', + redirect: auth === 'bearer' ? 'follow' : 'manual', }); } catch (error) { // Connection reset or timeout: worth one more try, since every call @@ -143,7 +156,12 @@ export class SchulcloudClient { if (response.ok) return response; const body = await response.text().catch(() => ''); - const error = new SchulcloudApiError(response.status, url.pathname + url.search, body); + // A pre-signed URL's query string is its credential, so it never goes + // into an error message; nor does the storage host's error body. + const error = + auth === 'none' + ? new SchulcloudApiError(response.status, `${url.host} (pre-signed download)`, '') + : new SchulcloudApiError(response.status, url.pathname + url.search, body); // A crawl issues hundreds of requests and the instance answers some of // them with a 503 front-page when it decides we are going too fast. @@ -175,6 +193,11 @@ export class SchulcloudClient { async getBytes(path: string, fallbackName: string): Promise { const url = this.url(path); const response = await this.request(url, '*/*'); + return this.readCapped(response, fallbackName); + } + + /** Reads a response body up to `maxDownloadBytes`, flagging anything cut off. */ + private async readCapped(response: Response, fallbackName: string): Promise { const limit = this.config.maxDownloadBytes; const chunks: Buffer[] = []; @@ -550,6 +573,118 @@ export class SchulcloudClient { const body = await this.getJson>('/api/v3/groups', { limit: MAX_PAGE_SIZE }); return body.data ?? []; } + + // --- the "Dateien" file manager ------------------------------------------ + // + // Persönliche Dateien, Kurs-Dateien, Team-Dateien and Geteilte Dateien live in + // the legacy file system, a different store from files-storage: listing a + // course through /api/v3/file answers 0 files for a course holding dozens. + // Its Feathers service is not in the public ingress, so the only way in is + // the legacy client — HTML pages for listings, one JSON route for downloads. + // See core/legacy-files.ts for the parsing and the path model. + + /** + * One file-manager page, as HTML. + * + * **Only the listing routes are reachable here, by construction.** Several of + * the legacy client's GET routes write: `GET /files/share/` mints a share + * token when the file has none, and `GET /files/file?share=…` grants the + * caller a permission on someone else's file. A GET-only client is therefore + * not read-only against this surface by itself; the allowlist is what makes + * the invariant hold. + */ + async getFileManagerPage(path: string): Promise { + if (!FILE_MANAGER_PAGE.test(path)) { + throw new Error(`refusing file-manager path outside the listing routes: ${path}`); + } + const response = await this.legacyRequest(path, 'text/html'); + return response.text(); + } + + /** + * A pre-signed download URL for one legacy file. + * + * `name` only sets the download's filename; the server checks read access on + * the id. The route is `/files/signedurl` rather than `/files/file`, which + * answers the same thing as a redirect but also accepts `share`, the + * parameter that writes. + */ + async getFileManagerSignedUrl(fileId: string, name: string): Promise { + if (!/^[0-9a-f]{24}$/i.test(fileId)) throw new Error(`not a file id: ${fileId}`); + const query = new URLSearchParams({ file: fileId, name: name || fileId }); + const response = await this.legacyRequest(`/files/signedurl?${query.toString()}`, 'application/json'); + // The server's error path *returns* its Forbidden rather than throwing it, + // so a refused file arrives as a 200 whose body has no url. + const body = (await response.json().catch(() => ({}))) as { url?: unknown; message?: unknown }; + if (typeof body.url !== 'string' || !body.url) { + throw new SchulcloudApiError(403, '/files/signedurl', typeof body.message === 'string' ? body.message : 'no download url'); + } + return body.url; + } + + /** Downloads one legacy file: signed URL, then the bytes, capped like every download. */ + async downloadFileManagerFile(fileId: string, name: string): Promise { + const signed = await this.getFileManagerSignedUrl(fileId, name); + const response = await this.openSignedUrl(signed); + return this.readCapped(response, name); + } + + /** + * Opens a pre-signed storage URL — with no credentials at all. + * + * The URL names another host (live: an S3 endpoint at the storage provider), + * so neither the bearer nor the cookie may go with it. It must also be + * https whenever the instance is, which keeps a URL the server hands back from + * pointing this process at a plaintext service on its own network. + */ + async openSignedUrl(signedUrl: string): Promise { + const target = checkSignedUrl(signedUrl, this.config.baseUrl); + return this.request(target, '*/*', 'none'); + } + + private async legacyRequest(path: string, accept: string): Promise { + try { + return await this.request(this.url(path), accept, 'cookie'); + } catch (error) { + // The legacy client answers a rejected cookie with a redirect to its + // login page. Report it as what it is, so tools say "token expired" + // rather than "HTTP 302". + if (error instanceof SchulcloudApiError && error.status >= 300 && error.status < 400) { + throw new SchulcloudApiError(401, path, 'redirected to login: the session is not accepted'); + } + throw error; + } + } +} + +/** + * The file-manager listing routes, and nothing else. + * + * Folders are addressed by id alone — `/files/courses/{course}/{folder}` holds + * one folder segment however deep the folder is — so every listing fits one of + * these shapes. `/files/my/{a}/{b}` exists too, but lists `b` exactly as + * `/files/my/{b}` does, so it is not needed. + */ +const FILE_MANAGER_PAGE = + /^\/files\/(?:(?:my|courses|teams|shared)\/|my\/[0-9a-f]{24}|(?:courses|teams)\/[0-9a-f]{24}(?:\/[0-9a-f]{24})?)$/i; + +/** Validates a pre-signed URL before anything is sent to it. Exported for testing. */ +export function checkSignedUrl(signedUrl: string, baseUrl: string): URL { + let target: URL; + try { + target = new URL(signedUrl); + } catch { + throw new Error('the download url the server returned is not a url'); + } + const instanceIsHttps = new URL(baseUrl).protocol === 'https:'; + const allowed = instanceIsHttps ? ['https:'] : ['https:', 'http:']; + if (!allowed.includes(target.protocol)) { + throw new Error(`refusing a ${target.protocol} download url from an ${instanceIsHttps ? 'https' : 'http'} instance`); + } + if (target.username || target.password) { + throw new Error('refusing a download url that carries credentials'); + } + return target; } /** diff --git a/src/core/crawl.ts b/src/core/crawl.ts index 94436aa..39b5c6a 100644 --- a/src/core/crawl.ts +++ b/src/core/crawl.ts @@ -2,6 +2,7 @@ import type { Config } from '../config.ts'; import { assembleBoard, type AssembledBoard } from './board.ts'; import type { SchulcloudClient } from './client.ts'; import { fetchHomeworkPage } from './homework-page.ts'; +import { FileManager, type DirectoryRef, type FmFile, type WalkEntry } from './legacy-files.ts'; import { fetchLessonTaskLinks, withScrapedIds } from './lesson-page.ts'; import { htmlToText, normalizeObjectId } from './text.ts'; import type { CourseMetadata, FileParentType, FileRecord, TaskContent } from './types.ts'; @@ -35,6 +36,11 @@ export interface Breadcrumb { /** Column → card, for board files. */ columnTitle?: string; cardTitle?: string; + /** + * Folder names from the file manager, outermost first. Unlike boards, its + * trees have no fixed depth, so they cannot be squeezed into the titles above. + */ + folders?: string[]; } export interface CrawledFile { @@ -43,6 +49,13 @@ export interface CrawledFile { parentType: FileParentType; parentId: string; at: Breadcrumb; + /** + * Which store holds the bytes. The file manager is not files-storage: its + * ids mean nothing to /api/v3/file, so downloads must be routed by this. + */ + source?: 'files-storage' | 'file-manager'; + /** The file-manager path fs_read takes, for files from there. */ + fsPath?: string; } /** @@ -150,6 +163,12 @@ export interface CrawlOptions { * about X" to be searchable, which is otherwise impossible. */ includePersonalFiles?: boolean; + /** + * Walk the file manager ("Dateien") too: Kurs-Dateien for every course, and on + * a full crawl Persönliche Dateien, Team-Dateien and Geteilte Dateien. One + * page load per folder — on the account this was built for, about 160. + */ + includeFileManager?: boolean; /** * Read the text of collaborative text editor (Etherpad) pads, which needs a * second credentialled hop outside the API. Omit to leave pads unread. @@ -207,10 +226,24 @@ export async function crawl(client: SchulcloudClient, options: CrawlOptions): Pr // outside that scope forward untouched. const rooms: CrawledRoom[] = options.courseIds ? [] : await crawlRooms(client, options, includeFiles, files, failures); + if (includeFiles && options.includeFileManager) { + const titles = new Map(crawled.map((entry) => [entry.course.id, entry.title])); + await crawlFileManager(client, options, titles, files, failures); + } + // Traversal order is nondeterministic under concurrency; sort so that two // crawls of unchanged content produce identical snapshots. crawled.sort((a, b) => a.course.id.localeCompare(b.course.id)); rooms.sort((a, b) => a.id.localeCompare(b.id)); + // One file can be reachable twice — a course file someone also shared with + // you appears under /shared as well — and the index keys nodes by id. Keep + // the first place it was found, which the traversal order makes the most + // specific one. + const seenFiles = new Set(); + const uniqueFiles = files.filter((file) => (seenFiles.has(file.record.id) ? false : (seenFiles.add(file.record.id), true))); + files.length = 0; + files.push(...uniqueFiles); + files.sort((a, b) => a.record.id.localeCompare(b.record.id)); submissions.sort((a, b) => a.id.localeCompare(b.id)); @@ -540,6 +573,128 @@ async function collectSubmissions( } } +/** + * Walks the file manager and records every file in it. + * + * Kurs-Dateien are filed under their course, so a per-course refresh replaces + * exactly that course's files and the store's scope rules carry the rest + * forward. The areas that belong to no course — personal, team, shared — are + * walked only on a full crawl, for the same reason. + * + * A fresh FileManager rather than the process-wide one: its listing cache is + * right for an interactive ls-then-read, and wrong for a crawl whose whole + * point is to see the current state. + */ +async function crawlFileManager( + client: SchulcloudClient, + options: CrawlOptions, + courseTitles: Map, + files: CrawledFile[], + failures: { courseId: string; boardId?: string; reason: string }[], +): Promise { + const manager = new FileManager(client); + const roots: { ref: DirectoryRef; path: string; courseId: string; at: Omit }[] = []; + + if (options.courseIds) { + for (const courseId of options.courseIds) { + const title = courseTitles.get(courseId) ?? courseId; + roots.push({ + ref: { area: 'courses', ownerId: courseId }, + path: `/courses/${title}`, + courseId, + at: { courseId, courseTitle: title, containerTitle: 'Kurs-Dateien' }, + }); + } + } else { + const owners = async (area: 'courses' | 'teams') => { + try { + return (await manager.list({ area })).directories; + } catch (error) { + failures.push({ courseId: '', reason: `file manager /${area}: ${error instanceof Error ? error.message : String(error)}` }); + return []; + } + }; + for (const course of await owners('courses')) { + const title = courseTitles.get(course.id) ?? course.name; + roots.push({ + ref: { area: 'courses', ownerId: course.id }, + path: `/courses/${course.name}`, + courseId: course.id, + at: { courseId: course.id, courseTitle: title, containerTitle: 'Kurs-Dateien' }, + }); + } + for (const team of await owners('teams')) { + roots.push({ + ref: { area: 'teams', ownerId: team.id }, + path: `/teams/${team.name}`, + courseId: '', + at: { courseId: '', courseTitle: 'Team-Dateien', containerTitle: team.name }, + }); + } + roots.push({ ref: { area: 'my' }, path: '/my', courseId: '', at: { courseId: '', courseTitle: 'Persönliche Dateien' } }); + roots.push({ ref: { area: 'shared' }, path: '/shared', courseId: '', at: { courseId: '', courseTitle: 'Geteilte Dateien' } }); + } + + // Sequential roots, modest concurrency within each: the instance answers a + // burst with 503s, and this walk is the largest single part of a crawl. + for (const root of roots) { + const result = await manager.walk( + { path: root.path, ref: root.ref }, + { maxDepth: 25, maxDirectories: 5000, concurrency: 2 }, + ); + for (const failure of result.failures) { + failures.push({ courseId: root.courseId, reason: `file manager ${failure.path}: ${failure.reason}` }); + } + if (result.truncated) { + failures.push({ courseId: root.courseId, reason: `file manager ${root.path}: stopped at 5000 folders` }); + } + + // Folder names come from the parent chain, never from splitting a path: + // names contain "/" in real data. + const folders = new Map([[root.path, []]]); + const ordered = [...result.entries].sort((a, b) => a.depth - b.depth); + for (const entry of ordered) { + if (entry.directory) { + const parent = folders.get(entry.parentPath); + if (parent) folders.set(entry.path, [...parent, entry.directory.name]); + } + } + for (const entry of ordered) { + if (!entry.file) continue; + files.push(fileManagerRecord(entry, entry.file, root, folders.get(entry.parentPath) ?? [])); + } + } +} + +function fileManagerRecord( + entry: WalkEntry, + file: FmFile, + root: { ref: DirectoryRef; courseId: string; at: Omit }, + folders: string[], +): CrawledFile { + const parentType: FileParentType = root.ref.area === 'courses' ? 'courses' : 'users'; + return { + // Shaped like a files-storage record so the store, the mirror and the + // manifest need no second code path; `source` is what keeps the two apart. + record: { + id: file.id, + name: file.name, + parentId: entry.parent.folderId ?? entry.parent.ownerId ?? '', + parentType, + url: '', + size: file.size, + mimeType: file.mimeType ?? 'application/octet-stream', + securityCheckStatus: file.blocked ? 'blocked' : 'verified', + previewStatus: '', + }, + parentType, + parentId: entry.parent.folderId ?? entry.parent.ownerId ?? '', + at: { ...root.at, folders }, + source: 'file-manager', + fsPath: entry.path, + }; +} + async function listFiles( client: SchulcloudClient, schoolId: string, diff --git a/src/core/extract.ts b/src/core/extract.ts index 9a0bcc5..aedc309 100644 --- a/src/core/extract.ts +++ b/src/core/extract.ts @@ -140,7 +140,10 @@ function hasTextLayer(bytes: Buffer): boolean { async function extractPdf(bytes: Buffer): Promise { const { extractText, getDocumentProxy } = await import('unpdf'); - const document = await getDocumentProxy(new Uint8Array(bytes)); + // verbosity 0 = errors only. pdf.js otherwise prints "Warning: TT: undefined + // function" for every font hint it skips — harmless, but a crawl of the file + // manager extracts hundreds of PDFs, and that buries the log in noise. + const document = await getDocumentProxy(new Uint8Array(bytes), { verbosity: 0 }); const { text } = await extractText(document, { mergePages: true }); return Array.isArray(text) ? text.join('\n\n') : text; } diff --git a/src/core/legacy-files.ts b/src/core/legacy-files.ts new file mode 100644 index 0000000..1a40d27 --- /dev/null +++ b/src/core/legacy-files.ts @@ -0,0 +1,500 @@ +import type { DownloadedFile, SchulcloudClient } from './client.ts'; +import { decodeEntities } from './text.ts'; + +/** + * The "Dateien" file manager — Persönliche Dateien, Kurs-Dateien, Team-Dateien + * and Geteilte Dateien — as one read-only filesystem. + * + * This is the legacy file system, a different store from files-storage + * (`/api/v3/file`), which holds board, topic and task attachments. The two do + * not overlap: asking files-storage for a course's files answers 0 for a course + * whose file manager holds dozens, and on the account this was built for 21 of + * 26 courses keep material here — some teachers use nothing else. + * + * Its Feathers service is not in the public ingress, so it is reached through + * the legacy client: server-rendered listing pages, parsed here, and one JSON + * route for pre-signed downloads (see the client). Two server quirks shape the + * design: + * + * - `GET /files/permittedDirectories/` looks like the obvious JSON source for + * the directory tree, but its query matches course folders on + * `refOwnerModel: 'courses'` while the records say `'course'`. It lists every + * course with **no** folders in any of them. Listings are the only complete + * view, which is also exactly what the file manager itself shows. + * - `GET /files/search/` runs an unindexed regex over every file record and + * times out (504) on the live instance, so finding is done by walking. + */ + +export type FileArea = 'my' | 'courses' | 'teams' | 'shared'; + +interface AreaInfo { + area: FileArea; + label: string; + /** Accepted spellings of the first path segment, compared lower-cased. */ + aliases: string[]; + summary: string; +} + +export const FILE_AREAS: AreaInfo[] = [ + { + area: 'my', + label: 'Persönliche Dateien', + aliases: ['my', 'persönliche dateien', 'persoenliche dateien', 'personal', 'meine dateien'], + summary: 'your own files', + }, + { + area: 'courses', + label: 'Kurs-Dateien', + aliases: ['courses', 'kurs-dateien', 'meine kurs-dateien', 'kursdateien', 'kurse'], + summary: 'one folder per course, holding what its teachers uploaded', + }, + { + area: 'teams', + label: 'Team-Dateien', + aliases: ['teams', 'team-dateien', 'meine team-dateien', 'teamdateien'], + summary: 'one folder per team', + }, + { + area: 'shared', + label: 'Geteilte Dateien', + aliases: ['shared', 'geteilte dateien', 'mit mir geteilt'], + summary: 'files other people shared with you, read-only and flat', + }, +]; + +export function areaInfo(area: FileArea): AreaInfo { + return FILE_AREAS.find((entry) => entry.area === area) as AreaInfo; +} + +export interface FmDirectory { + id: string; + name: string; +} + +export interface FmFile { + id: string; + name: string; + /** Bytes, as the listing reports it. */ + size: number; + /** Absent for a blocked file: the page withholds its viewer attributes. */ + mimeType?: string; + /** Rejected by the instance's virus scanner; it cannot be downloaded. */ + blocked: boolean; +} + +export interface FmListing { + directories: FmDirectory[]; + files: FmFile[]; +} + +/** A listing page's markup changed shape; never report that as an empty folder. */ +export class FileManagerMarkupError extends Error {} + +/** + * Parses one file-manager page. Exported for testing. + * + * Anchored on what the templates (`files/files.hbs`, `files/files-grid.hbs`) + * emit for the page's own scripts — `data-folder-id`, `data-file-id`, + * `data-file-name`, `data-file-size` — rather than on layout classes. + * + * Throws rather than returning an empty listing when the page is not a file + * manager page at all: "0 files" is precisely the wrong answer this module + * exists to fix, and a markup change must not quietly reproduce it. + */ +export function parseFileListing(html: string): FmListing { + if (!/class="route-files"/.test(html)) { + throw new FileManagerMarkupError('the page is not a file-manager listing (markup changed, or not logged in)'); + } + + const directories: FmDirectory[] = []; + const folderPattern = /]*\bopenfolder\b[^>]*)>([\s\S]*?)<\/button>/g; + for (const match of html.matchAll(folderPattern)) { + const id = /data-folder-id="([0-9a-f]{24})"/i.exec(match[1] ?? '')?.[1]; + if (!id) continue; + // The name is emitted unescaped (`{{{stripOnlyScript name}}}`) inside the + // title element, after an icon; strip the tags, then decode what is left. + const title = /]*card-title-directory[^>]*>([\s\S]*?)<\/strong>/.exec(match[2] ?? '')?.[1] ?? ''; + const name = decodeEntities(title.replace(/<[^>]+>/g, '')).replace(/\s+/g, ' ').trim(); + directories.push({ id, name: name || id }); + } + + const files: FmFile[] = []; + const cardPattern = /]*\bclass="card file\b([^"]*)"([^>]*)>/g; + const cards = [...html.matchAll(cardPattern)]; + cards.forEach((match, index) => { + const attributes = match[2] ?? ''; + const id = /data-file-id="([0-9a-f]{24})"/i.exec(attributes)?.[1]; + if (!id) return; + const name = decodeEntities(/data-file-name="([^"]*)"/.exec(attributes)?.[1] ?? ''); + const size = Number(/data-file-size="(\d*)"/.exec(attributes)?.[1] ?? ''); + // The viewer attributes sit further inside this card; look no further + // than the next card, so one file can never borrow another's type. + const start = (match.index ?? 0) + match[0].length; + const end = cards[index + 1]?.index ?? html.length; + const mimeType = /data-file-viewer-type="([^"]*)"/.exec(html.slice(start, end))?.[1]; + files.push({ + id, + name: name || id, + size: Number.isFinite(size) ? size : 0, + mimeType: mimeType ? decodeEntities(mimeType) : undefined, + blocked: /\bbtn-file-danger\b/.test(match[1] ?? ''), + }); + }); + + return { directories, files }; +} + +/** + * Where a directory is. `area` absent is the root; `ownerId` is the course or + * team (absent for `my` and `shared`); `folderId` absent is the owner's top. + */ +export interface DirectoryRef { + area?: FileArea; + ownerId?: string; + folderId?: string; +} + +export type FsNode = + | { kind: 'directory'; path: string; ref: DirectoryRef; name: string } + | { kind: 'file'; path: string; parent: DirectoryRef; file: FmFile }; + +export type FsErrorCode = 'not_found' | 'ambiguous' | 'not_a_directory' | 'not_a_file' | 'not_navigable'; + +export class FsError extends Error { + readonly code: FsErrorCode; + constructor(code: FsErrorCode, message: string) { + super(message); + this.code = code; + } +} + +/** The legacy page that lists a directory. */ +export function pageFor(ref: DirectoryRef): string | undefined { + if (!ref.area) return undefined; + if (ref.area === 'shared') return '/files/shared/'; + if (ref.area === 'my') return ref.folderId ? `/files/my/${ref.folderId}` : '/files/my/'; + if (!ref.ownerId) return `/files/${ref.area}/`; + return ref.folderId ? `/files/${ref.area}/${ref.ownerId}/${ref.folderId}` : `/files/${ref.area}/${ref.ownerId}`; +} + +/** The reference for a subdirectory found in `parent`'s listing. */ +export function childRef(parent: DirectoryRef, directory: FmDirectory): DirectoryRef { + if (!parent.area) return { area: directory.id as FileArea }; + if (parent.area === 'my') return { area: 'my', folderId: directory.id }; + if (parent.area === 'shared') { + // The file manager has no route that opens a shared folder — its owner is + // someone else, and every listing route is scoped to an owner. The UI's + // own link to one is a 404. + throw new FsError( + 'not_navigable', + `"${directory.name}" is a folder someone shared with you. The file manager cannot open shared folders ` + + '(not even in the browser); ask for the files themselves to be shared, or find them in the owning course.', + ); + } + if (!parent.ownerId) return { area: parent.area, ownerId: directory.id }; + return { area: parent.area, ownerId: parent.ownerId, folderId: directory.id }; +} + +/** Splits a path into raw segments. Empty segments (`//`, trailing `/`) drop out. */ +export function splitPath(path: string): string[] { + return path + .split('/') + .map((segment) => segment.trim()) + .filter((segment) => segment.length > 0); +} + +/** Joins names into a display path. Names keep any `/` they contain; see `resolve`. */ +export function joinPath(parent: string, name: string): string { + return `${parent === '/' ? '' : parent}/${name}`; +} + +function normalise(value: string): string { + return value.normalize('NFC').replace(/\s+/g, ' ').trim().toLocaleLowerCase('de'); +} + +export interface WalkEntry { + path: string; + /** The listed directory this entry came from; renderers group on it. */ + parentPath: string; + depth: number; + parent: DirectoryRef; + directory?: { ref: DirectoryRef; name: string; id: string }; + file?: FmFile; +} + +export interface WalkResult { + entries: WalkEntry[]; + /** Directories that were listed. */ + visited: number; + /** Set when the budget ran out before the walk finished. */ + truncated: boolean; + failures: { path: string; reason: string }[]; +} + +interface CachedListing { + at: number; + listing: Promise; +} + +/** A listing is reused this long: long enough for ls→read, short enough to stay live. */ +const LISTING_TTL_MS = 60_000; + +/** + * The file manager as a tree of paths: + * + * / the four areas + * /my/… Persönliche Dateien + * /courses//… Kurs-Dateien + * /teams//… Team-Dateien + * /shared/… Geteilte Dateien (flat) + * + * Names are the file manager's own. Because course names contain `/` in real + * data ("LF07 - FIA24A/B - Sb/Ha"), a path is not split naively: resolution + * tries joining consecutive segments into one name and backtracks when a + * shorter reading leads nowhere. Any segment may also be an id instead of a + * name, which is always unambiguous and is what the listings print alongside. + */ +export class FileManager { + private readonly client: SchulcloudClient; + private readonly cache = new Map(); + + constructor(client: SchulcloudClient) { + this.client = client; + } + + /** Lists one directory. The root is synthetic and costs nothing. */ + async list(ref: DirectoryRef): Promise { + const page = pageFor(ref); + if (!page) { + return { directories: FILE_AREAS.map((entry) => ({ id: entry.area, name: entry.area })), files: [] }; + } + + const cached = this.cache.get(page); + if (cached && Date.now() - cached.at < LISTING_TTL_MS) return cached.listing; + + const listing = this.client.getFileManagerPage(page).then(parseFileListing); + this.cache.set(page, { at: Date.now(), listing }); + // A failure must not be served from the cache for the next minute. + listing.catch(() => this.cache.delete(page)); + return listing; + } + + /** Resolves a path to a directory or a file. */ + async resolve(path: string): Promise { + const segments = splitPath(path); + if (segments.length === 0) return { kind: 'directory', path: '/', ref: {}, name: '/' }; + + const first = segments[0] as string; + const area = FILE_AREAS.find((entry) => entry.aliases.includes(normalise(first)) || entry.area === first); + if (!area) { + throw new FsError( + 'not_found', + `"${first}" is not a file area. The root holds ${FILE_AREAS.map((entry) => `/${entry.area} (${entry.label})`).join(', ')}.`, + ); + } + + const ref: DirectoryRef = { area: area.area }; + const found = await this.descend(ref, `/${area.area}`, segments.slice(1)); + return found; + } + + private async descend(ref: DirectoryRef, path: string, rest: string[]): Promise { + if (rest.length === 0) return { kind: 'directory', path, ref, name: path.split('/').pop() || '/' }; + + const listing = await this.list(ref); + const readings = candidateReadings(listing, rest); + if (readings.length === 0) throw notFound(listing, rest[0] as string, path); + + let lastError: FsError | undefined; + for (const reading of readings) { + if (reading.matches.length > 1) { + const options = reading.matches.map((match) => `"${match.entry.name}" (\`${match.entry.id}\`)`).join(', '); + throw new FsError( + 'ambiguous', + `${path} holds more than one entry named "${reading.name}": ${options}. Use the id as that path segment instead.`, + ); + } + const match = reading.matches[0] as { kind: 'directory'; entry: FmDirectory } | { kind: 'file'; entry: FmFile }; + const remaining = rest.slice(reading.consumed); + const nextPath = joinPath(path, match.entry.name); + + if (match.kind === 'file') { + if (remaining.length === 0) return { kind: 'file', path: nextPath, parent: ref, file: match.entry }; + lastError = new FsError('not_a_directory', `${nextPath} is a file, not a folder.`); + continue; + } + + try { + return await this.descend(childRef(ref, match.entry), nextPath, remaining); + } catch (error) { + // A shorter reading of a name containing "/" can lead nowhere while a + // longer one resolves; only give up once every reading has failed. + if (error instanceof FsError && error.code !== 'ambiguous' && error.code !== 'not_navigable') { + lastError = error; + continue; + } + throw error; + } + } + throw lastError ?? notFound(listing, rest[0] as string, path); + } + + /** + * Walks a directory breadth-first, listing at most `maxDirectories` of them. + * + * Every listing is one page fetch, so the budget is the cost. A folder that + * cannot be read is recorded and skipped, never silently dropped. + */ + async walk( + start: { path: string; ref: DirectoryRef }, + options: { maxDepth: number; maxDirectories: number; concurrency?: number }, + ): Promise { + const entries: WalkEntry[] = []; + const failures: { path: string; reason: string }[] = []; + let visited = 0; + let truncated = false; + + let frontier: { path: string; ref: DirectoryRef; depth: number }[] = [{ ...start, depth: 0 }]; + while (frontier.length > 0) { + const next: typeof frontier = []; + const batch = frontier; + frontier = []; + + const concurrency = Math.max(1, options.concurrency ?? 3); + for (let i = 0; i < batch.length; i += concurrency) { + const slice = batch.slice(i, i + concurrency); + await Promise.all( + slice.map(async (node) => { + // The root is synthetic and costs no request, so it does not count. + if (pageFor(node.ref)) { + if (visited >= options.maxDirectories) { + truncated = true; + return; + } + visited++; + } + let listing: FmListing; + try { + listing = await this.list(node.ref); + } catch (error) { + failures.push({ path: node.path, reason: error instanceof Error ? error.message : String(error) }); + return; + } + // The root's areas keep their defined order; real folders sort by name. + const directories = pageFor(node.ref) ? sortByName(listing.directories) : listing.directories; + for (const directory of directories) { + const path = joinPath(node.path, directory.name); + let ref: DirectoryRef | undefined; + try { + ref = childRef(node.ref, directory); + } catch { + ref = undefined; // shared folders: listed, never opened + } + entries.push({ + path, + parentPath: node.path, + depth: node.depth + 1, + parent: node.ref, + directory: { ref: ref ?? node.ref, name: directory.name, id: directory.id }, + }); + if (ref && node.depth + 1 < options.maxDepth) next.push({ path, ref, depth: node.depth + 1 }); + } + for (const file of sortByName(listing.files)) { + entries.push({ + path: joinPath(node.path, file.name), + parentPath: node.path, + depth: node.depth + 1, + parent: node.ref, + file, + }); + } + }), + ); + } + frontier = next; + } + + // Unsorted across directories on purpose: sorting whole path strings + // interleaves a folder's children with a sibling that shares its prefix + // ("Sub/…" against "Sub - Kopie"). Renderers group on `parentPath`. + return { entries, visited, truncated, failures }; + } + + download(file: Pick): Promise { + return this.client.downloadFileManagerFile(file.id, file.name); + } +} + +type Match = { kind: 'directory'; entry: FmDirectory } | { kind: 'file'; entry: FmFile }; + +/** + * Every way the next path segments can name an entry, shortest first. + * + * `rest[0]`, then `rest[0]/rest[1]`, and so on — so a course called + * "LF07 - FIA24A/B - Sb/Ha" resolves even when typed plainly. An id segment + * matches too. Exact names are preferred over case-insensitive ones. + */ +function candidateReadings(listing: FmListing, rest: string[]): { name: string; consumed: number; matches: Match[] }[] { + const all: Match[] = [ + ...listing.directories.map((entry) => ({ kind: 'directory' as const, entry })), + ...listing.files.map((entry) => ({ kind: 'file' as const, entry })), + ]; + + const first = rest[0] as string; + if (/^[0-9a-f]{24}$/i.test(first)) { + const byId = all.filter((match) => match.entry.id.toLowerCase() === first.toLowerCase()); + if (byId.length > 0) return [{ name: first, consumed: 1, matches: byId }]; + } + + const readings: { name: string; consumed: number; matches: Match[] }[] = []; + for (const strict of [true, false]) { + for (let take = 1; take <= rest.length; take++) { + const name = rest.slice(0, take).join('/'); + const matches = all.filter((match) => + strict ? match.entry.name.trim() === name : normalise(match.entry.name) === normalise(name), + ); + if (matches.length > 0 && !readings.some((reading) => reading.consumed === take)) { + readings.push({ name, consumed: take, matches }); + } + } + if (readings.length > 0) break; + } + return readings; +} + +function notFound(listing: FmListing, segment: string, path: string): FsError { + const names = [...listing.directories.map((entry) => `${entry.name}/`), ...listing.files.map((entry) => entry.name)]; + const needle = normalise(segment); + const close = names.filter((name) => normalise(name).includes(needle) || needle.includes(normalise(name).replace(/\/$/, ''))); + const hint = + close.length > 0 + ? ` Did you mean: ${close.slice(0, 5).map((name) => `"${name}"`).join(', ')}?` + : names.length > 0 + ? ` It holds ${names.length} entr${names.length === 1 ? 'y' : 'ies'}; list it with fs_list.` + : ' It is empty.'; + return new FsError('not_found', `No "${segment}" in ${path}.${hint}`); +} + +function sortByName(items: T[]): T[] { + return [...items].sort((a, b) => compareNames(a.name, b.name)); +} + +/** Name order as a person expects it: German collation, "Blatt 2" before "Blatt 10". */ +export function compareNames(a: string, b: string): number { + return a.localeCompare(b, 'de', { numeric: true, sensitivity: 'base' }); +} + +/** + * Case-insensitive substring; or, when the pattern uses * or ?, a glob that + * must match the whole name, as `find -name` does. Exported for testing. + */ +export function nameMatcher(pattern: string): (name: string) => boolean { + const needle = pattern.normalize('NFC').toLocaleLowerCase('de'); + if (!/[*?]/.test(needle)) return (name) => name.normalize('NFC').toLocaleLowerCase('de').includes(needle); + const source = needle + .split('') + .map((char) => (char === '*' ? '.*' : char === '?' ? '.' : char.replace(/[.+^${}()|[\]\\]/g, '\\$&'))) + .join(''); + const regex = new RegExp(`^${source}$`, 'i'); + return (name) => regex.test(name.normalize('NFC')); +} diff --git a/src/core/paths.ts b/src/core/paths.ts index 9034c19..81c58e7 100644 --- a/src/core/paths.ts +++ b/src/core/paths.ts @@ -74,7 +74,7 @@ export function safeComponent(raw: string, fallback = 'untitled'): string { * same card, which the API permits. */ export function mirrorPath(at: Breadcrumb, fileName: string, fileId: string): string { - const parts = [at.courseTitle, at.containerTitle, at.cardTitle] + const parts = [at.courseTitle, at.containerTitle, ...(at.folders ?? []), at.cardTitle] .filter((part): part is string => Boolean(part && part.trim())) .map((part) => safeComponent(part)); diff --git a/src/http/api.ts b/src/http/api.ts index 1aec8da..3cdaf9e 100644 --- a/src/http/api.ts +++ b/src/http/api.ts @@ -2,6 +2,16 @@ import { createReadStream } from 'node:fs'; import { stat } from 'node:fs/promises'; import { Readable } from 'node:stream'; import express, { type Request, type Response, type Router } from 'express'; +import { SchulcloudApiError } from '../core/client.ts'; +import { + compareNames, + FileManagerMarkupError, + FsError, + nameMatcher, + type FmFile, + type FsErrorCode, + type WalkEntry, +} from '../core/legacy-files.ts'; import { resolveWithin } from '../core/paths.ts'; import type { Services } from '../services.ts'; @@ -64,6 +74,113 @@ export function createApiRouter(services: Services): Router { } }); + // --- the file manager ("Dateien"), as a filesystem ---------------------- + // + // Live, not from the index: these answer what the file manager holds now, and + // need no database. Paths are the same ones the MCP fs_* tools print. + + router.get('/fs/list', async (req: Request, res: Response) => { + try { + const node = await services.files.resolve(stringParam(req.query.path) ?? '/'); + if (node.kind === 'file') return res.json({ path: node.path, kind: 'file', file: node.file }); + const listing = await services.files.list(node.ref); + return res.json({ + path: node.path, + kind: 'directory', + area: node.ref.area ?? null, + directories: listing.directories.map((entry) => ({ ...entry, path: childPath(node.path, entry.name) })), + files: listing.files.map((entry) => ({ ...entry, path: childPath(node.path, entry.name) })), + }); + } catch (error) { + return fsFail(res, error, 'fs list'); + } + }); + + router.get('/fs/tree', async (req: Request, res: Response) => { + try { + const node = await services.files.resolve(stringParam(req.query.path) ?? '/'); + if (node.kind === 'file') return res.json({ path: node.path, kind: 'file', file: node.file }); + const result = await services.files.walk(node, { + maxDepth: boundedInt(req.query.depth, 3, 1, 12), + maxDirectories: boundedInt(req.query.maxFolders, 200, 1, 1000), + }); + return res.json({ + path: node.path, + kind: 'directory', + entries: result.entries.map(treeEntry), + visited: result.visited, + truncated: result.truncated, + failures: result.failures, + }); + } catch (error) { + return fsFail(res, error, 'fs tree'); + } + }); + + router.get('/fs/find', async (req: Request, res: Response) => { + const name = stringParam(req.query.name); + if (!name) return res.status(400).json({ error: 'bad_request', message: 'Give name.' }); + try { + const node = await services.files.resolve(stringParam(req.query.path) ?? '/'); + if (node.kind === 'file') return res.json({ path: node.path, kind: 'file', matches: [] }); + const type = stringParam(req.query.type) ?? 'any'; + const matches = nameMatcher(name); + const result = await services.files.walk(node, { + maxDepth: 12, + maxDirectories: boundedInt(req.query.maxFolders, 400, 1, 1000), + }); + return res.json({ + path: node.path, + kind: 'directory', + matches: result.entries + .filter((entry) => (type === 'file' ? entry.file : type === 'folder' ? entry.directory : true)) + .filter((entry) => matches((entry.file ?? entry.directory)?.name ?? '')) + .sort((a, b) => compareNames(a.path, b.path)) + .map(treeEntry), + visited: result.visited, + truncated: result.truncated, + failures: result.failures, + }); + } catch (error) { + return fsFail(res, error, 'fs find'); + } + }); + + router.get('/fs/file', async (req: Request, res: Response) => { + try { + const path = stringParam(req.query.path); + const id = stringParam(req.query.id); + let file: Pick & Partial; + if (path) { + const node = await services.files.resolve(path); + if (node.kind !== 'file') return res.status(400).json({ error: 'not_a_file', message: `${node.path} is a folder.` }); + file = node.file; + } else if (id && /^[0-9a-f]{24}$/i.test(id)) { + file = { id, name: stringParam(req.query.name) ?? id }; + } else { + return res.status(400).json({ error: 'bad_request', message: 'Give path, or id (and name).' }); + } + if (file.blocked) { + return res.status(403).json({ error: 'blocked', message: 'The instance virus scanner blocked this file.' }); + } + + // Streamed straight through rather than buffered: the CLI uses this for + // whole folders, and videos routinely exceed any sensible in-memory cap. + const signed = await services.client.getFileManagerSignedUrl(file.id, file.name); + const upstream = await services.client.openSignedUrl(signed); + if (!upstream.body) return res.status(502).json({ error: 'upstream_failed', message: 'empty response' }); + + res.setHeader('Content-Type', file.mimeType || upstream.headers.get('content-type') || 'application/octet-stream'); + res.setHeader('Content-Disposition', contentDisposition(file.name)); + const length = upstream.headers.get('content-length'); + if (length) res.setHeader('Content-Length', length); + res.setHeader('X-Schulcloud-Source', 'file-manager'); + Readable.fromWeb(upstream.body as never).pipe(res); + } catch (error) { + return fsFail(res, error, 'fs file'); + } + }); + /** * Streams one file. Served from the local mirror when present; otherwise * proxied live, which is what keeps files too large to mirror reachable. @@ -91,6 +208,8 @@ export function createApiRouter(services: Services): Router { }); } } + const known = await services.store.fileSource(fileId); + if (known?.source === 'file-manager') return await proxyFileManager(services, fileId, known, res); return await proxyLive(services, fileId, res); } catch (error) { return fail(res, error, 'file'); @@ -101,6 +220,27 @@ export function createApiRouter(services: Services): Router { } /** Falls back to Schulcloud for anything not in the mirror, streaming through. */ +/** Streams a file-manager file live, via its pre-signed URL; no credentials leave for the storage host. */ +async function proxyFileManager( + services: Services, + fileId: string, + known: { name: string; mimeType: string; size: number }, + res: Response, +): Promise { + const signed = await services.client.getFileManagerSignedUrl(fileId, known.name); + const upstream = await services.client.openSignedUrl(signed); + if (!upstream.body) { + res.status(502).json({ error: 'upstream_failed', message: 'empty response' }); + return; + } + res.setHeader('Content-Type', known.mimeType || 'application/octet-stream'); + res.setHeader('Content-Disposition', contentDisposition(known.name)); + const length = upstream.headers.get('content-length'); + if (length) res.setHeader('Content-Length', length); + res.setHeader('X-Schulcloud-Source', 'live'); + Readable.fromWeb(upstream.body as never).pipe(res); +} + async function proxyLive(services: Services, fileId: string, res: Response): Promise { const record = await services.client.getFileRecord(fileId); if (record.securityCheckStatus === 'blocked') { @@ -143,3 +283,64 @@ function fail(res: Response, error: unknown, what: string): void { console.error(`[schulcloud-mcp] ${what} failed:`, error); if (!res.headersSent) res.status(500).json({ error: 'internal_error' }); } + +function stringParam(value: unknown): string | undefined { + const first = Array.isArray(value) ? value[0] : value; + return typeof first === 'string' && first.length > 0 ? first : undefined; +} + +function boundedInt(value: unknown, fallback: number, min: number, max: number): number { + const parsed = Number.parseInt(stringParam(value) ?? '', 10); + return Number.isFinite(parsed) ? Math.min(max, Math.max(min, parsed)) : fallback; +} + +function childPath(parent: string, name: string): string { + return `${parent === '/' ? '' : parent}/${name}`; +} + +/** A walk entry as JSON; `name` travels separately because names may contain "/". */ +function treeEntry(entry: WalkEntry) { + if (entry.directory) { + return { type: 'directory', path: entry.path, parentPath: entry.parentPath, depth: entry.depth, id: entry.directory.id, name: entry.directory.name }; + } + const file = entry.file as FmFile; + return { + type: 'file', + path: entry.path, + parentPath: entry.parentPath, + depth: entry.depth, + id: file.id, + name: file.name, + size: file.size, + mimeType: file.mimeType ?? null, + blocked: file.blocked, + }; +} + +const FS_STATUS: Record = { + not_found: 404, + ambiguous: 409, + not_a_directory: 400, + not_a_file: 400, + not_navigable: 422, +}; + +function fsFail(res: Response, error: unknown, what: string): void { + if (res.headersSent) return; + if (error instanceof FsError) { + res.status(FS_STATUS[error.code]).json({ error: error.code, message: error.message }); + return; + } + if (error instanceof FileManagerMarkupError) { + res.status(502).json({ error: 'markup_changed', message: error.message }); + return; + } + if (error instanceof SchulcloudApiError) { + // 401 upstream is the Pi's session, not the caller's token: say which. + const status = error.status === 401 ? 502 : error.status === 403 || error.status === 404 ? error.status : 502; + const message = error.status === 401 ? 'The Schulcloud session has expired on the server.' : error.message; + res.status(status).json({ error: 'upstream_failed', message }); + return; + } + fail(res, error, what); +} diff --git a/src/indexer/indexer.ts b/src/indexer/indexer.ts index eec0610..098de69 100644 --- a/src/indexer/indexer.ts +++ b/src/indexer/indexer.ts @@ -117,6 +117,7 @@ export class Indexer { includeLessonContents: true, includeFiles: true, includePersonalFiles: this.config.indexPersonalFiles, + includeFileManager: this.config.indexFileManager, config: this.config, }); @@ -192,7 +193,11 @@ export class Indexer { } try { - const downloaded = await this.client.downloadFile(file.record); + // The file manager's ids mean nothing to files-storage; route by store. + const downloaded = + file.source === 'file-manager' + ? await this.client.downloadFileManagerFile(file.record.id, file.record.name) + : await this.client.downloadFile(file.record); const relative = paths.get(entry.fileId); if (!relative) return; const absolute = resolveWithin(this.config.mirrorDir, relative); diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 8dcc81e..79b5244 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -4,6 +4,7 @@ import { ServerContext } from '../context.ts'; import type { Services } from '../services.ts'; import { registerContentTools } from './tools/content.ts'; import { registerFileTools } from './tools/files.ts'; +import { registerFilesystemTools } from './tools/filesystem.ts'; import { registerOverviewTools } from './tools/overview.ts'; import { registerRawTool } from './tools/raw.ts'; import { registerIndexTools } from './tools/index-tools.ts'; @@ -26,6 +27,11 @@ How the content is organised, and the usual path through it: - **Tasks** ("Aufgaben") — homework. list_tasks across all courses, get_task for one. - **Files** hang off boards, lessons and tasks. Every listing shows file ids; download_file fetches one and extracts its text (PDF, Word, Excel, PowerPoint, OpenDocument) or returns an image inline. +- **The file manager ("Dateien")** is a separate store with a real folder tree, browsed with the fs_* tools: + /my (Persönliche Dateien), /courses/ (Kurs-Dateien), /teams/ (Team-Dateien) and /shared + (Geteilte Dateien). **Many teachers put their material only here**, so when a course page looks empty or the + worksheets are not on its boards, look in /courses/. fs_list and fs_tree browse, fs_find finds by + name, fs_read opens a file. list_files and download_file do not see these files. - **Submissions** ("Abgaben") — what the user handed in. get_task shows that task's submission: the files, the graded flag, the grade, what the user wrote, and the teacher's written feedback. A grade is a percentage (0-100) or absent — there is no textual grade — and teachers often grade with the written @@ -51,6 +57,7 @@ export function createServer(config: Config, services?: Services): { server: Mcp registerContentTools(server, context); registerRoomTools(server, context); registerFileTools(server, context); + registerFilesystemTools(server, context); registerSearchTool(server, context); registerSubmissionTools(server, context); registerIndexTools(server, context); diff --git a/src/mcp/tools/content.ts b/src/mcp/tools/content.ts index b09f30c..9447b53 100644 --- a/src/mcp/tools/content.ts +++ b/src/mcp/tools/content.ts @@ -7,6 +7,7 @@ import { dueLabel, formatDate, heading, htmlToText, joinSections, normalizeObjec import { assembleBoard, type AssembledBoard, type AssembledElement } from '../../core/board.ts'; import { forEachLimited } from '../../core/crawl.ts'; import { fetchLessonPadText } from '../../core/etherpad.ts'; +import type { FmListing } from '../../core/legacy-files.ts'; import { fetchLessonTaskLinks, withScrapedIds } from '../../core/lesson-page.ts'; import type { CourseBoardResponse, @@ -38,17 +39,21 @@ export function registerContentTools(server: McpServer, context: ServerContext): }, async ({ courseId }) => { try { - const [board, legacy] = await Promise.all([ + const [board, legacy, courseFiles] = await Promise.all([ context.client.getCourseBoard(courseId), // The v3 projection carries no description, teachers, members or // timetable; /api/v1/courses still does. Optional on purpose — it // is a legacy route, so its absence must cost detail, not the call. context.client.getLegacyCourse(courseId).catch(() => undefined), + // The course's file-manager area is a different store from the page. + // Teachers who only upload files there leave the page itself empty, + // and reporting "empty" then sends the reader away from the material. + context.files.list({ area: 'courses', ownerId: courseId }).catch(() => undefined), ]); const teachers = legacy ? await context.resolveNames([...(legacy.teacherIds ?? []), ...(legacy.substitutionIds ?? [])]) : { names: [], unresolved: 0 }; - return text(formatCourseBoard(board, legacy, teachers)); + return text(formatCourseBoard(board, legacy, teachers, courseFiles)); } catch (error) { return toToolError(error, `read course ${courseId}`); } @@ -266,6 +271,7 @@ function formatCourseBoard( board: CourseBoardResponse, legacy?: LegacyCourse, teachers: { names: string[]; unresolved: number } = { names: [], unresolved: 0 }, + courseFiles?: FmListing, ): string { const boards: string[] = []; const lessons: string[] = []; @@ -293,8 +299,18 @@ function formatCourseBoard( formatCourseTimes(legacy?.times), ]); + const filesSection = formatCourseFiles(board.roomId, courseFiles); + if (boards.length + lessons.length + tasks.length === 0) { - return joinSections([heading(2, board.title), about, 'This course page is empty.']); + return joinSections([ + heading(2, board.title), + `Course id: \`${board.roomId}\``, + about, + filesSection + ? 'No boards, topics or tasks on the course page — the material is in the course files instead.' + : 'This course page is empty, and the course has no files in the file manager either.', + filesSection, + ]); } return joinSections([ @@ -304,6 +320,25 @@ function formatCourseBoard( boards.length > 0 && joinSections([heading(3, `Boards (${boards.length})`), boards.join('\n'), 'Read one with get_board.']), lessons.length > 0 && joinSections([heading(3, `Topics (${lessons.length})`), lessons.join('\n'), 'Read one with get_lesson.']), tasks.length > 0 && joinSections([heading(3, `Tasks (${tasks.length})`), tasks.join('\n'), 'Read one with get_task.']), + filesSection, + ]); +} + +/** + * The course's own file-manager area ("Kurs-Dateien"), when it holds anything. + * + * Only the top level is fetched — one page — so this says how much is there + * and where, rather than listing it; fs_tree does that. + */ +function formatCourseFiles(courseId: string, listing: FmListing | undefined): string | undefined { + if (!listing || listing.directories.length + listing.files.length === 0) return undefined; + const names = [...listing.directories.map((entry) => `${entry.name}/`), ...listing.files.map((entry) => entry.name)]; + const shown = names.slice(0, 8).map((name) => `- ${name}`).join('\n'); + return joinSections([ + heading(3, 'Course files (Kurs-Dateien)'), + `${listing.directories.length} folder(s) and ${listing.files.length} file(s) at the top level, newest first:`, + shown + (names.length > 8 ? `\n- … and ${names.length - 8} more` : ''), + `See everything with fs_tree path "/courses/${courseId}", read one with fs_read.`, ]); } diff --git a/src/mcp/tools/files.ts b/src/mcp/tools/files.ts index 0880aab..fd12c3b 100644 --- a/src/mcp/tools/files.ts +++ b/src/mcp/tools/files.ts @@ -2,6 +2,7 @@ import type { CallToolResult } from '@modelcontextprotocol/sdk/types.js'; import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; import { z } from 'zod'; import type { ServerContext } from '../../context.ts'; +import type { DownloadedFile } from '../../core/client.ts'; import { extractContent, formatBytes } from '../../core/extract.ts'; import { formatDate, heading, joinSections } from '../../core/text.ts'; import { FILE_PARENT_TYPES, type FileParentType } from '../../core/types.ts'; @@ -16,9 +17,11 @@ export function registerFileTools(server: McpServer, context: ServerContext): vo { title: 'List files of an entity', description: - 'Files attached to one entity. Most of the time you do not need this — get_board, get_lesson and ' + - 'get_task already list their own attachments. Reach for it to enumerate a course\'s own file area, ' + - 'or a single board element\'s files (parentType "boardnodes", parentId = the element id).', + 'Attachments on one entity in files-storage: a board element, a topic, a task, a submission. Most of the ' + + 'time you do not need this — get_board, get_lesson and get_task already list their own attachments. ' + + '**Not for a course\'s files, personal files, team files or shared files**: those live in the file ' + + 'manager ("Dateien"), a separate store this tool cannot see — it answers 0 for a course holding dozens of ' + + 'worksheets. Use fs_list, fs_tree, fs_find and fs_read for them.', inputSchema: { parentType: z .enum(FILE_PARENT_TYPES as [FileParentType, ...FileParentType[]]) @@ -61,9 +64,10 @@ export function registerFileTools(server: McpServer, context: ServerContext): vo { title: 'Download and read a file', description: - 'Fetches a file and returns its contents. PDFs, Word, Excel, PowerPoint and OpenDocument files are ' + - 'extracted to text; images come back inline so you can look at them; anything else reports its type. ' + - 'Pass raw=true to get base64 bytes instead of extracted text.', + 'Fetches a board, topic or task attachment and returns its contents. PDFs, Word, Excel, PowerPoint and ' + + 'OpenDocument files are extracted to text; images come back inline so you can look at them; anything ' + + 'else reports its type. Pass raw=true to get base64 bytes instead of extracted text. For files from the ' + + 'file manager (Persönliche Dateien, Kurs-Dateien, Team-Dateien, Geteilte Dateien) use fs_read instead.', inputSchema: { fileId: z.string().describe('File record id, from get_board, get_task, get_lesson or list_files.'), raw: z @@ -116,76 +120,99 @@ export function registerFileTools(server: McpServer, context: ServerContext): vo .join('\n'), ].join('\n\n'); - if (raw) { - return text( - joinSections([ - header, - `Base64 (${file.bytes.length} bytes):`, - '```', - file.bytes.toString('base64'), - '```', - ]), - ); - } - - const extraction = await extractContent( - file.bytes, - file.mimeType || record.mimeType, - record.name, - maxChars ?? context.config.maxExtractedChars, - ); - - if (extraction.kind === 'image' && extraction.image) { - const result: CallToolResult = { - content: [ - { type: 'text', text: joinSections([header, extraction.note]) }, - { type: 'image', data: extraction.image.base64, mimeType: extraction.image.mimeType }, - ], - }; - return result; - } - - if (extraction.kind === 'text') { - const body = extraction.text?.trim(); - return text( - joinSections([ - header, - extraction.note, - body ? joinSections([heading(3, 'Contents'), body]) : '_(the file contains no extractable text)_', - ]), - ); - } - - // Nothing extractable — but files-storage may still be able to render - // the file as a picture. That is the whole answer for an image-only - // PDF: its pages *are* pictures, so a rasterised preview is readable - // where the bytes are not, and it needs no OCR on our side. - if (record.previewStatus === 'preview_possible') { - const preview = await context.client.getFilePreview(record, 500).catch(() => undefined); - if (preview && preview.mimeType.startsWith('image/')) { - const result: CallToolResult = { - content: [ - { - type: 'text', - text: joinSections([ - header, - // The note ends by suggesting raw bytes, which is no longer the - // best answer once a readable rendering is attached. - extraction.note.replace(' Use download_file with raw=true to get the bytes.', ''), - "Showing the instance's own rendered preview below, which is readable as a picture.", - ]), - }, - { type: 'image', data: preview.bytes.toString('base64'), mimeType: preview.mimeType }, - ], - }; - return result; - } - } - - return text(joinSections([header, extraction.note])); + return await renderFileContent(context, header, file, { + name: record.name, + mimeType: record.mimeType, + raw, + maxChars, + // Nothing extractable — but files-storage may still be able to render + // the file as a picture. That is the whole answer for an image-only + // PDF: its pages *are* pictures, so a rasterised preview is readable + // where the bytes are not, and it needs no OCR on our side. + fallbackImage: + record.previewStatus === 'preview_possible' + ? async () => { + const preview = await context.client.getFilePreview(record, 500).catch(() => undefined); + return preview && preview.mimeType.startsWith('image/') ? preview : undefined; + } + : undefined, + }); } catch (error) { return toToolError(error, `download file ${fileId}`); } }, ); } + +/** + * Renders a downloaded file for a tool result: base64 when asked for, an image + * inline, extracted text, or — for a format with no extractor — its note. + * + * Shared by download_file (files-storage) and fs_read (the file manager), which + * differ only in how the bytes were obtained and in what the header says. + * `fallbackImage` is download_file's preview route; the file manager has none. + */ +export async function renderFileContent( + context: ServerContext, + header: string, + file: DownloadedFile, + options: { + name: string; + mimeType?: string; + raw: boolean; + maxChars?: number; + fallbackImage?: () => Promise; + }, +): Promise { + if (options.raw) { + return text(joinSections([header, `Base64 (${file.bytes.length} bytes):`, '```', file.bytes.toString('base64'), '```'])); + } + + const extraction = await extractContent( + file.bytes, + file.mimeType && file.mimeType !== 'application/octet-stream' ? file.mimeType : (options.mimeType ?? file.mimeType), + options.name, + options.maxChars ?? context.config.maxExtractedChars, + ); + + if (extraction.kind === 'image' && extraction.image) { + return { + content: [ + { type: 'text', text: joinSections([header, extraction.note]) }, + { type: 'image', data: extraction.image.base64, mimeType: extraction.image.mimeType }, + ], + }; + } + + if (extraction.kind === 'text') { + const body = extraction.text?.trim(); + return text( + joinSections([ + header, + extraction.note, + body ? joinSections([heading(3, 'Contents'), body]) : '_(the file contains no extractable text)_', + ]), + ); + } + + const image = await options.fallbackImage?.(); + if (image) { + return { + content: [ + { + type: 'text', + text: joinSections([ + header, + // The note ends by suggesting raw bytes, which is no longer the + // best answer once a readable rendering is attached. + extraction.note.replace(/ Use \w+ with raw=true to get the bytes\./, ''), + "Showing the instance's own rendered preview below, which is readable as a picture.", + ]), + }, + { type: 'image', data: image.bytes.toString('base64'), mimeType: image.mimeType }, + ], + }; + } + + return text(joinSections([header, extraction.note])); +} diff --git a/src/mcp/tools/filesystem.ts b/src/mcp/tools/filesystem.ts new file mode 100644 index 0000000..59150ab --- /dev/null +++ b/src/mcp/tools/filesystem.ts @@ -0,0 +1,401 @@ +import type { CallToolResult } from '@modelcontextprotocol/sdk/types.js'; +import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; +import { z } from 'zod'; +import type { ServerContext } from '../../context.ts'; +import { formatBytes } from '../../core/extract.ts'; +import { + areaInfo, + compareNames, + FILE_AREAS, + FileManagerMarkupError, + FsError, + nameMatcher, + type DirectoryRef, + type FmFile, + type FsNode, + type WalkEntry, +} from '../../core/legacy-files.ts'; +import { heading, joinSections } from '../../core/text.ts'; +import { renderFileContent } from './files.ts'; +import { failure, text, toToolError } from './result.ts'; + +const READ_ONLY = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true }; + +/** + * The "Dateien" file manager as filesystem tools: ls, tree, find, read. + * + * Deliberately separate from list_files / download_file, which read + * files-storage — board, topic and task attachments. The two stores do not + * overlap, and conflating them is how a course holding dozens of worksheets + * came to be reported as having 0 files. + */ + +const AREA_NOTE = + 'The file manager ("Dateien") is separate from course pages and holds four areas: ' + + '/my (Persönliche Dateien), /courses/ (Kurs-Dateien), /teams/ (Team-Dateien) and ' + + '/shared (Geteilte Dateien). Many teachers keep their material only in Kurs-Dateien, so a course whose page ' + + 'looks empty often has its worksheets here.'; + +const PATH_NOTE = + 'Paths use the names shown in listings, e.g. "/courses/FIA24B - LF2 (Rh)/Handlungssituation". Names may ' + + 'contain "/" and still resolve; any segment may also be the id printed next to it, which is never ambiguous.'; + +export function registerFilesystemTools(server: McpServer, context: ServerContext): void { + server.registerTool( + 'fs_list', + { + title: 'List a folder in the file manager', + description: + `Lists one folder of the Schulcloud file manager, like \`ls\`. ${AREA_NOTE} Start at "/" or ` + + '"/courses" to see what exists. Given a file path instead, shows that file\'s details. ' + + `${PATH_NOTE} Not for attachments on boards, topics or tasks — get_board, get_lesson and get_task list ` + + 'those, and download_file reads them.', + inputSchema: { + path: z.string().default('/').describe('Folder to list, e.g. "/", "/courses", "/courses//".'), + }, + annotations: READ_ONLY, + }, + async ({ path }) => { + try { + const node = await context.files.resolve(path); + if (node.kind === 'file') return text(describeFile(node)); + return text(await listDirectory(context, node)); + } catch (error) { + return fsError(error, `list ${path}`); + } + }, + ); + + server.registerTool( + 'fs_tree', + { + title: 'Show a folder tree in the file manager', + description: + `Everything below a folder of the Schulcloud file manager, as an indented tree, like \`tree\`. ${AREA_NOTE} ` + + 'Use it to get an overview of a course\'s files in one call — "/courses/" — or of all course ' + + 'files at a shallow depth. Each folder costs one page load, so the walk stops at `maxFolders` and says ' + + 'so; narrow the path or lower the depth rather than raising the limit. To look for a name, fs_find is ' + + `cheaper. ${PATH_NOTE}`, + inputSchema: { + path: z.string().default('/').describe('Folder to start from.'), + depth: z.number().int().min(1).max(8).default(3).describe('How many levels below the folder to show.'), + maxFolders: z + .number() + .int() + .min(1) + .max(400) + .default(80) + .describe('Stop after listing this many folders.'), + }, + annotations: READ_ONLY, + }, + async ({ path, depth, maxFolders }) => { + try { + const node = await context.files.resolve(path); + if (node.kind === 'file') return text(describeFile(node)); + const result = await context.files.walk(node, { maxDepth: depth, maxDirectories: maxFolders }); + return text(renderTree(node, result.entries, { depth, maxFolders, ...result })); + } catch (error) { + return fsError(error, `walk ${path}`); + } + }, + ); + + server.registerTool( + 'fs_find', + { + title: 'Find files by name in the file manager', + description: + `Finds files and folders by name anywhere below a folder of the Schulcloud file manager, like \`find\`. ` + + `${AREA_NOTE} Without wildcards it matches any part of the name, case-insensitively. With "*" or "?" the ` + + 'whole name must match, as with find -name — so "*.docx", or "*Erben*" for names containing Erben. Scope it with ' + + '`path` (e.g. "/courses/") whenever you know the course: searching all of /courses walks every ' + + 'folder of every course. This matches names only — to search inside documents use search, which ' + + `covers file-manager files once they are indexed. ${PATH_NOTE}`, + inputSchema: { + name: z + .string() + .min(1) + .describe('Part of the name ("Erbrecht"), or a whole-name pattern with * and ? ("*.docx", "*Erben*").'), + path: z.string().default('/').describe('Folder to search below. Default: every area.'), + type: z.enum(['any', 'file', 'folder']).default('any').describe('Only files, only folders, or both.'), + maxFolders: z + .number() + .int() + .min(1) + .max(600) + .default(250) + .describe('Stop after listing this many folders.'), + }, + annotations: READ_ONLY, + }, + async ({ name, path, type, maxFolders }) => { + try { + const node = await context.files.resolve(path); + if (node.kind === 'file') return text(describeFile(node)); + const matcher = nameMatcher(name); + const result = await context.files.walk(node, { maxDepth: 12, maxDirectories: maxFolders }); + const hits = result.entries + .filter((entry) => (type === 'file' ? entry.file : type === 'folder' ? entry.directory : true)) + .filter((entry) => matcher((entry.file ?? entry.directory)?.name ?? '')) + .sort((a, b) => compareNames(a.path, b.path)); + + const scope = `${result.visited} folder(s) searched`; + const notes = [ + result.truncated + ? `_Stopped after ${maxFolders} folders, so there may be more matches. Narrow \`path\` to one course._` + : undefined, + failureNote(result.failures), + ]; + if (hits.length === 0) { + return text(joinSections([`No names matching "${name}" below ${node.path} (${scope}).`, ...notes])); + } + return text( + joinSections([ + heading(2, `${hits.length} match(es) for "${name}" below ${node.path}`), + hits.map((entry) => entryLine(entry, { fullPath: true })).join('\n'), + `_${scope}._ Read a file with fs_read, open a folder with fs_list.`, + ...notes, + ]), + ); + } catch (error) { + return fsError(error, `search ${path}`); + } + }, + ); + + server.registerTool( + 'fs_read', + { + title: 'Read a file from the file manager', + description: + `Fetches one file from the Schulcloud file manager and returns its contents, like \`cat\`. ${AREA_NOTE} ` + + 'PDFs, Word, Excel, PowerPoint and OpenDocument files are extracted to text; images come back inline so ' + + 'you can look at them; anything else reports its type. Pass raw=true for base64 bytes. Give the path ' + + 'from a listing, or the file id and name. Not for board, topic or task attachments — use download_file ' + + `for those. ${PATH_NOTE}`, + inputSchema: { + path: z.string().optional().describe('File path, e.g. "/courses///Arbeitsblatt.pdf".'), + fileId: z.string().optional().describe('File id from a listing, instead of a path.'), + name: z.string().optional().describe('The file name, when giving fileId; used to recognise the format.'), + raw: z.boolean().default(false).describe('Return base64-encoded bytes instead of extracted text.'), + maxChars: z + .number() + .int() + .min(500) + .max(500_000) + .optional() + .describe('Override the character limit on extracted text.'), + }, + annotations: READ_ONLY, + }, + async ({ path, fileId, name, raw, maxChars }) => { + try { + let file: Pick & Partial; + let where: string; + if (path) { + const node = await context.files.resolve(path); + if (node.kind !== 'file') { + return failure(`${node.path} is a folder, not a file. List it with fs_list, or use fs_tree.`); + } + file = node.file; + where = node.path; + } else if (fileId) { + if (!/^[0-9a-f]{24}$/i.test(fileId)) return failure(`"${fileId}" is not a file id.`); + file = { id: fileId, name: name?.trim() || fileId }; + where = `file \`${fileId}\``; + } else { + return failure('Give either `path` or `fileId`.'); + } + + // The instance scans uploads; a file it rejected is not served. + if (file.blocked) { + return failure(`"${file.name}" was blocked by the instance's virus scanner and will not be downloaded.`); + } + + const downloaded = await context.files.download(file); + const header = [ + heading(2, file.name), + [ + `- Path: ${where}`, + `- File id: \`${file.id}\``, + `- Type: ${file.mimeType ?? downloaded.mimeType}`, + `- Size: ${formatBytes(file.size ?? downloaded.bytes.length)}`, + downloaded.truncated + ? `- **Download was capped at ${formatBytes(context.config.maxDownloadBytes)}; content is incomplete.**` + : undefined, + ] + .filter(Boolean) + .join('\n'), + ].join('\n\n'); + + return await renderFileContent(context, header, downloaded, { + name: file.name, + mimeType: file.mimeType, + raw, + maxChars, + }); + } catch (error) { + return fsError(error, `read ${path ?? fileId}`); + } + }, + ); +} + +async function listDirectory(context: ServerContext, node: Extract): Promise { + if (!node.ref.area) { + return joinSections([ + heading(2, '/ — the file manager ("Dateien")'), + FILE_AREAS.map((entry) => `- **/${entry.area}/** — ${entry.label}: ${entry.summary}`).join('\n'), + 'Open one with fs_list, e.g. path "/courses". A course\'s own files are under "/courses/".', + ]); + } + + const listing = await context.files.list(node.ref); + const area = areaInfo(node.ref.area); + const isOwnerList = (node.ref.area === 'courses' || node.ref.area === 'teams') && !node.ref.ownerId; + const directories = [...listing.directories].sort((a, b) => compareNames(a.name, b.name)); + const files = [...listing.files].sort((a, b) => compareNames(a.name, b.name)); + + const title = heading(2, `${node.path} — ${area.label}`); + if (directories.length === 0 && files.length === 0) { + return joinSections([ + title, + isOwnerList + ? `No ${node.ref.area === 'courses' ? 'courses' : 'teams'} with a file area.` + : node.ref.area === 'shared' + ? 'Nothing has been shared with you.' + : 'This folder is empty.', + ]); + } + + const lines = [ + ...directories.map((directory) => `- **${directory.name}/** (\`${directory.id}\`)`), + ...files.map((file) => `- ${fileLine(file)}`), + ]; + const bytes = files.reduce((sum, file) => sum + file.size, 0); + const summary = isOwnerList + ? `${directories.length} ${node.ref.area === 'courses' ? 'course' : 'team'}(s). Their files are inside; fs_tree with depth 2 shows which hold any.` + : `${directories.length} folder(s), ${files.length} file(s)${files.length ? `, ${formatBytes(bytes)}` : ''}.`; + + return joinSections([ + title, + lines.join('\n'), + summary, + node.ref.area === 'shared' && directories.length > 0 + ? '_Shared folders cannot be opened — the file manager has no route for them, not even in the browser._' + : undefined, + 'Open a folder with fs_list (its name or id appended to this path), read a file with fs_read.', + ]); +} + +function describeFile(node: Extract): string { + return joinSections([ + heading(2, node.file.name), + [ + `- Path: ${node.path}`, + `- File id: \`${node.file.id}\``, + `- Type: ${node.file.mimeType ?? 'unknown'}`, + `- Size: ${formatBytes(node.file.size)}`, + node.file.blocked ? '- **Blocked by the instance virus scanner; it cannot be downloaded.**' : undefined, + ] + .filter(Boolean) + .join('\n'), + node.file.blocked ? undefined : 'Read it with fs_read.', + ]); +} + +function fileLine(file: FmFile): string { + const type = file.mimeType ? `, ${file.mimeType}` : ''; + const blocked = file.blocked ? ' **[blocked by virus scan]**' : ''; + return `${file.name} — ${formatBytes(file.size)}${type} (\`${file.id}\`)${blocked}`; +} + +function entryLine(entry: WalkEntry, options: { fullPath: boolean }): string { + const label = options.fullPath ? entry.path : (entry.file ?? entry.directory)?.name; + if (entry.directory) return `- **${label}/** (\`${entry.directory.id}\`)`; + if (entry.file) return `- ${fileLine({ ...entry.file, name: label ?? entry.file.name })}`; + return `- ${label}`; +} + +/** + * An indented tree, built from each entry's parent rather than from sorting + * path strings — see `FileManager.walk` for why that distinction matters. + */ +function renderTree( + root: { path: string; ref: DirectoryRef }, + entries: WalkEntry[], + info: { depth: number; maxFolders: number; visited: number; truncated: boolean; failures: { path: string; reason: string }[] }, +): string { + const children = new Map(); + for (const entry of entries) { + const list = children.get(entry.parentPath) ?? []; + list.push(entry); + children.set(entry.parentPath, list); + } + + const lines: string[] = []; + let files = 0; + let folders = 0; + let bytes = 0; + const visit = (path: string, indent: string) => { + const kids = children.get(path) ?? []; + // The areas under "/" keep their own order (personal, courses, teams, + // shared); everything else sorts folders first, then by name. + if (path !== '/') { + kids.sort((a, b) => { + if (Boolean(a.directory) !== Boolean(b.directory)) return a.directory ? -1 : 1; + return compareNames(a.path, b.path); + }); + } + for (const kid of kids) { + if (kid.directory) { + folders++; + // An area's "id" is its slug, not an id anything accepts; leave it out. + const id = /^[0-9a-f]{24}$/i.test(kid.directory.id) ? ` \`${kid.directory.id}\`` : ''; + lines.push(`${indent}${kid.directory.name}/${id}`); + visit(kid.path, `${indent} `); + } else if (kid.file) { + files++; + bytes += kid.file.size; + const blocked = kid.file.blocked ? ' [blocked]' : ''; + lines.push(`${indent}${kid.file.name} (${formatBytes(kid.file.size)}) \`${kid.file.id}\`${blocked}`); + } + } + }; + visit(root.path, ''); + + const area = root.ref.area ? ` — ${areaInfo(root.ref.area).label}` : ''; + if (lines.length === 0) { + return joinSections([heading(2, `${root.path}${area}`), 'Nothing below this folder.', failureNote(info.failures)]); + } + + return joinSections([ + heading(2, `${root.path}${area}`), + ['```', ...lines, '```'].join('\n'), + `${folders} folder(s), ${files} file(s), ${formatBytes(bytes)} — ${info.visited} folder(s) listed, ${info.depth} level(s) deep.`, + info.truncated + ? `_Stopped after listing ${info.maxFolders} folders; the tree is incomplete. Start deeper, e.g. at one course._` + : undefined, + failureNote(info.failures), + 'Read a file with fs_read (path = this folder plus the names above).', + ]); +} + +function failureNote(failures: { path: string; reason: string }[]): string | undefined { + if (failures.length === 0) return undefined; + const shown = failures.slice(0, 5).map((entry) => `${entry.path} (${entry.reason})`).join('; '); + return `_Could not list ${failures.length} folder(s): ${shown}${failures.length > 5 ? '; …' : ''}._`; +} + +function fsError(error: unknown, action: string): CallToolResult { + if (error instanceof FsError) return failure(error.message); + if (error instanceof FileManagerMarkupError) { + return failure( + `Could not ${action}: the file manager page did not look like a file listing. Either the session is no ` + + 'longer accepted (check whoami) or the page markup changed.', + ); + } + return toToolError(error, action); +} + diff --git a/src/mcp/tools/search.ts b/src/mcp/tools/search.ts index d08e69d..993fbc4 100644 --- a/src/mcp/tools/search.ts +++ b/src/mcp/tools/search.ts @@ -1,6 +1,7 @@ import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; import { z } from 'zod'; import type { ServerContext } from '../../context.ts'; +import { FILE_AREAS } from '../../core/legacy-files.ts'; import { crawl } from '../../core/crawl.ts'; import { searchSnapshot, type Hit } from '../../core/match.ts'; import { formatDate, heading, joinSections } from '../../core/text.ts'; @@ -144,14 +145,37 @@ function targetIdFor(hit: SearchResult): string { return hit.nodeId; } +/** Where to go next for a hit. File-manager files are read by path, not by download_file. */ +function nextStep(hit: SearchResult): string { + if (hit.kind === 'file' && hit.meta?.source === 'file-manager') { + const fsPath = typeof hit.meta.fsPath === 'string' ? hit.meta.fsPath : undefined; + return fsPath + ? ` → \`fs_read\` with path \`${fsPath}\`` + : ` → \`fs_read\` with fileId \`${hit.nodeId}\` and name \`${hit.title}\``; + } + // A submission has no id of its own that any tool takes: get_task is + // reached through the *task*, so point at that rather than at the + // submission id, which would simply 404. + return ` → \`${TOOL_FOR[hit.kind] ?? 'api_get'}\` with id \`${targetIdFor(hit)}\``; +} + +/** "Kurs-Dateien, " or the area's own name, from the file's fs path. */ +function fileManagerPlace(hit: SearchResult): string { + const area = typeof hit.meta?.fsPath === 'string' ? hit.meta.fsPath.split('/')[1] : undefined; + const known = FILE_AREAS.find((entry) => entry.area === area); + if (!known) return 'the file manager'; + return known.area === 'courses' && hit.courseTitle ? `${known.label}, ${hit.courseTitle}` : known.label; +} + function formatIndexed(hit: SearchResult): string { + const where = + hit.kind === 'file' && hit.meta?.source === 'file-manager' + ? `file in ${fileManagerPlace(hit)}` + : `${hit.kind} in ${hit.courseTitle || hit.path}`; return [ - `- **${hit.title}** — ${hit.kind} in ${hit.courseTitle || hit.path}`, + `- **${hit.title}** — ${where}`, hit.snippet && hit.snippet !== hit.title ? ` ${hit.snippet}` : undefined, - // A submission has no id of its own that any tool takes: get_task is - // reached through the *task*, so point at that rather than at the - // submission id, which would simply 404. - ` → \`${TOOL_FOR[hit.kind] ?? 'api_get'}\` with id \`${targetIdFor(hit)}\``, + nextStep(hit), ] .filter(Boolean) .join('\n'); diff --git a/src/services.ts b/src/services.ts index 2d6a3f0..338abe8 100644 --- a/src/services.ts +++ b/src/services.ts @@ -1,5 +1,6 @@ import type { Config } from './config.ts'; import { SchulcloudClient } from './core/client.ts'; +import { FileManager } from './core/legacy-files.ts'; import { Indexer } from './indexer/indexer.ts'; import { Store } from './store/store.ts'; @@ -14,12 +15,19 @@ import { Store } from './store/store.ts'; export interface Services { config: Config; client: SchulcloudClient; + /** + * The "Dateien" file manager. Process-wide so its short listing cache is + * shared: an `ls` in one MCP session and a `schulcloud fs get` from the CLI + * then cost one page fetch between them, not two. + */ + files: FileManager; store: Store | undefined; indexer: Indexer | undefined; } export async function createServices(config: Config): Promise { const client = new SchulcloudClient(config); + const files = new FileManager(client); const store = await Store.open(config.databaseUrl); const indexer = store ? new Indexer(client, store, config) : undefined; @@ -29,7 +37,7 @@ export async function createServices(config: Config): Promise { '/files, /manifest and refresh_index are unavailable. Set DATABASE_URL to enable them.', ); } - return { config, client, store, indexer }; + return { config, client, files, store, indexer }; } export async function closeServices(services: Services): Promise { diff --git a/src/store/store.ts b/src/store/store.ts index 74600a5..cc27f17 100644 --- a/src/store/store.ts +++ b/src/store/store.ts @@ -396,6 +396,31 @@ export class Store { ); } + /** + * Where a file's bytes live, from the latest crawl. + * + * The file manager and files-storage use the same id shape but not the same + * ids, so a live fetch has to know which one to ask. + */ + async fileSource( + fileId: string, + ): Promise<{ source: 'files-storage' | 'file-manager'; name: string; mimeType: string; size: number } | undefined> { + const crawlId = await this.latestCrawlId(); + if (crawlId === undefined) return undefined; + const { rows } = await this.db.query<{ title: string; meta: Record }>( + `SELECT title, meta FROM nodes WHERE crawl_id = $1 AND kind = 'file' AND node_id = $2`, + [crawlId, fileId], + ); + const row = rows[0]; + if (!row) return undefined; + return { + source: row.meta.source === 'file-manager' ? 'file-manager' : 'files-storage', + name: row.title, + mimeType: typeof row.meta.mimeType === 'string' ? row.meta.mimeType : 'application/octet-stream', + size: typeof row.meta.size === 'number' ? row.meta.size : 0, + }; + } + async mirrorEntry(fileId: string): Promise<{ path: string; size: number; name: string; mimeType: string } | undefined> { const { rows } = await this.db.query<{ mirror_path: string | null; mirror_size: string | null; name: string; mime_type: string | null }>( `SELECT mirror_path, mirror_size, name, mime_type FROM file_texts WHERE file_id = $1`, @@ -612,6 +637,8 @@ function fileNode(file: CrawledFile): StoredNode { parentId: file.parentId, securityCheckStatus: file.record.securityCheckStatus, at: file.at, + source: file.source ?? 'files-storage', + ...(file.fsPath ? { fsPath: file.fsPath } : {}), }, // File records are less immutable than they look: `PATCH /file/rename/{id}` // changes the name in place, keeping the id and the size, and teachers do diff --git a/test/legacy-files.test.ts b/test/legacy-files.test.ts new file mode 100644 index 0000000..e527f7a --- /dev/null +++ b/test/legacy-files.test.ts @@ -0,0 +1,291 @@ +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { checkSignedUrl, type SchulcloudClient } from '../src/core/client.ts'; +import { + childRef, + FileManager, + FileManagerMarkupError, + FsError, + pageFor, + parseFileListing, + splitPath, +} from '../src/core/legacy-files.ts'; + +/** + * Fixtures follow the legacy client's templates as served (files/files.hbs and + * files/files-grid.hbs, client 33.40): a folder is an `openfolder` button whose + * name sits unescaped after an icon; a file is a `card file` div carrying its + * name, size and id as data attributes, with the viewer type further inside. + */ +const folder = (id: string, name: string) => ` +
+ +
`; + +const file = (id: string, name: string, size: number, type = 'application/pdf', blocked = false) => ` +
+
+
+
+ ${name} +
+
+
+
`; + +const page = (folders: string[], files: string[]) => ` +
+
+${folders.length ? `
${folders.join('')}
` : ''} +
${files.join('')}
+
`; + +const id = (n: number) => n.toString(16).padStart(24, '0'); + +describe('parseFileListing', () => { + it('reads folders and files with their ids, sizes and types', () => { + const listing = parseFileListing( + page([folder(id(1), 'Handlungssituation')], [file(id(2), '13_AB_Wareneingang.pdf', 171941)]), + ); + assert.deepEqual(listing.directories, [{ id: id(1), name: 'Handlungssituation' }]); + assert.deepEqual(listing.files, [ + { id: id(2), name: '13_AB_Wareneingang.pdf', size: 171941, mimeType: 'application/pdf', blocked: false }, + ]); + }); + + it('decodes entities in both folder and file names', () => { + const listing = parseFileListing( + page([folder(id(1), 'Lager & Logistik')], [file(id(2), 'A&B "final".pdf', 10)]), + ); + assert.equal(listing.directories[0]?.name, 'Lager & Logistik'); + assert.equal(listing.files[0]?.name, 'A&B "final".pdf'); + }); + + it('marks a file the virus scanner blocked, which also has no viewer type', () => { + const listing = parseFileListing(page([], [file(id(3), 'bad.exe', 5, undefined, true), file(id(4), 'ok.png', 7, 'image/png')])); + assert.equal(listing.files[0]?.blocked, true); + assert.equal(listing.files[0]?.mimeType, undefined); + // The next card's type must not leak into the blocked one. + assert.equal(listing.files[1]?.mimeType, 'image/png'); + }); + + it('returns an empty listing for an empty folder', () => { + assert.deepEqual(parseFileListing(page([], [])), { directories: [], files: [] }); + }); + + it('refuses a page that is not a file-manager listing rather than reporting 0 files', () => { + assert.throws(() => parseFileListing('Anmelden'), FileManagerMarkupError); + }); +}); + +describe('pageFor and childRef', () => { + it('builds each listing route', () => { + assert.equal(pageFor({}), undefined); + assert.equal(pageFor({ area: 'my' }), '/files/my/'); + assert.equal(pageFor({ area: 'my', folderId: id(1) }), `/files/my/${id(1)}`); + assert.equal(pageFor({ area: 'courses' }), '/files/courses/'); + assert.equal(pageFor({ area: 'courses', ownerId: id(2) }), `/files/courses/${id(2)}`); + assert.equal(pageFor({ area: 'teams', ownerId: id(2), folderId: id(3) }), `/files/teams/${id(2)}/${id(3)}`); + assert.equal(pageFor({ area: 'shared' }), '/files/shared/'); + }); + + it('addresses a nested folder by its own id, whatever its depth', () => { + const course = childRef({ area: 'courses' }, { id: id(2), name: 'Kurs' }); + const top = childRef(course, { id: id(3), name: 'A' }); + const deep = childRef(top, { id: id(4), name: 'B' }); + assert.deepEqual(deep, { area: 'courses', ownerId: id(2), folderId: id(4) }); + }); + + it('refuses to open a shared folder, which the file manager cannot do either', () => { + assert.throws( + () => childRef({ area: 'shared' }, { id: id(5), name: 'Ordner' }), + (error: unknown) => error instanceof FsError && error.code === 'not_navigable', + ); + }); +}); + +describe('splitPath', () => { + it('drops empty segments and surrounding spaces', () => { + assert.deepEqual(splitPath('//courses/ Kurs A /x.pdf/'), ['courses', 'Kurs A', 'x.pdf']); + assert.deepEqual(splitPath('/'), []); + }); +}); + +/** A client that serves canned listing pages and records what was asked for. */ +function fakeClient(pages: Record) { + const requested: string[] = []; + const client = { + getFileManagerPage: async (path: string) => { + requested.push(path); + const html = pages[path]; + if (html === undefined) throw new Error(`no fixture for ${path}`); + return html; + }, + } as unknown as SchulcloudClient; + return { client, requested }; +} + +describe('FileManager.resolve', () => { + const course = id(10); + const slashCourse = id(11); + const sub = id(12); + const pdf = id(13); + const pages = { + '/files/courses/': page([folder(course, 'FIA24B - LF2 (Rh)'), folder(slashCourse, 'LF07 - FIA24A/B - Sb/Ha')], []), + [`/files/courses/${course}`]: page([folder(sub, 'Handlungssituation')], [file(pdf, 'Mahnwesen.pdf', 100)]), + [`/files/courses/${course}/${sub}`]: page([], [file(id(14), 'Lager.pdf', 50)]), + [`/files/courses/${slashCourse}`]: page([], [file(id(15), 'MQTT.docx', 70, 'application/vnd.openxmlformats-officedocument.wordprocessingml.document')]), + '/files/my/': page([], []), + }; + + it('resolves the root and each area without a request for the root', async () => { + const { client, requested } = fakeClient(pages); + const fs = new FileManager(client); + assert.deepEqual(await fs.resolve('/'), { kind: 'directory', path: '/', ref: {}, name: '/' }); + const area = await fs.resolve('/Kurs-Dateien'); + assert.equal(area.kind, 'directory'); + assert.deepEqual(area.kind === 'directory' && area.ref, { area: 'courses' }); + assert.deepEqual(requested, []); + }); + + it('walks names down to a file in a nested folder', async () => { + const { client } = fakeClient(pages); + const node = await new FileManager(client).resolve('/courses/FIA24B - LF2 (Rh)/Handlungssituation/Lager.pdf'); + assert.equal(node.kind, 'file'); + assert.equal(node.kind === 'file' && node.file.id, id(14)); + assert.equal(node.path, '/courses/FIA24B - LF2 (Rh)/Handlungssituation/Lager.pdf'); + }); + + it('resolves a course whose name contains slashes, typed plainly', async () => { + const { client } = fakeClient(pages); + const node = await new FileManager(client).resolve('/courses/LF07 - FIA24A/B - Sb/Ha/MQTT.docx'); + assert.equal(node.kind === 'file' && node.file.id, id(15)); + }); + + it('accepts ids as path segments', async () => { + const { client } = fakeClient(pages); + const node = await new FileManager(client).resolve(`/courses/${course}/${pdf}`); + assert.equal(node.kind === 'file' && node.file.name, 'Mahnwesen.pdf'); + }); + + it('falls back to a case-insensitive match', async () => { + const { client } = fakeClient(pages); + const node = await new FileManager(client).resolve('/courses/fia24b - lf2 (rh)/handlungssituation'); + assert.equal(node.kind, 'directory'); + }); + + it('says what is there when a name is not found', async () => { + const { client } = fakeClient(pages); + await assert.rejects( + new FileManager(client).resolve('/courses/FIA24B - LF2 (Rh)/Handlung'), + (error: unknown) => error instanceof FsError && error.code === 'not_found' && /Handlungssituation/.test(error.message), + ); + }); + + it('refuses to treat a file as a folder', async () => { + const { client } = fakeClient(pages); + await assert.rejects( + new FileManager(client).resolve('/courses/FIA24B - LF2 (Rh)/Mahnwesen.pdf/more'), + (error: unknown) => error instanceof FsError && error.code === 'not_a_directory', + ); + }); + + it('reports an ambiguous name with the ids to use instead', async () => { + const twins = { + '/files/my/': page([folder(id(20), 'Kopie'), folder(id(21), 'Kopie')], []), + }; + const { client } = fakeClient(twins); + await assert.rejects( + new FileManager(client).resolve('/my/Kopie'), + (error: unknown) => + error instanceof FsError && error.code === 'ambiguous' && error.message.includes(id(20)) && error.message.includes(id(21)), + ); + }); + + it('rejects an unknown area and names the real ones', async () => { + const { client } = fakeClient(pages); + await assert.rejects( + new FileManager(client).resolve('/Downloads/x'), + (error: unknown) => error instanceof FsError && /\/courses/.test(error.message), + ); + }); + + it('caches a listing, so ls followed by read costs no second request', async () => { + const { client, requested } = fakeClient(pages); + const fs = new FileManager(client); + await fs.resolve('/courses/FIA24B - LF2 (Rh)/Mahnwesen.pdf'); + await fs.resolve('/courses/FIA24B - LF2 (Rh)/Mahnwesen.pdf'); + assert.equal(requested.filter((path) => path === '/files/courses/').length, 1); + }); +}); + +describe('FileManager.walk', () => { + it('stays within its folder budget and says so', async () => { + const course = id(30); + const pages = { + '/files/courses/': page([folder(course, 'Kurs')], []), + [`/files/courses/${course}`]: page([folder(id(31), 'A'), folder(id(32), 'B')], []), + [`/files/courses/${course}/${id(31)}`]: page([], [file(id(33), 'a.pdf', 1)]), + [`/files/courses/${course}/${id(32)}`]: page([], [file(id(34), 'b.pdf', 1)]), + }; + const { client } = fakeClient(pages); + const result = await new FileManager(client).walk( + { path: '/courses', ref: { area: 'courses' } }, + { maxDepth: 5, maxDirectories: 2 }, + ); + assert.equal(result.visited, 2); + assert.equal(result.truncated, true); + }); + + it('records a folder it could not read instead of dropping it', async () => { + const course = id(40); + const pages = { '/files/courses/': page([folder(course, 'Kurs')], []) }; + const { client } = fakeClient(pages); + const result = await new FileManager(client).walk( + { path: '/courses', ref: { area: 'courses' } }, + { maxDepth: 3, maxDirectories: 10 }, + ); + assert.equal(result.failures.length, 1); + assert.equal(result.failures[0]?.path, '/courses/Kurs'); + }); +}); + +describe('the file-manager allowlist', () => { + // Some legacy GET routes write. The client must refuse them before any request. + const client = { + config: { baseUrl: 'https://example.org', jwt: 'x', requestTimeoutMs: 1000, maxDownloadBytes: 10 }, + }; + + it('refuses the routes that write despite being GETs', async () => { + const { SchulcloudClient } = await import('../src/core/client.ts'); + const real = new SchulcloudClient(client.config as never); + for (const path of ['/files/share/?file=' + id(1), `/files/file?file=${id(1)}&share=abc`, `/files/fileModel/${id(1)}/proxy`, '/files/search/?q=x', '/files/permittedDirectories/']) { + await assert.rejects(real.getFileManagerPage(path), /refusing file-manager path/, path); + } + }); +}); + +describe('checkSignedUrl', () => { + it('accepts an https storage url from an https instance', () => { + assert.equal(checkSignedUrl('https://s3.example.com/bucket/key?X-Amz-Signature=1', 'https://schulcloud.example').host, 's3.example.com'); + }); + + it('refuses plaintext from an https instance, and credentials in the url', () => { + assert.throws(() => checkSignedUrl('http://10.0.0.1/x', 'https://schulcloud.example'), /refusing a http:/); + assert.throws(() => checkSignedUrl('https://user:pw@s3.example.com/x', 'https://schulcloud.example'), /credentials/); + assert.throws(() => checkSignedUrl('file:///etc/passwd', 'http://localhost:4400'), /refusing a file:/); + }); + + it('allows http when the instance itself is http, as the local one is', () => { + assert.equal(checkSignedUrl('http://localhost:9900/x', 'http://localhost:4400').protocol, 'http:'); + }); +});