Browse the file manager ("Dateien") as a filesystem
Many teachers never use topics or boards; their material sits in the course's file area, and the tools answered "0 files" for courses holding dozens of worksheets — 21 of 26 courses on the live account. Persönliche, Kurs-, Team- and Geteilte Dateien live in the legacy file store, not in files-storage, and its service is not in the public ingress. The only way in is the legacy client: HTML listings, and GET /files/signedurl for a pre-signed download. core/legacy-files.ts turns that into one path tree — /my, /courses/<course>, /teams/<team>, /shared — resolving names that contain "/", ids anywhere in a path, and wrong or ambiguous names with a message saying what is there. A listing that does not parse throws; it never reads as an empty folder. Some of the legacy client's GET routes write (GET /files/share/ mints a share token), so getFileManagerPage allows only the listing routes, by pattern. Signed URLs are fetched with no credentials and must be https. - MCP: fs_list, fs_tree, fs_find and fs_read; get_course lists course files. - CLI: schulcloud fs ls, tree, find and get, recursive and resumable. - API: /api/fs/list, tree, find and file. - Index: the crawl walks the file manager (INDEX_FILE_MANAGER, on by default), so search covers the text inside those files and sync mirrors them under <course>/Kurs-Dateien. The local instance gains a fixture for all four areas. It needed a loopback, so signed URLs open from the host, and a pre-created bucket, since MinIO does not implement PutBucketCors. 135 tests. Smoke 55/55 live; 57/57 and 55/55 on the local instance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -52,6 +52,12 @@ DATABASE_URL=postgresql://schulcloud:schulcloud@postgres:5432/schulcloud
|
|||||||
# three extra requests per task on a full crawl, so it is off by default.
|
# three extra requests per task on a full crawl, so it is off by default.
|
||||||
# INDEX_PERSONAL_FILES=false
|
# INDEX_PERSONAL_FILES=false
|
||||||
|
|
||||||
|
# Also walk the file manager ("Dateien") when crawling: Kurs-Dateien for every
|
||||||
|
# course, plus Persönliche, Team- and Geteilte Dateien on a full crawl. Many
|
||||||
|
# teachers keep their material only there, so this is on by default. One page
|
||||||
|
# load per folder — about 160 on a 26-course account.
|
||||||
|
# INDEX_FILE_MANAGER=true
|
||||||
|
|
||||||
# How often to re-crawl on a timer, in ms. Default 21600000 (6h). 0 = on demand
|
# How often to re-crawl on a timer, in ms. Default 21600000 (6h). 0 = on demand
|
||||||
# only. A re-crawl of unchanged content downloads nothing, because Schulcloud
|
# only. A re-crawl of unchanged content downloads nothing, because Schulcloud
|
||||||
# file records are immutable.
|
# file records are immutable.
|
||||||
|
|||||||
28
CLAUDE.md
28
CLAUDE.md
@@ -39,8 +39,8 @@ index.
|
|||||||
read-only with respect to Schulcloud. Run `smoke` after touching `src/core/`,
|
read-only with respect to Schulcloud. Run `smoke` after touching `src/core/`,
|
||||||
`src/mcp/` or `src/http/` — the unit tests cover only pure functions.
|
`src/mcp/` or `src/http/` — the unit tests cover only pure functions.
|
||||||
|
|
||||||
Run smoke **both ways**: with `DATABASE_URL` set (34 checks, index-backed) and
|
Run smoke **both ways**: with `DATABASE_URL` set (57 checks, index-backed) and
|
||||||
without (32 checks, live-only). The degradation path is a supported mode, not a
|
without (55 checks, live-only). The degradation path is a supported mode, not a
|
||||||
fallback nobody exercises.
|
fallback nobody exercises.
|
||||||
|
|
||||||
Store tests need a database and skip without one:
|
Store tests need a database and skip without one:
|
||||||
@@ -71,6 +71,9 @@ bin/cli.ts ──HTTP──────────┘ cli/{config,client,sync
|
|||||||
- **`crawl.ts`** — the one traversal. Search, the indexer, the what-changed
|
- **`crawl.ts`** — the one traversal. Search, the indexer, the what-changed
|
||||||
diff and the file mirror all need it; keep it here, not in a tool.
|
diff and the file mirror all need it; keep it here, not in a tool.
|
||||||
- `paths.ts` — the security boundary for mirrored filenames. See Invariants.
|
- `paths.ts` — the security boundary for mirrored filenames. See Invariants.
|
||||||
|
- `legacy-files.ts` — the file manager ("Dateien": Persönliche, Kurs-, Team-,
|
||||||
|
Geteilte Dateien) as one path tree, parsed from the legacy client's pages.
|
||||||
|
A separate store from files-storage; the `fs_*` tools and `/api/fs` sit on it.
|
||||||
- **`store/`** — crawl generations, identity diffs, `german` + `pg_trgm` FTS.
|
- **`store/`** — crawl generations, identity diffs, `german` + `pg_trgm` FTS.
|
||||||
`Store.open` returns `undefined` when Postgres is down; callers degrade.
|
`Store.open` returns `undefined` when Postgres is down; callers degrade.
|
||||||
- **`indexer/`** — crawl → persist → mirror bytes → extract text → index.
|
- **`indexer/`** — crawl → persist → mirror bytes → extract text → index.
|
||||||
@@ -92,7 +95,12 @@ bin/cli.ts ──HTTP──────────┘ cli/{config,client,sync
|
|||||||
`GET` except `extendSession` (the keepalive's `refresh-session` call, which
|
`GET` except `extendSession` (the keepalive's `refresh-session` call, which
|
||||||
touches only our own session and is not exposed as a tool, so no model-driven
|
touches only our own session and is not exposed as a tool, so no model-driven
|
||||||
call can be a POST). `api_get` rejects non-`/api/` paths and anything carrying a
|
call can be a POST). `api_get` rejects non-`/api/` paths and anything carrying a
|
||||||
scheme or host. `refresh_index` and `POST /api/refresh` write only to the Pi's
|
scheme or host. **Against the legacy client, GET-only is not enough:**
|
||||||
|
`GET /files/share/` mints a share token and `GET /files/file?share=` grants a
|
||||||
|
permission, so `getFileManagerPage` allows only the listing routes, by pattern —
|
||||||
|
widen that pattern only with a route you have read the handler of. A pre-signed
|
||||||
|
download URL is fetched with **no** credentials: it names another host, and
|
||||||
|
neither the bearer nor the `jwt` cookie may go with it. `refresh_index` and `POST /api/refresh` write only to the Pi's
|
||||||
own index and mirror — every upstream call they make is still a GET.
|
own index and mirror — every upstream call they make is still a GET.
|
||||||
|
|
||||||
**Filenames from Schulcloud are untrusted paths.** Course titles, card titles
|
**Filenames from Schulcloud are untrusted paths.** Course titles, card titles
|
||||||
@@ -217,10 +225,22 @@ These cost real time to discover; `docs/API.md` has the full list with evidence.
|
|||||||
upstream: interactive exercises are H5P elements, whose `contentId` is the
|
upstream: interactive exercises are H5P elements, whose `contentId` is the
|
||||||
only handle onto the content, or external (LTI) tools behind
|
only handle onto the content, or external (LTI) tools behind
|
||||||
`contextExternalToolId`. Say that rather than looking for a quiz API.
|
`contextExternalToolId`. Say that rather than looking for a quiz API.
|
||||||
|
- **The file manager is a third store, reachable only as HTML.** Persönliche,
|
||||||
|
Kurs-, Team- and Geteilte Dateien live in the legacy `files` collection, not in
|
||||||
|
files-storage: `list_files` answers 0 for a course holding dozens of
|
||||||
|
worksheets, and 21 of 26 live courses keep material there. Its Feathers service
|
||||||
|
is not in the ingress, so listings are parsed from `/files/{my,courses,teams,shared}`
|
||||||
|
pages and downloads go through `GET /files/signedurl` (JSON). Folders are
|
||||||
|
addressed by id alone — `/files/courses/{course}/{folder}` at any depth.
|
||||||
|
`permittedDirectories` returns every course with **no** folders (it queries
|
||||||
|
`refOwnerModel: 'courses'`, records say `'course'`), and `/files/search/` 504s,
|
||||||
|
so walk listings. Course names contain `/`; resolve by joining segments. A
|
||||||
|
listing page that does not parse must throw, never read as an empty folder —
|
||||||
|
"0 files" is the bug this exists to fix. `docs/API.md` has the evidence.
|
||||||
- **Teams cannot be read at any version.** v3 exposes only
|
- **Teams cannot be read at any version.** v3 exposes only
|
||||||
`GET /team/{teamId}/news`; upstream `main`'s teams controller is write-only
|
`GET /team/{teamId}/news`; upstream `main`'s teams controller is write-only
|
||||||
(`POST :teamId/create-room`). `/teams` is the legacy client's HTML page, not
|
(`POST :teamId/create-room`). `/teams` is the legacy client's HTML page, not
|
||||||
an API. This one is genuinely unavailable, not merely uncovered.
|
an API. A team's *files* are reachable, through the file manager (`/teams/<team>`).
|
||||||
- **`exp` (30 days) is not the session lifetime.** The binding limit is a Valkey
|
- **`exp` (30 days) is not the session lifetime.** The binding limit is a Valkey
|
||||||
whitelist entry with a `JWT_TIMEOUT_SECONDS` TTL (7200s; live value at
|
whitelist entry with a `JWT_TIMEOUT_SECONDS` TTL (7200s; live value at
|
||||||
`GET /api/v3/config/public`) that every authenticated request re-sets.
|
`GET /api/v3/config/public`) that every authenticated request re-sets.
|
||||||
|
|||||||
14
README.md
14
README.md
@@ -17,7 +17,7 @@ instance, not inferred from the upstream source.
|
|||||||
> *"Find the material about Verschlüsselung and explain the Caesar cipher worksheet."*
|
> *"Find the material about Verschlüsselung and explain the Caesar cipher worksheet."*
|
||||||
> *"Summarise the routing lesson from the LF10 course."*
|
> *"Summarise the routing lesson from the LF10 course."*
|
||||||
|
|
||||||
Twenty tools, all read-only:
|
Twenty-four tools, all read-only:
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -30,8 +30,12 @@ Twenty tools, all read-only:
|
|||||||
| `list_tasks` | homework across all courses, by due date |
|
| `list_tasks` | homework across all courses, by due date |
|
||||||
| `get_task` | one task: description, due date, attachments, **and your submission** |
|
| `get_task` | one task: description, due date, attachments, **and your submission** |
|
||||||
| `list_submissions` | what you handed in, and what is still ungraded |
|
| `list_submissions` | what you handed in, and what is still ungraded |
|
||||||
| `list_files` | files attached to any entity |
|
| `list_files` | files attached to a board element, topic, task or submission |
|
||||||
| `download_file` | fetch a file and extract its text, or view an image |
|
| `download_file` | fetch such an attachment and extract its text, or view an image |
|
||||||
|
| `fs_list` | list a folder of the file manager — Persönliche, Kurs-, Team-, Geteilte Dateien |
|
||||||
|
| `fs_tree` | everything below a file-manager folder, as a tree |
|
||||||
|
| `fs_find` | find file-manager files and folders by name |
|
||||||
|
| `fs_read` | read a file-manager file, extracted like `download_file` |
|
||||||
| `search` | keyword search across everything — **including the text inside PDFs and Office files** |
|
| `search` | keyword search across everything — **including the text inside PDFs and Office files** |
|
||||||
| `refresh_index` | re-read Schulcloud now, per course or in full |
|
| `refresh_index` | re-read Schulcloud now, per course or in full |
|
||||||
| `what_changed` | what appeared, changed or vanished since a date |
|
| `what_changed` | what appeared, changed or vanished since a date |
|
||||||
@@ -54,6 +58,8 @@ schulcloud login --server https://mcp.example.org --token <token>
|
|||||||
schulcloud sync --dry-run # see what would be mirrored
|
schulcloud sync --dry-run # see what would be mirrored
|
||||||
schulcloud sync # mirror coursework to ~/Schulcloud
|
schulcloud sync # mirror coursework to ~/Schulcloud
|
||||||
schulcloud refresh --course <id>
|
schulcloud refresh --course <id>
|
||||||
|
schulcloud fs tree /courses # browse the file manager ("Dateien")
|
||||||
|
schulcloud fs get "/courses/<course>/<folder>"
|
||||||
```
|
```
|
||||||
|
|
||||||
It talks only to the Pi and holds no Schulcloud credential — see
|
It talks only to the Pi and holds no Schulcloud credential — see
|
||||||
@@ -151,7 +157,7 @@ npm run typecheck
|
|||||||
```
|
```
|
||||||
|
|
||||||
`npm run smoke` starts the HTTP server, connects a real MCP client over
|
`npm run smoke` starts the HTTP server, connects a real MCP client over
|
||||||
Streamable HTTP and exercises every tool against the live account — 30 checks
|
Streamable HTTP and exercises every tool against the live account — 55 checks (57 with the index)
|
||||||
covering the auth gate, the protocol handshake, every content chain, file
|
covering the auth gate, the protocol handshake, every content chain, file
|
||||||
extraction, `api_get`'s guard rails and error handling.
|
extraction, `api_get`'s guard rails and error handling.
|
||||||
|
|
||||||
|
|||||||
63
docs/API.md
63
docs/API.md
@@ -225,6 +225,69 @@ hops and without Etherpad's API key; `core/etherpad.ts` does this. The url is
|
|||||||
built from the server's `ETHERPAD__PAD_URI`, so it must be checked against the
|
built from the server's `ETHERPAD__PAD_URI`, so it must be checked against the
|
||||||
instance host before the session cookie is sent to it.
|
instance host before the session cookie is sent to it.
|
||||||
|
|
||||||
|
## The file manager ("Dateien") is a third store
|
||||||
|
|
||||||
|
Persönliche Dateien, Kurs-Dateien, Team-Dateien and Geteilte Dateien are the
|
||||||
|
**legacy file system**: a `files` collection with real folders (`isDirectory`,
|
||||||
|
`parent`, `owner`, `refOwnerModel`), served by the legacy Feathers
|
||||||
|
`fileStorage` service. It shares nothing with files-storage. Asking
|
||||||
|
`/api/v3/file/list/school/{school}/courses/{courseId}` answers **0** for a
|
||||||
|
course whose file manager holds dozens of worksheets — measured: 21 of 26
|
||||||
|
courses on the live account keep files here, some nothing else.
|
||||||
|
|
||||||
|
**Its service is not in the public ingress**, so the only way in is the legacy
|
||||||
|
client. Verified live:
|
||||||
|
|
||||||
|
| Route (legacy client, `jwt` cookie) | Returns | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `GET /files/my/` , `/files/my/{folder}` | HTML listing | personal root / one folder |
|
||||||
|
| `GET /files/courses/` | HTML listing | the courses, *as folders* (ids = course ids) |
|
||||||
|
| `GET /files/courses/{course}` , `/files/courses/{course}/{folder}` | HTML listing | **one** folder segment at any depth |
|
||||||
|
| `GET /files/teams/…` | HTML listing | same shape as courses |
|
||||||
|
| `GET /files/shared/` | HTML listing | flat; shared *folders* cannot be opened (no route; the UI's link 404s) |
|
||||||
|
| `GET /files/signedurl?file={id}&name={name}` | `{"url": …}` | pre-signed S3 url, **another host** (live: `s3.hidrive.strato.com`) |
|
||||||
|
| `GET /files/permittedDirectories/` | JSON tree | **lists every course with no folders in any** — see below |
|
||||||
|
| `GET /files/search/?q=` | HTML | **504** on live: an unindexed regex over every file record |
|
||||||
|
|
||||||
|
Listings are parsed on the attributes the page's own scripts use:
|
||||||
|
`data-folder-id` with the name inside `.card-title-directory` (emitted
|
||||||
|
**unescaped**, `{{{stripOnlyScript name}}}`), and `data-file-id`,
|
||||||
|
`data-file-name`, `data-file-size`, `data-file-viewer-type` on each
|
||||||
|
`.card.file`. A blocked file carries `btn-file-danger` and no viewer type. No
|
||||||
|
dates are rendered.
|
||||||
|
|
||||||
|
**`permittedDirectories` is broken for courses.** The directory service's
|
||||||
|
query matches course folders on `refOwnerModel: 'courses'`; the records say
|
||||||
|
`'course'`. Live result: 26 courses, 0 folders; personal folders come through.
|
||||||
|
Listings are the only complete view — which is also what the UI shows.
|
||||||
|
|
||||||
|
**Some GET routes write.** `GET /files/share/?file=` mints a share token when
|
||||||
|
the file has none (`PATCH /fileStorage/shared/{id}`), and
|
||||||
|
`GET /files/file?…&share=…` grants the caller a permission on the file.
|
||||||
|
`GET /files/fileModel/{id}/proxy` forwards to the latter. A GET-only client is
|
||||||
|
therefore *not* read-only against this surface by itself: `core/client.ts`
|
||||||
|
allows only the listing routes and `/files/signedurl`, by pattern.
|
||||||
|
|
||||||
|
**The signed-url service returns its error instead of throwing it**
|
||||||
|
(`.catch((err) => new Forbidden(err))`), so a refused file is a 200 whose body
|
||||||
|
has no `url`.
|
||||||
|
|
||||||
|
**Course names contain `/`** in real data ("LF07 - FIA24A/B - Sb/Ha",
|
||||||
|
"FIA24/FIP24 IT LF12"), and a real file is literally called `..docx`. Paths
|
||||||
|
built from names cannot be split naively; `core/legacy-files.ts` resolves by
|
||||||
|
trying joined segments, and accepts ids as segments.
|
||||||
|
|
||||||
|
Two more, seen while building the local fixture:
|
||||||
|
|
||||||
|
- **`getRefOwnerModel(owner)` answers "a course, or else `teams`".** Any owner
|
||||||
|
id that is not a course — a *user* included — is recorded as a team's. The
|
||||||
|
upload page never sends an owner for personal files (`data-owner=""`), so the
|
||||||
|
server defaults to the creator and records `user`; send the user id and every
|
||||||
|
later permission check dereferences a team that does not exist.
|
||||||
|
- **The file permission service writes `refOwnerModel`** where the "shared with
|
||||||
|
me" query reads `refPermModel`, so a share made through it never appears
|
||||||
|
under Geteilte Dateien. The share-link flow patches `/files/{id}` directly.
|
||||||
|
|
||||||
## Re-verifying after an upstream release
|
## Re-verifying after an upstream release
|
||||||
|
|
||||||
`npm run probe` re-checks every assumption above against the live instance and
|
`npm run probe` re-checks every assumption above against the live instance and
|
||||||
|
|||||||
33
docs/CLI.md
33
docs/CLI.md
@@ -50,6 +50,39 @@ alongside courses, with their files under the room's name rather than a course's
|
|||||||
what changed: that is a handful of requests, where a full re-crawl reads every
|
what changed: that is a handful of requests, where a full re-crawl reads every
|
||||||
course. The server refuses a repeat within a minute unless you pass `--force`.
|
course. The server refuses a repeat within a minute unless you pass `--force`.
|
||||||
|
|
||||||
|
### The file manager (`fs`)
|
||||||
|
|
||||||
|
The Schulcloud file manager ("Dateien") — Persönliche, Kurs-, Team- and
|
||||||
|
Geteilte Dateien — browsed like a filesystem, live:
|
||||||
|
|
||||||
|
```
|
||||||
|
schulcloud fs ls [path] [--long]
|
||||||
|
schulcloud fs tree [path] [--depth <n>] [--max-folders <n>]
|
||||||
|
schulcloud fs find <name> [--path <path>] [--type file|folder] [--long]
|
||||||
|
schulcloud fs get <path> [--out <path>] [--force] [--jobs <n>]
|
||||||
|
```
|
||||||
|
|
||||||
|
```console
|
||||||
|
$ schulcloud fs ls /courses
|
||||||
|
$ schulcloud fs tree "/courses/FIA24B - SK (Rh)"
|
||||||
|
$ schulcloud fs find "*Erben*" --path /courses
|
||||||
|
$ schulcloud fs get "/courses/FIA24B - SK (Rh)/02_Erbrecht" --out ~/Erbrecht
|
||||||
|
```
|
||||||
|
|
||||||
|
The tree is `/my`, `/courses/<course>`, `/teams/<team>` and `/shared`; the
|
||||||
|
German names ("/Kurs-Dateien") work too. Names may contain `/` — course names
|
||||||
|
often do — and still resolve; any segment may also be an id from `--long`.
|
||||||
|
|
||||||
|
`fs find` matches any part of a name, or, given `*` or `?`, the whole name as
|
||||||
|
`find -name` does. `fs get` on a folder downloads everything below it, keeps
|
||||||
|
the structure, and skips files already present at the same size — so re-running
|
||||||
|
it resumes. Each folder is one page load on the server, so large trees take a
|
||||||
|
while.
|
||||||
|
|
||||||
|
`sync` mirrors these files too, under `<course>/Kurs-Dateien/…`,
|
||||||
|
`Persönliche Dateien/…`, `Team-Dateien/<team>/…` and `Geteilte Dateien/`, once
|
||||||
|
the server's index includes them (`INDEX_FILE_MANAGER`, on by default).
|
||||||
|
|
||||||
## How sync works
|
## How sync works
|
||||||
|
|
||||||
It is a **one-way mirror, not a two-way sync**, and that follows from the data
|
It is a **one-way mirror, not a two-way sync**, and that follows from the data
|
||||||
|
|||||||
@@ -103,9 +103,9 @@ node dist/bin/cli.js sync
|
|||||||
## Run the test suites
|
## Run the test suites
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm test # 58 offline tests
|
npm test # 135 offline tests
|
||||||
npm run smoke # end-to-end against the live instance, live-only mode
|
npm run smoke # end-to-end against the live instance, live-only mode
|
||||||
DATABASE_URL=… npm run smoke # end-to-end with the index (34 checks)
|
DATABASE_URL=… npm run smoke # end-to-end with the index (57 checks)
|
||||||
```
|
```
|
||||||
|
|
||||||
Store tests need a database and skip without one:
|
Store tests need a database and skip without one:
|
||||||
|
|||||||
@@ -89,6 +89,7 @@ Every one of these is marked in the env files at the line it affects.
|
|||||||
|
|
||||||
```
|
```
|
||||||
docker-compose.yml the stack; profiles: (default) | tools | av | preview
|
docker-compose.yml the stack; profiles: (default) | tools | av | preview
|
||||||
|
(minio-loopback: see "Simulating a teacher")
|
||||||
env/ one file per service, all values local-only
|
env/ one file per service, all values local-only
|
||||||
proxy/nginx.conf GENERATED — the single origin, see scripts/gen-proxy-conf.py
|
proxy/nginx.conf GENERATED — the single origin, see scripts/gen-proxy-conf.py
|
||||||
scripts/seed.sh loads the demo school via the management app
|
scripts/seed.sh loads the demo school via the management app
|
||||||
@@ -183,14 +184,40 @@ node scripts/simulate-teacher.mjs create # course, room, topic, task, board,
|
|||||||
# Etherpad pad, folder, files
|
# Etherpad pad, folder, files
|
||||||
node scripts/simulate-teacher.mjs update # rename and rewrite all of it
|
node scripts/simulate-teacher.mjs update # rename and rewrite all of it
|
||||||
node scripts/simulate-teacher.mjs delete # remove it again
|
node scripts/simulate-teacher.mjs delete # remove it again
|
||||||
|
node scripts/simulate-teacher.mjs files # only the file-manager part, onto an existing fixture
|
||||||
```
|
```
|
||||||
|
|
||||||
|
It also fills the **file manager** ("Dateien") — the legacy file system behind
|
||||||
|
Persönliche, Kurs-, Team- and Geteilte Dateien, a separate store from the board
|
||||||
|
files above — with a course folder tree ("Arbeitsblätter/Woche 1"), a team
|
||||||
|
folder, a folder in the demo student's own files, and a teacher's file shared
|
||||||
|
read-only with the student. The student is added to the adopted team so the
|
||||||
|
team files are visible from the account under test. Every file carries a unique
|
||||||
|
search term, so search over the index can be checked per area.
|
||||||
|
|
||||||
|
Legacy file uploads only work because of two changes to the stack:
|
||||||
|
|
||||||
|
- **`minio-loopback`.** The legacy service signs upload and download URLs for
|
||||||
|
its storage provider's single endpoint, and uses that endpoint for its own S3
|
||||||
|
calls. `minio:9000` works inside the compose network and nowhere else, so the
|
||||||
|
browser and the MCP server on the host were handed URLs they could not open.
|
||||||
|
`seed.sh` now registers `http://localhost:9900`, and this socat sidecar,
|
||||||
|
sharing the api container's network namespace, forwards that address to
|
||||||
|
MinIO — the same url then works from the api, the browser and the host.
|
||||||
|
After recreating `api`, recreate `minio-loopback` too: it lives in api's
|
||||||
|
network namespace.
|
||||||
|
- **The school bucket is created up front.** The legacy service makes
|
||||||
|
`bucket-<schoolId>` on first upload and then calls `PutBucketCors`, which
|
||||||
|
MinIO does not implement, so the first upload failed with *"A header you
|
||||||
|
provided implies functionality that is not implemented"*. `minio-init.sh`
|
||||||
|
creates the demo school's bucket, and an existing bucket skips both calls.
|
||||||
|
|
||||||
Ids are kept in `.simulate-teacher.json` between phases, so the MCP server can
|
Ids are kept in `.simulate-teacher.json` between phases, so the MCP server can
|
||||||
be pointed at the instance in between:
|
be pointed at the instance in between:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
eval "$(./scripts/mcp-env.sh)" # as the demo student
|
eval "$(./scripts/mcp-env.sh)" # as the demo student
|
||||||
cd .. && npm run smoke # 39 checks against the local instance
|
cd .. && npm run smoke # 57 checks against the local instance
|
||||||
```
|
```
|
||||||
|
|
||||||
`mcp-env.sh` points the index at its own database, `schulcloud_local`, and the
|
`mcp-env.sh` points the index at its own database, `schulcloud_local`, and the
|
||||||
|
|||||||
@@ -154,6 +154,24 @@ services:
|
|||||||
minio: {condition: service_healthy}
|
minio: {condition: service_healthy}
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
|
minio-loopback:
|
||||||
|
# The legacy file service (the file manager: Persönliche/Kurs-/Team-Dateien)
|
||||||
|
# signs upload and download URLs for its storage provider's one endpoint, and
|
||||||
|
# the same endpoint serves its own S3 calls. "minio:9000" works inside the
|
||||||
|
# compose network and nowhere else, so a browser or the MCP server on this
|
||||||
|
# machine was handed URLs it could not open: uploads through the UI failed
|
||||||
|
# and downloads could not be tested. seed.sh registers the provider as
|
||||||
|
# http://localhost:9900 instead, and this forwards that address to MinIO from
|
||||||
|
# inside the api container's own network namespace — so the one URL now
|
||||||
|
# works from the api, from the browser and from the host alike.
|
||||||
|
image: alpine/socat:1.8.0.1
|
||||||
|
network_mode: "service:api"
|
||||||
|
command: ["TCP-LISTEN:9900,fork,reuseaddr,bind=127.0.0.1", "TCP:minio:9000"]
|
||||||
|
depends_on:
|
||||||
|
api: {condition: service_started}
|
||||||
|
minio: {condition: service_healthy}
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
file-preview:
|
file-preview:
|
||||||
# Generates thumbnails via ImageMagick, driven off RabbitMQ. Optional: with
|
# Generates thumbnails via ImageMagick, driven off RabbitMQ. Optional: with
|
||||||
# it absent, files still upload and download, they just have no preview.
|
# it absent, files still upload and download, they just have no preview.
|
||||||
|
|||||||
@@ -11,9 +11,16 @@ for bucket in \
|
|||||||
h5p-content-bucket ` # h5p-editor content` \
|
h5p-content-bucket ` # h5p-editor content` \
|
||||||
h5p-library-bucket ` # h5p content types` \
|
h5p-library-bucket ` # h5p content types` \
|
||||||
ydocs ` # tldraw whiteboard documents` \
|
ydocs ` # tldraw whiteboard documents` \
|
||||||
fwu-content # FWU media, unused but cheap to create
|
fwu-content ` # FWU media, unused but cheap to create` \
|
||||||
|
bucket-5f2987e020834114b8efd6f6 # legacy file manager, demo school (see below)
|
||||||
do
|
do
|
||||||
mc mb --ignore-existing "local/$bucket"
|
mc mb --ignore-existing "local/$bucket"
|
||||||
done
|
done
|
||||||
|
|
||||||
|
# The legacy file manager keeps one bucket per school, "bucket-<schoolId>", and
|
||||||
|
# creates it on first upload — then calls PutBucketCors, which MinIO does not
|
||||||
|
# implement, so the first upload fails with "A header you provided implies
|
||||||
|
# functionality that is not implemented". A bucket that already exists skips
|
||||||
|
# both calls. 5f2987e020834114b8efd6f6 is the demo school's fixed seed id.
|
||||||
|
|
||||||
mc ls local
|
mc ls local
|
||||||
|
|||||||
@@ -39,7 +39,10 @@ SECRET=$(curl -fsS -X POST "$MGMT/encrypt-plain-text" \
|
|||||||
const id = ObjectId('62949a4003839b6162aa566b');
|
const id = ObjectId('62949a4003839b6162aa566b');
|
||||||
db.storageproviders.replaceOne({ _id: id }, {
|
db.storageproviders.replaceOne({ _id: id }, {
|
||||||
_id: id, isShared: true, region: 'eu-central-1', type: 'S3',
|
_id: id, isShared: true, region: 'eu-central-1', type: 'S3',
|
||||||
endpointUrl: 'http://minio:9000',
|
// Not minio:9000: this one endpoint also goes into every signed URL, and
|
||||||
|
// those are opened by the browser and the MCP server on the host. The
|
||||||
|
// minio-loopback service makes localhost:9900 reach MinIO from the api too.
|
||||||
|
endpointUrl: 'http://localhost:9900',
|
||||||
accessKeyId: 'miniouser',
|
accessKeyId: 'miniouser',
|
||||||
secretAccessKey: '$SECRET',
|
secretAccessKey: '$SECRET',
|
||||||
maxBuckets: 150, freeBuckets: 138,
|
maxBuckets: 150, freeBuckets: 138,
|
||||||
|
|||||||
@@ -11,6 +11,7 @@
|
|||||||
* node scripts/simulate-teacher.mjs create # build the fixture, print ids
|
* node scripts/simulate-teacher.mjs create # build the fixture, print ids
|
||||||
* node scripts/simulate-teacher.mjs update # rename/edit everything it made
|
* node scripts/simulate-teacher.mjs update # rename/edit everything it made
|
||||||
* node scripts/simulate-teacher.mjs delete # remove it again
|
* node scripts/simulate-teacher.mjs delete # remove it again
|
||||||
|
* node scripts/simulate-teacher.mjs files # (re)build only the file-manager part
|
||||||
*
|
*
|
||||||
* State lives in .simulate-teacher.json so the phases can be run one at a time
|
* State lives in .simulate-teacher.json so the phases can be run one at a time
|
||||||
* with MCP checks in between.
|
* with MCP checks in between.
|
||||||
@@ -295,8 +296,9 @@ async function create() {
|
|||||||
log(`second board left unpublished on purpose: ${draft.id}`);
|
log(`second board left unpublished on purpose: ${draft.id}`);
|
||||||
|
|
||||||
saveState(s);
|
saveState(s);
|
||||||
|
await fileManager();
|
||||||
console.log(`\nstate written to ${STATE}`);
|
console.log(`\nstate written to ${STATE}`);
|
||||||
summary(s);
|
summary(loadState());
|
||||||
}
|
}
|
||||||
|
|
||||||
/** files-storage attaches bytes to a board node (element) id, not to the card. */
|
/** files-storage attaches bytes to a board node (element) id, not to the card. */
|
||||||
@@ -307,6 +309,146 @@ async function upload(parentId, name, type, body) {
|
|||||||
return record.id;
|
return record.id;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------- file manager ---
|
||||||
|
//
|
||||||
|
// The file manager ("Dateien": Persönliche, Kurs-, Team- and Geteilte Dateien)
|
||||||
|
// is the legacy file system, a different store from files-storage above, with a
|
||||||
|
// real folder tree. Many teachers use nothing else, so the MCP server's fs_*
|
||||||
|
// tools need content there. Its services are only reachable on the server's own
|
||||||
|
// port, like the other /api/v1 writes in this script.
|
||||||
|
|
||||||
|
const STUDENT_PASSWORD = process.env.SIM_STUDENT_PASSWORD ?? 'schulcloud';
|
||||||
|
const TEAM_MEMBER_ROLE = '5bb5c190fb457b1c3c0c7e0f'; // "teammember" in the seed
|
||||||
|
|
||||||
|
/** Runs `fn` signed in as another account, then restores the teacher. */
|
||||||
|
async function asUser(email, password, fn) {
|
||||||
|
const saved = { jwt, me };
|
||||||
|
const res = await fetch(`${API}/api/v3/authentication/local`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ username: email, password }),
|
||||||
|
});
|
||||||
|
if (!res.ok) throw new Error(`login as ${email} failed: ${res.status}`);
|
||||||
|
jwt = (await res.json()).accessToken;
|
||||||
|
me = await v3('GET', '/me');
|
||||||
|
try {
|
||||||
|
return await fn();
|
||||||
|
} finally {
|
||||||
|
({ jwt, me } = saved);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function legacyDir(name, owner, parent) {
|
||||||
|
return (await v1('POST', '/fileStorage/directories', { name, owner, parent }))._id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The browser's own upload sequence: a signed PUT url, the bytes, then the
|
||||||
|
* record. The url comes from the server, so it is held to the same localhost
|
||||||
|
* rule as everything else this script writes to.
|
||||||
|
*/
|
||||||
|
async function legacyUpload({ owner, parent, name, type, body }) {
|
||||||
|
const bytes = Buffer.from(body);
|
||||||
|
const signed = await v1('POST', '/fileStorage/signedUrl', { parent, filename: name, fileType: type });
|
||||||
|
const { hostname } = new URL(signed.url);
|
||||||
|
if (hostname !== '127.0.0.1' && hostname !== 'localhost') {
|
||||||
|
throw new Error(`refusing to upload to ${hostname}: not a localhost address (see minio-loopback in docker-compose.yml)`);
|
||||||
|
}
|
||||||
|
const put = await fetch(signed.url, { method: 'PUT', headers: signed.header, body: bytes });
|
||||||
|
if (!put.ok) throw new Error(`PUT ${name} to storage -> ${put.status} ${(await put.text()).slice(0, 200)}`);
|
||||||
|
const record = await v1('POST', '/fileStorage', {
|
||||||
|
name,
|
||||||
|
owner,
|
||||||
|
parent,
|
||||||
|
type,
|
||||||
|
size: bytes.length,
|
||||||
|
storageFileName: signed.header['x-amz-meta-flat-name'],
|
||||||
|
});
|
||||||
|
return record._id;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fileManager() {
|
||||||
|
const s = loadState();
|
||||||
|
if (!s.courseId) throw new Error('run `create` first: the file-manager fixture lives in its course');
|
||||||
|
const keep = (key, value) => {
|
||||||
|
s[key] = value;
|
||||||
|
saveState(s);
|
||||||
|
};
|
||||||
|
const student = await studentId();
|
||||||
|
|
||||||
|
step('file manager: Kurs-Dateien');
|
||||||
|
keep('fmCourseRootFileId', await legacyUpload({
|
||||||
|
owner: s.courseId, name: 'Kursplan.txt', type: 'text/plain',
|
||||||
|
body: 'Kursplan Biologie\n\nThemen: Zelle, Gewebe, Organe. Stichwort: Photosynthese-Lichtreaktion.\n',
|
||||||
|
}));
|
||||||
|
keep('fmCourseDirId', await legacyDir('Arbeitsblätter', s.courseId));
|
||||||
|
keep('fmCourseFileId', await legacyUpload({
|
||||||
|
owner: s.courseId, parent: s.fmCourseDirId, name: 'Blatt 1 - Zellorganellen.txt', type: 'text/plain',
|
||||||
|
body: 'Blatt 1: Zellorganellen\n\nBeschrifte die Mitochondrienmembran und das endoplasmatische Retikulum.\n',
|
||||||
|
}));
|
||||||
|
keep('fmCourseSubDirId', await legacyDir('Woche 1', s.courseId, s.fmCourseDirId));
|
||||||
|
keep('fmCourseDeepFileId', await legacyUpload({
|
||||||
|
owner: s.courseId, parent: s.fmCourseSubDirId, name: 'Blatt 2 - Gewebe.txt', type: 'text/plain',
|
||||||
|
body: 'Blatt 2: Gewebe\n\nVergleiche Epithelgewebe und Bindegewebe.\n',
|
||||||
|
}));
|
||||||
|
log(`course root file, folder "Arbeitsblätter" with a file, and "Woche 1" nested inside it`);
|
||||||
|
|
||||||
|
if (s.teamId) {
|
||||||
|
step('file manager: Team-Dateien');
|
||||||
|
// Teams cannot be created (see the README), and the adopted one does not
|
||||||
|
// include the demo student, whose view is the one under test.
|
||||||
|
const team = await v1('GET', `/teams/${s.teamId}`);
|
||||||
|
if (!team.userIds.some((entry) => String(entry.userId?._id ?? entry.userId) === student)) {
|
||||||
|
const userIds = team.userIds.map((entry) => ({
|
||||||
|
userId: String(entry.userId?._id ?? entry.userId),
|
||||||
|
role: String(entry.role?._id ?? entry.role),
|
||||||
|
schoolId: String(entry.schoolId?._id ?? entry.schoolId),
|
||||||
|
}));
|
||||||
|
userIds.push({ userId: student, role: TEAM_MEMBER_ROLE, schoolId: me.school.id });
|
||||||
|
await v1('PATCH', `/teams/${s.teamId}`, { userIds });
|
||||||
|
keep('fmStudentAddedToTeam', true);
|
||||||
|
log('demo student added to the team');
|
||||||
|
}
|
||||||
|
keep('fmTeamDirId', await legacyDir('Projekt', s.teamId));
|
||||||
|
keep('fmTeamFileId', await legacyUpload({
|
||||||
|
owner: s.teamId, parent: s.fmTeamDirId, name: 'Projektplan.txt', type: 'text/plain',
|
||||||
|
body: 'Projektplan\n\nMeilenstein Chlorophyll bis Freitag.\n',
|
||||||
|
}));
|
||||||
|
log('team folder "Projekt" with a file');
|
||||||
|
}
|
||||||
|
|
||||||
|
step('file manager: Persönliche Dateien (the student\'s own)');
|
||||||
|
// No `owner` for personal files, exactly as the upload page sends none: the
|
||||||
|
// server decides the owner model as "a course, or else a team", so passing a
|
||||||
|
// user id records the folder as a team's, and every later permission check on
|
||||||
|
// it then dereferences a team that does not exist.
|
||||||
|
await asUser(STUDENT_EMAIL, STUDENT_PASSWORD, async () => {
|
||||||
|
keep('fmStudentDirId', await legacyDir('Notizen'));
|
||||||
|
keep('fmStudentFileId', await legacyUpload({
|
||||||
|
parent: s.fmStudentDirId, name: 'Lernzettel.txt', type: 'text/plain',
|
||||||
|
body: 'Lernzettel\n\nRibosomenfabrik: Proteinbiosynthese am rauen ER.\n',
|
||||||
|
}));
|
||||||
|
});
|
||||||
|
log('student folder "Notizen" with a file');
|
||||||
|
|
||||||
|
step('file manager: Geteilte Dateien');
|
||||||
|
keep('fmSharedFileId', await legacyUpload({
|
||||||
|
name: 'Geteilt vom Lehrer.txt', type: 'text/plain',
|
||||||
|
body: 'Zusatzmaterial\n\nDie Zellkernhuelle trennt Kernplasma und Zytoplasma.\n',
|
||||||
|
}));
|
||||||
|
// What accepting a share link does in the legacy client: a read-only user
|
||||||
|
// permission. Not the permission service, which writes `refOwnerModel` where
|
||||||
|
// the "shared with me" query reads `refPermModel`, so its shares never show.
|
||||||
|
const shared = await v1('GET', `/files/${s.fmSharedFileId}`);
|
||||||
|
await v1('PATCH', `/files/${s.fmSharedFileId}`, {
|
||||||
|
permissions: [
|
||||||
|
...shared.permissions,
|
||||||
|
{ refId: student, refPermModel: 'user', read: true, write: false, delete: false, create: false },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
log('teacher file shared read-only with the student');
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------- update ---
|
// ---------------------------------------------------------------- update ---
|
||||||
|
|
||||||
async function update() {
|
async function update() {
|
||||||
@@ -348,6 +490,17 @@ async function update() {
|
|||||||
await files('PATCH', `/rename/${s.fileId}`, { fileName: 'nervensystem-notiz-v2.txt' });
|
await files('PATCH', `/rename/${s.fileId}`, { fileName: 'nervensystem-notiz-v2.txt' });
|
||||||
log('file renamed');
|
log('file renamed');
|
||||||
|
|
||||||
|
if (s.fmCourseFileId) {
|
||||||
|
step('file manager edits');
|
||||||
|
await v1('POST', '/fileStorage/rename', { id: s.fmCourseFileId, newName: 'Blatt 1 - Zellorganellen (korrigiert).txt' });
|
||||||
|
log('course file renamed');
|
||||||
|
s.fmCourseAddedFileId = await legacyUpload({
|
||||||
|
owner: s.courseId, parent: s.fmCourseSubDirId, name: 'Blatt 3 - Organe.txt', type: 'text/plain',
|
||||||
|
body: 'Blatt 3: Organe\n\nNeuer Suchbegriff: Nephronschleife.\n',
|
||||||
|
});
|
||||||
|
log('course file added in "Woche 1" (new search term: Nephronschleife)');
|
||||||
|
}
|
||||||
|
|
||||||
saveState({ ...s, updated: true });
|
saveState({ ...s, updated: true });
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -376,6 +529,46 @@ async function remove() {
|
|||||||
['second room', () => v3('DELETE', `/rooms/${s.roomWithoutStudentId}`)],
|
['second room', () => v3('DELETE', `/rooms/${s.roomWithoutStudentId}`)],
|
||||||
['course', () => v1('DELETE', `/courses/${s.courseId}`)],
|
['course', () => v1('DELETE', `/courses/${s.courseId}`)],
|
||||||
];
|
];
|
||||||
|
// File-manager content goes first, while its course and team still exist.
|
||||||
|
const fileManagerTries = [
|
||||||
|
['file-manager course files', async () => {
|
||||||
|
for (const id of [s.fmCourseAddedFileId, s.fmCourseDeepFileId, s.fmCourseFileId, s.fmCourseRootFileId]) {
|
||||||
|
if (id) await v1('DELETE', `/fileStorage?_id=${id}`);
|
||||||
|
}
|
||||||
|
}],
|
||||||
|
['file-manager course folders', async () => {
|
||||||
|
for (const id of [s.fmCourseSubDirId, s.fmCourseDirId]) if (id) await v1('DELETE', `/fileStorage/directories?_id=${id}`);
|
||||||
|
}],
|
||||||
|
['file-manager team content', async () => {
|
||||||
|
if (s.fmTeamFileId) await v1('DELETE', `/fileStorage?_id=${s.fmTeamFileId}`);
|
||||||
|
if (s.fmTeamDirId) await v1('DELETE', `/fileStorage/directories?_id=${s.fmTeamDirId}`);
|
||||||
|
}],
|
||||||
|
['shared file', async () => {
|
||||||
|
if (s.fmSharedFileId) await v1('DELETE', `/fileStorage?_id=${s.fmSharedFileId}`);
|
||||||
|
}],
|
||||||
|
['student personal files', async () => {
|
||||||
|
if (!s.fmStudentFileId && !s.fmStudentDirId) return;
|
||||||
|
await asUser(STUDENT_EMAIL, STUDENT_PASSWORD, async () => {
|
||||||
|
if (s.fmStudentFileId) await v1('DELETE', `/fileStorage?_id=${s.fmStudentFileId}`);
|
||||||
|
if (s.fmStudentDirId) await v1('DELETE', `/fileStorage/directories?_id=${s.fmStudentDirId}`);
|
||||||
|
});
|
||||||
|
}],
|
||||||
|
['student team membership', async () => {
|
||||||
|
if (!s.fmStudentAddedToTeam || !s.teamId) return;
|
||||||
|
const student = await studentId();
|
||||||
|
const team = await v1('GET', `/teams/${s.teamId}`);
|
||||||
|
const userIds = team.userIds
|
||||||
|
.map((entry) => ({
|
||||||
|
userId: String(entry.userId?._id ?? entry.userId),
|
||||||
|
role: String(entry.role?._id ?? entry.role),
|
||||||
|
schoolId: String(entry.schoolId?._id ?? entry.schoolId),
|
||||||
|
}))
|
||||||
|
.filter((entry) => entry.userId !== student);
|
||||||
|
await v1('PATCH', `/teams/${s.teamId}`, { userIds });
|
||||||
|
}],
|
||||||
|
];
|
||||||
|
tries.unshift(...fileManagerTries);
|
||||||
|
|
||||||
for (const [what, fn] of tries) {
|
for (const [what, fn] of tries) {
|
||||||
try {
|
try {
|
||||||
await fn();
|
await fn();
|
||||||
@@ -398,8 +591,11 @@ await login();
|
|||||||
if (phase === 'create') await create();
|
if (phase === 'create') await create();
|
||||||
else if (phase === 'update') await update();
|
else if (phase === 'update') await update();
|
||||||
else if (phase === 'delete') await remove();
|
else if (phase === 'delete') await remove();
|
||||||
else if (phase === 'show') summary(loadState());
|
else if (phase === 'files') {
|
||||||
|
await fileManager();
|
||||||
|
summary(loadState());
|
||||||
|
} else if (phase === 'show') summary(loadState());
|
||||||
else {
|
else {
|
||||||
console.error(`unknown phase ${phase}; expected create | update | delete | show`);
|
console.error(`unknown phase ${phase}; expected create | update | delete | files | show`);
|
||||||
process.exit(2);
|
process.exit(2);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -223,6 +223,63 @@ console.log('\n== classes and groups ==');
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
console.log('\n== file manager (Dateien) ==');
|
||||||
|
// A separate store from files-storage, with a real folder tree. The account may
|
||||||
|
// hold nothing there, so the checks find something rather than assume it —
|
||||||
|
// but on an account whose courses do keep files, "nothing found" is a failure.
|
||||||
|
{
|
||||||
|
const root = await call('fs_list', { path: '/' });
|
||||||
|
check('fs_list / names the four areas', !root.isError && /\/courses\//.test(root.text) && /\/shared\//.test(root.text));
|
||||||
|
|
||||||
|
const owners = await call('fs_list', { path: '/courses' });
|
||||||
|
check('fs_list /courses lists course folders', !owners.isError, owners.text.split('\n').find((line) => /course\(s\)/.test(line)));
|
||||||
|
const ownerIds = [...owners.text.matchAll(/\*\*.*?\/\*\* \(`([0-9a-f]{24})`\)/g)].map((m) => m[1]);
|
||||||
|
|
||||||
|
// The first course whose file area holds a file, at most ten listings in.
|
||||||
|
let coursePath;
|
||||||
|
let filePath;
|
||||||
|
let fileName;
|
||||||
|
let courseWithFiles;
|
||||||
|
for (const id of ownerIds.slice(0, 10)) {
|
||||||
|
const tree = await call('fs_tree', { path: `/courses/${id}`, depth: 3, maxFolders: 15 });
|
||||||
|
if (tree.isError) continue;
|
||||||
|
const heading = tree.text.match(/^## (\/courses\/.+?) — /m)?.[1];
|
||||||
|
const line = tree.text.match(/^\s*([^\n]+?\.(?:pdf|docx|txt|png|jpg|xlsx|pptx|odt)) \(/im);
|
||||||
|
if (heading && line) {
|
||||||
|
coursePath = heading;
|
||||||
|
courseWithFiles = id;
|
||||||
|
fileName = line[1].trim();
|
||||||
|
// The tree gives names; fs_find recovers the full path to read.
|
||||||
|
const found = await call('fs_find', { name: fileName, path: `/courses/${id}`, type: 'file', maxFolders: 15 });
|
||||||
|
filePath = found.text.match(/^- (\/courses\/.+?) — /m)?.[1];
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (coursePath && filePath) {
|
||||||
|
check('fs_tree shows a course file area', true, coursePath);
|
||||||
|
check('fs_find finds a file by name', Boolean(filePath), filePath);
|
||||||
|
const read = await call('fs_read', { path: filePath, maxChars: 800 });
|
||||||
|
check(
|
||||||
|
'fs_read fetches a file-manager file and reports an extraction outcome',
|
||||||
|
!read.isError && /File id:/.test(read.text),
|
||||||
|
read.text.split('\n').find((l) => /extracted|image|no extractable|image-only|Word|PDF/.test(l)) ?? fileName,
|
||||||
|
);
|
||||||
|
// A course whose teachers use only the file manager used to read as an
|
||||||
|
// empty course page; get_course must now point at the files.
|
||||||
|
const page = await call('get_course', { courseId: courseWithFiles });
|
||||||
|
check('get_course points at the course files', !page.isError && /Course files \(Kurs-Dateien\)/.test(page.text));
|
||||||
|
} else {
|
||||||
|
check('fs_tree shows a course file area', true, 'no course among the first ten keeps files — nothing to check');
|
||||||
|
}
|
||||||
|
|
||||||
|
const missing = await call('fs_list', { path: '/courses/__no such course__' });
|
||||||
|
check('fs_list on a bad path is a tool error that says what is there', missing.isError && /No "|not a file area|Did you mean/.test(missing.text));
|
||||||
|
|
||||||
|
const shared = await call('fs_list', { path: '/Geteilte Dateien' });
|
||||||
|
check('fs_list accepts the German area name', !shared.isError, shared.text.split('\n')[0]);
|
||||||
|
}
|
||||||
|
|
||||||
console.log('\n== rooms ==');
|
console.log('\n== rooms ==');
|
||||||
// Rooms ("Räume") are a separate space from courses. An account in none is
|
// Rooms ("Räume") are a separate space from courses. An account in none is
|
||||||
// normal — and is exactly the state that hid this whole feature — so the check
|
// normal — and is exactly the state that hid this whole feature — so the check
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import { pipeline } from 'node:stream/promises';
|
|||||||
import { ApiClient, ApiError } from '../cli/client.ts';
|
import { ApiClient, ApiError } from '../cli/client.ts';
|
||||||
import { defaultSyncDir, loadCliConfig, saveCliConfig, configPath } from '../cli/config.ts';
|
import { defaultSyncDir, loadCliConfig, saveCliConfig, configPath } from '../cli/config.ts';
|
||||||
import { formatBytes } from '../core/extract.ts';
|
import { formatBytes } from '../core/extract.ts';
|
||||||
|
import { fsFind, fsGet, fsList, fsTree } from '../cli/fs.ts';
|
||||||
import { sync, type SyncEvent } from '../cli/sync.ts';
|
import { sync, type SyncEvent } from '../cli/sync.ts';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -26,6 +27,16 @@ const USAGE = `schulcloud — browse and mirror your Schulcloud files
|
|||||||
schulcloud sync [--dry-run] [--full] [--prune] [--dir <path>] [--jobs <n>]
|
schulcloud sync [--dry-run] [--full] [--prune] [--dir <path>] [--jobs <n>]
|
||||||
schulcloud refresh [--course <id>] [--force]
|
schulcloud refresh [--course <id>] [--force]
|
||||||
|
|
||||||
|
The file manager ("Dateien") — /my, /courses/<course>, /teams/<team>, /shared:
|
||||||
|
|
||||||
|
schulcloud fs ls [path] [--long]
|
||||||
|
schulcloud fs tree [path] [--depth <n>] [--max-folders <n>]
|
||||||
|
schulcloud fs find <name> [--path <path>] [--type file|folder] [--long]
|
||||||
|
schulcloud fs get <path> [--out <path>] [--force] [--jobs <n>]
|
||||||
|
|
||||||
|
fs get downloads a file, or a folder with everything below it. Names may contain
|
||||||
|
"/" and still resolve; any path segment can also be an id from "fs ls --long".
|
||||||
|
|
||||||
--course takes a course or a room id: rooms ("Räume") mirror alongside courses
|
--course takes a course or a room id: rooms ("Räume") mirror alongside courses
|
||||||
and their files sit under the room's name.
|
and their files sit under the room's name.
|
||||||
|
|
||||||
@@ -58,6 +69,8 @@ async function main(argv: string[]): Promise<number> {
|
|||||||
return runSync(flags);
|
return runSync(flags);
|
||||||
case 'refresh':
|
case 'refresh':
|
||||||
return refresh(flags);
|
return refresh(flags);
|
||||||
|
case 'fs':
|
||||||
|
return fileManager(flags);
|
||||||
default:
|
default:
|
||||||
process.stderr.write(`Unknown command "${command}".\n\n${USAGE}`);
|
process.stderr.write(`Unknown command "${command}".\n\n${USAGE}`);
|
||||||
return 2;
|
return 2;
|
||||||
@@ -154,6 +167,41 @@ async function get(flags: Flags): Promise<number> {
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function fileManager(flags: Flags): Promise<number> {
|
||||||
|
const [sub, ...args] = flags._ as string[];
|
||||||
|
const api = new ApiClient(await loadCliConfig());
|
||||||
|
const out = (line: string) => process.stdout.write(`${line}\n`);
|
||||||
|
const long = Boolean(flags.long);
|
||||||
|
switch (sub) {
|
||||||
|
case 'ls':
|
||||||
|
return fsList(api, args[0] ?? '/', long, out);
|
||||||
|
case 'tree':
|
||||||
|
return fsTree(api, args[0] ?? '/', Number(flags.depth ?? 3), Number(flags['max-folders'] ?? 200), out);
|
||||||
|
case 'find': {
|
||||||
|
if (!args[0]) {
|
||||||
|
process.stderr.write('fs find needs a name, e.g.: schulcloud fs find Erbrecht --path /courses\n');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
const type = flags.type === 'folder' || flags.type === 'file' ? String(flags.type) : 'any';
|
||||||
|
return fsFind(api, args[0], String(flags.path ?? '/'), type, Number(flags['max-folders'] ?? 400), long, out);
|
||||||
|
}
|
||||||
|
case 'get':
|
||||||
|
if (!args[0]) {
|
||||||
|
process.stderr.write('fs get needs a path, e.g.: schulcloud fs get "/courses/<course>/<folder>"\n');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
return fsGet(
|
||||||
|
api,
|
||||||
|
args[0],
|
||||||
|
{ out: flags.out ? String(flags.out) : undefined, force: Boolean(flags.force), jobs: Number(flags.jobs ?? 3) },
|
||||||
|
out,
|
||||||
|
);
|
||||||
|
default:
|
||||||
|
process.stderr.write(`Unknown fs command "${sub ?? ''}". Use ls, tree, find or get.\n\n${USAGE}`);
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function runSync(flags: Flags): Promise<number> {
|
async function runSync(flags: Flags): Promise<number> {
|
||||||
const config = await loadCliConfig();
|
const config = await loadCliConfig();
|
||||||
const root = flags.dir ? resolve(String(flags.dir)) : config.syncDir;
|
const root = flags.dir ? resolve(String(flags.dir)) : config.syncDir;
|
||||||
|
|||||||
@@ -26,6 +26,39 @@ export interface Manifest {
|
|||||||
entries: ManifestEntry[];
|
entries: ManifestEntry[];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** One entry of a file-manager tree or search, as /api/fs returns it. */
|
||||||
|
export interface FsEntry {
|
||||||
|
type: 'directory' | 'file';
|
||||||
|
path: string;
|
||||||
|
parentPath: string;
|
||||||
|
depth: number;
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
size?: number;
|
||||||
|
mimeType?: string | null;
|
||||||
|
blocked?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FsListing {
|
||||||
|
path: string;
|
||||||
|
kind: 'directory' | 'file';
|
||||||
|
area?: string | null;
|
||||||
|
directories?: { id: string; name: string; path: string }[];
|
||||||
|
files?: { id: string; name: string; path: string; size: number; mimeType?: string; blocked: boolean }[];
|
||||||
|
file?: { id: string; name: string; size: number; mimeType?: string; blocked: boolean };
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FsWalk {
|
||||||
|
path: string;
|
||||||
|
kind: 'directory' | 'file';
|
||||||
|
entries?: FsEntry[];
|
||||||
|
matches?: FsEntry[];
|
||||||
|
file?: FsListing['file'];
|
||||||
|
visited?: number;
|
||||||
|
truncated?: boolean;
|
||||||
|
failures?: { path: string; reason: string }[];
|
||||||
|
}
|
||||||
|
|
||||||
export class ApiError extends Error {
|
export class ApiError extends Error {
|
||||||
readonly status: number;
|
readonly status: number;
|
||||||
|
|
||||||
@@ -83,6 +116,28 @@ export class ApiClient {
|
|||||||
async file(fileId: string): Promise<Response> {
|
async file(fileId: string): Promise<Response> {
|
||||||
return this.request(`/api/files/${encodeURIComponent(fileId)}`);
|
return this.request(`/api/files/${encodeURIComponent(fileId)}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// --- the file manager ------------------------------------------------------
|
||||||
|
|
||||||
|
async fsList(path: string): Promise<FsListing> {
|
||||||
|
return (await (await this.request(`/api/fs/list?${new URLSearchParams({ path })}`)).json()) as FsListing;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fsTree(path: string, depth: number, maxFolders: number): Promise<FsWalk> {
|
||||||
|
const query = new URLSearchParams({ path, depth: String(depth), maxFolders: String(maxFolders) });
|
||||||
|
return (await (await this.request(`/api/fs/tree?${query}`)).json()) as FsWalk;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fsFind(name: string, path: string, type: string, maxFolders: number): Promise<FsWalk> {
|
||||||
|
const query = new URLSearchParams({ name, path, type, maxFolders: String(maxFolders) });
|
||||||
|
return (await (await this.request(`/api/fs/find?${query}`)).json()) as FsWalk;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Streams one file-manager file's bytes, by path or by id. */
|
||||||
|
async fsFile(target: { path: string } | { id: string; name: string }): Promise<Response> {
|
||||||
|
const query = 'path' in target ? new URLSearchParams({ path: target.path }) : new URLSearchParams(target);
|
||||||
|
return this.request(`/api/fs/file?${query}`);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function describe(status: number, detail: string, server: string): string {
|
function describe(status: number, detail: string, server: string): string {
|
||||||
@@ -90,5 +145,7 @@ function describe(status: number, detail: string, server: string): string {
|
|||||||
if (status === 503) return 'The server is running without an index, so this command is unavailable. Set DATABASE_URL on the server.';
|
if (status === 503) return 'The server is running without an index, so this command is unavailable. Set DATABASE_URL on the server.';
|
||||||
if (status === 409) return detail || 'The sync cursor is unknown to the server. Run a full sync with --full.';
|
if (status === 409) return detail || 'The sync cursor is unknown to the server. Run a full sync with --full.';
|
||||||
if (status === 429) return detail || 'Refreshed too recently — wait a moment, or pass --force.';
|
if (status === 429) return detail || 'Refreshed too recently — wait a moment, or pass --force.';
|
||||||
|
// The file manager's own errors already say what was not found and what is there.
|
||||||
|
if ((status === 400 || status === 404 || status === 422) && detail) return detail;
|
||||||
return detail ? `HTTP ${status}: ${detail}` : `HTTP ${status}`;
|
return detail ? `HTTP ${status}: ${detail}` : `HTTP ${status}`;
|
||||||
}
|
}
|
||||||
|
|||||||
218
src/cli/fs.ts
Normal file
218
src/cli/fs.ts
Normal file
@@ -0,0 +1,218 @@
|
|||||||
|
import { createWriteStream } from 'node:fs';
|
||||||
|
import { mkdir, rename, stat, unlink } from 'node:fs/promises';
|
||||||
|
import { dirname, resolve } from 'node:path';
|
||||||
|
import { Readable } from 'node:stream';
|
||||||
|
import { pipeline } from 'node:stream/promises';
|
||||||
|
import { formatBytes } from '../core/extract.ts';
|
||||||
|
import { resolveWithin, safeComponent } from '../core/paths.ts';
|
||||||
|
import type { ApiClient, FsEntry } from './client.ts';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `schulcloud fs` — the file manager ("Dateien") from the command line.
|
||||||
|
*
|
||||||
|
* The same tree the MCP fs_* tools show: /my, /courses/<course>, /teams/<team>,
|
||||||
|
* /shared. Everything goes through the server's /api/fs routes; nothing here
|
||||||
|
* talks to Schulcloud.
|
||||||
|
*/
|
||||||
|
|
||||||
|
type Out = (line: string) => void;
|
||||||
|
|
||||||
|
export async function fsList(api: ApiClient, path: string, long: boolean, out: Out): Promise<number> {
|
||||||
|
const listing = await api.fsList(path);
|
||||||
|
if (listing.kind === 'file' && listing.file) {
|
||||||
|
out(`${listing.path}`);
|
||||||
|
out(` ${formatBytes(listing.file.size)} ${listing.file.mimeType ?? 'unknown type'} ${listing.file.id}${listing.file.blocked ? ' [blocked]' : ''}`);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const directories = listing.directories ?? [];
|
||||||
|
const files = listing.files ?? [];
|
||||||
|
out(listing.path);
|
||||||
|
for (const directory of directories) out(` ${directory.name}/${long ? ` ${directory.id}` : ''}`);
|
||||||
|
for (const file of files) {
|
||||||
|
const detail = long ? ` ${formatBytes(file.size).padStart(9)} ${file.id}${file.mimeType ? ` ${file.mimeType}` : ''}` : '';
|
||||||
|
out(` ${file.name}${detail}${file.blocked ? ' [blocked]' : ''}`);
|
||||||
|
}
|
||||||
|
if (directories.length + files.length === 0) out(' (empty)');
|
||||||
|
out(`${directories.length} folder(s), ${files.length} file(s)`);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function fsTree(api: ApiClient, path: string, depth: number, maxFolders: number, out: Out): Promise<number> {
|
||||||
|
const walk = await api.fsTree(path, depth, maxFolders);
|
||||||
|
if (walk.kind === 'file') {
|
||||||
|
out(`${walk.path} is a file.`);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
const entries = walk.entries ?? [];
|
||||||
|
const children = groupByParent(entries);
|
||||||
|
|
||||||
|
out(walk.path);
|
||||||
|
let files = 0;
|
||||||
|
let bytes = 0;
|
||||||
|
const visit = (parent: string, prefix: string) => {
|
||||||
|
const kids = children.get(parent) ?? [];
|
||||||
|
kids.forEach((kid, index) => {
|
||||||
|
const last = index === kids.length - 1;
|
||||||
|
const branch = last ? '└── ' : '├── ';
|
||||||
|
if (kid.type === 'directory') {
|
||||||
|
out(`${prefix}${branch}${kid.name}/`);
|
||||||
|
visit(kid.path, `${prefix}${last ? ' ' : '│ '}`);
|
||||||
|
} else {
|
||||||
|
files++;
|
||||||
|
bytes += kid.size ?? 0;
|
||||||
|
out(`${prefix}${branch}${kid.name} (${formatBytes(kid.size ?? 0)})${kid.blocked ? ' [blocked]' : ''}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
};
|
||||||
|
visit(walk.path, '');
|
||||||
|
|
||||||
|
out(`\n${files} file(s), ${formatBytes(bytes)} — ${walk.visited ?? 0} folder(s) listed`);
|
||||||
|
if (walk.truncated) out(`Stopped after ${maxFolders} folders; pass --max-folders or start deeper.`);
|
||||||
|
for (const failure of walk.failures ?? []) out(`could not list ${failure.path}: ${failure.reason}`);
|
||||||
|
return walk.failures?.length ? 1 : 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function fsFind(
|
||||||
|
api: ApiClient,
|
||||||
|
name: string,
|
||||||
|
path: string,
|
||||||
|
type: string,
|
||||||
|
maxFolders: number,
|
||||||
|
long: boolean,
|
||||||
|
out: Out,
|
||||||
|
): Promise<number> {
|
||||||
|
const result = await api.fsFind(name, path, type, maxFolders);
|
||||||
|
for (const match of result.matches ?? []) {
|
||||||
|
const suffix = match.type === 'directory' ? '/' : '';
|
||||||
|
const detail = long && match.type === 'file' ? ` ${formatBytes(match.size ?? 0)} ${match.id}` : long ? ` ${match.id}` : '';
|
||||||
|
out(`${match.path}${suffix}${detail}`);
|
||||||
|
}
|
||||||
|
const count = result.matches?.length ?? 0;
|
||||||
|
process.stderr.write(`${count} match(es), ${result.visited ?? 0} folder(s) searched\n`);
|
||||||
|
if (result.truncated) process.stderr.write(`Stopped after ${maxFolders} folders; there may be more. Narrow --path.\n`);
|
||||||
|
return count > 0 ? 0 : 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Downloads a file, or a whole folder recursively.
|
||||||
|
*
|
||||||
|
* A folder lands under `--out` (default: a directory named after it) with the
|
||||||
|
* file manager's structure. Every component is a name from Schulcloud and so
|
||||||
|
* untrusted: each goes through `safeComponent`, and the joined path through
|
||||||
|
* `resolveWithin`, exactly as sync does.
|
||||||
|
*/
|
||||||
|
export async function fsGet(
|
||||||
|
api: ApiClient,
|
||||||
|
path: string,
|
||||||
|
options: { out?: string; force: boolean; jobs: number },
|
||||||
|
out: Out,
|
||||||
|
): Promise<number> {
|
||||||
|
const target = await api.fsList(path);
|
||||||
|
|
||||||
|
if (target.kind === 'file' && target.file) {
|
||||||
|
const destination = options.out ? resolve(options.out) : resolve(safeComponent(target.file.name, target.file.id));
|
||||||
|
if (target.file.blocked) {
|
||||||
|
process.stderr.write(`${target.path}: blocked by the instance virus scanner; not downloaded.\n`);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
await downloadTo(api, { path: target.path }, destination);
|
||||||
|
out(destination);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const rootName = target.path === '/' ? 'Dateien' : (target.path.split('/').pop() ?? 'Dateien');
|
||||||
|
const root = options.out ? resolve(options.out) : resolve(safeComponent(rootName, 'Dateien'));
|
||||||
|
process.stderr.write(`Listing ${target.path} …\n`);
|
||||||
|
const walk = await api.fsTree(target.path, 12, 1000);
|
||||||
|
const entries = walk.entries ?? [];
|
||||||
|
|
||||||
|
// Rebuild each file's name segments from its parents rather than splitting
|
||||||
|
// its path: names may contain "/", which would otherwise invent folders.
|
||||||
|
const segments = new Map<string, string[]>([[walk.path, []]]);
|
||||||
|
for (const entry of [...entries].sort((a, b) => a.depth - b.depth)) {
|
||||||
|
const parent = segments.get(entry.parentPath);
|
||||||
|
if (parent) segments.set(entry.path, [...parent, entry.name]);
|
||||||
|
}
|
||||||
|
|
||||||
|
const files = entries.filter((entry) => entry.type === 'file');
|
||||||
|
let downloaded = 0;
|
||||||
|
let skipped = 0;
|
||||||
|
let failed = 0;
|
||||||
|
let bytes = 0;
|
||||||
|
|
||||||
|
const queue = [...files];
|
||||||
|
const worker = async () => {
|
||||||
|
for (let entry = queue.shift(); entry; entry = queue.shift()) {
|
||||||
|
const parts = segments.get(entry.path);
|
||||||
|
if (!parts) continue;
|
||||||
|
const relative = parts.map((part) => safeComponent(part)).join('/');
|
||||||
|
const destination = resolveWithin(root, relative);
|
||||||
|
if (entry.blocked) {
|
||||||
|
process.stderr.write(` blocked ${relative}\n`);
|
||||||
|
skipped++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
// Unchanged by size: re-running a folder download resumes rather than repeats.
|
||||||
|
const existing = await stat(destination).catch(() => undefined);
|
||||||
|
if (!options.force && existing?.isFile() && existing.size === entry.size) {
|
||||||
|
skipped++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
await downloadTo(api, { id: entry.id, name: entry.name }, destination);
|
||||||
|
downloaded++;
|
||||||
|
bytes += entry.size ?? 0;
|
||||||
|
process.stderr.write(` get ${relative}\n`);
|
||||||
|
} catch (error) {
|
||||||
|
failed++;
|
||||||
|
process.stderr.write(` FAILED ${relative}: ${(error as Error).message}\n`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
await Promise.all(Array.from({ length: Math.max(1, options.jobs) }, worker));
|
||||||
|
|
||||||
|
out(root);
|
||||||
|
process.stderr.write(
|
||||||
|
`Downloaded ${downloaded} file(s) (${formatBytes(bytes)}), ${skipped} skipped` +
|
||||||
|
`${failed ? `, FAILED ${failed}` : ''} — ${walk.visited ?? 0} folder(s) listed\n`,
|
||||||
|
);
|
||||||
|
if (walk.truncated) process.stderr.write('The folder is larger than one listing pass; some files were not reached.\n');
|
||||||
|
for (const failure of walk.failures ?? []) process.stderr.write(`could not list ${failure.path}: ${failure.reason}\n`);
|
||||||
|
return failed > 0 || (walk.failures?.length ?? 0) > 0 ? 1 : 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function downloadTo(api: ApiClient, target: { path: string } | { id: string; name: string }, destination: string) {
|
||||||
|
const response = await api.fsFile(target);
|
||||||
|
if (!response.body) throw new Error('empty response body');
|
||||||
|
await mkdir(dirname(destination), { recursive: true });
|
||||||
|
// A temporary neighbour, renamed into place, so an interrupted download never
|
||||||
|
// leaves a half-file that the size check would later accept as complete.
|
||||||
|
const temp = `${destination}.part`;
|
||||||
|
try {
|
||||||
|
await pipeline(Readable.fromWeb(response.body as never), createWriteStream(temp));
|
||||||
|
await rename(temp, destination);
|
||||||
|
} catch (error) {
|
||||||
|
await unlink(temp).catch(() => {});
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function groupByParent(entries: FsEntry[]): Map<string, FsEntry[]> {
|
||||||
|
const children = new Map<string, FsEntry[]>();
|
||||||
|
for (const entry of entries) {
|
||||||
|
const list = children.get(entry.parentPath) ?? [];
|
||||||
|
list.push(entry);
|
||||||
|
children.set(entry.parentPath, list);
|
||||||
|
}
|
||||||
|
for (const list of children.values()) {
|
||||||
|
list.sort((a, b) =>
|
||||||
|
a.type !== b.type
|
||||||
|
? a.type === 'directory'
|
||||||
|
? -1
|
||||||
|
: 1
|
||||||
|
: a.name.localeCompare(b.name, 'de', { numeric: true, sensitivity: 'base' }),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return children;
|
||||||
|
}
|
||||||
@@ -37,6 +37,8 @@ export interface Config {
|
|||||||
mirrorMaxBytes: number;
|
mirrorMaxBytes: number;
|
||||||
/** Index personal files and submitted/returned work as well as course content. */
|
/** Index personal files and submitted/returned work as well as course content. */
|
||||||
indexPersonalFiles: boolean;
|
indexPersonalFiles: boolean;
|
||||||
|
/** Walk the file manager (Kurs-, Persönliche, Team- and Geteilte Dateien) when crawling. */
|
||||||
|
indexFileManager: boolean;
|
||||||
/** How often to re-crawl on a timer. Zero = only on demand. */
|
/** How often to re-crawl on a timer. Zero = only on demand. */
|
||||||
crawlIntervalMs: number;
|
crawlIntervalMs: number;
|
||||||
}
|
}
|
||||||
@@ -95,6 +97,9 @@ export function loadConfig(): Config {
|
|||||||
// cost of a full crawl. Worth turning on to make your own handed-in work
|
// cost of a full crawl. Worth turning on to make your own handed-in work
|
||||||
// searchable, which no other route offers.
|
// searchable, which no other route offers.
|
||||||
indexPersonalFiles: bool('INDEX_PERSONAL_FILES', false),
|
indexPersonalFiles: bool('INDEX_PERSONAL_FILES', false),
|
||||||
|
// On by default: many teachers keep their material only in Kurs-Dateien,
|
||||||
|
// so an index without it misses whole courses. One page load per folder.
|
||||||
|
indexFileManager: bool('INDEX_FILE_MANAGER', true),
|
||||||
crawlIntervalMs: intAllowingZero('CRAWL_INTERVAL_MS', 6 * 60 * 60_000),
|
crawlIntervalMs: intAllowingZero('CRAWL_INTERVAL_MS', 6 * 60 * 60_000),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import type { Config } from './config.ts';
|
import type { Config } from './config.ts';
|
||||||
import { SchulcloudClient } from './core/client.ts';
|
import { SchulcloudClient } from './core/client.ts';
|
||||||
|
import { FileManager } from './core/legacy-files.ts';
|
||||||
import type { LegacyUser, MeResponse } from './core/types.ts';
|
import type { LegacyUser, MeResponse } from './core/types.ts';
|
||||||
import type { Indexer } from './indexer/indexer.ts';
|
import type { Indexer } from './indexer/indexer.ts';
|
||||||
import type { Store } from './store/store.ts';
|
import type { Store } from './store/store.ts';
|
||||||
@@ -14,6 +15,8 @@ import type { Store } from './store/store.ts';
|
|||||||
export class ServerContext {
|
export class ServerContext {
|
||||||
readonly config: Config;
|
readonly config: Config;
|
||||||
readonly client: SchulcloudClient;
|
readonly client: SchulcloudClient;
|
||||||
|
/** The "Dateien" file manager; shared across sessions when the process provides one. */
|
||||||
|
readonly files: FileManager;
|
||||||
/** Shared across sessions; undefined when running without an index. */
|
/** Shared across sessions; undefined when running without an index. */
|
||||||
readonly store: Store | undefined;
|
readonly store: Store | undefined;
|
||||||
readonly indexer: Indexer | undefined;
|
readonly indexer: Indexer | undefined;
|
||||||
@@ -30,9 +33,13 @@ export class ServerContext {
|
|||||||
*/
|
*/
|
||||||
private readonly userNames = new Map<string, Promise<string | undefined>>();
|
private readonly userNames = new Map<string, Promise<string | undefined>>();
|
||||||
|
|
||||||
constructor(config: Config, shared?: { client?: SchulcloudClient; store?: Store; indexer?: Indexer }) {
|
constructor(
|
||||||
|
config: Config,
|
||||||
|
shared?: { client?: SchulcloudClient; files?: FileManager; store?: Store; indexer?: Indexer },
|
||||||
|
) {
|
||||||
this.config = config;
|
this.config = config;
|
||||||
this.client = shared?.client ?? new SchulcloudClient(config);
|
this.client = shared?.client ?? new SchulcloudClient(config);
|
||||||
|
this.files = shared?.files ?? new FileManager(this.client);
|
||||||
this.store = shared?.store;
|
this.store = shared?.store;
|
||||||
this.indexer = shared?.indexer;
|
this.indexer = shared?.indexer;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -119,8 +119,21 @@ export class SchulcloudClient {
|
|||||||
return url;
|
return url;
|
||||||
}
|
}
|
||||||
|
|
||||||
private async request(url: URL, accept: string): Promise<Response> {
|
/**
|
||||||
|
* One upstream GET, with retries for the transient failures.
|
||||||
|
*
|
||||||
|
* `auth` picks how the session travels. The v3 API takes it as a bearer
|
||||||
|
* token; the legacy client's pages take it only as the `jwt` cookie; and a
|
||||||
|
* pre-signed storage URL must get **nothing** — it lives on another host, and
|
||||||
|
* the session token has no business leaving this instance. Anything but the
|
||||||
|
* bearer form is fetched with redirects off, so a login bounce or a hop to a
|
||||||
|
* third host is seen rather than silently followed with credentials attached.
|
||||||
|
*/
|
||||||
|
private async request(url: URL, accept: string, auth: 'bearer' | 'cookie' | 'none' = 'bearer'): Promise<Response> {
|
||||||
let lastError: unknown;
|
let lastError: unknown;
|
||||||
|
const headers: Record<string, string> = { Accept: accept };
|
||||||
|
if (auth === 'bearer') headers.Authorization = `Bearer ${this.config.jwt}`;
|
||||||
|
if (auth === 'cookie') headers.Cookie = `jwt=${this.config.jwt}`;
|
||||||
|
|
||||||
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
|
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
|
||||||
if (attempt > 0) await delay(backoffMs(attempt));
|
if (attempt > 0) await delay(backoffMs(attempt));
|
||||||
@@ -128,9 +141,9 @@ export class SchulcloudClient {
|
|||||||
let response: Response;
|
let response: Response;
|
||||||
try {
|
try {
|
||||||
response = await fetch(url, {
|
response = await fetch(url, {
|
||||||
headers: { Authorization: `Bearer ${this.config.jwt}`, Accept: accept },
|
headers,
|
||||||
signal: AbortSignal.timeout(this.config.requestTimeoutMs),
|
signal: AbortSignal.timeout(this.config.requestTimeoutMs),
|
||||||
redirect: 'follow',
|
redirect: auth === 'bearer' ? 'follow' : 'manual',
|
||||||
});
|
});
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
// Connection reset or timeout: worth one more try, since every call
|
// Connection reset or timeout: worth one more try, since every call
|
||||||
@@ -143,7 +156,12 @@ export class SchulcloudClient {
|
|||||||
if (response.ok) return response;
|
if (response.ok) return response;
|
||||||
|
|
||||||
const body = await response.text().catch(() => '');
|
const body = await response.text().catch(() => '');
|
||||||
const error = new SchulcloudApiError(response.status, url.pathname + url.search, body);
|
// A pre-signed URL's query string is its credential, so it never goes
|
||||||
|
// into an error message; nor does the storage host's error body.
|
||||||
|
const error =
|
||||||
|
auth === 'none'
|
||||||
|
? new SchulcloudApiError(response.status, `${url.host} (pre-signed download)`, '')
|
||||||
|
: new SchulcloudApiError(response.status, url.pathname + url.search, body);
|
||||||
|
|
||||||
// A crawl issues hundreds of requests and the instance answers some of
|
// A crawl issues hundreds of requests and the instance answers some of
|
||||||
// them with a 503 front-page when it decides we are going too fast.
|
// them with a 503 front-page when it decides we are going too fast.
|
||||||
@@ -175,6 +193,11 @@ export class SchulcloudClient {
|
|||||||
async getBytes(path: string, fallbackName: string): Promise<DownloadedFile> {
|
async getBytes(path: string, fallbackName: string): Promise<DownloadedFile> {
|
||||||
const url = this.url(path);
|
const url = this.url(path);
|
||||||
const response = await this.request(url, '*/*');
|
const response = await this.request(url, '*/*');
|
||||||
|
return this.readCapped(response, fallbackName);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Reads a response body up to `maxDownloadBytes`, flagging anything cut off. */
|
||||||
|
private async readCapped(response: Response, fallbackName: string): Promise<DownloadedFile> {
|
||||||
const limit = this.config.maxDownloadBytes;
|
const limit = this.config.maxDownloadBytes;
|
||||||
|
|
||||||
const chunks: Buffer[] = [];
|
const chunks: Buffer[] = [];
|
||||||
@@ -550,6 +573,118 @@ export class SchulcloudClient {
|
|||||||
const body = await this.getJson<Paginated<GroupItem>>('/api/v3/groups', { limit: MAX_PAGE_SIZE });
|
const body = await this.getJson<Paginated<GroupItem>>('/api/v3/groups', { limit: MAX_PAGE_SIZE });
|
||||||
return body.data ?? [];
|
return body.data ?? [];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// --- the "Dateien" file manager ------------------------------------------
|
||||||
|
//
|
||||||
|
// Persönliche Dateien, Kurs-Dateien, Team-Dateien and Geteilte Dateien live in
|
||||||
|
// the legacy file system, a different store from files-storage: listing a
|
||||||
|
// course through /api/v3/file answers 0 files for a course holding dozens.
|
||||||
|
// Its Feathers service is not in the public ingress, so the only way in is
|
||||||
|
// the legacy client — HTML pages for listings, one JSON route for downloads.
|
||||||
|
// See core/legacy-files.ts for the parsing and the path model.
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One file-manager page, as HTML.
|
||||||
|
*
|
||||||
|
* **Only the listing routes are reachable here, by construction.** Several of
|
||||||
|
* the legacy client's GET routes write: `GET /files/share/` mints a share
|
||||||
|
* token when the file has none, and `GET /files/file?share=…` grants the
|
||||||
|
* caller a permission on someone else's file. A GET-only client is therefore
|
||||||
|
* not read-only against this surface by itself; the allowlist is what makes
|
||||||
|
* the invariant hold.
|
||||||
|
*/
|
||||||
|
async getFileManagerPage(path: string): Promise<string> {
|
||||||
|
if (!FILE_MANAGER_PAGE.test(path)) {
|
||||||
|
throw new Error(`refusing file-manager path outside the listing routes: ${path}`);
|
||||||
|
}
|
||||||
|
const response = await this.legacyRequest(path, 'text/html');
|
||||||
|
return response.text();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A pre-signed download URL for one legacy file.
|
||||||
|
*
|
||||||
|
* `name` only sets the download's filename; the server checks read access on
|
||||||
|
* the id. The route is `/files/signedurl` rather than `/files/file`, which
|
||||||
|
* answers the same thing as a redirect but also accepts `share`, the
|
||||||
|
* parameter that writes.
|
||||||
|
*/
|
||||||
|
async getFileManagerSignedUrl(fileId: string, name: string): Promise<string> {
|
||||||
|
if (!/^[0-9a-f]{24}$/i.test(fileId)) throw new Error(`not a file id: ${fileId}`);
|
||||||
|
const query = new URLSearchParams({ file: fileId, name: name || fileId });
|
||||||
|
const response = await this.legacyRequest(`/files/signedurl?${query.toString()}`, 'application/json');
|
||||||
|
// The server's error path *returns* its Forbidden rather than throwing it,
|
||||||
|
// so a refused file arrives as a 200 whose body has no url.
|
||||||
|
const body = (await response.json().catch(() => ({}))) as { url?: unknown; message?: unknown };
|
||||||
|
if (typeof body.url !== 'string' || !body.url) {
|
||||||
|
throw new SchulcloudApiError(403, '/files/signedurl', typeof body.message === 'string' ? body.message : 'no download url');
|
||||||
|
}
|
||||||
|
return body.url;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Downloads one legacy file: signed URL, then the bytes, capped like every download. */
|
||||||
|
async downloadFileManagerFile(fileId: string, name: string): Promise<DownloadedFile> {
|
||||||
|
const signed = await this.getFileManagerSignedUrl(fileId, name);
|
||||||
|
const response = await this.openSignedUrl(signed);
|
||||||
|
return this.readCapped(response, name);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Opens a pre-signed storage URL — with no credentials at all.
|
||||||
|
*
|
||||||
|
* The URL names another host (live: an S3 endpoint at the storage provider),
|
||||||
|
* so neither the bearer nor the cookie may go with it. It must also be
|
||||||
|
* https whenever the instance is, which keeps a URL the server hands back from
|
||||||
|
* pointing this process at a plaintext service on its own network.
|
||||||
|
*/
|
||||||
|
async openSignedUrl(signedUrl: string): Promise<Response> {
|
||||||
|
const target = checkSignedUrl(signedUrl, this.config.baseUrl);
|
||||||
|
return this.request(target, '*/*', 'none');
|
||||||
|
}
|
||||||
|
|
||||||
|
private async legacyRequest(path: string, accept: string): Promise<Response> {
|
||||||
|
try {
|
||||||
|
return await this.request(this.url(path), accept, 'cookie');
|
||||||
|
} catch (error) {
|
||||||
|
// The legacy client answers a rejected cookie with a redirect to its
|
||||||
|
// login page. Report it as what it is, so tools say "token expired"
|
||||||
|
// rather than "HTTP 302".
|
||||||
|
if (error instanceof SchulcloudApiError && error.status >= 300 && error.status < 400) {
|
||||||
|
throw new SchulcloudApiError(401, path, 'redirected to login: the session is not accepted');
|
||||||
|
}
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The file-manager listing routes, and nothing else.
|
||||||
|
*
|
||||||
|
* Folders are addressed by id alone — `/files/courses/{course}/{folder}` holds
|
||||||
|
* one folder segment however deep the folder is — so every listing fits one of
|
||||||
|
* these shapes. `/files/my/{a}/{b}` exists too, but lists `b` exactly as
|
||||||
|
* `/files/my/{b}` does, so it is not needed.
|
||||||
|
*/
|
||||||
|
const FILE_MANAGER_PAGE =
|
||||||
|
/^\/files\/(?:(?:my|courses|teams|shared)\/|my\/[0-9a-f]{24}|(?:courses|teams)\/[0-9a-f]{24}(?:\/[0-9a-f]{24})?)$/i;
|
||||||
|
|
||||||
|
/** Validates a pre-signed URL before anything is sent to it. Exported for testing. */
|
||||||
|
export function checkSignedUrl(signedUrl: string, baseUrl: string): URL {
|
||||||
|
let target: URL;
|
||||||
|
try {
|
||||||
|
target = new URL(signedUrl);
|
||||||
|
} catch {
|
||||||
|
throw new Error('the download url the server returned is not a url');
|
||||||
|
}
|
||||||
|
const instanceIsHttps = new URL(baseUrl).protocol === 'https:';
|
||||||
|
const allowed = instanceIsHttps ? ['https:'] : ['https:', 'http:'];
|
||||||
|
if (!allowed.includes(target.protocol)) {
|
||||||
|
throw new Error(`refusing a ${target.protocol} download url from an ${instanceIsHttps ? 'https' : 'http'} instance`);
|
||||||
|
}
|
||||||
|
if (target.username || target.password) {
|
||||||
|
throw new Error('refusing a download url that carries credentials');
|
||||||
|
}
|
||||||
|
return target;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import type { Config } from '../config.ts';
|
|||||||
import { assembleBoard, type AssembledBoard } from './board.ts';
|
import { assembleBoard, type AssembledBoard } from './board.ts';
|
||||||
import type { SchulcloudClient } from './client.ts';
|
import type { SchulcloudClient } from './client.ts';
|
||||||
import { fetchHomeworkPage } from './homework-page.ts';
|
import { fetchHomeworkPage } from './homework-page.ts';
|
||||||
|
import { FileManager, type DirectoryRef, type FmFile, type WalkEntry } from './legacy-files.ts';
|
||||||
import { fetchLessonTaskLinks, withScrapedIds } from './lesson-page.ts';
|
import { fetchLessonTaskLinks, withScrapedIds } from './lesson-page.ts';
|
||||||
import { htmlToText, normalizeObjectId } from './text.ts';
|
import { htmlToText, normalizeObjectId } from './text.ts';
|
||||||
import type { CourseMetadata, FileParentType, FileRecord, TaskContent } from './types.ts';
|
import type { CourseMetadata, FileParentType, FileRecord, TaskContent } from './types.ts';
|
||||||
@@ -35,6 +36,11 @@ export interface Breadcrumb {
|
|||||||
/** Column → card, for board files. */
|
/** Column → card, for board files. */
|
||||||
columnTitle?: string;
|
columnTitle?: string;
|
||||||
cardTitle?: string;
|
cardTitle?: string;
|
||||||
|
/**
|
||||||
|
* Folder names from the file manager, outermost first. Unlike boards, its
|
||||||
|
* trees have no fixed depth, so they cannot be squeezed into the titles above.
|
||||||
|
*/
|
||||||
|
folders?: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface CrawledFile {
|
export interface CrawledFile {
|
||||||
@@ -43,6 +49,13 @@ export interface CrawledFile {
|
|||||||
parentType: FileParentType;
|
parentType: FileParentType;
|
||||||
parentId: string;
|
parentId: string;
|
||||||
at: Breadcrumb;
|
at: Breadcrumb;
|
||||||
|
/**
|
||||||
|
* Which store holds the bytes. The file manager is not files-storage: its
|
||||||
|
* ids mean nothing to /api/v3/file, so downloads must be routed by this.
|
||||||
|
*/
|
||||||
|
source?: 'files-storage' | 'file-manager';
|
||||||
|
/** The file-manager path fs_read takes, for files from there. */
|
||||||
|
fsPath?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -150,6 +163,12 @@ export interface CrawlOptions {
|
|||||||
* about X" to be searchable, which is otherwise impossible.
|
* about X" to be searchable, which is otherwise impossible.
|
||||||
*/
|
*/
|
||||||
includePersonalFiles?: boolean;
|
includePersonalFiles?: boolean;
|
||||||
|
/**
|
||||||
|
* Walk the file manager ("Dateien") too: Kurs-Dateien for every course, and on
|
||||||
|
* a full crawl Persönliche Dateien, Team-Dateien and Geteilte Dateien. One
|
||||||
|
* page load per folder — on the account this was built for, about 160.
|
||||||
|
*/
|
||||||
|
includeFileManager?: boolean;
|
||||||
/**
|
/**
|
||||||
* Read the text of collaborative text editor (Etherpad) pads, which needs a
|
* Read the text of collaborative text editor (Etherpad) pads, which needs a
|
||||||
* second credentialled hop outside the API. Omit to leave pads unread.
|
* second credentialled hop outside the API. Omit to leave pads unread.
|
||||||
@@ -207,10 +226,24 @@ export async function crawl(client: SchulcloudClient, options: CrawlOptions): Pr
|
|||||||
// outside that scope forward untouched.
|
// outside that scope forward untouched.
|
||||||
const rooms: CrawledRoom[] = options.courseIds ? [] : await crawlRooms(client, options, includeFiles, files, failures);
|
const rooms: CrawledRoom[] = options.courseIds ? [] : await crawlRooms(client, options, includeFiles, files, failures);
|
||||||
|
|
||||||
|
if (includeFiles && options.includeFileManager) {
|
||||||
|
const titles = new Map(crawled.map((entry) => [entry.course.id, entry.title]));
|
||||||
|
await crawlFileManager(client, options, titles, files, failures);
|
||||||
|
}
|
||||||
|
|
||||||
// Traversal order is nondeterministic under concurrency; sort so that two
|
// Traversal order is nondeterministic under concurrency; sort so that two
|
||||||
// crawls of unchanged content produce identical snapshots.
|
// crawls of unchanged content produce identical snapshots.
|
||||||
crawled.sort((a, b) => a.course.id.localeCompare(b.course.id));
|
crawled.sort((a, b) => a.course.id.localeCompare(b.course.id));
|
||||||
rooms.sort((a, b) => a.id.localeCompare(b.id));
|
rooms.sort((a, b) => a.id.localeCompare(b.id));
|
||||||
|
// One file can be reachable twice — a course file someone also shared with
|
||||||
|
// you appears under /shared as well — and the index keys nodes by id. Keep
|
||||||
|
// the first place it was found, which the traversal order makes the most
|
||||||
|
// specific one.
|
||||||
|
const seenFiles = new Set<string>();
|
||||||
|
const uniqueFiles = files.filter((file) => (seenFiles.has(file.record.id) ? false : (seenFiles.add(file.record.id), true)));
|
||||||
|
files.length = 0;
|
||||||
|
files.push(...uniqueFiles);
|
||||||
|
|
||||||
files.sort((a, b) => a.record.id.localeCompare(b.record.id));
|
files.sort((a, b) => a.record.id.localeCompare(b.record.id));
|
||||||
submissions.sort((a, b) => a.id.localeCompare(b.id));
|
submissions.sort((a, b) => a.id.localeCompare(b.id));
|
||||||
|
|
||||||
@@ -540,6 +573,128 @@ async function collectSubmissions(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Walks the file manager and records every file in it.
|
||||||
|
*
|
||||||
|
* Kurs-Dateien are filed under their course, so a per-course refresh replaces
|
||||||
|
* exactly that course's files and the store's scope rules carry the rest
|
||||||
|
* forward. The areas that belong to no course — personal, team, shared — are
|
||||||
|
* walked only on a full crawl, for the same reason.
|
||||||
|
*
|
||||||
|
* A fresh FileManager rather than the process-wide one: its listing cache is
|
||||||
|
* right for an interactive ls-then-read, and wrong for a crawl whose whole
|
||||||
|
* point is to see the current state.
|
||||||
|
*/
|
||||||
|
async function crawlFileManager(
|
||||||
|
client: SchulcloudClient,
|
||||||
|
options: CrawlOptions,
|
||||||
|
courseTitles: Map<string, string>,
|
||||||
|
files: CrawledFile[],
|
||||||
|
failures: { courseId: string; boardId?: string; reason: string }[],
|
||||||
|
): Promise<void> {
|
||||||
|
const manager = new FileManager(client);
|
||||||
|
const roots: { ref: DirectoryRef; path: string; courseId: string; at: Omit<Breadcrumb, 'folders'> }[] = [];
|
||||||
|
|
||||||
|
if (options.courseIds) {
|
||||||
|
for (const courseId of options.courseIds) {
|
||||||
|
const title = courseTitles.get(courseId) ?? courseId;
|
||||||
|
roots.push({
|
||||||
|
ref: { area: 'courses', ownerId: courseId },
|
||||||
|
path: `/courses/${title}`,
|
||||||
|
courseId,
|
||||||
|
at: { courseId, courseTitle: title, containerTitle: 'Kurs-Dateien' },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
const owners = async (area: 'courses' | 'teams') => {
|
||||||
|
try {
|
||||||
|
return (await manager.list({ area })).directories;
|
||||||
|
} catch (error) {
|
||||||
|
failures.push({ courseId: '', reason: `file manager /${area}: ${error instanceof Error ? error.message : String(error)}` });
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
};
|
||||||
|
for (const course of await owners('courses')) {
|
||||||
|
const title = courseTitles.get(course.id) ?? course.name;
|
||||||
|
roots.push({
|
||||||
|
ref: { area: 'courses', ownerId: course.id },
|
||||||
|
path: `/courses/${course.name}`,
|
||||||
|
courseId: course.id,
|
||||||
|
at: { courseId: course.id, courseTitle: title, containerTitle: 'Kurs-Dateien' },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
for (const team of await owners('teams')) {
|
||||||
|
roots.push({
|
||||||
|
ref: { area: 'teams', ownerId: team.id },
|
||||||
|
path: `/teams/${team.name}`,
|
||||||
|
courseId: '',
|
||||||
|
at: { courseId: '', courseTitle: 'Team-Dateien', containerTitle: team.name },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
roots.push({ ref: { area: 'my' }, path: '/my', courseId: '', at: { courseId: '', courseTitle: 'Persönliche Dateien' } });
|
||||||
|
roots.push({ ref: { area: 'shared' }, path: '/shared', courseId: '', at: { courseId: '', courseTitle: 'Geteilte Dateien' } });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sequential roots, modest concurrency within each: the instance answers a
|
||||||
|
// burst with 503s, and this walk is the largest single part of a crawl.
|
||||||
|
for (const root of roots) {
|
||||||
|
const result = await manager.walk(
|
||||||
|
{ path: root.path, ref: root.ref },
|
||||||
|
{ maxDepth: 25, maxDirectories: 5000, concurrency: 2 },
|
||||||
|
);
|
||||||
|
for (const failure of result.failures) {
|
||||||
|
failures.push({ courseId: root.courseId, reason: `file manager ${failure.path}: ${failure.reason}` });
|
||||||
|
}
|
||||||
|
if (result.truncated) {
|
||||||
|
failures.push({ courseId: root.courseId, reason: `file manager ${root.path}: stopped at 5000 folders` });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Folder names come from the parent chain, never from splitting a path:
|
||||||
|
// names contain "/" in real data.
|
||||||
|
const folders = new Map<string, string[]>([[root.path, []]]);
|
||||||
|
const ordered = [...result.entries].sort((a, b) => a.depth - b.depth);
|
||||||
|
for (const entry of ordered) {
|
||||||
|
if (entry.directory) {
|
||||||
|
const parent = folders.get(entry.parentPath);
|
||||||
|
if (parent) folders.set(entry.path, [...parent, entry.directory.name]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const entry of ordered) {
|
||||||
|
if (!entry.file) continue;
|
||||||
|
files.push(fileManagerRecord(entry, entry.file, root, folders.get(entry.parentPath) ?? []));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function fileManagerRecord(
|
||||||
|
entry: WalkEntry,
|
||||||
|
file: FmFile,
|
||||||
|
root: { ref: DirectoryRef; courseId: string; at: Omit<Breadcrumb, 'folders'> },
|
||||||
|
folders: string[],
|
||||||
|
): CrawledFile {
|
||||||
|
const parentType: FileParentType = root.ref.area === 'courses' ? 'courses' : 'users';
|
||||||
|
return {
|
||||||
|
// Shaped like a files-storage record so the store, the mirror and the
|
||||||
|
// manifest need no second code path; `source` is what keeps the two apart.
|
||||||
|
record: {
|
||||||
|
id: file.id,
|
||||||
|
name: file.name,
|
||||||
|
parentId: entry.parent.folderId ?? entry.parent.ownerId ?? '',
|
||||||
|
parentType,
|
||||||
|
url: '',
|
||||||
|
size: file.size,
|
||||||
|
mimeType: file.mimeType ?? 'application/octet-stream',
|
||||||
|
securityCheckStatus: file.blocked ? 'blocked' : 'verified',
|
||||||
|
previewStatus: '',
|
||||||
|
},
|
||||||
|
parentType,
|
||||||
|
parentId: entry.parent.folderId ?? entry.parent.ownerId ?? '',
|
||||||
|
at: { ...root.at, folders },
|
||||||
|
source: 'file-manager',
|
||||||
|
fsPath: entry.path,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
async function listFiles(
|
async function listFiles(
|
||||||
client: SchulcloudClient,
|
client: SchulcloudClient,
|
||||||
schoolId: string,
|
schoolId: string,
|
||||||
|
|||||||
@@ -140,7 +140,10 @@ function hasTextLayer(bytes: Buffer): boolean {
|
|||||||
|
|
||||||
async function extractPdf(bytes: Buffer): Promise<string> {
|
async function extractPdf(bytes: Buffer): Promise<string> {
|
||||||
const { extractText, getDocumentProxy } = await import('unpdf');
|
const { extractText, getDocumentProxy } = await import('unpdf');
|
||||||
const document = await getDocumentProxy(new Uint8Array(bytes));
|
// verbosity 0 = errors only. pdf.js otherwise prints "Warning: TT: undefined
|
||||||
|
// function" for every font hint it skips — harmless, but a crawl of the file
|
||||||
|
// manager extracts hundreds of PDFs, and that buries the log in noise.
|
||||||
|
const document = await getDocumentProxy(new Uint8Array(bytes), { verbosity: 0 });
|
||||||
const { text } = await extractText(document, { mergePages: true });
|
const { text } = await extractText(document, { mergePages: true });
|
||||||
return Array.isArray(text) ? text.join('\n\n') : text;
|
return Array.isArray(text) ? text.join('\n\n') : text;
|
||||||
}
|
}
|
||||||
|
|||||||
500
src/core/legacy-files.ts
Normal file
500
src/core/legacy-files.ts
Normal file
@@ -0,0 +1,500 @@
|
|||||||
|
import type { DownloadedFile, SchulcloudClient } from './client.ts';
|
||||||
|
import { decodeEntities } from './text.ts';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The "Dateien" file manager — Persönliche Dateien, Kurs-Dateien, Team-Dateien
|
||||||
|
* and Geteilte Dateien — as one read-only filesystem.
|
||||||
|
*
|
||||||
|
* This is the legacy file system, a different store from files-storage
|
||||||
|
* (`/api/v3/file`), which holds board, topic and task attachments. The two do
|
||||||
|
* not overlap: asking files-storage for a course's files answers 0 for a course
|
||||||
|
* whose file manager holds dozens, and on the account this was built for 21 of
|
||||||
|
* 26 courses keep material here — some teachers use nothing else.
|
||||||
|
*
|
||||||
|
* Its Feathers service is not in the public ingress, so it is reached through
|
||||||
|
* the legacy client: server-rendered listing pages, parsed here, and one JSON
|
||||||
|
* route for pre-signed downloads (see the client). Two server quirks shape the
|
||||||
|
* design:
|
||||||
|
*
|
||||||
|
* - `GET /files/permittedDirectories/` looks like the obvious JSON source for
|
||||||
|
* the directory tree, but its query matches course folders on
|
||||||
|
* `refOwnerModel: 'courses'` while the records say `'course'`. It lists every
|
||||||
|
* course with **no** folders in any of them. Listings are the only complete
|
||||||
|
* view, which is also exactly what the file manager itself shows.
|
||||||
|
* - `GET /files/search/` runs an unindexed regex over every file record and
|
||||||
|
* times out (504) on the live instance, so finding is done by walking.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type FileArea = 'my' | 'courses' | 'teams' | 'shared';
|
||||||
|
|
||||||
|
interface AreaInfo {
|
||||||
|
area: FileArea;
|
||||||
|
label: string;
|
||||||
|
/** Accepted spellings of the first path segment, compared lower-cased. */
|
||||||
|
aliases: string[];
|
||||||
|
summary: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const FILE_AREAS: AreaInfo[] = [
|
||||||
|
{
|
||||||
|
area: 'my',
|
||||||
|
label: 'Persönliche Dateien',
|
||||||
|
aliases: ['my', 'persönliche dateien', 'persoenliche dateien', 'personal', 'meine dateien'],
|
||||||
|
summary: 'your own files',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
area: 'courses',
|
||||||
|
label: 'Kurs-Dateien',
|
||||||
|
aliases: ['courses', 'kurs-dateien', 'meine kurs-dateien', 'kursdateien', 'kurse'],
|
||||||
|
summary: 'one folder per course, holding what its teachers uploaded',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
area: 'teams',
|
||||||
|
label: 'Team-Dateien',
|
||||||
|
aliases: ['teams', 'team-dateien', 'meine team-dateien', 'teamdateien'],
|
||||||
|
summary: 'one folder per team',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
area: 'shared',
|
||||||
|
label: 'Geteilte Dateien',
|
||||||
|
aliases: ['shared', 'geteilte dateien', 'mit mir geteilt'],
|
||||||
|
summary: 'files other people shared with you, read-only and flat',
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
export function areaInfo(area: FileArea): AreaInfo {
|
||||||
|
return FILE_AREAS.find((entry) => entry.area === area) as AreaInfo;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FmDirectory {
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FmFile {
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
/** Bytes, as the listing reports it. */
|
||||||
|
size: number;
|
||||||
|
/** Absent for a blocked file: the page withholds its viewer attributes. */
|
||||||
|
mimeType?: string;
|
||||||
|
/** Rejected by the instance's virus scanner; it cannot be downloaded. */
|
||||||
|
blocked: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FmListing {
|
||||||
|
directories: FmDirectory[];
|
||||||
|
files: FmFile[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A listing page's markup changed shape; never report that as an empty folder. */
|
||||||
|
export class FileManagerMarkupError extends Error {}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parses one file-manager page. Exported for testing.
|
||||||
|
*
|
||||||
|
* Anchored on what the templates (`files/files.hbs`, `files/files-grid.hbs`)
|
||||||
|
* emit for the page's own scripts — `data-folder-id`, `data-file-id`,
|
||||||
|
* `data-file-name`, `data-file-size` — rather than on layout classes.
|
||||||
|
*
|
||||||
|
* Throws rather than returning an empty listing when the page is not a file
|
||||||
|
* manager page at all: "0 files" is precisely the wrong answer this module
|
||||||
|
* exists to fix, and a markup change must not quietly reproduce it.
|
||||||
|
*/
|
||||||
|
export function parseFileListing(html: string): FmListing {
|
||||||
|
if (!/class="route-files"/.test(html)) {
|
||||||
|
throw new FileManagerMarkupError('the page is not a file-manager listing (markup changed, or not logged in)');
|
||||||
|
}
|
||||||
|
|
||||||
|
const directories: FmDirectory[] = [];
|
||||||
|
const folderPattern = /<button\b([^>]*\bopenfolder\b[^>]*)>([\s\S]*?)<\/button>/g;
|
||||||
|
for (const match of html.matchAll(folderPattern)) {
|
||||||
|
const id = /data-folder-id="([0-9a-f]{24})"/i.exec(match[1] ?? '')?.[1];
|
||||||
|
if (!id) continue;
|
||||||
|
// The name is emitted unescaped (`{{{stripOnlyScript name}}}`) inside the
|
||||||
|
// title element, after an icon; strip the tags, then decode what is left.
|
||||||
|
const title = /<strong\b[^>]*card-title-directory[^>]*>([\s\S]*?)<\/strong>/.exec(match[2] ?? '')?.[1] ?? '';
|
||||||
|
const name = decodeEntities(title.replace(/<[^>]+>/g, '')).replace(/\s+/g, ' ').trim();
|
||||||
|
directories.push({ id, name: name || id });
|
||||||
|
}
|
||||||
|
|
||||||
|
const files: FmFile[] = [];
|
||||||
|
const cardPattern = /<div\b[^>]*\bclass="card file\b([^"]*)"([^>]*)>/g;
|
||||||
|
const cards = [...html.matchAll(cardPattern)];
|
||||||
|
cards.forEach((match, index) => {
|
||||||
|
const attributes = match[2] ?? '';
|
||||||
|
const id = /data-file-id="([0-9a-f]{24})"/i.exec(attributes)?.[1];
|
||||||
|
if (!id) return;
|
||||||
|
const name = decodeEntities(/data-file-name="([^"]*)"/.exec(attributes)?.[1] ?? '');
|
||||||
|
const size = Number(/data-file-size="(\d*)"/.exec(attributes)?.[1] ?? '');
|
||||||
|
// The viewer attributes sit further inside this card; look no further
|
||||||
|
// than the next card, so one file can never borrow another's type.
|
||||||
|
const start = (match.index ?? 0) + match[0].length;
|
||||||
|
const end = cards[index + 1]?.index ?? html.length;
|
||||||
|
const mimeType = /data-file-viewer-type="([^"]*)"/.exec(html.slice(start, end))?.[1];
|
||||||
|
files.push({
|
||||||
|
id,
|
||||||
|
name: name || id,
|
||||||
|
size: Number.isFinite(size) ? size : 0,
|
||||||
|
mimeType: mimeType ? decodeEntities(mimeType) : undefined,
|
||||||
|
blocked: /\bbtn-file-danger\b/.test(match[1] ?? ''),
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
return { directories, files };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Where a directory is. `area` absent is the root; `ownerId` is the course or
|
||||||
|
* team (absent for `my` and `shared`); `folderId` absent is the owner's top.
|
||||||
|
*/
|
||||||
|
export interface DirectoryRef {
|
||||||
|
area?: FileArea;
|
||||||
|
ownerId?: string;
|
||||||
|
folderId?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type FsNode =
|
||||||
|
| { kind: 'directory'; path: string; ref: DirectoryRef; name: string }
|
||||||
|
| { kind: 'file'; path: string; parent: DirectoryRef; file: FmFile };
|
||||||
|
|
||||||
|
export type FsErrorCode = 'not_found' | 'ambiguous' | 'not_a_directory' | 'not_a_file' | 'not_navigable';
|
||||||
|
|
||||||
|
export class FsError extends Error {
|
||||||
|
readonly code: FsErrorCode;
|
||||||
|
constructor(code: FsErrorCode, message: string) {
|
||||||
|
super(message);
|
||||||
|
this.code = code;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The legacy page that lists a directory. */
|
||||||
|
export function pageFor(ref: DirectoryRef): string | undefined {
|
||||||
|
if (!ref.area) return undefined;
|
||||||
|
if (ref.area === 'shared') return '/files/shared/';
|
||||||
|
if (ref.area === 'my') return ref.folderId ? `/files/my/${ref.folderId}` : '/files/my/';
|
||||||
|
if (!ref.ownerId) return `/files/${ref.area}/`;
|
||||||
|
return ref.folderId ? `/files/${ref.area}/${ref.ownerId}/${ref.folderId}` : `/files/${ref.area}/${ref.ownerId}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The reference for a subdirectory found in `parent`'s listing. */
|
||||||
|
export function childRef(parent: DirectoryRef, directory: FmDirectory): DirectoryRef {
|
||||||
|
if (!parent.area) return { area: directory.id as FileArea };
|
||||||
|
if (parent.area === 'my') return { area: 'my', folderId: directory.id };
|
||||||
|
if (parent.area === 'shared') {
|
||||||
|
// The file manager has no route that opens a shared folder — its owner is
|
||||||
|
// someone else, and every listing route is scoped to an owner. The UI's
|
||||||
|
// own link to one is a 404.
|
||||||
|
throw new FsError(
|
||||||
|
'not_navigable',
|
||||||
|
`"${directory.name}" is a folder someone shared with you. The file manager cannot open shared folders ` +
|
||||||
|
'(not even in the browser); ask for the files themselves to be shared, or find them in the owning course.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (!parent.ownerId) return { area: parent.area, ownerId: directory.id };
|
||||||
|
return { area: parent.area, ownerId: parent.ownerId, folderId: directory.id };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Splits a path into raw segments. Empty segments (`//`, trailing `/`) drop out. */
|
||||||
|
export function splitPath(path: string): string[] {
|
||||||
|
return path
|
||||||
|
.split('/')
|
||||||
|
.map((segment) => segment.trim())
|
||||||
|
.filter((segment) => segment.length > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Joins names into a display path. Names keep any `/` they contain; see `resolve`. */
|
||||||
|
export function joinPath(parent: string, name: string): string {
|
||||||
|
return `${parent === '/' ? '' : parent}/${name}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalise(value: string): string {
|
||||||
|
return value.normalize('NFC').replace(/\s+/g, ' ').trim().toLocaleLowerCase('de');
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface WalkEntry {
|
||||||
|
path: string;
|
||||||
|
/** The listed directory this entry came from; renderers group on it. */
|
||||||
|
parentPath: string;
|
||||||
|
depth: number;
|
||||||
|
parent: DirectoryRef;
|
||||||
|
directory?: { ref: DirectoryRef; name: string; id: string };
|
||||||
|
file?: FmFile;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface WalkResult {
|
||||||
|
entries: WalkEntry[];
|
||||||
|
/** Directories that were listed. */
|
||||||
|
visited: number;
|
||||||
|
/** Set when the budget ran out before the walk finished. */
|
||||||
|
truncated: boolean;
|
||||||
|
failures: { path: string; reason: string }[];
|
||||||
|
}
|
||||||
|
|
||||||
|
interface CachedListing {
|
||||||
|
at: number;
|
||||||
|
listing: Promise<FmListing>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A listing is reused this long: long enough for ls→read, short enough to stay live. */
|
||||||
|
const LISTING_TTL_MS = 60_000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The file manager as a tree of paths:
|
||||||
|
*
|
||||||
|
* / the four areas
|
||||||
|
* /my/… Persönliche Dateien
|
||||||
|
* /courses/<course>/… Kurs-Dateien
|
||||||
|
* /teams/<team>/… Team-Dateien
|
||||||
|
* /shared/… Geteilte Dateien (flat)
|
||||||
|
*
|
||||||
|
* Names are the file manager's own. Because course names contain `/` in real
|
||||||
|
* data ("LF07 - FIA24A/B - Sb/Ha"), a path is not split naively: resolution
|
||||||
|
* tries joining consecutive segments into one name and backtracks when a
|
||||||
|
* shorter reading leads nowhere. Any segment may also be an id instead of a
|
||||||
|
* name, which is always unambiguous and is what the listings print alongside.
|
||||||
|
*/
|
||||||
|
export class FileManager {
|
||||||
|
private readonly client: SchulcloudClient;
|
||||||
|
private readonly cache = new Map<string, CachedListing>();
|
||||||
|
|
||||||
|
constructor(client: SchulcloudClient) {
|
||||||
|
this.client = client;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Lists one directory. The root is synthetic and costs nothing. */
|
||||||
|
async list(ref: DirectoryRef): Promise<FmListing> {
|
||||||
|
const page = pageFor(ref);
|
||||||
|
if (!page) {
|
||||||
|
return { directories: FILE_AREAS.map((entry) => ({ id: entry.area, name: entry.area })), files: [] };
|
||||||
|
}
|
||||||
|
|
||||||
|
const cached = this.cache.get(page);
|
||||||
|
if (cached && Date.now() - cached.at < LISTING_TTL_MS) return cached.listing;
|
||||||
|
|
||||||
|
const listing = this.client.getFileManagerPage(page).then(parseFileListing);
|
||||||
|
this.cache.set(page, { at: Date.now(), listing });
|
||||||
|
// A failure must not be served from the cache for the next minute.
|
||||||
|
listing.catch(() => this.cache.delete(page));
|
||||||
|
return listing;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Resolves a path to a directory or a file. */
|
||||||
|
async resolve(path: string): Promise<FsNode> {
|
||||||
|
const segments = splitPath(path);
|
||||||
|
if (segments.length === 0) return { kind: 'directory', path: '/', ref: {}, name: '/' };
|
||||||
|
|
||||||
|
const first = segments[0] as string;
|
||||||
|
const area = FILE_AREAS.find((entry) => entry.aliases.includes(normalise(first)) || entry.area === first);
|
||||||
|
if (!area) {
|
||||||
|
throw new FsError(
|
||||||
|
'not_found',
|
||||||
|
`"${first}" is not a file area. The root holds ${FILE_AREAS.map((entry) => `/${entry.area} (${entry.label})`).join(', ')}.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const ref: DirectoryRef = { area: area.area };
|
||||||
|
const found = await this.descend(ref, `/${area.area}`, segments.slice(1));
|
||||||
|
return found;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async descend(ref: DirectoryRef, path: string, rest: string[]): Promise<FsNode> {
|
||||||
|
if (rest.length === 0) return { kind: 'directory', path, ref, name: path.split('/').pop() || '/' };
|
||||||
|
|
||||||
|
const listing = await this.list(ref);
|
||||||
|
const readings = candidateReadings(listing, rest);
|
||||||
|
if (readings.length === 0) throw notFound(listing, rest[0] as string, path);
|
||||||
|
|
||||||
|
let lastError: FsError | undefined;
|
||||||
|
for (const reading of readings) {
|
||||||
|
if (reading.matches.length > 1) {
|
||||||
|
const options = reading.matches.map((match) => `"${match.entry.name}" (\`${match.entry.id}\`)`).join(', ');
|
||||||
|
throw new FsError(
|
||||||
|
'ambiguous',
|
||||||
|
`${path} holds more than one entry named "${reading.name}": ${options}. Use the id as that path segment instead.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const match = reading.matches[0] as { kind: 'directory'; entry: FmDirectory } | { kind: 'file'; entry: FmFile };
|
||||||
|
const remaining = rest.slice(reading.consumed);
|
||||||
|
const nextPath = joinPath(path, match.entry.name);
|
||||||
|
|
||||||
|
if (match.kind === 'file') {
|
||||||
|
if (remaining.length === 0) return { kind: 'file', path: nextPath, parent: ref, file: match.entry };
|
||||||
|
lastError = new FsError('not_a_directory', `${nextPath} is a file, not a folder.`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
return await this.descend(childRef(ref, match.entry), nextPath, remaining);
|
||||||
|
} catch (error) {
|
||||||
|
// A shorter reading of a name containing "/" can lead nowhere while a
|
||||||
|
// longer one resolves; only give up once every reading has failed.
|
||||||
|
if (error instanceof FsError && error.code !== 'ambiguous' && error.code !== 'not_navigable') {
|
||||||
|
lastError = error;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw lastError ?? notFound(listing, rest[0] as string, path);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Walks a directory breadth-first, listing at most `maxDirectories` of them.
|
||||||
|
*
|
||||||
|
* Every listing is one page fetch, so the budget is the cost. A folder that
|
||||||
|
* cannot be read is recorded and skipped, never silently dropped.
|
||||||
|
*/
|
||||||
|
async walk(
|
||||||
|
start: { path: string; ref: DirectoryRef },
|
||||||
|
options: { maxDepth: number; maxDirectories: number; concurrency?: number },
|
||||||
|
): Promise<WalkResult> {
|
||||||
|
const entries: WalkEntry[] = [];
|
||||||
|
const failures: { path: string; reason: string }[] = [];
|
||||||
|
let visited = 0;
|
||||||
|
let truncated = false;
|
||||||
|
|
||||||
|
let frontier: { path: string; ref: DirectoryRef; depth: number }[] = [{ ...start, depth: 0 }];
|
||||||
|
while (frontier.length > 0) {
|
||||||
|
const next: typeof frontier = [];
|
||||||
|
const batch = frontier;
|
||||||
|
frontier = [];
|
||||||
|
|
||||||
|
const concurrency = Math.max(1, options.concurrency ?? 3);
|
||||||
|
for (let i = 0; i < batch.length; i += concurrency) {
|
||||||
|
const slice = batch.slice(i, i + concurrency);
|
||||||
|
await Promise.all(
|
||||||
|
slice.map(async (node) => {
|
||||||
|
// The root is synthetic and costs no request, so it does not count.
|
||||||
|
if (pageFor(node.ref)) {
|
||||||
|
if (visited >= options.maxDirectories) {
|
||||||
|
truncated = true;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
visited++;
|
||||||
|
}
|
||||||
|
let listing: FmListing;
|
||||||
|
try {
|
||||||
|
listing = await this.list(node.ref);
|
||||||
|
} catch (error) {
|
||||||
|
failures.push({ path: node.path, reason: error instanceof Error ? error.message : String(error) });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// The root's areas keep their defined order; real folders sort by name.
|
||||||
|
const directories = pageFor(node.ref) ? sortByName(listing.directories) : listing.directories;
|
||||||
|
for (const directory of directories) {
|
||||||
|
const path = joinPath(node.path, directory.name);
|
||||||
|
let ref: DirectoryRef | undefined;
|
||||||
|
try {
|
||||||
|
ref = childRef(node.ref, directory);
|
||||||
|
} catch {
|
||||||
|
ref = undefined; // shared folders: listed, never opened
|
||||||
|
}
|
||||||
|
entries.push({
|
||||||
|
path,
|
||||||
|
parentPath: node.path,
|
||||||
|
depth: node.depth + 1,
|
||||||
|
parent: node.ref,
|
||||||
|
directory: { ref: ref ?? node.ref, name: directory.name, id: directory.id },
|
||||||
|
});
|
||||||
|
if (ref && node.depth + 1 < options.maxDepth) next.push({ path, ref, depth: node.depth + 1 });
|
||||||
|
}
|
||||||
|
for (const file of sortByName(listing.files)) {
|
||||||
|
entries.push({
|
||||||
|
path: joinPath(node.path, file.name),
|
||||||
|
parentPath: node.path,
|
||||||
|
depth: node.depth + 1,
|
||||||
|
parent: node.ref,
|
||||||
|
file,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
frontier = next;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Unsorted across directories on purpose: sorting whole path strings
|
||||||
|
// interleaves a folder's children with a sibling that shares its prefix
|
||||||
|
// ("Sub/…" against "Sub - Kopie"). Renderers group on `parentPath`.
|
||||||
|
return { entries, visited, truncated, failures };
|
||||||
|
}
|
||||||
|
|
||||||
|
download(file: Pick<FmFile, 'id' | 'name'>): Promise<DownloadedFile> {
|
||||||
|
return this.client.downloadFileManagerFile(file.id, file.name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type Match = { kind: 'directory'; entry: FmDirectory } | { kind: 'file'; entry: FmFile };
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Every way the next path segments can name an entry, shortest first.
|
||||||
|
*
|
||||||
|
* `rest[0]`, then `rest[0]/rest[1]`, and so on — so a course called
|
||||||
|
* "LF07 - FIA24A/B - Sb/Ha" resolves even when typed plainly. An id segment
|
||||||
|
* matches too. Exact names are preferred over case-insensitive ones.
|
||||||
|
*/
|
||||||
|
function candidateReadings(listing: FmListing, rest: string[]): { name: string; consumed: number; matches: Match[] }[] {
|
||||||
|
const all: Match[] = [
|
||||||
|
...listing.directories.map((entry) => ({ kind: 'directory' as const, entry })),
|
||||||
|
...listing.files.map((entry) => ({ kind: 'file' as const, entry })),
|
||||||
|
];
|
||||||
|
|
||||||
|
const first = rest[0] as string;
|
||||||
|
if (/^[0-9a-f]{24}$/i.test(first)) {
|
||||||
|
const byId = all.filter((match) => match.entry.id.toLowerCase() === first.toLowerCase());
|
||||||
|
if (byId.length > 0) return [{ name: first, consumed: 1, matches: byId }];
|
||||||
|
}
|
||||||
|
|
||||||
|
const readings: { name: string; consumed: number; matches: Match[] }[] = [];
|
||||||
|
for (const strict of [true, false]) {
|
||||||
|
for (let take = 1; take <= rest.length; take++) {
|
||||||
|
const name = rest.slice(0, take).join('/');
|
||||||
|
const matches = all.filter((match) =>
|
||||||
|
strict ? match.entry.name.trim() === name : normalise(match.entry.name) === normalise(name),
|
||||||
|
);
|
||||||
|
if (matches.length > 0 && !readings.some((reading) => reading.consumed === take)) {
|
||||||
|
readings.push({ name, consumed: take, matches });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (readings.length > 0) break;
|
||||||
|
}
|
||||||
|
return readings;
|
||||||
|
}
|
||||||
|
|
||||||
|
function notFound(listing: FmListing, segment: string, path: string): FsError {
|
||||||
|
const names = [...listing.directories.map((entry) => `${entry.name}/`), ...listing.files.map((entry) => entry.name)];
|
||||||
|
const needle = normalise(segment);
|
||||||
|
const close = names.filter((name) => normalise(name).includes(needle) || needle.includes(normalise(name).replace(/\/$/, '')));
|
||||||
|
const hint =
|
||||||
|
close.length > 0
|
||||||
|
? ` Did you mean: ${close.slice(0, 5).map((name) => `"${name}"`).join(', ')}?`
|
||||||
|
: names.length > 0
|
||||||
|
? ` It holds ${names.length} entr${names.length === 1 ? 'y' : 'ies'}; list it with fs_list.`
|
||||||
|
: ' It is empty.';
|
||||||
|
return new FsError('not_found', `No "${segment}" in ${path}.${hint}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function sortByName<T extends { name: string }>(items: T[]): T[] {
|
||||||
|
return [...items].sort((a, b) => compareNames(a.name, b.name));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Name order as a person expects it: German collation, "Blatt 2" before "Blatt 10". */
|
||||||
|
export function compareNames(a: string, b: string): number {
|
||||||
|
return a.localeCompare(b, 'de', { numeric: true, sensitivity: 'base' });
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Case-insensitive substring; or, when the pattern uses * or ?, a glob that
|
||||||
|
* must match the whole name, as `find -name` does. Exported for testing.
|
||||||
|
*/
|
||||||
|
export function nameMatcher(pattern: string): (name: string) => boolean {
|
||||||
|
const needle = pattern.normalize('NFC').toLocaleLowerCase('de');
|
||||||
|
if (!/[*?]/.test(needle)) return (name) => name.normalize('NFC').toLocaleLowerCase('de').includes(needle);
|
||||||
|
const source = needle
|
||||||
|
.split('')
|
||||||
|
.map((char) => (char === '*' ? '.*' : char === '?' ? '.' : char.replace(/[.+^${}()|[\]\\]/g, '\\$&')))
|
||||||
|
.join('');
|
||||||
|
const regex = new RegExp(`^${source}$`, 'i');
|
||||||
|
return (name) => regex.test(name.normalize('NFC'));
|
||||||
|
}
|
||||||
@@ -74,7 +74,7 @@ export function safeComponent(raw: string, fallback = 'untitled'): string {
|
|||||||
* same card, which the API permits.
|
* same card, which the API permits.
|
||||||
*/
|
*/
|
||||||
export function mirrorPath(at: Breadcrumb, fileName: string, fileId: string): string {
|
export function mirrorPath(at: Breadcrumb, fileName: string, fileId: string): string {
|
||||||
const parts = [at.courseTitle, at.containerTitle, at.cardTitle]
|
const parts = [at.courseTitle, at.containerTitle, ...(at.folders ?? []), at.cardTitle]
|
||||||
.filter((part): part is string => Boolean(part && part.trim()))
|
.filter((part): part is string => Boolean(part && part.trim()))
|
||||||
.map((part) => safeComponent(part));
|
.map((part) => safeComponent(part));
|
||||||
|
|
||||||
|
|||||||
201
src/http/api.ts
201
src/http/api.ts
@@ -2,6 +2,16 @@ import { createReadStream } from 'node:fs';
|
|||||||
import { stat } from 'node:fs/promises';
|
import { stat } from 'node:fs/promises';
|
||||||
import { Readable } from 'node:stream';
|
import { Readable } from 'node:stream';
|
||||||
import express, { type Request, type Response, type Router } from 'express';
|
import express, { type Request, type Response, type Router } from 'express';
|
||||||
|
import { SchulcloudApiError } from '../core/client.ts';
|
||||||
|
import {
|
||||||
|
compareNames,
|
||||||
|
FileManagerMarkupError,
|
||||||
|
FsError,
|
||||||
|
nameMatcher,
|
||||||
|
type FmFile,
|
||||||
|
type FsErrorCode,
|
||||||
|
type WalkEntry,
|
||||||
|
} from '../core/legacy-files.ts';
|
||||||
import { resolveWithin } from '../core/paths.ts';
|
import { resolveWithin } from '../core/paths.ts';
|
||||||
import type { Services } from '../services.ts';
|
import type { Services } from '../services.ts';
|
||||||
|
|
||||||
@@ -64,6 +74,113 @@ export function createApiRouter(services: Services): Router {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// --- the file manager ("Dateien"), as a filesystem ----------------------
|
||||||
|
//
|
||||||
|
// Live, not from the index: these answer what the file manager holds now, and
|
||||||
|
// need no database. Paths are the same ones the MCP fs_* tools print.
|
||||||
|
|
||||||
|
router.get('/fs/list', async (req: Request, res: Response) => {
|
||||||
|
try {
|
||||||
|
const node = await services.files.resolve(stringParam(req.query.path) ?? '/');
|
||||||
|
if (node.kind === 'file') return res.json({ path: node.path, kind: 'file', file: node.file });
|
||||||
|
const listing = await services.files.list(node.ref);
|
||||||
|
return res.json({
|
||||||
|
path: node.path,
|
||||||
|
kind: 'directory',
|
||||||
|
area: node.ref.area ?? null,
|
||||||
|
directories: listing.directories.map((entry) => ({ ...entry, path: childPath(node.path, entry.name) })),
|
||||||
|
files: listing.files.map((entry) => ({ ...entry, path: childPath(node.path, entry.name) })),
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
return fsFail(res, error, 'fs list');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
router.get('/fs/tree', async (req: Request, res: Response) => {
|
||||||
|
try {
|
||||||
|
const node = await services.files.resolve(stringParam(req.query.path) ?? '/');
|
||||||
|
if (node.kind === 'file') return res.json({ path: node.path, kind: 'file', file: node.file });
|
||||||
|
const result = await services.files.walk(node, {
|
||||||
|
maxDepth: boundedInt(req.query.depth, 3, 1, 12),
|
||||||
|
maxDirectories: boundedInt(req.query.maxFolders, 200, 1, 1000),
|
||||||
|
});
|
||||||
|
return res.json({
|
||||||
|
path: node.path,
|
||||||
|
kind: 'directory',
|
||||||
|
entries: result.entries.map(treeEntry),
|
||||||
|
visited: result.visited,
|
||||||
|
truncated: result.truncated,
|
||||||
|
failures: result.failures,
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
return fsFail(res, error, 'fs tree');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
router.get('/fs/find', async (req: Request, res: Response) => {
|
||||||
|
const name = stringParam(req.query.name);
|
||||||
|
if (!name) return res.status(400).json({ error: 'bad_request', message: 'Give name.' });
|
||||||
|
try {
|
||||||
|
const node = await services.files.resolve(stringParam(req.query.path) ?? '/');
|
||||||
|
if (node.kind === 'file') return res.json({ path: node.path, kind: 'file', matches: [] });
|
||||||
|
const type = stringParam(req.query.type) ?? 'any';
|
||||||
|
const matches = nameMatcher(name);
|
||||||
|
const result = await services.files.walk(node, {
|
||||||
|
maxDepth: 12,
|
||||||
|
maxDirectories: boundedInt(req.query.maxFolders, 400, 1, 1000),
|
||||||
|
});
|
||||||
|
return res.json({
|
||||||
|
path: node.path,
|
||||||
|
kind: 'directory',
|
||||||
|
matches: result.entries
|
||||||
|
.filter((entry) => (type === 'file' ? entry.file : type === 'folder' ? entry.directory : true))
|
||||||
|
.filter((entry) => matches((entry.file ?? entry.directory)?.name ?? ''))
|
||||||
|
.sort((a, b) => compareNames(a.path, b.path))
|
||||||
|
.map(treeEntry),
|
||||||
|
visited: result.visited,
|
||||||
|
truncated: result.truncated,
|
||||||
|
failures: result.failures,
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
return fsFail(res, error, 'fs find');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
router.get('/fs/file', async (req: Request, res: Response) => {
|
||||||
|
try {
|
||||||
|
const path = stringParam(req.query.path);
|
||||||
|
const id = stringParam(req.query.id);
|
||||||
|
let file: Pick<FmFile, 'id' | 'name'> & Partial<FmFile>;
|
||||||
|
if (path) {
|
||||||
|
const node = await services.files.resolve(path);
|
||||||
|
if (node.kind !== 'file') return res.status(400).json({ error: 'not_a_file', message: `${node.path} is a folder.` });
|
||||||
|
file = node.file;
|
||||||
|
} else if (id && /^[0-9a-f]{24}$/i.test(id)) {
|
||||||
|
file = { id, name: stringParam(req.query.name) ?? id };
|
||||||
|
} else {
|
||||||
|
return res.status(400).json({ error: 'bad_request', message: 'Give path, or id (and name).' });
|
||||||
|
}
|
||||||
|
if (file.blocked) {
|
||||||
|
return res.status(403).json({ error: 'blocked', message: 'The instance virus scanner blocked this file.' });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Streamed straight through rather than buffered: the CLI uses this for
|
||||||
|
// whole folders, and videos routinely exceed any sensible in-memory cap.
|
||||||
|
const signed = await services.client.getFileManagerSignedUrl(file.id, file.name);
|
||||||
|
const upstream = await services.client.openSignedUrl(signed);
|
||||||
|
if (!upstream.body) return res.status(502).json({ error: 'upstream_failed', message: 'empty response' });
|
||||||
|
|
||||||
|
res.setHeader('Content-Type', file.mimeType || upstream.headers.get('content-type') || 'application/octet-stream');
|
||||||
|
res.setHeader('Content-Disposition', contentDisposition(file.name));
|
||||||
|
const length = upstream.headers.get('content-length');
|
||||||
|
if (length) res.setHeader('Content-Length', length);
|
||||||
|
res.setHeader('X-Schulcloud-Source', 'file-manager');
|
||||||
|
Readable.fromWeb(upstream.body as never).pipe(res);
|
||||||
|
} catch (error) {
|
||||||
|
return fsFail(res, error, 'fs file');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Streams one file. Served from the local mirror when present; otherwise
|
* Streams one file. Served from the local mirror when present; otherwise
|
||||||
* proxied live, which is what keeps files too large to mirror reachable.
|
* proxied live, which is what keeps files too large to mirror reachable.
|
||||||
@@ -91,6 +208,8 @@ export function createApiRouter(services: Services): Router {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
const known = await services.store.fileSource(fileId);
|
||||||
|
if (known?.source === 'file-manager') return await proxyFileManager(services, fileId, known, res);
|
||||||
return await proxyLive(services, fileId, res);
|
return await proxyLive(services, fileId, res);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return fail(res, error, 'file');
|
return fail(res, error, 'file');
|
||||||
@@ -101,6 +220,27 @@ export function createApiRouter(services: Services): Router {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** Falls back to Schulcloud for anything not in the mirror, streaming through. */
|
/** Falls back to Schulcloud for anything not in the mirror, streaming through. */
|
||||||
|
/** Streams a file-manager file live, via its pre-signed URL; no credentials leave for the storage host. */
|
||||||
|
async function proxyFileManager(
|
||||||
|
services: Services,
|
||||||
|
fileId: string,
|
||||||
|
known: { name: string; mimeType: string; size: number },
|
||||||
|
res: Response,
|
||||||
|
): Promise<void> {
|
||||||
|
const signed = await services.client.getFileManagerSignedUrl(fileId, known.name);
|
||||||
|
const upstream = await services.client.openSignedUrl(signed);
|
||||||
|
if (!upstream.body) {
|
||||||
|
res.status(502).json({ error: 'upstream_failed', message: 'empty response' });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
res.setHeader('Content-Type', known.mimeType || 'application/octet-stream');
|
||||||
|
res.setHeader('Content-Disposition', contentDisposition(known.name));
|
||||||
|
const length = upstream.headers.get('content-length');
|
||||||
|
if (length) res.setHeader('Content-Length', length);
|
||||||
|
res.setHeader('X-Schulcloud-Source', 'live');
|
||||||
|
Readable.fromWeb(upstream.body as never).pipe(res);
|
||||||
|
}
|
||||||
|
|
||||||
async function proxyLive(services: Services, fileId: string, res: Response): Promise<void> {
|
async function proxyLive(services: Services, fileId: string, res: Response): Promise<void> {
|
||||||
const record = await services.client.getFileRecord(fileId);
|
const record = await services.client.getFileRecord(fileId);
|
||||||
if (record.securityCheckStatus === 'blocked') {
|
if (record.securityCheckStatus === 'blocked') {
|
||||||
@@ -143,3 +283,64 @@ function fail(res: Response, error: unknown, what: string): void {
|
|||||||
console.error(`[schulcloud-mcp] ${what} failed:`, error);
|
console.error(`[schulcloud-mcp] ${what} failed:`, error);
|
||||||
if (!res.headersSent) res.status(500).json({ error: 'internal_error' });
|
if (!res.headersSent) res.status(500).json({ error: 'internal_error' });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function stringParam(value: unknown): string | undefined {
|
||||||
|
const first = Array.isArray(value) ? value[0] : value;
|
||||||
|
return typeof first === 'string' && first.length > 0 ? first : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
function boundedInt(value: unknown, fallback: number, min: number, max: number): number {
|
||||||
|
const parsed = Number.parseInt(stringParam(value) ?? '', 10);
|
||||||
|
return Number.isFinite(parsed) ? Math.min(max, Math.max(min, parsed)) : fallback;
|
||||||
|
}
|
||||||
|
|
||||||
|
function childPath(parent: string, name: string): string {
|
||||||
|
return `${parent === '/' ? '' : parent}/${name}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A walk entry as JSON; `name` travels separately because names may contain "/". */
|
||||||
|
function treeEntry(entry: WalkEntry) {
|
||||||
|
if (entry.directory) {
|
||||||
|
return { type: 'directory', path: entry.path, parentPath: entry.parentPath, depth: entry.depth, id: entry.directory.id, name: entry.directory.name };
|
||||||
|
}
|
||||||
|
const file = entry.file as FmFile;
|
||||||
|
return {
|
||||||
|
type: 'file',
|
||||||
|
path: entry.path,
|
||||||
|
parentPath: entry.parentPath,
|
||||||
|
depth: entry.depth,
|
||||||
|
id: file.id,
|
||||||
|
name: file.name,
|
||||||
|
size: file.size,
|
||||||
|
mimeType: file.mimeType ?? null,
|
||||||
|
blocked: file.blocked,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const FS_STATUS: Record<FsErrorCode, number> = {
|
||||||
|
not_found: 404,
|
||||||
|
ambiguous: 409,
|
||||||
|
not_a_directory: 400,
|
||||||
|
not_a_file: 400,
|
||||||
|
not_navigable: 422,
|
||||||
|
};
|
||||||
|
|
||||||
|
function fsFail(res: Response, error: unknown, what: string): void {
|
||||||
|
if (res.headersSent) return;
|
||||||
|
if (error instanceof FsError) {
|
||||||
|
res.status(FS_STATUS[error.code]).json({ error: error.code, message: error.message });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (error instanceof FileManagerMarkupError) {
|
||||||
|
res.status(502).json({ error: 'markup_changed', message: error.message });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (error instanceof SchulcloudApiError) {
|
||||||
|
// 401 upstream is the Pi's session, not the caller's token: say which.
|
||||||
|
const status = error.status === 401 ? 502 : error.status === 403 || error.status === 404 ? error.status : 502;
|
||||||
|
const message = error.status === 401 ? 'The Schulcloud session has expired on the server.' : error.message;
|
||||||
|
res.status(status).json({ error: 'upstream_failed', message });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
fail(res, error, what);
|
||||||
|
}
|
||||||
|
|||||||
@@ -117,6 +117,7 @@ export class Indexer {
|
|||||||
includeLessonContents: true,
|
includeLessonContents: true,
|
||||||
includeFiles: true,
|
includeFiles: true,
|
||||||
includePersonalFiles: this.config.indexPersonalFiles,
|
includePersonalFiles: this.config.indexPersonalFiles,
|
||||||
|
includeFileManager: this.config.indexFileManager,
|
||||||
config: this.config,
|
config: this.config,
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -192,7 +193,11 @@ export class Indexer {
|
|||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const downloaded = await this.client.downloadFile(file.record);
|
// The file manager's ids mean nothing to files-storage; route by store.
|
||||||
|
const downloaded =
|
||||||
|
file.source === 'file-manager'
|
||||||
|
? await this.client.downloadFileManagerFile(file.record.id, file.record.name)
|
||||||
|
: await this.client.downloadFile(file.record);
|
||||||
const relative = paths.get(entry.fileId);
|
const relative = paths.get(entry.fileId);
|
||||||
if (!relative) return;
|
if (!relative) return;
|
||||||
const absolute = resolveWithin(this.config.mirrorDir, relative);
|
const absolute = resolveWithin(this.config.mirrorDir, relative);
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import { ServerContext } from '../context.ts';
|
|||||||
import type { Services } from '../services.ts';
|
import type { Services } from '../services.ts';
|
||||||
import { registerContentTools } from './tools/content.ts';
|
import { registerContentTools } from './tools/content.ts';
|
||||||
import { registerFileTools } from './tools/files.ts';
|
import { registerFileTools } from './tools/files.ts';
|
||||||
|
import { registerFilesystemTools } from './tools/filesystem.ts';
|
||||||
import { registerOverviewTools } from './tools/overview.ts';
|
import { registerOverviewTools } from './tools/overview.ts';
|
||||||
import { registerRawTool } from './tools/raw.ts';
|
import { registerRawTool } from './tools/raw.ts';
|
||||||
import { registerIndexTools } from './tools/index-tools.ts';
|
import { registerIndexTools } from './tools/index-tools.ts';
|
||||||
@@ -26,6 +27,11 @@ How the content is organised, and the usual path through it:
|
|||||||
- **Tasks** ("Aufgaben") — homework. list_tasks across all courses, get_task for one.
|
- **Tasks** ("Aufgaben") — homework. list_tasks across all courses, get_task for one.
|
||||||
- **Files** hang off boards, lessons and tasks. Every listing shows file ids; download_file fetches one and
|
- **Files** hang off boards, lessons and tasks. Every listing shows file ids; download_file fetches one and
|
||||||
extracts its text (PDF, Word, Excel, PowerPoint, OpenDocument) or returns an image inline.
|
extracts its text (PDF, Word, Excel, PowerPoint, OpenDocument) or returns an image inline.
|
||||||
|
- **The file manager ("Dateien")** is a separate store with a real folder tree, browsed with the fs_* tools:
|
||||||
|
/my (Persönliche Dateien), /courses/<course> (Kurs-Dateien), /teams/<team> (Team-Dateien) and /shared
|
||||||
|
(Geteilte Dateien). **Many teachers put their material only here**, so when a course page looks empty or the
|
||||||
|
worksheets are not on its boards, look in /courses/<course name>. fs_list and fs_tree browse, fs_find finds by
|
||||||
|
name, fs_read opens a file. list_files and download_file do not see these files.
|
||||||
- **Submissions** ("Abgaben") — what the user handed in. get_task shows that task's submission: the files,
|
- **Submissions** ("Abgaben") — what the user handed in. get_task shows that task's submission: the files,
|
||||||
the graded flag, the grade, what the user wrote, and the teacher's written feedback. A grade is a
|
the graded flag, the grade, what the user wrote, and the teacher's written feedback. A grade is a
|
||||||
percentage (0-100) or absent — there is no textual grade — and teachers often grade with the written
|
percentage (0-100) or absent — there is no textual grade — and teachers often grade with the written
|
||||||
@@ -51,6 +57,7 @@ export function createServer(config: Config, services?: Services): { server: Mcp
|
|||||||
registerContentTools(server, context);
|
registerContentTools(server, context);
|
||||||
registerRoomTools(server, context);
|
registerRoomTools(server, context);
|
||||||
registerFileTools(server, context);
|
registerFileTools(server, context);
|
||||||
|
registerFilesystemTools(server, context);
|
||||||
registerSearchTool(server, context);
|
registerSearchTool(server, context);
|
||||||
registerSubmissionTools(server, context);
|
registerSubmissionTools(server, context);
|
||||||
registerIndexTools(server, context);
|
registerIndexTools(server, context);
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import { dueLabel, formatDate, heading, htmlToText, joinSections, normalizeObjec
|
|||||||
import { assembleBoard, type AssembledBoard, type AssembledElement } from '../../core/board.ts';
|
import { assembleBoard, type AssembledBoard, type AssembledElement } from '../../core/board.ts';
|
||||||
import { forEachLimited } from '../../core/crawl.ts';
|
import { forEachLimited } from '../../core/crawl.ts';
|
||||||
import { fetchLessonPadText } from '../../core/etherpad.ts';
|
import { fetchLessonPadText } from '../../core/etherpad.ts';
|
||||||
|
import type { FmListing } from '../../core/legacy-files.ts';
|
||||||
import { fetchLessonTaskLinks, withScrapedIds } from '../../core/lesson-page.ts';
|
import { fetchLessonTaskLinks, withScrapedIds } from '../../core/lesson-page.ts';
|
||||||
import type {
|
import type {
|
||||||
CourseBoardResponse,
|
CourseBoardResponse,
|
||||||
@@ -38,17 +39,21 @@ export function registerContentTools(server: McpServer, context: ServerContext):
|
|||||||
},
|
},
|
||||||
async ({ courseId }) => {
|
async ({ courseId }) => {
|
||||||
try {
|
try {
|
||||||
const [board, legacy] = await Promise.all([
|
const [board, legacy, courseFiles] = await Promise.all([
|
||||||
context.client.getCourseBoard(courseId),
|
context.client.getCourseBoard(courseId),
|
||||||
// The v3 projection carries no description, teachers, members or
|
// The v3 projection carries no description, teachers, members or
|
||||||
// timetable; /api/v1/courses still does. Optional on purpose — it
|
// timetable; /api/v1/courses still does. Optional on purpose — it
|
||||||
// is a legacy route, so its absence must cost detail, not the call.
|
// is a legacy route, so its absence must cost detail, not the call.
|
||||||
context.client.getLegacyCourse(courseId).catch(() => undefined),
|
context.client.getLegacyCourse(courseId).catch(() => undefined),
|
||||||
|
// The course's file-manager area is a different store from the page.
|
||||||
|
// Teachers who only upload files there leave the page itself empty,
|
||||||
|
// and reporting "empty" then sends the reader away from the material.
|
||||||
|
context.files.list({ area: 'courses', ownerId: courseId }).catch(() => undefined),
|
||||||
]);
|
]);
|
||||||
const teachers = legacy
|
const teachers = legacy
|
||||||
? await context.resolveNames([...(legacy.teacherIds ?? []), ...(legacy.substitutionIds ?? [])])
|
? await context.resolveNames([...(legacy.teacherIds ?? []), ...(legacy.substitutionIds ?? [])])
|
||||||
: { names: [], unresolved: 0 };
|
: { names: [], unresolved: 0 };
|
||||||
return text(formatCourseBoard(board, legacy, teachers));
|
return text(formatCourseBoard(board, legacy, teachers, courseFiles));
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return toToolError(error, `read course ${courseId}`);
|
return toToolError(error, `read course ${courseId}`);
|
||||||
}
|
}
|
||||||
@@ -266,6 +271,7 @@ function formatCourseBoard(
|
|||||||
board: CourseBoardResponse,
|
board: CourseBoardResponse,
|
||||||
legacy?: LegacyCourse,
|
legacy?: LegacyCourse,
|
||||||
teachers: { names: string[]; unresolved: number } = { names: [], unresolved: 0 },
|
teachers: { names: string[]; unresolved: number } = { names: [], unresolved: 0 },
|
||||||
|
courseFiles?: FmListing,
|
||||||
): string {
|
): string {
|
||||||
const boards: string[] = [];
|
const boards: string[] = [];
|
||||||
const lessons: string[] = [];
|
const lessons: string[] = [];
|
||||||
@@ -293,8 +299,18 @@ function formatCourseBoard(
|
|||||||
formatCourseTimes(legacy?.times),
|
formatCourseTimes(legacy?.times),
|
||||||
]);
|
]);
|
||||||
|
|
||||||
|
const filesSection = formatCourseFiles(board.roomId, courseFiles);
|
||||||
|
|
||||||
if (boards.length + lessons.length + tasks.length === 0) {
|
if (boards.length + lessons.length + tasks.length === 0) {
|
||||||
return joinSections([heading(2, board.title), about, 'This course page is empty.']);
|
return joinSections([
|
||||||
|
heading(2, board.title),
|
||||||
|
`Course id: \`${board.roomId}\``,
|
||||||
|
about,
|
||||||
|
filesSection
|
||||||
|
? 'No boards, topics or tasks on the course page — the material is in the course files instead.'
|
||||||
|
: 'This course page is empty, and the course has no files in the file manager either.',
|
||||||
|
filesSection,
|
||||||
|
]);
|
||||||
}
|
}
|
||||||
|
|
||||||
return joinSections([
|
return joinSections([
|
||||||
@@ -304,6 +320,25 @@ function formatCourseBoard(
|
|||||||
boards.length > 0 && joinSections([heading(3, `Boards (${boards.length})`), boards.join('\n'), 'Read one with get_board.']),
|
boards.length > 0 && joinSections([heading(3, `Boards (${boards.length})`), boards.join('\n'), 'Read one with get_board.']),
|
||||||
lessons.length > 0 && joinSections([heading(3, `Topics (${lessons.length})`), lessons.join('\n'), 'Read one with get_lesson.']),
|
lessons.length > 0 && joinSections([heading(3, `Topics (${lessons.length})`), lessons.join('\n'), 'Read one with get_lesson.']),
|
||||||
tasks.length > 0 && joinSections([heading(3, `Tasks (${tasks.length})`), tasks.join('\n'), 'Read one with get_task.']),
|
tasks.length > 0 && joinSections([heading(3, `Tasks (${tasks.length})`), tasks.join('\n'), 'Read one with get_task.']),
|
||||||
|
filesSection,
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The course's own file-manager area ("Kurs-Dateien"), when it holds anything.
|
||||||
|
*
|
||||||
|
* Only the top level is fetched — one page — so this says how much is there
|
||||||
|
* and where, rather than listing it; fs_tree does that.
|
||||||
|
*/
|
||||||
|
function formatCourseFiles(courseId: string, listing: FmListing | undefined): string | undefined {
|
||||||
|
if (!listing || listing.directories.length + listing.files.length === 0) return undefined;
|
||||||
|
const names = [...listing.directories.map((entry) => `${entry.name}/`), ...listing.files.map((entry) => entry.name)];
|
||||||
|
const shown = names.slice(0, 8).map((name) => `- ${name}`).join('\n');
|
||||||
|
return joinSections([
|
||||||
|
heading(3, 'Course files (Kurs-Dateien)'),
|
||||||
|
`${listing.directories.length} folder(s) and ${listing.files.length} file(s) at the top level, newest first:`,
|
||||||
|
shown + (names.length > 8 ? `\n- … and ${names.length - 8} more` : ''),
|
||||||
|
`See everything with fs_tree path "/courses/${courseId}", read one with fs_read.`,
|
||||||
]);
|
]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import type { CallToolResult } from '@modelcontextprotocol/sdk/types.js';
|
|||||||
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
|
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
|
||||||
import { z } from 'zod';
|
import { z } from 'zod';
|
||||||
import type { ServerContext } from '../../context.ts';
|
import type { ServerContext } from '../../context.ts';
|
||||||
|
import type { DownloadedFile } from '../../core/client.ts';
|
||||||
import { extractContent, formatBytes } from '../../core/extract.ts';
|
import { extractContent, formatBytes } from '../../core/extract.ts';
|
||||||
import { formatDate, heading, joinSections } from '../../core/text.ts';
|
import { formatDate, heading, joinSections } from '../../core/text.ts';
|
||||||
import { FILE_PARENT_TYPES, type FileParentType } from '../../core/types.ts';
|
import { FILE_PARENT_TYPES, type FileParentType } from '../../core/types.ts';
|
||||||
@@ -16,9 +17,11 @@ export function registerFileTools(server: McpServer, context: ServerContext): vo
|
|||||||
{
|
{
|
||||||
title: 'List files of an entity',
|
title: 'List files of an entity',
|
||||||
description:
|
description:
|
||||||
'Files attached to one entity. Most of the time you do not need this — get_board, get_lesson and ' +
|
'Attachments on one entity in files-storage: a board element, a topic, a task, a submission. Most of the ' +
|
||||||
'get_task already list their own attachments. Reach for it to enumerate a course\'s own file area, ' +
|
'time you do not need this — get_board, get_lesson and get_task already list their own attachments. ' +
|
||||||
'or a single board element\'s files (parentType "boardnodes", parentId = the element id).',
|
'**Not for a course\'s files, personal files, team files or shared files**: those live in the file ' +
|
||||||
|
'manager ("Dateien"), a separate store this tool cannot see — it answers 0 for a course holding dozens of ' +
|
||||||
|
'worksheets. Use fs_list, fs_tree, fs_find and fs_read for them.',
|
||||||
inputSchema: {
|
inputSchema: {
|
||||||
parentType: z
|
parentType: z
|
||||||
.enum(FILE_PARENT_TYPES as [FileParentType, ...FileParentType[]])
|
.enum(FILE_PARENT_TYPES as [FileParentType, ...FileParentType[]])
|
||||||
@@ -61,9 +64,10 @@ export function registerFileTools(server: McpServer, context: ServerContext): vo
|
|||||||
{
|
{
|
||||||
title: 'Download and read a file',
|
title: 'Download and read a file',
|
||||||
description:
|
description:
|
||||||
'Fetches a file and returns its contents. PDFs, Word, Excel, PowerPoint and OpenDocument files are ' +
|
'Fetches a board, topic or task attachment and returns its contents. PDFs, Word, Excel, PowerPoint and ' +
|
||||||
'extracted to text; images come back inline so you can look at them; anything else reports its type. ' +
|
'OpenDocument files are extracted to text; images come back inline so you can look at them; anything ' +
|
||||||
'Pass raw=true to get base64 bytes instead of extracted text.',
|
'else reports its type. Pass raw=true to get base64 bytes instead of extracted text. For files from the ' +
|
||||||
|
'file manager (Persönliche Dateien, Kurs-Dateien, Team-Dateien, Geteilte Dateien) use fs_read instead.',
|
||||||
inputSchema: {
|
inputSchema: {
|
||||||
fileId: z.string().describe('File record id, from get_board, get_task, get_lesson or list_files.'),
|
fileId: z.string().describe('File record id, from get_board, get_task, get_lesson or list_files.'),
|
||||||
raw: z
|
raw: z
|
||||||
@@ -116,76 +120,99 @@ export function registerFileTools(server: McpServer, context: ServerContext): vo
|
|||||||
.join('\n'),
|
.join('\n'),
|
||||||
].join('\n\n');
|
].join('\n\n');
|
||||||
|
|
||||||
if (raw) {
|
return await renderFileContent(context, header, file, {
|
||||||
return text(
|
name: record.name,
|
||||||
joinSections([
|
mimeType: record.mimeType,
|
||||||
header,
|
raw,
|
||||||
`Base64 (${file.bytes.length} bytes):`,
|
maxChars,
|
||||||
'```',
|
// Nothing extractable — but files-storage may still be able to render
|
||||||
file.bytes.toString('base64'),
|
// the file as a picture. That is the whole answer for an image-only
|
||||||
'```',
|
// PDF: its pages *are* pictures, so a rasterised preview is readable
|
||||||
]),
|
// where the bytes are not, and it needs no OCR on our side.
|
||||||
);
|
fallbackImage:
|
||||||
}
|
record.previewStatus === 'preview_possible'
|
||||||
|
? async () => {
|
||||||
const extraction = await extractContent(
|
const preview = await context.client.getFilePreview(record, 500).catch(() => undefined);
|
||||||
file.bytes,
|
return preview && preview.mimeType.startsWith('image/') ? preview : undefined;
|
||||||
file.mimeType || record.mimeType,
|
}
|
||||||
record.name,
|
: undefined,
|
||||||
maxChars ?? context.config.maxExtractedChars,
|
});
|
||||||
);
|
|
||||||
|
|
||||||
if (extraction.kind === 'image' && extraction.image) {
|
|
||||||
const result: CallToolResult = {
|
|
||||||
content: [
|
|
||||||
{ type: 'text', text: joinSections([header, extraction.note]) },
|
|
||||||
{ type: 'image', data: extraction.image.base64, mimeType: extraction.image.mimeType },
|
|
||||||
],
|
|
||||||
};
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (extraction.kind === 'text') {
|
|
||||||
const body = extraction.text?.trim();
|
|
||||||
return text(
|
|
||||||
joinSections([
|
|
||||||
header,
|
|
||||||
extraction.note,
|
|
||||||
body ? joinSections([heading(3, 'Contents'), body]) : '_(the file contains no extractable text)_',
|
|
||||||
]),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Nothing extractable — but files-storage may still be able to render
|
|
||||||
// the file as a picture. That is the whole answer for an image-only
|
|
||||||
// PDF: its pages *are* pictures, so a rasterised preview is readable
|
|
||||||
// where the bytes are not, and it needs no OCR on our side.
|
|
||||||
if (record.previewStatus === 'preview_possible') {
|
|
||||||
const preview = await context.client.getFilePreview(record, 500).catch(() => undefined);
|
|
||||||
if (preview && preview.mimeType.startsWith('image/')) {
|
|
||||||
const result: CallToolResult = {
|
|
||||||
content: [
|
|
||||||
{
|
|
||||||
type: 'text',
|
|
||||||
text: joinSections([
|
|
||||||
header,
|
|
||||||
// The note ends by suggesting raw bytes, which is no longer the
|
|
||||||
// best answer once a readable rendering is attached.
|
|
||||||
extraction.note.replace(' Use download_file with raw=true to get the bytes.', ''),
|
|
||||||
"Showing the instance's own rendered preview below, which is readable as a picture.",
|
|
||||||
]),
|
|
||||||
},
|
|
||||||
{ type: 'image', data: preview.bytes.toString('base64'), mimeType: preview.mimeType },
|
|
||||||
],
|
|
||||||
};
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return text(joinSections([header, extraction.note]));
|
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return toToolError(error, `download file ${fileId}`);
|
return toToolError(error, `download file ${fileId}`);
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Renders a downloaded file for a tool result: base64 when asked for, an image
|
||||||
|
* inline, extracted text, or — for a format with no extractor — its note.
|
||||||
|
*
|
||||||
|
* Shared by download_file (files-storage) and fs_read (the file manager), which
|
||||||
|
* differ only in how the bytes were obtained and in what the header says.
|
||||||
|
* `fallbackImage` is download_file's preview route; the file manager has none.
|
||||||
|
*/
|
||||||
|
export async function renderFileContent(
|
||||||
|
context: ServerContext,
|
||||||
|
header: string,
|
||||||
|
file: DownloadedFile,
|
||||||
|
options: {
|
||||||
|
name: string;
|
||||||
|
mimeType?: string;
|
||||||
|
raw: boolean;
|
||||||
|
maxChars?: number;
|
||||||
|
fallbackImage?: () => Promise<DownloadedFile | undefined>;
|
||||||
|
},
|
||||||
|
): Promise<CallToolResult> {
|
||||||
|
if (options.raw) {
|
||||||
|
return text(joinSections([header, `Base64 (${file.bytes.length} bytes):`, '```', file.bytes.toString('base64'), '```']));
|
||||||
|
}
|
||||||
|
|
||||||
|
const extraction = await extractContent(
|
||||||
|
file.bytes,
|
||||||
|
file.mimeType && file.mimeType !== 'application/octet-stream' ? file.mimeType : (options.mimeType ?? file.mimeType),
|
||||||
|
options.name,
|
||||||
|
options.maxChars ?? context.config.maxExtractedChars,
|
||||||
|
);
|
||||||
|
|
||||||
|
if (extraction.kind === 'image' && extraction.image) {
|
||||||
|
return {
|
||||||
|
content: [
|
||||||
|
{ type: 'text', text: joinSections([header, extraction.note]) },
|
||||||
|
{ type: 'image', data: extraction.image.base64, mimeType: extraction.image.mimeType },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (extraction.kind === 'text') {
|
||||||
|
const body = extraction.text?.trim();
|
||||||
|
return text(
|
||||||
|
joinSections([
|
||||||
|
header,
|
||||||
|
extraction.note,
|
||||||
|
body ? joinSections([heading(3, 'Contents'), body]) : '_(the file contains no extractable text)_',
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const image = await options.fallbackImage?.();
|
||||||
|
if (image) {
|
||||||
|
return {
|
||||||
|
content: [
|
||||||
|
{
|
||||||
|
type: 'text',
|
||||||
|
text: joinSections([
|
||||||
|
header,
|
||||||
|
// The note ends by suggesting raw bytes, which is no longer the
|
||||||
|
// best answer once a readable rendering is attached.
|
||||||
|
extraction.note.replace(/ Use \w+ with raw=true to get the bytes\./, ''),
|
||||||
|
"Showing the instance's own rendered preview below, which is readable as a picture.",
|
||||||
|
]),
|
||||||
|
},
|
||||||
|
{ type: 'image', data: image.bytes.toString('base64'), mimeType: image.mimeType },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return text(joinSections([header, extraction.note]));
|
||||||
|
}
|
||||||
|
|||||||
401
src/mcp/tools/filesystem.ts
Normal file
401
src/mcp/tools/filesystem.ts
Normal file
@@ -0,0 +1,401 @@
|
|||||||
|
import type { CallToolResult } from '@modelcontextprotocol/sdk/types.js';
|
||||||
|
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
|
||||||
|
import { z } from 'zod';
|
||||||
|
import type { ServerContext } from '../../context.ts';
|
||||||
|
import { formatBytes } from '../../core/extract.ts';
|
||||||
|
import {
|
||||||
|
areaInfo,
|
||||||
|
compareNames,
|
||||||
|
FILE_AREAS,
|
||||||
|
FileManagerMarkupError,
|
||||||
|
FsError,
|
||||||
|
nameMatcher,
|
||||||
|
type DirectoryRef,
|
||||||
|
type FmFile,
|
||||||
|
type FsNode,
|
||||||
|
type WalkEntry,
|
||||||
|
} from '../../core/legacy-files.ts';
|
||||||
|
import { heading, joinSections } from '../../core/text.ts';
|
||||||
|
import { renderFileContent } from './files.ts';
|
||||||
|
import { failure, text, toToolError } from './result.ts';
|
||||||
|
|
||||||
|
const READ_ONLY = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true };
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The "Dateien" file manager as filesystem tools: ls, tree, find, read.
|
||||||
|
*
|
||||||
|
* Deliberately separate from list_files / download_file, which read
|
||||||
|
* files-storage — board, topic and task attachments. The two stores do not
|
||||||
|
* overlap, and conflating them is how a course holding dozens of worksheets
|
||||||
|
* came to be reported as having 0 files.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const AREA_NOTE =
|
||||||
|
'The file manager ("Dateien") is separate from course pages and holds four areas: ' +
|
||||||
|
'/my (Persönliche Dateien), /courses/<course name> (Kurs-Dateien), /teams/<team name> (Team-Dateien) and ' +
|
||||||
|
'/shared (Geteilte Dateien). Many teachers keep their material only in Kurs-Dateien, so a course whose page ' +
|
||||||
|
'looks empty often has its worksheets here.';
|
||||||
|
|
||||||
|
const PATH_NOTE =
|
||||||
|
'Paths use the names shown in listings, e.g. "/courses/FIA24B - LF2 (Rh)/Handlungssituation". Names may ' +
|
||||||
|
'contain "/" and still resolve; any segment may also be the id printed next to it, which is never ambiguous.';
|
||||||
|
|
||||||
|
export function registerFilesystemTools(server: McpServer, context: ServerContext): void {
|
||||||
|
server.registerTool(
|
||||||
|
'fs_list',
|
||||||
|
{
|
||||||
|
title: 'List a folder in the file manager',
|
||||||
|
description:
|
||||||
|
`Lists one folder of the Schulcloud file manager, like \`ls\`. ${AREA_NOTE} Start at "/" or ` +
|
||||||
|
'"/courses" to see what exists. Given a file path instead, shows that file\'s details. ' +
|
||||||
|
`${PATH_NOTE} Not for attachments on boards, topics or tasks — get_board, get_lesson and get_task list ` +
|
||||||
|
'those, and download_file reads them.',
|
||||||
|
inputSchema: {
|
||||||
|
path: z.string().default('/').describe('Folder to list, e.g. "/", "/courses", "/courses/<course>/<folder>".'),
|
||||||
|
},
|
||||||
|
annotations: READ_ONLY,
|
||||||
|
},
|
||||||
|
async ({ path }) => {
|
||||||
|
try {
|
||||||
|
const node = await context.files.resolve(path);
|
||||||
|
if (node.kind === 'file') return text(describeFile(node));
|
||||||
|
return text(await listDirectory(context, node));
|
||||||
|
} catch (error) {
|
||||||
|
return fsError(error, `list ${path}`);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
server.registerTool(
|
||||||
|
'fs_tree',
|
||||||
|
{
|
||||||
|
title: 'Show a folder tree in the file manager',
|
||||||
|
description:
|
||||||
|
`Everything below a folder of the Schulcloud file manager, as an indented tree, like \`tree\`. ${AREA_NOTE} ` +
|
||||||
|
'Use it to get an overview of a course\'s files in one call — "/courses/<course>" — or of all course ' +
|
||||||
|
'files at a shallow depth. Each folder costs one page load, so the walk stops at `maxFolders` and says ' +
|
||||||
|
'so; narrow the path or lower the depth rather than raising the limit. To look for a name, fs_find is ' +
|
||||||
|
`cheaper. ${PATH_NOTE}`,
|
||||||
|
inputSchema: {
|
||||||
|
path: z.string().default('/').describe('Folder to start from.'),
|
||||||
|
depth: z.number().int().min(1).max(8).default(3).describe('How many levels below the folder to show.'),
|
||||||
|
maxFolders: z
|
||||||
|
.number()
|
||||||
|
.int()
|
||||||
|
.min(1)
|
||||||
|
.max(400)
|
||||||
|
.default(80)
|
||||||
|
.describe('Stop after listing this many folders.'),
|
||||||
|
},
|
||||||
|
annotations: READ_ONLY,
|
||||||
|
},
|
||||||
|
async ({ path, depth, maxFolders }) => {
|
||||||
|
try {
|
||||||
|
const node = await context.files.resolve(path);
|
||||||
|
if (node.kind === 'file') return text(describeFile(node));
|
||||||
|
const result = await context.files.walk(node, { maxDepth: depth, maxDirectories: maxFolders });
|
||||||
|
return text(renderTree(node, result.entries, { depth, maxFolders, ...result }));
|
||||||
|
} catch (error) {
|
||||||
|
return fsError(error, `walk ${path}`);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
server.registerTool(
|
||||||
|
'fs_find',
|
||||||
|
{
|
||||||
|
title: 'Find files by name in the file manager',
|
||||||
|
description:
|
||||||
|
`Finds files and folders by name anywhere below a folder of the Schulcloud file manager, like \`find\`. ` +
|
||||||
|
`${AREA_NOTE} Without wildcards it matches any part of the name, case-insensitively. With "*" or "?" the ` +
|
||||||
|
'whole name must match, as with find -name — so "*.docx", or "*Erben*" for names containing Erben. Scope it with ' +
|
||||||
|
'`path` (e.g. "/courses/<course>") whenever you know the course: searching all of /courses walks every ' +
|
||||||
|
'folder of every course. This matches names only — to search inside documents use search, which ' +
|
||||||
|
`covers file-manager files once they are indexed. ${PATH_NOTE}`,
|
||||||
|
inputSchema: {
|
||||||
|
name: z
|
||||||
|
.string()
|
||||||
|
.min(1)
|
||||||
|
.describe('Part of the name ("Erbrecht"), or a whole-name pattern with * and ? ("*.docx", "*Erben*").'),
|
||||||
|
path: z.string().default('/').describe('Folder to search below. Default: every area.'),
|
||||||
|
type: z.enum(['any', 'file', 'folder']).default('any').describe('Only files, only folders, or both.'),
|
||||||
|
maxFolders: z
|
||||||
|
.number()
|
||||||
|
.int()
|
||||||
|
.min(1)
|
||||||
|
.max(600)
|
||||||
|
.default(250)
|
||||||
|
.describe('Stop after listing this many folders.'),
|
||||||
|
},
|
||||||
|
annotations: READ_ONLY,
|
||||||
|
},
|
||||||
|
async ({ name, path, type, maxFolders }) => {
|
||||||
|
try {
|
||||||
|
const node = await context.files.resolve(path);
|
||||||
|
if (node.kind === 'file') return text(describeFile(node));
|
||||||
|
const matcher = nameMatcher(name);
|
||||||
|
const result = await context.files.walk(node, { maxDepth: 12, maxDirectories: maxFolders });
|
||||||
|
const hits = result.entries
|
||||||
|
.filter((entry) => (type === 'file' ? entry.file : type === 'folder' ? entry.directory : true))
|
||||||
|
.filter((entry) => matcher((entry.file ?? entry.directory)?.name ?? ''))
|
||||||
|
.sort((a, b) => compareNames(a.path, b.path));
|
||||||
|
|
||||||
|
const scope = `${result.visited} folder(s) searched`;
|
||||||
|
const notes = [
|
||||||
|
result.truncated
|
||||||
|
? `_Stopped after ${maxFolders} folders, so there may be more matches. Narrow \`path\` to one course._`
|
||||||
|
: undefined,
|
||||||
|
failureNote(result.failures),
|
||||||
|
];
|
||||||
|
if (hits.length === 0) {
|
||||||
|
return text(joinSections([`No names matching "${name}" below ${node.path} (${scope}).`, ...notes]));
|
||||||
|
}
|
||||||
|
return text(
|
||||||
|
joinSections([
|
||||||
|
heading(2, `${hits.length} match(es) for "${name}" below ${node.path}`),
|
||||||
|
hits.map((entry) => entryLine(entry, { fullPath: true })).join('\n'),
|
||||||
|
`_${scope}._ Read a file with fs_read, open a folder with fs_list.`,
|
||||||
|
...notes,
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
} catch (error) {
|
||||||
|
return fsError(error, `search ${path}`);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
server.registerTool(
|
||||||
|
'fs_read',
|
||||||
|
{
|
||||||
|
title: 'Read a file from the file manager',
|
||||||
|
description:
|
||||||
|
`Fetches one file from the Schulcloud file manager and returns its contents, like \`cat\`. ${AREA_NOTE} ` +
|
||||||
|
'PDFs, Word, Excel, PowerPoint and OpenDocument files are extracted to text; images come back inline so ' +
|
||||||
|
'you can look at them; anything else reports its type. Pass raw=true for base64 bytes. Give the path ' +
|
||||||
|
'from a listing, or the file id and name. Not for board, topic or task attachments — use download_file ' +
|
||||||
|
`for those. ${PATH_NOTE}`,
|
||||||
|
inputSchema: {
|
||||||
|
path: z.string().optional().describe('File path, e.g. "/courses/<course>/<folder>/Arbeitsblatt.pdf".'),
|
||||||
|
fileId: z.string().optional().describe('File id from a listing, instead of a path.'),
|
||||||
|
name: z.string().optional().describe('The file name, when giving fileId; used to recognise the format.'),
|
||||||
|
raw: z.boolean().default(false).describe('Return base64-encoded bytes instead of extracted text.'),
|
||||||
|
maxChars: z
|
||||||
|
.number()
|
||||||
|
.int()
|
||||||
|
.min(500)
|
||||||
|
.max(500_000)
|
||||||
|
.optional()
|
||||||
|
.describe('Override the character limit on extracted text.'),
|
||||||
|
},
|
||||||
|
annotations: READ_ONLY,
|
||||||
|
},
|
||||||
|
async ({ path, fileId, name, raw, maxChars }) => {
|
||||||
|
try {
|
||||||
|
let file: Pick<FmFile, 'id' | 'name'> & Partial<FmFile>;
|
||||||
|
let where: string;
|
||||||
|
if (path) {
|
||||||
|
const node = await context.files.resolve(path);
|
||||||
|
if (node.kind !== 'file') {
|
||||||
|
return failure(`${node.path} is a folder, not a file. List it with fs_list, or use fs_tree.`);
|
||||||
|
}
|
||||||
|
file = node.file;
|
||||||
|
where = node.path;
|
||||||
|
} else if (fileId) {
|
||||||
|
if (!/^[0-9a-f]{24}$/i.test(fileId)) return failure(`"${fileId}" is not a file id.`);
|
||||||
|
file = { id: fileId, name: name?.trim() || fileId };
|
||||||
|
where = `file \`${fileId}\``;
|
||||||
|
} else {
|
||||||
|
return failure('Give either `path` or `fileId`.');
|
||||||
|
}
|
||||||
|
|
||||||
|
// The instance scans uploads; a file it rejected is not served.
|
||||||
|
if (file.blocked) {
|
||||||
|
return failure(`"${file.name}" was blocked by the instance's virus scanner and will not be downloaded.`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const downloaded = await context.files.download(file);
|
||||||
|
const header = [
|
||||||
|
heading(2, file.name),
|
||||||
|
[
|
||||||
|
`- Path: ${where}`,
|
||||||
|
`- File id: \`${file.id}\``,
|
||||||
|
`- Type: ${file.mimeType ?? downloaded.mimeType}`,
|
||||||
|
`- Size: ${formatBytes(file.size ?? downloaded.bytes.length)}`,
|
||||||
|
downloaded.truncated
|
||||||
|
? `- **Download was capped at ${formatBytes(context.config.maxDownloadBytes)}; content is incomplete.**`
|
||||||
|
: undefined,
|
||||||
|
]
|
||||||
|
.filter(Boolean)
|
||||||
|
.join('\n'),
|
||||||
|
].join('\n\n');
|
||||||
|
|
||||||
|
return await renderFileContent(context, header, downloaded, {
|
||||||
|
name: file.name,
|
||||||
|
mimeType: file.mimeType,
|
||||||
|
raw,
|
||||||
|
maxChars,
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
return fsError(error, `read ${path ?? fileId}`);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function listDirectory(context: ServerContext, node: Extract<FsNode, { kind: 'directory' }>): Promise<string> {
|
||||||
|
if (!node.ref.area) {
|
||||||
|
return joinSections([
|
||||||
|
heading(2, '/ — the file manager ("Dateien")'),
|
||||||
|
FILE_AREAS.map((entry) => `- **/${entry.area}/** — ${entry.label}: ${entry.summary}`).join('\n'),
|
||||||
|
'Open one with fs_list, e.g. path "/courses". A course\'s own files are under "/courses/<course name>".',
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
const listing = await context.files.list(node.ref);
|
||||||
|
const area = areaInfo(node.ref.area);
|
||||||
|
const isOwnerList = (node.ref.area === 'courses' || node.ref.area === 'teams') && !node.ref.ownerId;
|
||||||
|
const directories = [...listing.directories].sort((a, b) => compareNames(a.name, b.name));
|
||||||
|
const files = [...listing.files].sort((a, b) => compareNames(a.name, b.name));
|
||||||
|
|
||||||
|
const title = heading(2, `${node.path} — ${area.label}`);
|
||||||
|
if (directories.length === 0 && files.length === 0) {
|
||||||
|
return joinSections([
|
||||||
|
title,
|
||||||
|
isOwnerList
|
||||||
|
? `No ${node.ref.area === 'courses' ? 'courses' : 'teams'} with a file area.`
|
||||||
|
: node.ref.area === 'shared'
|
||||||
|
? 'Nothing has been shared with you.'
|
||||||
|
: 'This folder is empty.',
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
const lines = [
|
||||||
|
...directories.map((directory) => `- **${directory.name}/** (\`${directory.id}\`)`),
|
||||||
|
...files.map((file) => `- ${fileLine(file)}`),
|
||||||
|
];
|
||||||
|
const bytes = files.reduce((sum, file) => sum + file.size, 0);
|
||||||
|
const summary = isOwnerList
|
||||||
|
? `${directories.length} ${node.ref.area === 'courses' ? 'course' : 'team'}(s). Their files are inside; fs_tree with depth 2 shows which hold any.`
|
||||||
|
: `${directories.length} folder(s), ${files.length} file(s)${files.length ? `, ${formatBytes(bytes)}` : ''}.`;
|
||||||
|
|
||||||
|
return joinSections([
|
||||||
|
title,
|
||||||
|
lines.join('\n'),
|
||||||
|
summary,
|
||||||
|
node.ref.area === 'shared' && directories.length > 0
|
||||||
|
? '_Shared folders cannot be opened — the file manager has no route for them, not even in the browser._'
|
||||||
|
: undefined,
|
||||||
|
'Open a folder with fs_list (its name or id appended to this path), read a file with fs_read.',
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
function describeFile(node: Extract<FsNode, { kind: 'file' }>): string {
|
||||||
|
return joinSections([
|
||||||
|
heading(2, node.file.name),
|
||||||
|
[
|
||||||
|
`- Path: ${node.path}`,
|
||||||
|
`- File id: \`${node.file.id}\``,
|
||||||
|
`- Type: ${node.file.mimeType ?? 'unknown'}`,
|
||||||
|
`- Size: ${formatBytes(node.file.size)}`,
|
||||||
|
node.file.blocked ? '- **Blocked by the instance virus scanner; it cannot be downloaded.**' : undefined,
|
||||||
|
]
|
||||||
|
.filter(Boolean)
|
||||||
|
.join('\n'),
|
||||||
|
node.file.blocked ? undefined : 'Read it with fs_read.',
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
function fileLine(file: FmFile): string {
|
||||||
|
const type = file.mimeType ? `, ${file.mimeType}` : '';
|
||||||
|
const blocked = file.blocked ? ' **[blocked by virus scan]**' : '';
|
||||||
|
return `${file.name} — ${formatBytes(file.size)}${type} (\`${file.id}\`)${blocked}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function entryLine(entry: WalkEntry, options: { fullPath: boolean }): string {
|
||||||
|
const label = options.fullPath ? entry.path : (entry.file ?? entry.directory)?.name;
|
||||||
|
if (entry.directory) return `- **${label}/** (\`${entry.directory.id}\`)`;
|
||||||
|
if (entry.file) return `- ${fileLine({ ...entry.file, name: label ?? entry.file.name })}`;
|
||||||
|
return `- ${label}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* An indented tree, built from each entry's parent rather than from sorting
|
||||||
|
* path strings — see `FileManager.walk` for why that distinction matters.
|
||||||
|
*/
|
||||||
|
function renderTree(
|
||||||
|
root: { path: string; ref: DirectoryRef },
|
||||||
|
entries: WalkEntry[],
|
||||||
|
info: { depth: number; maxFolders: number; visited: number; truncated: boolean; failures: { path: string; reason: string }[] },
|
||||||
|
): string {
|
||||||
|
const children = new Map<string, WalkEntry[]>();
|
||||||
|
for (const entry of entries) {
|
||||||
|
const list = children.get(entry.parentPath) ?? [];
|
||||||
|
list.push(entry);
|
||||||
|
children.set(entry.parentPath, list);
|
||||||
|
}
|
||||||
|
|
||||||
|
const lines: string[] = [];
|
||||||
|
let files = 0;
|
||||||
|
let folders = 0;
|
||||||
|
let bytes = 0;
|
||||||
|
const visit = (path: string, indent: string) => {
|
||||||
|
const kids = children.get(path) ?? [];
|
||||||
|
// The areas under "/" keep their own order (personal, courses, teams,
|
||||||
|
// shared); everything else sorts folders first, then by name.
|
||||||
|
if (path !== '/') {
|
||||||
|
kids.sort((a, b) => {
|
||||||
|
if (Boolean(a.directory) !== Boolean(b.directory)) return a.directory ? -1 : 1;
|
||||||
|
return compareNames(a.path, b.path);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
for (const kid of kids) {
|
||||||
|
if (kid.directory) {
|
||||||
|
folders++;
|
||||||
|
// An area's "id" is its slug, not an id anything accepts; leave it out.
|
||||||
|
const id = /^[0-9a-f]{24}$/i.test(kid.directory.id) ? ` \`${kid.directory.id}\`` : '';
|
||||||
|
lines.push(`${indent}${kid.directory.name}/${id}`);
|
||||||
|
visit(kid.path, `${indent} `);
|
||||||
|
} else if (kid.file) {
|
||||||
|
files++;
|
||||||
|
bytes += kid.file.size;
|
||||||
|
const blocked = kid.file.blocked ? ' [blocked]' : '';
|
||||||
|
lines.push(`${indent}${kid.file.name} (${formatBytes(kid.file.size)}) \`${kid.file.id}\`${blocked}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
visit(root.path, '');
|
||||||
|
|
||||||
|
const area = root.ref.area ? ` — ${areaInfo(root.ref.area).label}` : '';
|
||||||
|
if (lines.length === 0) {
|
||||||
|
return joinSections([heading(2, `${root.path}${area}`), 'Nothing below this folder.', failureNote(info.failures)]);
|
||||||
|
}
|
||||||
|
|
||||||
|
return joinSections([
|
||||||
|
heading(2, `${root.path}${area}`),
|
||||||
|
['```', ...lines, '```'].join('\n'),
|
||||||
|
`${folders} folder(s), ${files} file(s), ${formatBytes(bytes)} — ${info.visited} folder(s) listed, ${info.depth} level(s) deep.`,
|
||||||
|
info.truncated
|
||||||
|
? `_Stopped after listing ${info.maxFolders} folders; the tree is incomplete. Start deeper, e.g. at one course._`
|
||||||
|
: undefined,
|
||||||
|
failureNote(info.failures),
|
||||||
|
'Read a file with fs_read (path = this folder plus the names above).',
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
function failureNote(failures: { path: string; reason: string }[]): string | undefined {
|
||||||
|
if (failures.length === 0) return undefined;
|
||||||
|
const shown = failures.slice(0, 5).map((entry) => `${entry.path} (${entry.reason})`).join('; ');
|
||||||
|
return `_Could not list ${failures.length} folder(s): ${shown}${failures.length > 5 ? '; …' : ''}._`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function fsError(error: unknown, action: string): CallToolResult {
|
||||||
|
if (error instanceof FsError) return failure(error.message);
|
||||||
|
if (error instanceof FileManagerMarkupError) {
|
||||||
|
return failure(
|
||||||
|
`Could not ${action}: the file manager page did not look like a file listing. Either the session is no ` +
|
||||||
|
'longer accepted (check whoami) or the page markup changed.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return toToolError(error, action);
|
||||||
|
}
|
||||||
|
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
|
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
|
||||||
import { z } from 'zod';
|
import { z } from 'zod';
|
||||||
import type { ServerContext } from '../../context.ts';
|
import type { ServerContext } from '../../context.ts';
|
||||||
|
import { FILE_AREAS } from '../../core/legacy-files.ts';
|
||||||
import { crawl } from '../../core/crawl.ts';
|
import { crawl } from '../../core/crawl.ts';
|
||||||
import { searchSnapshot, type Hit } from '../../core/match.ts';
|
import { searchSnapshot, type Hit } from '../../core/match.ts';
|
||||||
import { formatDate, heading, joinSections } from '../../core/text.ts';
|
import { formatDate, heading, joinSections } from '../../core/text.ts';
|
||||||
@@ -144,14 +145,37 @@ function targetIdFor(hit: SearchResult): string {
|
|||||||
return hit.nodeId;
|
return hit.nodeId;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Where to go next for a hit. File-manager files are read by path, not by download_file. */
|
||||||
|
function nextStep(hit: SearchResult): string {
|
||||||
|
if (hit.kind === 'file' && hit.meta?.source === 'file-manager') {
|
||||||
|
const fsPath = typeof hit.meta.fsPath === 'string' ? hit.meta.fsPath : undefined;
|
||||||
|
return fsPath
|
||||||
|
? ` → \`fs_read\` with path \`${fsPath}\``
|
||||||
|
: ` → \`fs_read\` with fileId \`${hit.nodeId}\` and name \`${hit.title}\``;
|
||||||
|
}
|
||||||
|
// A submission has no id of its own that any tool takes: get_task is
|
||||||
|
// reached through the *task*, so point at that rather than at the
|
||||||
|
// submission id, which would simply 404.
|
||||||
|
return ` → \`${TOOL_FOR[hit.kind] ?? 'api_get'}\` with id \`${targetIdFor(hit)}\``;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** "Kurs-Dateien, <course>" or the area's own name, from the file's fs path. */
|
||||||
|
function fileManagerPlace(hit: SearchResult): string {
|
||||||
|
const area = typeof hit.meta?.fsPath === 'string' ? hit.meta.fsPath.split('/')[1] : undefined;
|
||||||
|
const known = FILE_AREAS.find((entry) => entry.area === area);
|
||||||
|
if (!known) return 'the file manager';
|
||||||
|
return known.area === 'courses' && hit.courseTitle ? `${known.label}, ${hit.courseTitle}` : known.label;
|
||||||
|
}
|
||||||
|
|
||||||
function formatIndexed(hit: SearchResult): string {
|
function formatIndexed(hit: SearchResult): string {
|
||||||
|
const where =
|
||||||
|
hit.kind === 'file' && hit.meta?.source === 'file-manager'
|
||||||
|
? `file in ${fileManagerPlace(hit)}`
|
||||||
|
: `${hit.kind} in ${hit.courseTitle || hit.path}`;
|
||||||
return [
|
return [
|
||||||
`- **${hit.title}** — ${hit.kind} in ${hit.courseTitle || hit.path}`,
|
`- **${hit.title}** — ${where}`,
|
||||||
hit.snippet && hit.snippet !== hit.title ? ` ${hit.snippet}` : undefined,
|
hit.snippet && hit.snippet !== hit.title ? ` ${hit.snippet}` : undefined,
|
||||||
// A submission has no id of its own that any tool takes: get_task is
|
nextStep(hit),
|
||||||
// reached through the *task*, so point at that rather than at the
|
|
||||||
// submission id, which would simply 404.
|
|
||||||
` → \`${TOOL_FOR[hit.kind] ?? 'api_get'}\` with id \`${targetIdFor(hit)}\``,
|
|
||||||
]
|
]
|
||||||
.filter(Boolean)
|
.filter(Boolean)
|
||||||
.join('\n');
|
.join('\n');
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import type { Config } from './config.ts';
|
import type { Config } from './config.ts';
|
||||||
import { SchulcloudClient } from './core/client.ts';
|
import { SchulcloudClient } from './core/client.ts';
|
||||||
|
import { FileManager } from './core/legacy-files.ts';
|
||||||
import { Indexer } from './indexer/indexer.ts';
|
import { Indexer } from './indexer/indexer.ts';
|
||||||
import { Store } from './store/store.ts';
|
import { Store } from './store/store.ts';
|
||||||
|
|
||||||
@@ -14,12 +15,19 @@ import { Store } from './store/store.ts';
|
|||||||
export interface Services {
|
export interface Services {
|
||||||
config: Config;
|
config: Config;
|
||||||
client: SchulcloudClient;
|
client: SchulcloudClient;
|
||||||
|
/**
|
||||||
|
* The "Dateien" file manager. Process-wide so its short listing cache is
|
||||||
|
* shared: an `ls` in one MCP session and a `schulcloud fs get` from the CLI
|
||||||
|
* then cost one page fetch between them, not two.
|
||||||
|
*/
|
||||||
|
files: FileManager;
|
||||||
store: Store | undefined;
|
store: Store | undefined;
|
||||||
indexer: Indexer | undefined;
|
indexer: Indexer | undefined;
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function createServices(config: Config): Promise<Services> {
|
export async function createServices(config: Config): Promise<Services> {
|
||||||
const client = new SchulcloudClient(config);
|
const client = new SchulcloudClient(config);
|
||||||
|
const files = new FileManager(client);
|
||||||
const store = await Store.open(config.databaseUrl);
|
const store = await Store.open(config.databaseUrl);
|
||||||
const indexer = store ? new Indexer(client, store, config) : undefined;
|
const indexer = store ? new Indexer(client, store, config) : undefined;
|
||||||
|
|
||||||
@@ -29,7 +37,7 @@ export async function createServices(config: Config): Promise<Services> {
|
|||||||
'/files, /manifest and refresh_index are unavailable. Set DATABASE_URL to enable them.',
|
'/files, /manifest and refresh_index are unavailable. Set DATABASE_URL to enable them.',
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
return { config, client, store, indexer };
|
return { config, client, files, store, indexer };
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function closeServices(services: Services): Promise<void> {
|
export async function closeServices(services: Services): Promise<void> {
|
||||||
|
|||||||
@@ -396,6 +396,31 @@ export class Store {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Where a file's bytes live, from the latest crawl.
|
||||||
|
*
|
||||||
|
* The file manager and files-storage use the same id shape but not the same
|
||||||
|
* ids, so a live fetch has to know which one to ask.
|
||||||
|
*/
|
||||||
|
async fileSource(
|
||||||
|
fileId: string,
|
||||||
|
): Promise<{ source: 'files-storage' | 'file-manager'; name: string; mimeType: string; size: number } | undefined> {
|
||||||
|
const crawlId = await this.latestCrawlId();
|
||||||
|
if (crawlId === undefined) return undefined;
|
||||||
|
const { rows } = await this.db.query<{ title: string; meta: Record<string, unknown> }>(
|
||||||
|
`SELECT title, meta FROM nodes WHERE crawl_id = $1 AND kind = 'file' AND node_id = $2`,
|
||||||
|
[crawlId, fileId],
|
||||||
|
);
|
||||||
|
const row = rows[0];
|
||||||
|
if (!row) return undefined;
|
||||||
|
return {
|
||||||
|
source: row.meta.source === 'file-manager' ? 'file-manager' : 'files-storage',
|
||||||
|
name: row.title,
|
||||||
|
mimeType: typeof row.meta.mimeType === 'string' ? row.meta.mimeType : 'application/octet-stream',
|
||||||
|
size: typeof row.meta.size === 'number' ? row.meta.size : 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
async mirrorEntry(fileId: string): Promise<{ path: string; size: number; name: string; mimeType: string } | undefined> {
|
async mirrorEntry(fileId: string): Promise<{ path: string; size: number; name: string; mimeType: string } | undefined> {
|
||||||
const { rows } = await this.db.query<{ mirror_path: string | null; mirror_size: string | null; name: string; mime_type: string | null }>(
|
const { rows } = await this.db.query<{ mirror_path: string | null; mirror_size: string | null; name: string; mime_type: string | null }>(
|
||||||
`SELECT mirror_path, mirror_size, name, mime_type FROM file_texts WHERE file_id = $1`,
|
`SELECT mirror_path, mirror_size, name, mime_type FROM file_texts WHERE file_id = $1`,
|
||||||
@@ -612,6 +637,8 @@ function fileNode(file: CrawledFile): StoredNode {
|
|||||||
parentId: file.parentId,
|
parentId: file.parentId,
|
||||||
securityCheckStatus: file.record.securityCheckStatus,
|
securityCheckStatus: file.record.securityCheckStatus,
|
||||||
at: file.at,
|
at: file.at,
|
||||||
|
source: file.source ?? 'files-storage',
|
||||||
|
...(file.fsPath ? { fsPath: file.fsPath } : {}),
|
||||||
},
|
},
|
||||||
// File records are less immutable than they look: `PATCH /file/rename/{id}`
|
// File records are less immutable than they look: `PATCH /file/rename/{id}`
|
||||||
// changes the name in place, keeping the id and the size, and teachers do
|
// changes the name in place, keeping the id and the size, and teachers do
|
||||||
|
|||||||
291
test/legacy-files.test.ts
Normal file
291
test/legacy-files.test.ts
Normal file
@@ -0,0 +1,291 @@
|
|||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { describe, it } from 'node:test';
|
||||||
|
import { checkSignedUrl, type SchulcloudClient } from '../src/core/client.ts';
|
||||||
|
import {
|
||||||
|
childRef,
|
||||||
|
FileManager,
|
||||||
|
FileManagerMarkupError,
|
||||||
|
FsError,
|
||||||
|
pageFor,
|
||||||
|
parseFileListing,
|
||||||
|
splitPath,
|
||||||
|
} from '../src/core/legacy-files.ts';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fixtures follow the legacy client's templates as served (files/files.hbs and
|
||||||
|
* files/files-grid.hbs, client 33.40): a folder is an `openfolder` button whose
|
||||||
|
* name sits unescaped after an icon; a file is a `card file` div carrying its
|
||||||
|
* name, size and id as data attributes, with the viewer type further inside.
|
||||||
|
*/
|
||||||
|
const folder = (id: string, name: string) => `
|
||||||
|
<div class="col-xs-12">
|
||||||
|
<button class="card card-block folder openfolder" data-testid="folder-open-button" data-folder-id="${id}" aria-label="
|
||||||
|
files.label.clickToOpenDir">
|
||||||
|
<strong class="card-title-directory">
|
||||||
|
<i class="fa fa-folder" data-testid="folder-icon" aria-hidden="true"></i> ${name}
|
||||||
|
</strong>
|
||||||
|
</button>
|
||||||
|
</div>`;
|
||||||
|
|
||||||
|
const file = (id: string, name: string, size: number, type = 'application/pdf', blocked = false) => `
|
||||||
|
<div class="col-sm-12" data-testid="files-section">
|
||||||
|
<div class="card file ${blocked ? 'btn-file-danger' : ''}" data-href="/files/file?file=${name}" data-file-name="${name}" data-file-size="${size}" data-file-id="${id}">
|
||||||
|
<div class="openFile">
|
||||||
|
<div class="card-block
|
||||||
|
${blocked ? '' : ' fileviewer'}"
|
||||||
|
${blocked ? '' : ` data-file-viewer-type="${type}"`}
|
||||||
|
${blocked ? '' : ` data-file-viewer-id="${id}"`}
|
||||||
|
tabindex="0" role="button">
|
||||||
|
<a class="col-sm-10 title" data-testid="file-title">${name}</a>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>`;
|
||||||
|
|
||||||
|
const page = (folders: string[], files: string[]) => `
|
||||||
|
<html><body><main id="main-content" class="container-fluid">
|
||||||
|
<div class="route-files">
|
||||||
|
${folders.length ? `<section class="directories"><div class="row">${folders.join('')}</div></section>` : ''}
|
||||||
|
<section class="files"><div class="row">${files.join('')}</div></section>
|
||||||
|
</div></main></body></html>`;
|
||||||
|
|
||||||
|
const id = (n: number) => n.toString(16).padStart(24, '0');
|
||||||
|
|
||||||
|
describe('parseFileListing', () => {
|
||||||
|
it('reads folders and files with their ids, sizes and types', () => {
|
||||||
|
const listing = parseFileListing(
|
||||||
|
page([folder(id(1), 'Handlungssituation')], [file(id(2), '13_AB_Wareneingang.pdf', 171941)]),
|
||||||
|
);
|
||||||
|
assert.deepEqual(listing.directories, [{ id: id(1), name: 'Handlungssituation' }]);
|
||||||
|
assert.deepEqual(listing.files, [
|
||||||
|
{ id: id(2), name: '13_AB_Wareneingang.pdf', size: 171941, mimeType: 'application/pdf', blocked: false },
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('decodes entities in both folder and file names', () => {
|
||||||
|
const listing = parseFileListing(
|
||||||
|
page([folder(id(1), 'Lager & Logistik')], [file(id(2), 'A&B "final".pdf', 10)]),
|
||||||
|
);
|
||||||
|
assert.equal(listing.directories[0]?.name, 'Lager & Logistik');
|
||||||
|
assert.equal(listing.files[0]?.name, 'A&B "final".pdf');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('marks a file the virus scanner blocked, which also has no viewer type', () => {
|
||||||
|
const listing = parseFileListing(page([], [file(id(3), 'bad.exe', 5, undefined, true), file(id(4), 'ok.png', 7, 'image/png')]));
|
||||||
|
assert.equal(listing.files[0]?.blocked, true);
|
||||||
|
assert.equal(listing.files[0]?.mimeType, undefined);
|
||||||
|
// The next card's type must not leak into the blocked one.
|
||||||
|
assert.equal(listing.files[1]?.mimeType, 'image/png');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns an empty listing for an empty folder', () => {
|
||||||
|
assert.deepEqual(parseFileListing(page([], [])), { directories: [], files: [] });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('refuses a page that is not a file-manager listing rather than reporting 0 files', () => {
|
||||||
|
assert.throws(() => parseFileListing('<html><body>Anmelden</body></html>'), FileManagerMarkupError);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('pageFor and childRef', () => {
|
||||||
|
it('builds each listing route', () => {
|
||||||
|
assert.equal(pageFor({}), undefined);
|
||||||
|
assert.equal(pageFor({ area: 'my' }), '/files/my/');
|
||||||
|
assert.equal(pageFor({ area: 'my', folderId: id(1) }), `/files/my/${id(1)}`);
|
||||||
|
assert.equal(pageFor({ area: 'courses' }), '/files/courses/');
|
||||||
|
assert.equal(pageFor({ area: 'courses', ownerId: id(2) }), `/files/courses/${id(2)}`);
|
||||||
|
assert.equal(pageFor({ area: 'teams', ownerId: id(2), folderId: id(3) }), `/files/teams/${id(2)}/${id(3)}`);
|
||||||
|
assert.equal(pageFor({ area: 'shared' }), '/files/shared/');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('addresses a nested folder by its own id, whatever its depth', () => {
|
||||||
|
const course = childRef({ area: 'courses' }, { id: id(2), name: 'Kurs' });
|
||||||
|
const top = childRef(course, { id: id(3), name: 'A' });
|
||||||
|
const deep = childRef(top, { id: id(4), name: 'B' });
|
||||||
|
assert.deepEqual(deep, { area: 'courses', ownerId: id(2), folderId: id(4) });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('refuses to open a shared folder, which the file manager cannot do either', () => {
|
||||||
|
assert.throws(
|
||||||
|
() => childRef({ area: 'shared' }, { id: id(5), name: 'Ordner' }),
|
||||||
|
(error: unknown) => error instanceof FsError && error.code === 'not_navigable',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('splitPath', () => {
|
||||||
|
it('drops empty segments and surrounding spaces', () => {
|
||||||
|
assert.deepEqual(splitPath('//courses/ Kurs A /x.pdf/'), ['courses', 'Kurs A', 'x.pdf']);
|
||||||
|
assert.deepEqual(splitPath('/'), []);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
/** A client that serves canned listing pages and records what was asked for. */
|
||||||
|
function fakeClient(pages: Record<string, string>) {
|
||||||
|
const requested: string[] = [];
|
||||||
|
const client = {
|
||||||
|
getFileManagerPage: async (path: string) => {
|
||||||
|
requested.push(path);
|
||||||
|
const html = pages[path];
|
||||||
|
if (html === undefined) throw new Error(`no fixture for ${path}`);
|
||||||
|
return html;
|
||||||
|
},
|
||||||
|
} as unknown as SchulcloudClient;
|
||||||
|
return { client, requested };
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('FileManager.resolve', () => {
|
||||||
|
const course = id(10);
|
||||||
|
const slashCourse = id(11);
|
||||||
|
const sub = id(12);
|
||||||
|
const pdf = id(13);
|
||||||
|
const pages = {
|
||||||
|
'/files/courses/': page([folder(course, 'FIA24B - LF2 (Rh)'), folder(slashCourse, 'LF07 - FIA24A/B - Sb/Ha')], []),
|
||||||
|
[`/files/courses/${course}`]: page([folder(sub, 'Handlungssituation')], [file(pdf, 'Mahnwesen.pdf', 100)]),
|
||||||
|
[`/files/courses/${course}/${sub}`]: page([], [file(id(14), 'Lager.pdf', 50)]),
|
||||||
|
[`/files/courses/${slashCourse}`]: page([], [file(id(15), 'MQTT.docx', 70, 'application/vnd.openxmlformats-officedocument.wordprocessingml.document')]),
|
||||||
|
'/files/my/': page([], []),
|
||||||
|
};
|
||||||
|
|
||||||
|
it('resolves the root and each area without a request for the root', async () => {
|
||||||
|
const { client, requested } = fakeClient(pages);
|
||||||
|
const fs = new FileManager(client);
|
||||||
|
assert.deepEqual(await fs.resolve('/'), { kind: 'directory', path: '/', ref: {}, name: '/' });
|
||||||
|
const area = await fs.resolve('/Kurs-Dateien');
|
||||||
|
assert.equal(area.kind, 'directory');
|
||||||
|
assert.deepEqual(area.kind === 'directory' && area.ref, { area: 'courses' });
|
||||||
|
assert.deepEqual(requested, []);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('walks names down to a file in a nested folder', async () => {
|
||||||
|
const { client } = fakeClient(pages);
|
||||||
|
const node = await new FileManager(client).resolve('/courses/FIA24B - LF2 (Rh)/Handlungssituation/Lager.pdf');
|
||||||
|
assert.equal(node.kind, 'file');
|
||||||
|
assert.equal(node.kind === 'file' && node.file.id, id(14));
|
||||||
|
assert.equal(node.path, '/courses/FIA24B - LF2 (Rh)/Handlungssituation/Lager.pdf');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('resolves a course whose name contains slashes, typed plainly', async () => {
|
||||||
|
const { client } = fakeClient(pages);
|
||||||
|
const node = await new FileManager(client).resolve('/courses/LF07 - FIA24A/B - Sb/Ha/MQTT.docx');
|
||||||
|
assert.equal(node.kind === 'file' && node.file.id, id(15));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('accepts ids as path segments', async () => {
|
||||||
|
const { client } = fakeClient(pages);
|
||||||
|
const node = await new FileManager(client).resolve(`/courses/${course}/${pdf}`);
|
||||||
|
assert.equal(node.kind === 'file' && node.file.name, 'Mahnwesen.pdf');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to a case-insensitive match', async () => {
|
||||||
|
const { client } = fakeClient(pages);
|
||||||
|
const node = await new FileManager(client).resolve('/courses/fia24b - lf2 (rh)/handlungssituation');
|
||||||
|
assert.equal(node.kind, 'directory');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('says what is there when a name is not found', async () => {
|
||||||
|
const { client } = fakeClient(pages);
|
||||||
|
await assert.rejects(
|
||||||
|
new FileManager(client).resolve('/courses/FIA24B - LF2 (Rh)/Handlung'),
|
||||||
|
(error: unknown) => error instanceof FsError && error.code === 'not_found' && /Handlungssituation/.test(error.message),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('refuses to treat a file as a folder', async () => {
|
||||||
|
const { client } = fakeClient(pages);
|
||||||
|
await assert.rejects(
|
||||||
|
new FileManager(client).resolve('/courses/FIA24B - LF2 (Rh)/Mahnwesen.pdf/more'),
|
||||||
|
(error: unknown) => error instanceof FsError && error.code === 'not_a_directory',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reports an ambiguous name with the ids to use instead', async () => {
|
||||||
|
const twins = {
|
||||||
|
'/files/my/': page([folder(id(20), 'Kopie'), folder(id(21), 'Kopie')], []),
|
||||||
|
};
|
||||||
|
const { client } = fakeClient(twins);
|
||||||
|
await assert.rejects(
|
||||||
|
new FileManager(client).resolve('/my/Kopie'),
|
||||||
|
(error: unknown) =>
|
||||||
|
error instanceof FsError && error.code === 'ambiguous' && error.message.includes(id(20)) && error.message.includes(id(21)),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('rejects an unknown area and names the real ones', async () => {
|
||||||
|
const { client } = fakeClient(pages);
|
||||||
|
await assert.rejects(
|
||||||
|
new FileManager(client).resolve('/Downloads/x'),
|
||||||
|
(error: unknown) => error instanceof FsError && /\/courses/.test(error.message),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('caches a listing, so ls followed by read costs no second request', async () => {
|
||||||
|
const { client, requested } = fakeClient(pages);
|
||||||
|
const fs = new FileManager(client);
|
||||||
|
await fs.resolve('/courses/FIA24B - LF2 (Rh)/Mahnwesen.pdf');
|
||||||
|
await fs.resolve('/courses/FIA24B - LF2 (Rh)/Mahnwesen.pdf');
|
||||||
|
assert.equal(requested.filter((path) => path === '/files/courses/').length, 1);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('FileManager.walk', () => {
|
||||||
|
it('stays within its folder budget and says so', async () => {
|
||||||
|
const course = id(30);
|
||||||
|
const pages = {
|
||||||
|
'/files/courses/': page([folder(course, 'Kurs')], []),
|
||||||
|
[`/files/courses/${course}`]: page([folder(id(31), 'A'), folder(id(32), 'B')], []),
|
||||||
|
[`/files/courses/${course}/${id(31)}`]: page([], [file(id(33), 'a.pdf', 1)]),
|
||||||
|
[`/files/courses/${course}/${id(32)}`]: page([], [file(id(34), 'b.pdf', 1)]),
|
||||||
|
};
|
||||||
|
const { client } = fakeClient(pages);
|
||||||
|
const result = await new FileManager(client).walk(
|
||||||
|
{ path: '/courses', ref: { area: 'courses' } },
|
||||||
|
{ maxDepth: 5, maxDirectories: 2 },
|
||||||
|
);
|
||||||
|
assert.equal(result.visited, 2);
|
||||||
|
assert.equal(result.truncated, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('records a folder it could not read instead of dropping it', async () => {
|
||||||
|
const course = id(40);
|
||||||
|
const pages = { '/files/courses/': page([folder(course, 'Kurs')], []) };
|
||||||
|
const { client } = fakeClient(pages);
|
||||||
|
const result = await new FileManager(client).walk(
|
||||||
|
{ path: '/courses', ref: { area: 'courses' } },
|
||||||
|
{ maxDepth: 3, maxDirectories: 10 },
|
||||||
|
);
|
||||||
|
assert.equal(result.failures.length, 1);
|
||||||
|
assert.equal(result.failures[0]?.path, '/courses/Kurs');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('the file-manager allowlist', () => {
|
||||||
|
// Some legacy GET routes write. The client must refuse them before any request.
|
||||||
|
const client = {
|
||||||
|
config: { baseUrl: 'https://example.org', jwt: 'x', requestTimeoutMs: 1000, maxDownloadBytes: 10 },
|
||||||
|
};
|
||||||
|
|
||||||
|
it('refuses the routes that write despite being GETs', async () => {
|
||||||
|
const { SchulcloudClient } = await import('../src/core/client.ts');
|
||||||
|
const real = new SchulcloudClient(client.config as never);
|
||||||
|
for (const path of ['/files/share/?file=' + id(1), `/files/file?file=${id(1)}&share=abc`, `/files/fileModel/${id(1)}/proxy`, '/files/search/?q=x', '/files/permittedDirectories/']) {
|
||||||
|
await assert.rejects(real.getFileManagerPage(path), /refusing file-manager path/, path);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('checkSignedUrl', () => {
|
||||||
|
it('accepts an https storage url from an https instance', () => {
|
||||||
|
assert.equal(checkSignedUrl('https://s3.example.com/bucket/key?X-Amz-Signature=1', 'https://schulcloud.example').host, 's3.example.com');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('refuses plaintext from an https instance, and credentials in the url', () => {
|
||||||
|
assert.throws(() => checkSignedUrl('http://10.0.0.1/x', 'https://schulcloud.example'), /refusing a http:/);
|
||||||
|
assert.throws(() => checkSignedUrl('https://user:pw@s3.example.com/x', 'https://schulcloud.example'), /credentials/);
|
||||||
|
assert.throws(() => checkSignedUrl('file:///etc/passwd', 'http://localhost:4400'), /refusing a file:/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('allows http when the instance itself is http, as the local one is', () => {
|
||||||
|
assert.equal(checkSignedUrl('http://localhost:9900/x', 'http://localhost:4400').protocol, 'http:');
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user