Browse the file manager ("Dateien") as a filesystem

Many teachers never use topics or boards; their material sits in the
course's file area, and the tools answered "0 files" for courses holding
dozens of worksheets — 21 of 26 courses on the live account. Persönliche,
Kurs-, Team- and Geteilte Dateien live in the legacy file store, not in
files-storage, and its service is not in the public ingress. The only way in
is the legacy client: HTML listings, and GET /files/signedurl for a
pre-signed download.

core/legacy-files.ts turns that into one path tree — /my, /courses/<course>,
/teams/<team>, /shared — resolving names that contain "/", ids anywhere in a
path, and wrong or ambiguous names with a message saying what is there. A
listing that does not parse throws; it never reads as an empty folder.

Some of the legacy client's GET routes write (GET /files/share/ mints a
share token), so getFileManagerPage allows only the listing routes, by
pattern. Signed URLs are fetched with no credentials and must be https.

- MCP: fs_list, fs_tree, fs_find and fs_read; get_course lists course files.
- CLI: schulcloud fs ls, tree, find and get, recursive and resumable.
- API: /api/fs/list, tree, find and file.
- Index: the crawl walks the file manager (INDEX_FILE_MANAGER, on by
  default), so search covers the text inside those files and sync mirrors
  them under <course>/Kurs-Dateien.

The local instance gains a fixture for all four areas. It needed a loopback,
so signed URLs open from the host, and a pre-created bucket, since MinIO
does not implement PutBucketCors.

135 tests. Smoke 55/55 live; 57/57 and 55/55 on the local instance.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-09-16 20:19:16 +02:00
parent 10c6544579
commit bed3923902
32 changed files with 2696 additions and 106 deletions

View File

@@ -52,6 +52,12 @@ DATABASE_URL=postgresql://schulcloud:schulcloud@postgres:5432/schulcloud
# three extra requests per task on a full crawl, so it is off by default.
# INDEX_PERSONAL_FILES=false
# Also walk the file manager ("Dateien") when crawling: Kurs-Dateien for every
# course, plus Persönliche, Team- and Geteilte Dateien on a full crawl. Many
# teachers keep their material only there, so this is on by default. One page
# load per folder — about 160 on a 26-course account.
# INDEX_FILE_MANAGER=true
# How often to re-crawl on a timer, in ms. Default 21600000 (6h). 0 = on demand
# only. A re-crawl of unchanged content downloads nothing, because Schulcloud
# file records are immutable.

View File

@@ -39,8 +39,8 @@ index.
read-only with respect to Schulcloud. Run `smoke` after touching `src/core/`,
`src/mcp/` or `src/http/` — the unit tests cover only pure functions.
Run smoke **both ways**: with `DATABASE_URL` set (34 checks, index-backed) and
without (32 checks, live-only). The degradation path is a supported mode, not a
Run smoke **both ways**: with `DATABASE_URL` set (57 checks, index-backed) and
without (55 checks, live-only). The degradation path is a supported mode, not a
fallback nobody exercises.
Store tests need a database and skip without one:
@@ -71,6 +71,9 @@ bin/cli.ts ──HTTP──────────┘ cli/{config,client,sync
- **`crawl.ts`** — the one traversal. Search, the indexer, the what-changed
diff and the file mirror all need it; keep it here, not in a tool.
- `paths.ts` — the security boundary for mirrored filenames. See Invariants.
- `legacy-files.ts` — the file manager ("Dateien": Persönliche, Kurs-, Team-,
Geteilte Dateien) as one path tree, parsed from the legacy client's pages.
A separate store from files-storage; the `fs_*` tools and `/api/fs` sit on it.
- **`store/`** — crawl generations, identity diffs, `german` + `pg_trgm` FTS.
`Store.open` returns `undefined` when Postgres is down; callers degrade.
- **`indexer/`** — crawl → persist → mirror bytes → extract text → index.
@@ -92,7 +95,12 @@ bin/cli.ts ──HTTP──────────┘ cli/{config,client,sync
`GET` except `extendSession` (the keepalive's `refresh-session` call, which
touches only our own session and is not exposed as a tool, so no model-driven
call can be a POST). `api_get` rejects non-`/api/` paths and anything carrying a
scheme or host. `refresh_index` and `POST /api/refresh` write only to the Pi's
scheme or host. **Against the legacy client, GET-only is not enough:**
`GET /files/share/` mints a share token and `GET /files/file?share=` grants a
permission, so `getFileManagerPage` allows only the listing routes, by pattern —
widen that pattern only with a route you have read the handler of. A pre-signed
download URL is fetched with **no** credentials: it names another host, and
neither the bearer nor the `jwt` cookie may go with it. `refresh_index` and `POST /api/refresh` write only to the Pi's
own index and mirror — every upstream call they make is still a GET.
**Filenames from Schulcloud are untrusted paths.** Course titles, card titles
@@ -217,10 +225,22 @@ These cost real time to discover; `docs/API.md` has the full list with evidence.
upstream: interactive exercises are H5P elements, whose `contentId` is the
only handle onto the content, or external (LTI) tools behind
`contextExternalToolId`. Say that rather than looking for a quiz API.
- **The file manager is a third store, reachable only as HTML.** Persönliche,
Kurs-, Team- and Geteilte Dateien live in the legacy `files` collection, not in
files-storage: `list_files` answers 0 for a course holding dozens of
worksheets, and 21 of 26 live courses keep material there. Its Feathers service
is not in the ingress, so listings are parsed from `/files/{my,courses,teams,shared}`
pages and downloads go through `GET /files/signedurl` (JSON). Folders are
addressed by id alone — `/files/courses/{course}/{folder}` at any depth.
`permittedDirectories` returns every course with **no** folders (it queries
`refOwnerModel: 'courses'`, records say `'course'`), and `/files/search/` 504s,
so walk listings. Course names contain `/`; resolve by joining segments. A
listing page that does not parse must throw, never read as an empty folder —
"0 files" is the bug this exists to fix. `docs/API.md` has the evidence.
- **Teams cannot be read at any version.** v3 exposes only
`GET /team/{teamId}/news`; upstream `main`'s teams controller is write-only
(`POST :teamId/create-room`). `/teams` is the legacy client's HTML page, not
an API. This one is genuinely unavailable, not merely uncovered.
an API. A team's *files* are reachable, through the file manager (`/teams/<team>`).
- **`exp` (30 days) is not the session lifetime.** The binding limit is a Valkey
whitelist entry with a `JWT_TIMEOUT_SECONDS` TTL (7200s; live value at
`GET /api/v3/config/public`) that every authenticated request re-sets.

View File

@@ -17,7 +17,7 @@ instance, not inferred from the upstream source.
> *"Find the material about Verschlüsselung and explain the Caesar cipher worksheet."*
> *"Summarise the routing lesson from the LF10 course."*
Twenty tools, all read-only:
Twenty-four tools, all read-only:
| | |
|---|---|
@@ -30,8 +30,12 @@ Twenty tools, all read-only:
| `list_tasks` | homework across all courses, by due date |
| `get_task` | one task: description, due date, attachments, **and your submission** |
| `list_submissions` | what you handed in, and what is still ungraded |
| `list_files` | files attached to any entity |
| `download_file` | fetch a file and extract its text, or view an image |
| `list_files` | files attached to a board element, topic, task or submission |
| `download_file` | fetch such an attachment and extract its text, or view an image |
| `fs_list` | list a folder of the file manager — Persönliche, Kurs-, Team-, Geteilte Dateien |
| `fs_tree` | everything below a file-manager folder, as a tree |
| `fs_find` | find file-manager files and folders by name |
| `fs_read` | read a file-manager file, extracted like `download_file` |
| `search` | keyword search across everything — **including the text inside PDFs and Office files** |
| `refresh_index` | re-read Schulcloud now, per course or in full |
| `what_changed` | what appeared, changed or vanished since a date |
@@ -54,6 +58,8 @@ schulcloud login --server https://mcp.example.org --token <token>
schulcloud sync --dry-run # see what would be mirrored
schulcloud sync # mirror coursework to ~/Schulcloud
schulcloud refresh --course <id>
schulcloud fs tree /courses # browse the file manager ("Dateien")
schulcloud fs get "/courses/<course>/<folder>"
```
It talks only to the Pi and holds no Schulcloud credential — see
@@ -151,7 +157,7 @@ npm run typecheck
```
`npm run smoke` starts the HTTP server, connects a real MCP client over
Streamable HTTP and exercises every tool against the live account — 30 checks
Streamable HTTP and exercises every tool against the live account — 55 checks (57 with the index)
covering the auth gate, the protocol handshake, every content chain, file
extraction, `api_get`'s guard rails and error handling.

View File

@@ -225,6 +225,69 @@ hops and without Etherpad's API key; `core/etherpad.ts` does this. The url is
built from the server's `ETHERPAD__PAD_URI`, so it must be checked against the
instance host before the session cookie is sent to it.
## The file manager ("Dateien") is a third store
Persönliche Dateien, Kurs-Dateien, Team-Dateien and Geteilte Dateien are the
**legacy file system**: a `files` collection with real folders (`isDirectory`,
`parent`, `owner`, `refOwnerModel`), served by the legacy Feathers
`fileStorage` service. It shares nothing with files-storage. Asking
`/api/v3/file/list/school/{school}/courses/{courseId}` answers **0** for a
course whose file manager holds dozens of worksheets — measured: 21 of 26
courses on the live account keep files here, some nothing else.
**Its service is not in the public ingress**, so the only way in is the legacy
client. Verified live:
| Route (legacy client, `jwt` cookie) | Returns | Notes |
|---|---|---|
| `GET /files/my/` , `/files/my/{folder}` | HTML listing | personal root / one folder |
| `GET /files/courses/` | HTML listing | the courses, *as folders* (ids = course ids) |
| `GET /files/courses/{course}` , `/files/courses/{course}/{folder}` | HTML listing | **one** folder segment at any depth |
| `GET /files/teams/…` | HTML listing | same shape as courses |
| `GET /files/shared/` | HTML listing | flat; shared *folders* cannot be opened (no route; the UI's link 404s) |
| `GET /files/signedurl?file={id}&name={name}` | `{"url": …}` | pre-signed S3 url, **another host** (live: `s3.hidrive.strato.com`) |
| `GET /files/permittedDirectories/` | JSON tree | **lists every course with no folders in any** — see below |
| `GET /files/search/?q=` | HTML | **504** on live: an unindexed regex over every file record |
Listings are parsed on the attributes the page's own scripts use:
`data-folder-id` with the name inside `.card-title-directory` (emitted
**unescaped**, `{{{stripOnlyScript name}}}`), and `data-file-id`,
`data-file-name`, `data-file-size`, `data-file-viewer-type` on each
`.card.file`. A blocked file carries `btn-file-danger` and no viewer type. No
dates are rendered.
**`permittedDirectories` is broken for courses.** The directory service's
query matches course folders on `refOwnerModel: 'courses'`; the records say
`'course'`. Live result: 26 courses, 0 folders; personal folders come through.
Listings are the only complete view — which is also what the UI shows.
**Some GET routes write.** `GET /files/share/?file=` mints a share token when
the file has none (`PATCH /fileStorage/shared/{id}`), and
`GET /files/file?…&share=…` grants the caller a permission on the file.
`GET /files/fileModel/{id}/proxy` forwards to the latter. A GET-only client is
therefore *not* read-only against this surface by itself: `core/client.ts`
allows only the listing routes and `/files/signedurl`, by pattern.
**The signed-url service returns its error instead of throwing it**
(`.catch((err) => new Forbidden(err))`), so a refused file is a 200 whose body
has no `url`.
**Course names contain `/`** in real data ("LF07 - FIA24A/B - Sb/Ha",
"FIA24/FIP24 IT LF12"), and a real file is literally called `..docx`. Paths
built from names cannot be split naively; `core/legacy-files.ts` resolves by
trying joined segments, and accepts ids as segments.
Two more, seen while building the local fixture:
- **`getRefOwnerModel(owner)` answers "a course, or else `teams`".** Any owner
id that is not a course — a *user* included — is recorded as a team's. The
upload page never sends an owner for personal files (`data-owner=""`), so the
server defaults to the creator and records `user`; send the user id and every
later permission check dereferences a team that does not exist.
- **The file permission service writes `refOwnerModel`** where the "shared with
me" query reads `refPermModel`, so a share made through it never appears
under Geteilte Dateien. The share-link flow patches `/files/{id}` directly.
## Re-verifying after an upstream release
`npm run probe` re-checks every assumption above against the live instance and

View File

@@ -50,6 +50,39 @@ alongside courses, with their files under the room's name rather than a course's
what changed: that is a handful of requests, where a full re-crawl reads every
course. The server refuses a repeat within a minute unless you pass `--force`.
### The file manager (`fs`)
The Schulcloud file manager ("Dateien") — Persönliche, Kurs-, Team- and
Geteilte Dateien — browsed like a filesystem, live:
```
schulcloud fs ls [path] [--long]
schulcloud fs tree [path] [--depth <n>] [--max-folders <n>]
schulcloud fs find <name> [--path <path>] [--type file|folder] [--long]
schulcloud fs get <path> [--out <path>] [--force] [--jobs <n>]
```
```console
$ schulcloud fs ls /courses
$ schulcloud fs tree "/courses/FIA24B - SK (Rh)"
$ schulcloud fs find "*Erben*" --path /courses
$ schulcloud fs get "/courses/FIA24B - SK (Rh)/02_Erbrecht" --out ~/Erbrecht
```
The tree is `/my`, `/courses/<course>`, `/teams/<team>` and `/shared`; the
German names ("/Kurs-Dateien") work too. Names may contain `/` — course names
often do — and still resolve; any segment may also be an id from `--long`.
`fs find` matches any part of a name, or, given `*` or `?`, the whole name as
`find -name` does. `fs get` on a folder downloads everything below it, keeps
the structure, and skips files already present at the same size — so re-running
it resumes. Each folder is one page load on the server, so large trees take a
while.
`sync` mirrors these files too, under `<course>/Kurs-Dateien/…`,
`Persönliche Dateien/…`, `Team-Dateien/<team>/…` and `Geteilte Dateien/`, once
the server's index includes them (`INDEX_FILE_MANAGER`, on by default).
## How sync works
It is a **one-way mirror, not a two-way sync**, and that follows from the data

View File

@@ -103,9 +103,9 @@ node dist/bin/cli.js sync
## Run the test suites
```bash
npm test # 58 offline tests
npm test # 135 offline tests
npm run smoke # end-to-end against the live instance, live-only mode
DATABASE_URL=… npm run smoke # end-to-end with the index (34 checks)
DATABASE_URL=… npm run smoke # end-to-end with the index (57 checks)
```
Store tests need a database and skip without one:

View File

@@ -89,6 +89,7 @@ Every one of these is marked in the env files at the line it affects.
```
docker-compose.yml the stack; profiles: (default) | tools | av | preview
(minio-loopback: see "Simulating a teacher")
env/ one file per service, all values local-only
proxy/nginx.conf GENERATED — the single origin, see scripts/gen-proxy-conf.py
scripts/seed.sh loads the demo school via the management app
@@ -183,14 +184,40 @@ node scripts/simulate-teacher.mjs create # course, room, topic, task, board,
# Etherpad pad, folder, files
node scripts/simulate-teacher.mjs update # rename and rewrite all of it
node scripts/simulate-teacher.mjs delete # remove it again
node scripts/simulate-teacher.mjs files # only the file-manager part, onto an existing fixture
```
It also fills the **file manager** ("Dateien") — the legacy file system behind
Persönliche, Kurs-, Team- and Geteilte Dateien, a separate store from the board
files above — with a course folder tree ("Arbeitsblätter/Woche 1"), a team
folder, a folder in the demo student's own files, and a teacher's file shared
read-only with the student. The student is added to the adopted team so the
team files are visible from the account under test. Every file carries a unique
search term, so search over the index can be checked per area.
Legacy file uploads only work because of two changes to the stack:
- **`minio-loopback`.** The legacy service signs upload and download URLs for
its storage provider's single endpoint, and uses that endpoint for its own S3
calls. `minio:9000` works inside the compose network and nowhere else, so the
browser and the MCP server on the host were handed URLs they could not open.
`seed.sh` now registers `http://localhost:9900`, and this socat sidecar,
sharing the api container's network namespace, forwards that address to
MinIO — the same url then works from the api, the browser and the host.
After recreating `api`, recreate `minio-loopback` too: it lives in api's
network namespace.
- **The school bucket is created up front.** The legacy service makes
`bucket-<schoolId>` on first upload and then calls `PutBucketCors`, which
MinIO does not implement, so the first upload failed with *"A header you
provided implies functionality that is not implemented"*. `minio-init.sh`
creates the demo school's bucket, and an existing bucket skips both calls.
Ids are kept in `.simulate-teacher.json` between phases, so the MCP server can
be pointed at the instance in between:
```bash
eval "$(./scripts/mcp-env.sh)" # as the demo student
cd .. && npm run smoke # 39 checks against the local instance
cd .. && npm run smoke # 57 checks against the local instance
```
`mcp-env.sh` points the index at its own database, `schulcloud_local`, and the

View File

@@ -154,6 +154,24 @@ services:
minio: {condition: service_healthy}
restart: unless-stopped
minio-loopback:
# The legacy file service (the file manager: Persönliche/Kurs-/Team-Dateien)
# signs upload and download URLs for its storage provider's one endpoint, and
# the same endpoint serves its own S3 calls. "minio:9000" works inside the
# compose network and nowhere else, so a browser or the MCP server on this
# machine was handed URLs it could not open: uploads through the UI failed
# and downloads could not be tested. seed.sh registers the provider as
# http://localhost:9900 instead, and this forwards that address to MinIO from
# inside the api container's own network namespace — so the one URL now
# works from the api, from the browser and from the host alike.
image: alpine/socat:1.8.0.1
network_mode: "service:api"
command: ["TCP-LISTEN:9900,fork,reuseaddr,bind=127.0.0.1", "TCP:minio:9000"]
depends_on:
api: {condition: service_started}
minio: {condition: service_healthy}
restart: unless-stopped
file-preview:
# Generates thumbnails via ImageMagick, driven off RabbitMQ. Optional: with
# it absent, files still upload and download, they just have no preview.

View File

@@ -11,9 +11,16 @@ for bucket in \
h5p-content-bucket ` # h5p-editor content` \
h5p-library-bucket ` # h5p content types` \
ydocs ` # tldraw whiteboard documents` \
fwu-content # FWU media, unused but cheap to create
fwu-content ` # FWU media, unused but cheap to create` \
bucket-5f2987e020834114b8efd6f6 # legacy file manager, demo school (see below)
do
mc mb --ignore-existing "local/$bucket"
done
# The legacy file manager keeps one bucket per school, "bucket-<schoolId>", and
# creates it on first upload — then calls PutBucketCors, which MinIO does not
# implement, so the first upload fails with "A header you provided implies
# functionality that is not implemented". A bucket that already exists skips
# both calls. 5f2987e020834114b8efd6f6 is the demo school's fixed seed id.
mc ls local

View File

@@ -39,7 +39,10 @@ SECRET=$(curl -fsS -X POST "$MGMT/encrypt-plain-text" \
const id = ObjectId('62949a4003839b6162aa566b');
db.storageproviders.replaceOne({ _id: id }, {
_id: id, isShared: true, region: 'eu-central-1', type: 'S3',
endpointUrl: 'http://minio:9000',
// Not minio:9000: this one endpoint also goes into every signed URL, and
// those are opened by the browser and the MCP server on the host. The
// minio-loopback service makes localhost:9900 reach MinIO from the api too.
endpointUrl: 'http://localhost:9900',
accessKeyId: 'miniouser',
secretAccessKey: '$SECRET',
maxBuckets: 150, freeBuckets: 138,

View File

@@ -11,6 +11,7 @@
* node scripts/simulate-teacher.mjs create # build the fixture, print ids
* node scripts/simulate-teacher.mjs update # rename/edit everything it made
* node scripts/simulate-teacher.mjs delete # remove it again
* node scripts/simulate-teacher.mjs files # (re)build only the file-manager part
*
* State lives in .simulate-teacher.json so the phases can be run one at a time
* with MCP checks in between.
@@ -295,8 +296,9 @@ async function create() {
log(`second board left unpublished on purpose: ${draft.id}`);
saveState(s);
await fileManager();
console.log(`\nstate written to ${STATE}`);
summary(s);
summary(loadState());
}
/** files-storage attaches bytes to a board node (element) id, not to the card. */
@@ -307,6 +309,146 @@ async function upload(parentId, name, type, body) {
return record.id;
}
// ---------------------------------------------------------- file manager ---
//
// The file manager ("Dateien": Persönliche, Kurs-, Team- and Geteilte Dateien)
// is the legacy file system, a different store from files-storage above, with a
// real folder tree. Many teachers use nothing else, so the MCP server's fs_*
// tools need content there. Its services are only reachable on the server's own
// port, like the other /api/v1 writes in this script.
const STUDENT_PASSWORD = process.env.SIM_STUDENT_PASSWORD ?? 'schulcloud';
const TEAM_MEMBER_ROLE = '5bb5c190fb457b1c3c0c7e0f'; // "teammember" in the seed
/** Runs `fn` signed in as another account, then restores the teacher. */
async function asUser(email, password, fn) {
const saved = { jwt, me };
const res = await fetch(`${API}/api/v3/authentication/local`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ username: email, password }),
});
if (!res.ok) throw new Error(`login as ${email} failed: ${res.status}`);
jwt = (await res.json()).accessToken;
me = await v3('GET', '/me');
try {
return await fn();
} finally {
({ jwt, me } = saved);
}
}
async function legacyDir(name, owner, parent) {
return (await v1('POST', '/fileStorage/directories', { name, owner, parent }))._id;
}
/**
* The browser's own upload sequence: a signed PUT url, the bytes, then the
* record. The url comes from the server, so it is held to the same localhost
* rule as everything else this script writes to.
*/
async function legacyUpload({ owner, parent, name, type, body }) {
const bytes = Buffer.from(body);
const signed = await v1('POST', '/fileStorage/signedUrl', { parent, filename: name, fileType: type });
const { hostname } = new URL(signed.url);
if (hostname !== '127.0.0.1' && hostname !== 'localhost') {
throw new Error(`refusing to upload to ${hostname}: not a localhost address (see minio-loopback in docker-compose.yml)`);
}
const put = await fetch(signed.url, { method: 'PUT', headers: signed.header, body: bytes });
if (!put.ok) throw new Error(`PUT ${name} to storage -> ${put.status} ${(await put.text()).slice(0, 200)}`);
const record = await v1('POST', '/fileStorage', {
name,
owner,
parent,
type,
size: bytes.length,
storageFileName: signed.header['x-amz-meta-flat-name'],
});
return record._id;
}
async function fileManager() {
const s = loadState();
if (!s.courseId) throw new Error('run `create` first: the file-manager fixture lives in its course');
const keep = (key, value) => {
s[key] = value;
saveState(s);
};
const student = await studentId();
step('file manager: Kurs-Dateien');
keep('fmCourseRootFileId', await legacyUpload({
owner: s.courseId, name: 'Kursplan.txt', type: 'text/plain',
body: 'Kursplan Biologie\n\nThemen: Zelle, Gewebe, Organe. Stichwort: Photosynthese-Lichtreaktion.\n',
}));
keep('fmCourseDirId', await legacyDir('Arbeitsblätter', s.courseId));
keep('fmCourseFileId', await legacyUpload({
owner: s.courseId, parent: s.fmCourseDirId, name: 'Blatt 1 - Zellorganellen.txt', type: 'text/plain',
body: 'Blatt 1: Zellorganellen\n\nBeschrifte die Mitochondrienmembran und das endoplasmatische Retikulum.\n',
}));
keep('fmCourseSubDirId', await legacyDir('Woche 1', s.courseId, s.fmCourseDirId));
keep('fmCourseDeepFileId', await legacyUpload({
owner: s.courseId, parent: s.fmCourseSubDirId, name: 'Blatt 2 - Gewebe.txt', type: 'text/plain',
body: 'Blatt 2: Gewebe\n\nVergleiche Epithelgewebe und Bindegewebe.\n',
}));
log(`course root file, folder "Arbeitsblätter" with a file, and "Woche 1" nested inside it`);
if (s.teamId) {
step('file manager: Team-Dateien');
// Teams cannot be created (see the README), and the adopted one does not
// include the demo student, whose view is the one under test.
const team = await v1('GET', `/teams/${s.teamId}`);
if (!team.userIds.some((entry) => String(entry.userId?._id ?? entry.userId) === student)) {
const userIds = team.userIds.map((entry) => ({
userId: String(entry.userId?._id ?? entry.userId),
role: String(entry.role?._id ?? entry.role),
schoolId: String(entry.schoolId?._id ?? entry.schoolId),
}));
userIds.push({ userId: student, role: TEAM_MEMBER_ROLE, schoolId: me.school.id });
await v1('PATCH', `/teams/${s.teamId}`, { userIds });
keep('fmStudentAddedToTeam', true);
log('demo student added to the team');
}
keep('fmTeamDirId', await legacyDir('Projekt', s.teamId));
keep('fmTeamFileId', await legacyUpload({
owner: s.teamId, parent: s.fmTeamDirId, name: 'Projektplan.txt', type: 'text/plain',
body: 'Projektplan\n\nMeilenstein Chlorophyll bis Freitag.\n',
}));
log('team folder "Projekt" with a file');
}
step('file manager: Persönliche Dateien (the student\'s own)');
// No `owner` for personal files, exactly as the upload page sends none: the
// server decides the owner model as "a course, or else a team", so passing a
// user id records the folder as a team's, and every later permission check on
// it then dereferences a team that does not exist.
await asUser(STUDENT_EMAIL, STUDENT_PASSWORD, async () => {
keep('fmStudentDirId', await legacyDir('Notizen'));
keep('fmStudentFileId', await legacyUpload({
parent: s.fmStudentDirId, name: 'Lernzettel.txt', type: 'text/plain',
body: 'Lernzettel\n\nRibosomenfabrik: Proteinbiosynthese am rauen ER.\n',
}));
});
log('student folder "Notizen" with a file');
step('file manager: Geteilte Dateien');
keep('fmSharedFileId', await legacyUpload({
name: 'Geteilt vom Lehrer.txt', type: 'text/plain',
body: 'Zusatzmaterial\n\nDie Zellkernhuelle trennt Kernplasma und Zytoplasma.\n',
}));
// What accepting a share link does in the legacy client: a read-only user
// permission. Not the permission service, which writes `refOwnerModel` where
// the "shared with me" query reads `refPermModel`, so its shares never show.
const shared = await v1('GET', `/files/${s.fmSharedFileId}`);
await v1('PATCH', `/files/${s.fmSharedFileId}`, {
permissions: [
...shared.permissions,
{ refId: student, refPermModel: 'user', read: true, write: false, delete: false, create: false },
],
});
log('teacher file shared read-only with the student');
}
// ---------------------------------------------------------------- update ---
async function update() {
@@ -348,6 +490,17 @@ async function update() {
await files('PATCH', `/rename/${s.fileId}`, { fileName: 'nervensystem-notiz-v2.txt' });
log('file renamed');
if (s.fmCourseFileId) {
step('file manager edits');
await v1('POST', '/fileStorage/rename', { id: s.fmCourseFileId, newName: 'Blatt 1 - Zellorganellen (korrigiert).txt' });
log('course file renamed');
s.fmCourseAddedFileId = await legacyUpload({
owner: s.courseId, parent: s.fmCourseSubDirId, name: 'Blatt 3 - Organe.txt', type: 'text/plain',
body: 'Blatt 3: Organe\n\nNeuer Suchbegriff: Nephronschleife.\n',
});
log('course file added in "Woche 1" (new search term: Nephronschleife)');
}
saveState({ ...s, updated: true });
}
@@ -376,6 +529,46 @@ async function remove() {
['second room', () => v3('DELETE', `/rooms/${s.roomWithoutStudentId}`)],
['course', () => v1('DELETE', `/courses/${s.courseId}`)],
];
// File-manager content goes first, while its course and team still exist.
const fileManagerTries = [
['file-manager course files', async () => {
for (const id of [s.fmCourseAddedFileId, s.fmCourseDeepFileId, s.fmCourseFileId, s.fmCourseRootFileId]) {
if (id) await v1('DELETE', `/fileStorage?_id=${id}`);
}
}],
['file-manager course folders', async () => {
for (const id of [s.fmCourseSubDirId, s.fmCourseDirId]) if (id) await v1('DELETE', `/fileStorage/directories?_id=${id}`);
}],
['file-manager team content', async () => {
if (s.fmTeamFileId) await v1('DELETE', `/fileStorage?_id=${s.fmTeamFileId}`);
if (s.fmTeamDirId) await v1('DELETE', `/fileStorage/directories?_id=${s.fmTeamDirId}`);
}],
['shared file', async () => {
if (s.fmSharedFileId) await v1('DELETE', `/fileStorage?_id=${s.fmSharedFileId}`);
}],
['student personal files', async () => {
if (!s.fmStudentFileId && !s.fmStudentDirId) return;
await asUser(STUDENT_EMAIL, STUDENT_PASSWORD, async () => {
if (s.fmStudentFileId) await v1('DELETE', `/fileStorage?_id=${s.fmStudentFileId}`);
if (s.fmStudentDirId) await v1('DELETE', `/fileStorage/directories?_id=${s.fmStudentDirId}`);
});
}],
['student team membership', async () => {
if (!s.fmStudentAddedToTeam || !s.teamId) return;
const student = await studentId();
const team = await v1('GET', `/teams/${s.teamId}`);
const userIds = team.userIds
.map((entry) => ({
userId: String(entry.userId?._id ?? entry.userId),
role: String(entry.role?._id ?? entry.role),
schoolId: String(entry.schoolId?._id ?? entry.schoolId),
}))
.filter((entry) => entry.userId !== student);
await v1('PATCH', `/teams/${s.teamId}`, { userIds });
}],
];
tries.unshift(...fileManagerTries);
for (const [what, fn] of tries) {
try {
await fn();
@@ -398,8 +591,11 @@ await login();
if (phase === 'create') await create();
else if (phase === 'update') await update();
else if (phase === 'delete') await remove();
else if (phase === 'show') summary(loadState());
else if (phase === 'files') {
await fileManager();
summary(loadState());
} else if (phase === 'show') summary(loadState());
else {
console.error(`unknown phase ${phase}; expected create | update | delete | show`);
console.error(`unknown phase ${phase}; expected create | update | delete | files | show`);
process.exit(2);
}

View File

@@ -223,6 +223,63 @@ console.log('\n== classes and groups ==');
);
}
console.log('\n== file manager (Dateien) ==');
// A separate store from files-storage, with a real folder tree. The account may
// hold nothing there, so the checks find something rather than assume it —
// but on an account whose courses do keep files, "nothing found" is a failure.
{
const root = await call('fs_list', { path: '/' });
check('fs_list / names the four areas', !root.isError && /\/courses\//.test(root.text) && /\/shared\//.test(root.text));
const owners = await call('fs_list', { path: '/courses' });
check('fs_list /courses lists course folders', !owners.isError, owners.text.split('\n').find((line) => /course\(s\)/.test(line)));
const ownerIds = [...owners.text.matchAll(/\*\*.*?\/\*\* \(`([0-9a-f]{24})`\)/g)].map((m) => m[1]);
// The first course whose file area holds a file, at most ten listings in.
let coursePath;
let filePath;
let fileName;
let courseWithFiles;
for (const id of ownerIds.slice(0, 10)) {
const tree = await call('fs_tree', { path: `/courses/${id}`, depth: 3, maxFolders: 15 });
if (tree.isError) continue;
const heading = tree.text.match(/^## (\/courses\/.+?) — /m)?.[1];
const line = tree.text.match(/^\s*([^\n]+?\.(?:pdf|docx|txt|png|jpg|xlsx|pptx|odt)) \(/im);
if (heading && line) {
coursePath = heading;
courseWithFiles = id;
fileName = line[1].trim();
// The tree gives names; fs_find recovers the full path to read.
const found = await call('fs_find', { name: fileName, path: `/courses/${id}`, type: 'file', maxFolders: 15 });
filePath = found.text.match(/^- (\/courses\/.+?) — /m)?.[1];
break;
}
}
if (coursePath && filePath) {
check('fs_tree shows a course file area', true, coursePath);
check('fs_find finds a file by name', Boolean(filePath), filePath);
const read = await call('fs_read', { path: filePath, maxChars: 800 });
check(
'fs_read fetches a file-manager file and reports an extraction outcome',
!read.isError && /File id:/.test(read.text),
read.text.split('\n').find((l) => /extracted|image|no extractable|image-only|Word|PDF/.test(l)) ?? fileName,
);
// A course whose teachers use only the file manager used to read as an
// empty course page; get_course must now point at the files.
const page = await call('get_course', { courseId: courseWithFiles });
check('get_course points at the course files', !page.isError && /Course files \(Kurs-Dateien\)/.test(page.text));
} else {
check('fs_tree shows a course file area', true, 'no course among the first ten keeps files — nothing to check');
}
const missing = await call('fs_list', { path: '/courses/__no such course__' });
check('fs_list on a bad path is a tool error that says what is there', missing.isError && /No "|not a file area|Did you mean/.test(missing.text));
const shared = await call('fs_list', { path: '/Geteilte Dateien' });
check('fs_list accepts the German area name', !shared.isError, shared.text.split('\n')[0]);
}
console.log('\n== rooms ==');
// Rooms ("Räume") are a separate space from courses. An account in none is
// normal — and is exactly the state that hid this whole feature — so the check

View File

@@ -7,6 +7,7 @@ import { pipeline } from 'node:stream/promises';
import { ApiClient, ApiError } from '../cli/client.ts';
import { defaultSyncDir, loadCliConfig, saveCliConfig, configPath } from '../cli/config.ts';
import { formatBytes } from '../core/extract.ts';
import { fsFind, fsGet, fsList, fsTree } from '../cli/fs.ts';
import { sync, type SyncEvent } from '../cli/sync.ts';
/**
@@ -26,6 +27,16 @@ const USAGE = `schulcloud — browse and mirror your Schulcloud files
schulcloud sync [--dry-run] [--full] [--prune] [--dir <path>] [--jobs <n>]
schulcloud refresh [--course <id>] [--force]
The file manager ("Dateien") — /my, /courses/<course>, /teams/<team>, /shared:
schulcloud fs ls [path] [--long]
schulcloud fs tree [path] [--depth <n>] [--max-folders <n>]
schulcloud fs find <name> [--path <path>] [--type file|folder] [--long]
schulcloud fs get <path> [--out <path>] [--force] [--jobs <n>]
fs get downloads a file, or a folder with everything below it. Names may contain
"/" and still resolve; any path segment can also be an id from "fs ls --long".
--course takes a course or a room id: rooms ("Räume") mirror alongside courses
and their files sit under the room's name.
@@ -58,6 +69,8 @@ async function main(argv: string[]): Promise<number> {
return runSync(flags);
case 'refresh':
return refresh(flags);
case 'fs':
return fileManager(flags);
default:
process.stderr.write(`Unknown command "${command}".\n\n${USAGE}`);
return 2;
@@ -154,6 +167,41 @@ async function get(flags: Flags): Promise<number> {
return 0;
}
async function fileManager(flags: Flags): Promise<number> {
const [sub, ...args] = flags._ as string[];
const api = new ApiClient(await loadCliConfig());
const out = (line: string) => process.stdout.write(`${line}\n`);
const long = Boolean(flags.long);
switch (sub) {
case 'ls':
return fsList(api, args[0] ?? '/', long, out);
case 'tree':
return fsTree(api, args[0] ?? '/', Number(flags.depth ?? 3), Number(flags['max-folders'] ?? 200), out);
case 'find': {
if (!args[0]) {
process.stderr.write('fs find needs a name, e.g.: schulcloud fs find Erbrecht --path /courses\n');
return 2;
}
const type = flags.type === 'folder' || flags.type === 'file' ? String(flags.type) : 'any';
return fsFind(api, args[0], String(flags.path ?? '/'), type, Number(flags['max-folders'] ?? 400), long, out);
}
case 'get':
if (!args[0]) {
process.stderr.write('fs get needs a path, e.g.: schulcloud fs get "/courses/<course>/<folder>"\n');
return 2;
}
return fsGet(
api,
args[0],
{ out: flags.out ? String(flags.out) : undefined, force: Boolean(flags.force), jobs: Number(flags.jobs ?? 3) },
out,
);
default:
process.stderr.write(`Unknown fs command "${sub ?? ''}". Use ls, tree, find or get.\n\n${USAGE}`);
return 2;
}
}
async function runSync(flags: Flags): Promise<number> {
const config = await loadCliConfig();
const root = flags.dir ? resolve(String(flags.dir)) : config.syncDir;

View File

@@ -26,6 +26,39 @@ export interface Manifest {
entries: ManifestEntry[];
}
/** One entry of a file-manager tree or search, as /api/fs returns it. */
export interface FsEntry {
type: 'directory' | 'file';
path: string;
parentPath: string;
depth: number;
id: string;
name: string;
size?: number;
mimeType?: string | null;
blocked?: boolean;
}
export interface FsListing {
path: string;
kind: 'directory' | 'file';
area?: string | null;
directories?: { id: string; name: string; path: string }[];
files?: { id: string; name: string; path: string; size: number; mimeType?: string; blocked: boolean }[];
file?: { id: string; name: string; size: number; mimeType?: string; blocked: boolean };
}
export interface FsWalk {
path: string;
kind: 'directory' | 'file';
entries?: FsEntry[];
matches?: FsEntry[];
file?: FsListing['file'];
visited?: number;
truncated?: boolean;
failures?: { path: string; reason: string }[];
}
export class ApiError extends Error {
readonly status: number;
@@ -83,6 +116,28 @@ export class ApiClient {
async file(fileId: string): Promise<Response> {
return this.request(`/api/files/${encodeURIComponent(fileId)}`);
}
// --- the file manager ------------------------------------------------------
async fsList(path: string): Promise<FsListing> {
return (await (await this.request(`/api/fs/list?${new URLSearchParams({ path })}`)).json()) as FsListing;
}
async fsTree(path: string, depth: number, maxFolders: number): Promise<FsWalk> {
const query = new URLSearchParams({ path, depth: String(depth), maxFolders: String(maxFolders) });
return (await (await this.request(`/api/fs/tree?${query}`)).json()) as FsWalk;
}
async fsFind(name: string, path: string, type: string, maxFolders: number): Promise<FsWalk> {
const query = new URLSearchParams({ name, path, type, maxFolders: String(maxFolders) });
return (await (await this.request(`/api/fs/find?${query}`)).json()) as FsWalk;
}
/** Streams one file-manager file's bytes, by path or by id. */
async fsFile(target: { path: string } | { id: string; name: string }): Promise<Response> {
const query = 'path' in target ? new URLSearchParams({ path: target.path }) : new URLSearchParams(target);
return this.request(`/api/fs/file?${query}`);
}
}
function describe(status: number, detail: string, server: string): string {
@@ -90,5 +145,7 @@ function describe(status: number, detail: string, server: string): string {
if (status === 503) return 'The server is running without an index, so this command is unavailable. Set DATABASE_URL on the server.';
if (status === 409) return detail || 'The sync cursor is unknown to the server. Run a full sync with --full.';
if (status === 429) return detail || 'Refreshed too recently — wait a moment, or pass --force.';
// The file manager's own errors already say what was not found and what is there.
if ((status === 400 || status === 404 || status === 422) && detail) return detail;
return detail ? `HTTP ${status}: ${detail}` : `HTTP ${status}`;
}

218
src/cli/fs.ts Normal file
View File

@@ -0,0 +1,218 @@
import { createWriteStream } from 'node:fs';
import { mkdir, rename, stat, unlink } from 'node:fs/promises';
import { dirname, resolve } from 'node:path';
import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { formatBytes } from '../core/extract.ts';
import { resolveWithin, safeComponent } from '../core/paths.ts';
import type { ApiClient, FsEntry } from './client.ts';
/**
* `schulcloud fs` — the file manager ("Dateien") from the command line.
*
* The same tree the MCP fs_* tools show: /my, /courses/<course>, /teams/<team>,
* /shared. Everything goes through the server's /api/fs routes; nothing here
* talks to Schulcloud.
*/
type Out = (line: string) => void;
export async function fsList(api: ApiClient, path: string, long: boolean, out: Out): Promise<number> {
const listing = await api.fsList(path);
if (listing.kind === 'file' && listing.file) {
out(`${listing.path}`);
out(` ${formatBytes(listing.file.size)} ${listing.file.mimeType ?? 'unknown type'} ${listing.file.id}${listing.file.blocked ? ' [blocked]' : ''}`);
return 0;
}
const directories = listing.directories ?? [];
const files = listing.files ?? [];
out(listing.path);
for (const directory of directories) out(` ${directory.name}/${long ? ` ${directory.id}` : ''}`);
for (const file of files) {
const detail = long ? ` ${formatBytes(file.size).padStart(9)} ${file.id}${file.mimeType ? ` ${file.mimeType}` : ''}` : '';
out(` ${file.name}${detail}${file.blocked ? ' [blocked]' : ''}`);
}
if (directories.length + files.length === 0) out(' (empty)');
out(`${directories.length} folder(s), ${files.length} file(s)`);
return 0;
}
export async function fsTree(api: ApiClient, path: string, depth: number, maxFolders: number, out: Out): Promise<number> {
const walk = await api.fsTree(path, depth, maxFolders);
if (walk.kind === 'file') {
out(`${walk.path} is a file.`);
return 0;
}
const entries = walk.entries ?? [];
const children = groupByParent(entries);
out(walk.path);
let files = 0;
let bytes = 0;
const visit = (parent: string, prefix: string) => {
const kids = children.get(parent) ?? [];
kids.forEach((kid, index) => {
const last = index === kids.length - 1;
const branch = last ? '└── ' : '├── ';
if (kid.type === 'directory') {
out(`${prefix}${branch}${kid.name}/`);
visit(kid.path, `${prefix}${last ? ' ' : '│ '}`);
} else {
files++;
bytes += kid.size ?? 0;
out(`${prefix}${branch}${kid.name} (${formatBytes(kid.size ?? 0)})${kid.blocked ? ' [blocked]' : ''}`);
}
});
};
visit(walk.path, '');
out(`\n${files} file(s), ${formatBytes(bytes)}${walk.visited ?? 0} folder(s) listed`);
if (walk.truncated) out(`Stopped after ${maxFolders} folders; pass --max-folders or start deeper.`);
for (const failure of walk.failures ?? []) out(`could not list ${failure.path}: ${failure.reason}`);
return walk.failures?.length ? 1 : 0;
}
export async function fsFind(
api: ApiClient,
name: string,
path: string,
type: string,
maxFolders: number,
long: boolean,
out: Out,
): Promise<number> {
const result = await api.fsFind(name, path, type, maxFolders);
for (const match of result.matches ?? []) {
const suffix = match.type === 'directory' ? '/' : '';
const detail = long && match.type === 'file' ? ` ${formatBytes(match.size ?? 0)} ${match.id}` : long ? ` ${match.id}` : '';
out(`${match.path}${suffix}${detail}`);
}
const count = result.matches?.length ?? 0;
process.stderr.write(`${count} match(es), ${result.visited ?? 0} folder(s) searched\n`);
if (result.truncated) process.stderr.write(`Stopped after ${maxFolders} folders; there may be more. Narrow --path.\n`);
return count > 0 ? 0 : 1;
}
/**
* Downloads a file, or a whole folder recursively.
*
* A folder lands under `--out` (default: a directory named after it) with the
* file manager's structure. Every component is a name from Schulcloud and so
* untrusted: each goes through `safeComponent`, and the joined path through
* `resolveWithin`, exactly as sync does.
*/
export async function fsGet(
api: ApiClient,
path: string,
options: { out?: string; force: boolean; jobs: number },
out: Out,
): Promise<number> {
const target = await api.fsList(path);
if (target.kind === 'file' && target.file) {
const destination = options.out ? resolve(options.out) : resolve(safeComponent(target.file.name, target.file.id));
if (target.file.blocked) {
process.stderr.write(`${target.path}: blocked by the instance virus scanner; not downloaded.\n`);
return 1;
}
await downloadTo(api, { path: target.path }, destination);
out(destination);
return 0;
}
const rootName = target.path === '/' ? 'Dateien' : (target.path.split('/').pop() ?? 'Dateien');
const root = options.out ? resolve(options.out) : resolve(safeComponent(rootName, 'Dateien'));
process.stderr.write(`Listing ${target.path}\n`);
const walk = await api.fsTree(target.path, 12, 1000);
const entries = walk.entries ?? [];
// Rebuild each file's name segments from its parents rather than splitting
// its path: names may contain "/", which would otherwise invent folders.
const segments = new Map<string, string[]>([[walk.path, []]]);
for (const entry of [...entries].sort((a, b) => a.depth - b.depth)) {
const parent = segments.get(entry.parentPath);
if (parent) segments.set(entry.path, [...parent, entry.name]);
}
const files = entries.filter((entry) => entry.type === 'file');
let downloaded = 0;
let skipped = 0;
let failed = 0;
let bytes = 0;
const queue = [...files];
const worker = async () => {
for (let entry = queue.shift(); entry; entry = queue.shift()) {
const parts = segments.get(entry.path);
if (!parts) continue;
const relative = parts.map((part) => safeComponent(part)).join('/');
const destination = resolveWithin(root, relative);
if (entry.blocked) {
process.stderr.write(` blocked ${relative}\n`);
skipped++;
continue;
}
// Unchanged by size: re-running a folder download resumes rather than repeats.
const existing = await stat(destination).catch(() => undefined);
if (!options.force && existing?.isFile() && existing.size === entry.size) {
skipped++;
continue;
}
try {
await downloadTo(api, { id: entry.id, name: entry.name }, destination);
downloaded++;
bytes += entry.size ?? 0;
process.stderr.write(` get ${relative}\n`);
} catch (error) {
failed++;
process.stderr.write(` FAILED ${relative}: ${(error as Error).message}\n`);
}
}
};
await Promise.all(Array.from({ length: Math.max(1, options.jobs) }, worker));
out(root);
process.stderr.write(
`Downloaded ${downloaded} file(s) (${formatBytes(bytes)}), ${skipped} skipped` +
`${failed ? `, FAILED ${failed}` : ''}${walk.visited ?? 0} folder(s) listed\n`,
);
if (walk.truncated) process.stderr.write('The folder is larger than one listing pass; some files were not reached.\n');
for (const failure of walk.failures ?? []) process.stderr.write(`could not list ${failure.path}: ${failure.reason}\n`);
return failed > 0 || (walk.failures?.length ?? 0) > 0 ? 1 : 0;
}
async function downloadTo(api: ApiClient, target: { path: string } | { id: string; name: string }, destination: string) {
const response = await api.fsFile(target);
if (!response.body) throw new Error('empty response body');
await mkdir(dirname(destination), { recursive: true });
// A temporary neighbour, renamed into place, so an interrupted download never
// leaves a half-file that the size check would later accept as complete.
const temp = `${destination}.part`;
try {
await pipeline(Readable.fromWeb(response.body as never), createWriteStream(temp));
await rename(temp, destination);
} catch (error) {
await unlink(temp).catch(() => {});
throw error;
}
}
function groupByParent(entries: FsEntry[]): Map<string, FsEntry[]> {
const children = new Map<string, FsEntry[]>();
for (const entry of entries) {
const list = children.get(entry.parentPath) ?? [];
list.push(entry);
children.set(entry.parentPath, list);
}
for (const list of children.values()) {
list.sort((a, b) =>
a.type !== b.type
? a.type === 'directory'
? -1
: 1
: a.name.localeCompare(b.name, 'de', { numeric: true, sensitivity: 'base' }),
);
}
return children;
}

View File

@@ -37,6 +37,8 @@ export interface Config {
mirrorMaxBytes: number;
/** Index personal files and submitted/returned work as well as course content. */
indexPersonalFiles: boolean;
/** Walk the file manager (Kurs-, Persönliche, Team- and Geteilte Dateien) when crawling. */
indexFileManager: boolean;
/** How often to re-crawl on a timer. Zero = only on demand. */
crawlIntervalMs: number;
}
@@ -95,6 +97,9 @@ export function loadConfig(): Config {
// cost of a full crawl. Worth turning on to make your own handed-in work
// searchable, which no other route offers.
indexPersonalFiles: bool('INDEX_PERSONAL_FILES', false),
// On by default: many teachers keep their material only in Kurs-Dateien,
// so an index without it misses whole courses. One page load per folder.
indexFileManager: bool('INDEX_FILE_MANAGER', true),
crawlIntervalMs: intAllowingZero('CRAWL_INTERVAL_MS', 6 * 60 * 60_000),
};
}

View File

@@ -1,5 +1,6 @@
import type { Config } from './config.ts';
import { SchulcloudClient } from './core/client.ts';
import { FileManager } from './core/legacy-files.ts';
import type { LegacyUser, MeResponse } from './core/types.ts';
import type { Indexer } from './indexer/indexer.ts';
import type { Store } from './store/store.ts';
@@ -14,6 +15,8 @@ import type { Store } from './store/store.ts';
export class ServerContext {
readonly config: Config;
readonly client: SchulcloudClient;
/** The "Dateien" file manager; shared across sessions when the process provides one. */
readonly files: FileManager;
/** Shared across sessions; undefined when running without an index. */
readonly store: Store | undefined;
readonly indexer: Indexer | undefined;
@@ -30,9 +33,13 @@ export class ServerContext {
*/
private readonly userNames = new Map<string, Promise<string | undefined>>();
constructor(config: Config, shared?: { client?: SchulcloudClient; store?: Store; indexer?: Indexer }) {
constructor(
config: Config,
shared?: { client?: SchulcloudClient; files?: FileManager; store?: Store; indexer?: Indexer },
) {
this.config = config;
this.client = shared?.client ?? new SchulcloudClient(config);
this.files = shared?.files ?? new FileManager(this.client);
this.store = shared?.store;
this.indexer = shared?.indexer;
}

View File

@@ -119,8 +119,21 @@ export class SchulcloudClient {
return url;
}
private async request(url: URL, accept: string): Promise<Response> {
/**
* One upstream GET, with retries for the transient failures.
*
* `auth` picks how the session travels. The v3 API takes it as a bearer
* token; the legacy client's pages take it only as the `jwt` cookie; and a
* pre-signed storage URL must get **nothing** — it lives on another host, and
* the session token has no business leaving this instance. Anything but the
* bearer form is fetched with redirects off, so a login bounce or a hop to a
* third host is seen rather than silently followed with credentials attached.
*/
private async request(url: URL, accept: string, auth: 'bearer' | 'cookie' | 'none' = 'bearer'): Promise<Response> {
let lastError: unknown;
const headers: Record<string, string> = { Accept: accept };
if (auth === 'bearer') headers.Authorization = `Bearer ${this.config.jwt}`;
if (auth === 'cookie') headers.Cookie = `jwt=${this.config.jwt}`;
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
if (attempt > 0) await delay(backoffMs(attempt));
@@ -128,9 +141,9 @@ export class SchulcloudClient {
let response: Response;
try {
response = await fetch(url, {
headers: { Authorization: `Bearer ${this.config.jwt}`, Accept: accept },
headers,
signal: AbortSignal.timeout(this.config.requestTimeoutMs),
redirect: 'follow',
redirect: auth === 'bearer' ? 'follow' : 'manual',
});
} catch (error) {
// Connection reset or timeout: worth one more try, since every call
@@ -143,7 +156,12 @@ export class SchulcloudClient {
if (response.ok) return response;
const body = await response.text().catch(() => '');
const error = new SchulcloudApiError(response.status, url.pathname + url.search, body);
// A pre-signed URL's query string is its credential, so it never goes
// into an error message; nor does the storage host's error body.
const error =
auth === 'none'
? new SchulcloudApiError(response.status, `${url.host} (pre-signed download)`, '')
: new SchulcloudApiError(response.status, url.pathname + url.search, body);
// A crawl issues hundreds of requests and the instance answers some of
// them with a 503 front-page when it decides we are going too fast.
@@ -175,6 +193,11 @@ export class SchulcloudClient {
async getBytes(path: string, fallbackName: string): Promise<DownloadedFile> {
const url = this.url(path);
const response = await this.request(url, '*/*');
return this.readCapped(response, fallbackName);
}
/** Reads a response body up to `maxDownloadBytes`, flagging anything cut off. */
private async readCapped(response: Response, fallbackName: string): Promise<DownloadedFile> {
const limit = this.config.maxDownloadBytes;
const chunks: Buffer[] = [];
@@ -550,6 +573,118 @@ export class SchulcloudClient {
const body = await this.getJson<Paginated<GroupItem>>('/api/v3/groups', { limit: MAX_PAGE_SIZE });
return body.data ?? [];
}
// --- the "Dateien" file manager ------------------------------------------
//
// Persönliche Dateien, Kurs-Dateien, Team-Dateien and Geteilte Dateien live in
// the legacy file system, a different store from files-storage: listing a
// course through /api/v3/file answers 0 files for a course holding dozens.
// Its Feathers service is not in the public ingress, so the only way in is
// the legacy client — HTML pages for listings, one JSON route for downloads.
// See core/legacy-files.ts for the parsing and the path model.
/**
* One file-manager page, as HTML.
*
* **Only the listing routes are reachable here, by construction.** Several of
* the legacy client's GET routes write: `GET /files/share/` mints a share
* token when the file has none, and `GET /files/file?share=…` grants the
* caller a permission on someone else's file. A GET-only client is therefore
* not read-only against this surface by itself; the allowlist is what makes
* the invariant hold.
*/
async getFileManagerPage(path: string): Promise<string> {
if (!FILE_MANAGER_PAGE.test(path)) {
throw new Error(`refusing file-manager path outside the listing routes: ${path}`);
}
const response = await this.legacyRequest(path, 'text/html');
return response.text();
}
/**
* A pre-signed download URL for one legacy file.
*
* `name` only sets the download's filename; the server checks read access on
* the id. The route is `/files/signedurl` rather than `/files/file`, which
* answers the same thing as a redirect but also accepts `share`, the
* parameter that writes.
*/
async getFileManagerSignedUrl(fileId: string, name: string): Promise<string> {
if (!/^[0-9a-f]{24}$/i.test(fileId)) throw new Error(`not a file id: ${fileId}`);
const query = new URLSearchParams({ file: fileId, name: name || fileId });
const response = await this.legacyRequest(`/files/signedurl?${query.toString()}`, 'application/json');
// The server's error path *returns* its Forbidden rather than throwing it,
// so a refused file arrives as a 200 whose body has no url.
const body = (await response.json().catch(() => ({}))) as { url?: unknown; message?: unknown };
if (typeof body.url !== 'string' || !body.url) {
throw new SchulcloudApiError(403, '/files/signedurl', typeof body.message === 'string' ? body.message : 'no download url');
}
return body.url;
}
/** Downloads one legacy file: signed URL, then the bytes, capped like every download. */
async downloadFileManagerFile(fileId: string, name: string): Promise<DownloadedFile> {
const signed = await this.getFileManagerSignedUrl(fileId, name);
const response = await this.openSignedUrl(signed);
return this.readCapped(response, name);
}
/**
* Opens a pre-signed storage URL — with no credentials at all.
*
* The URL names another host (live: an S3 endpoint at the storage provider),
* so neither the bearer nor the cookie may go with it. It must also be
* https whenever the instance is, which keeps a URL the server hands back from
* pointing this process at a plaintext service on its own network.
*/
async openSignedUrl(signedUrl: string): Promise<Response> {
const target = checkSignedUrl(signedUrl, this.config.baseUrl);
return this.request(target, '*/*', 'none');
}
private async legacyRequest(path: string, accept: string): Promise<Response> {
try {
return await this.request(this.url(path), accept, 'cookie');
} catch (error) {
// The legacy client answers a rejected cookie with a redirect to its
// login page. Report it as what it is, so tools say "token expired"
// rather than "HTTP 302".
if (error instanceof SchulcloudApiError && error.status >= 300 && error.status < 400) {
throw new SchulcloudApiError(401, path, 'redirected to login: the session is not accepted');
}
throw error;
}
}
}
/**
* The file-manager listing routes, and nothing else.
*
* Folders are addressed by id alone — `/files/courses/{course}/{folder}` holds
* one folder segment however deep the folder is — so every listing fits one of
* these shapes. `/files/my/{a}/{b}` exists too, but lists `b` exactly as
* `/files/my/{b}` does, so it is not needed.
*/
const FILE_MANAGER_PAGE =
/^\/files\/(?:(?:my|courses|teams|shared)\/|my\/[0-9a-f]{24}|(?:courses|teams)\/[0-9a-f]{24}(?:\/[0-9a-f]{24})?)$/i;
/** Validates a pre-signed URL before anything is sent to it. Exported for testing. */
export function checkSignedUrl(signedUrl: string, baseUrl: string): URL {
let target: URL;
try {
target = new URL(signedUrl);
} catch {
throw new Error('the download url the server returned is not a url');
}
const instanceIsHttps = new URL(baseUrl).protocol === 'https:';
const allowed = instanceIsHttps ? ['https:'] : ['https:', 'http:'];
if (!allowed.includes(target.protocol)) {
throw new Error(`refusing a ${target.protocol} download url from an ${instanceIsHttps ? 'https' : 'http'} instance`);
}
if (target.username || target.password) {
throw new Error('refusing a download url that carries credentials');
}
return target;
}
/**

View File

@@ -2,6 +2,7 @@ import type { Config } from '../config.ts';
import { assembleBoard, type AssembledBoard } from './board.ts';
import type { SchulcloudClient } from './client.ts';
import { fetchHomeworkPage } from './homework-page.ts';
import { FileManager, type DirectoryRef, type FmFile, type WalkEntry } from './legacy-files.ts';
import { fetchLessonTaskLinks, withScrapedIds } from './lesson-page.ts';
import { htmlToText, normalizeObjectId } from './text.ts';
import type { CourseMetadata, FileParentType, FileRecord, TaskContent } from './types.ts';
@@ -35,6 +36,11 @@ export interface Breadcrumb {
/** Column → card, for board files. */
columnTitle?: string;
cardTitle?: string;
/**
* Folder names from the file manager, outermost first. Unlike boards, its
* trees have no fixed depth, so they cannot be squeezed into the titles above.
*/
folders?: string[];
}
export interface CrawledFile {
@@ -43,6 +49,13 @@ export interface CrawledFile {
parentType: FileParentType;
parentId: string;
at: Breadcrumb;
/**
* Which store holds the bytes. The file manager is not files-storage: its
* ids mean nothing to /api/v3/file, so downloads must be routed by this.
*/
source?: 'files-storage' | 'file-manager';
/** The file-manager path fs_read takes, for files from there. */
fsPath?: string;
}
/**
@@ -150,6 +163,12 @@ export interface CrawlOptions {
* about X" to be searchable, which is otherwise impossible.
*/
includePersonalFiles?: boolean;
/**
* Walk the file manager ("Dateien") too: Kurs-Dateien for every course, and on
* a full crawl Persönliche Dateien, Team-Dateien and Geteilte Dateien. One
* page load per folder — on the account this was built for, about 160.
*/
includeFileManager?: boolean;
/**
* Read the text of collaborative text editor (Etherpad) pads, which needs a
* second credentialled hop outside the API. Omit to leave pads unread.
@@ -207,10 +226,24 @@ export async function crawl(client: SchulcloudClient, options: CrawlOptions): Pr
// outside that scope forward untouched.
const rooms: CrawledRoom[] = options.courseIds ? [] : await crawlRooms(client, options, includeFiles, files, failures);
if (includeFiles && options.includeFileManager) {
const titles = new Map(crawled.map((entry) => [entry.course.id, entry.title]));
await crawlFileManager(client, options, titles, files, failures);
}
// Traversal order is nondeterministic under concurrency; sort so that two
// crawls of unchanged content produce identical snapshots.
crawled.sort((a, b) => a.course.id.localeCompare(b.course.id));
rooms.sort((a, b) => a.id.localeCompare(b.id));
// One file can be reachable twice — a course file someone also shared with
// you appears under /shared as well — and the index keys nodes by id. Keep
// the first place it was found, which the traversal order makes the most
// specific one.
const seenFiles = new Set<string>();
const uniqueFiles = files.filter((file) => (seenFiles.has(file.record.id) ? false : (seenFiles.add(file.record.id), true)));
files.length = 0;
files.push(...uniqueFiles);
files.sort((a, b) => a.record.id.localeCompare(b.record.id));
submissions.sort((a, b) => a.id.localeCompare(b.id));
@@ -540,6 +573,128 @@ async function collectSubmissions(
}
}
/**
* Walks the file manager and records every file in it.
*
* Kurs-Dateien are filed under their course, so a per-course refresh replaces
* exactly that course's files and the store's scope rules carry the rest
* forward. The areas that belong to no course — personal, team, shared — are
* walked only on a full crawl, for the same reason.
*
* A fresh FileManager rather than the process-wide one: its listing cache is
* right for an interactive ls-then-read, and wrong for a crawl whose whole
* point is to see the current state.
*/
async function crawlFileManager(
client: SchulcloudClient,
options: CrawlOptions,
courseTitles: Map<string, string>,
files: CrawledFile[],
failures: { courseId: string; boardId?: string; reason: string }[],
): Promise<void> {
const manager = new FileManager(client);
const roots: { ref: DirectoryRef; path: string; courseId: string; at: Omit<Breadcrumb, 'folders'> }[] = [];
if (options.courseIds) {
for (const courseId of options.courseIds) {
const title = courseTitles.get(courseId) ?? courseId;
roots.push({
ref: { area: 'courses', ownerId: courseId },
path: `/courses/${title}`,
courseId,
at: { courseId, courseTitle: title, containerTitle: 'Kurs-Dateien' },
});
}
} else {
const owners = async (area: 'courses' | 'teams') => {
try {
return (await manager.list({ area })).directories;
} catch (error) {
failures.push({ courseId: '', reason: `file manager /${area}: ${error instanceof Error ? error.message : String(error)}` });
return [];
}
};
for (const course of await owners('courses')) {
const title = courseTitles.get(course.id) ?? course.name;
roots.push({
ref: { area: 'courses', ownerId: course.id },
path: `/courses/${course.name}`,
courseId: course.id,
at: { courseId: course.id, courseTitle: title, containerTitle: 'Kurs-Dateien' },
});
}
for (const team of await owners('teams')) {
roots.push({
ref: { area: 'teams', ownerId: team.id },
path: `/teams/${team.name}`,
courseId: '',
at: { courseId: '', courseTitle: 'Team-Dateien', containerTitle: team.name },
});
}
roots.push({ ref: { area: 'my' }, path: '/my', courseId: '', at: { courseId: '', courseTitle: 'Persönliche Dateien' } });
roots.push({ ref: { area: 'shared' }, path: '/shared', courseId: '', at: { courseId: '', courseTitle: 'Geteilte Dateien' } });
}
// Sequential roots, modest concurrency within each: the instance answers a
// burst with 503s, and this walk is the largest single part of a crawl.
for (const root of roots) {
const result = await manager.walk(
{ path: root.path, ref: root.ref },
{ maxDepth: 25, maxDirectories: 5000, concurrency: 2 },
);
for (const failure of result.failures) {
failures.push({ courseId: root.courseId, reason: `file manager ${failure.path}: ${failure.reason}` });
}
if (result.truncated) {
failures.push({ courseId: root.courseId, reason: `file manager ${root.path}: stopped at 5000 folders` });
}
// Folder names come from the parent chain, never from splitting a path:
// names contain "/" in real data.
const folders = new Map<string, string[]>([[root.path, []]]);
const ordered = [...result.entries].sort((a, b) => a.depth - b.depth);
for (const entry of ordered) {
if (entry.directory) {
const parent = folders.get(entry.parentPath);
if (parent) folders.set(entry.path, [...parent, entry.directory.name]);
}
}
for (const entry of ordered) {
if (!entry.file) continue;
files.push(fileManagerRecord(entry, entry.file, root, folders.get(entry.parentPath) ?? []));
}
}
}
function fileManagerRecord(
entry: WalkEntry,
file: FmFile,
root: { ref: DirectoryRef; courseId: string; at: Omit<Breadcrumb, 'folders'> },
folders: string[],
): CrawledFile {
const parentType: FileParentType = root.ref.area === 'courses' ? 'courses' : 'users';
return {
// Shaped like a files-storage record so the store, the mirror and the
// manifest need no second code path; `source` is what keeps the two apart.
record: {
id: file.id,
name: file.name,
parentId: entry.parent.folderId ?? entry.parent.ownerId ?? '',
parentType,
url: '',
size: file.size,
mimeType: file.mimeType ?? 'application/octet-stream',
securityCheckStatus: file.blocked ? 'blocked' : 'verified',
previewStatus: '',
},
parentType,
parentId: entry.parent.folderId ?? entry.parent.ownerId ?? '',
at: { ...root.at, folders },
source: 'file-manager',
fsPath: entry.path,
};
}
async function listFiles(
client: SchulcloudClient,
schoolId: string,

View File

@@ -140,7 +140,10 @@ function hasTextLayer(bytes: Buffer): boolean {
async function extractPdf(bytes: Buffer): Promise<string> {
const { extractText, getDocumentProxy } = await import('unpdf');
const document = await getDocumentProxy(new Uint8Array(bytes));
// verbosity 0 = errors only. pdf.js otherwise prints "Warning: TT: undefined
// function" for every font hint it skips — harmless, but a crawl of the file
// manager extracts hundreds of PDFs, and that buries the log in noise.
const document = await getDocumentProxy(new Uint8Array(bytes), { verbosity: 0 });
const { text } = await extractText(document, { mergePages: true });
return Array.isArray(text) ? text.join('\n\n') : text;
}

500
src/core/legacy-files.ts Normal file
View File

@@ -0,0 +1,500 @@
import type { DownloadedFile, SchulcloudClient } from './client.ts';
import { decodeEntities } from './text.ts';
/**
* The "Dateien" file manager — Persönliche Dateien, Kurs-Dateien, Team-Dateien
* and Geteilte Dateien — as one read-only filesystem.
*
* This is the legacy file system, a different store from files-storage
* (`/api/v3/file`), which holds board, topic and task attachments. The two do
* not overlap: asking files-storage for a course's files answers 0 for a course
* whose file manager holds dozens, and on the account this was built for 21 of
* 26 courses keep material here — some teachers use nothing else.
*
* Its Feathers service is not in the public ingress, so it is reached through
* the legacy client: server-rendered listing pages, parsed here, and one JSON
* route for pre-signed downloads (see the client). Two server quirks shape the
* design:
*
* - `GET /files/permittedDirectories/` looks like the obvious JSON source for
* the directory tree, but its query matches course folders on
* `refOwnerModel: 'courses'` while the records say `'course'`. It lists every
* course with **no** folders in any of them. Listings are the only complete
* view, which is also exactly what the file manager itself shows.
* - `GET /files/search/` runs an unindexed regex over every file record and
* times out (504) on the live instance, so finding is done by walking.
*/
export type FileArea = 'my' | 'courses' | 'teams' | 'shared';
interface AreaInfo {
area: FileArea;
label: string;
/** Accepted spellings of the first path segment, compared lower-cased. */
aliases: string[];
summary: string;
}
export const FILE_AREAS: AreaInfo[] = [
{
area: 'my',
label: 'Persönliche Dateien',
aliases: ['my', 'persönliche dateien', 'persoenliche dateien', 'personal', 'meine dateien'],
summary: 'your own files',
},
{
area: 'courses',
label: 'Kurs-Dateien',
aliases: ['courses', 'kurs-dateien', 'meine kurs-dateien', 'kursdateien', 'kurse'],
summary: 'one folder per course, holding what its teachers uploaded',
},
{
area: 'teams',
label: 'Team-Dateien',
aliases: ['teams', 'team-dateien', 'meine team-dateien', 'teamdateien'],
summary: 'one folder per team',
},
{
area: 'shared',
label: 'Geteilte Dateien',
aliases: ['shared', 'geteilte dateien', 'mit mir geteilt'],
summary: 'files other people shared with you, read-only and flat',
},
];
export function areaInfo(area: FileArea): AreaInfo {
return FILE_AREAS.find((entry) => entry.area === area) as AreaInfo;
}
export interface FmDirectory {
id: string;
name: string;
}
export interface FmFile {
id: string;
name: string;
/** Bytes, as the listing reports it. */
size: number;
/** Absent for a blocked file: the page withholds its viewer attributes. */
mimeType?: string;
/** Rejected by the instance's virus scanner; it cannot be downloaded. */
blocked: boolean;
}
export interface FmListing {
directories: FmDirectory[];
files: FmFile[];
}
/** A listing page's markup changed shape; never report that as an empty folder. */
export class FileManagerMarkupError extends Error {}
/**
* Parses one file-manager page. Exported for testing.
*
* Anchored on what the templates (`files/files.hbs`, `files/files-grid.hbs`)
* emit for the page's own scripts — `data-folder-id`, `data-file-id`,
* `data-file-name`, `data-file-size` — rather than on layout classes.
*
* Throws rather than returning an empty listing when the page is not a file
* manager page at all: "0 files" is precisely the wrong answer this module
* exists to fix, and a markup change must not quietly reproduce it.
*/
export function parseFileListing(html: string): FmListing {
if (!/class="route-files"/.test(html)) {
throw new FileManagerMarkupError('the page is not a file-manager listing (markup changed, or not logged in)');
}
const directories: FmDirectory[] = [];
const folderPattern = /<button\b([^>]*\bopenfolder\b[^>]*)>([\s\S]*?)<\/button>/g;
for (const match of html.matchAll(folderPattern)) {
const id = /data-folder-id="([0-9a-f]{24})"/i.exec(match[1] ?? '')?.[1];
if (!id) continue;
// The name is emitted unescaped (`{{{stripOnlyScript name}}}`) inside the
// title element, after an icon; strip the tags, then decode what is left.
const title = /<strong\b[^>]*card-title-directory[^>]*>([\s\S]*?)<\/strong>/.exec(match[2] ?? '')?.[1] ?? '';
const name = decodeEntities(title.replace(/<[^>]+>/g, '')).replace(/\s+/g, ' ').trim();
directories.push({ id, name: name || id });
}
const files: FmFile[] = [];
const cardPattern = /<div\b[^>]*\bclass="card file\b([^"]*)"([^>]*)>/g;
const cards = [...html.matchAll(cardPattern)];
cards.forEach((match, index) => {
const attributes = match[2] ?? '';
const id = /data-file-id="([0-9a-f]{24})"/i.exec(attributes)?.[1];
if (!id) return;
const name = decodeEntities(/data-file-name="([^"]*)"/.exec(attributes)?.[1] ?? '');
const size = Number(/data-file-size="(\d*)"/.exec(attributes)?.[1] ?? '');
// The viewer attributes sit further inside this card; look no further
// than the next card, so one file can never borrow another's type.
const start = (match.index ?? 0) + match[0].length;
const end = cards[index + 1]?.index ?? html.length;
const mimeType = /data-file-viewer-type="([^"]*)"/.exec(html.slice(start, end))?.[1];
files.push({
id,
name: name || id,
size: Number.isFinite(size) ? size : 0,
mimeType: mimeType ? decodeEntities(mimeType) : undefined,
blocked: /\bbtn-file-danger\b/.test(match[1] ?? ''),
});
});
return { directories, files };
}
/**
* Where a directory is. `area` absent is the root; `ownerId` is the course or
* team (absent for `my` and `shared`); `folderId` absent is the owner's top.
*/
export interface DirectoryRef {
area?: FileArea;
ownerId?: string;
folderId?: string;
}
export type FsNode =
| { kind: 'directory'; path: string; ref: DirectoryRef; name: string }
| { kind: 'file'; path: string; parent: DirectoryRef; file: FmFile };
export type FsErrorCode = 'not_found' | 'ambiguous' | 'not_a_directory' | 'not_a_file' | 'not_navigable';
export class FsError extends Error {
readonly code: FsErrorCode;
constructor(code: FsErrorCode, message: string) {
super(message);
this.code = code;
}
}
/** The legacy page that lists a directory. */
export function pageFor(ref: DirectoryRef): string | undefined {
if (!ref.area) return undefined;
if (ref.area === 'shared') return '/files/shared/';
if (ref.area === 'my') return ref.folderId ? `/files/my/${ref.folderId}` : '/files/my/';
if (!ref.ownerId) return `/files/${ref.area}/`;
return ref.folderId ? `/files/${ref.area}/${ref.ownerId}/${ref.folderId}` : `/files/${ref.area}/${ref.ownerId}`;
}
/** The reference for a subdirectory found in `parent`'s listing. */
export function childRef(parent: DirectoryRef, directory: FmDirectory): DirectoryRef {
if (!parent.area) return { area: directory.id as FileArea };
if (parent.area === 'my') return { area: 'my', folderId: directory.id };
if (parent.area === 'shared') {
// The file manager has no route that opens a shared folder — its owner is
// someone else, and every listing route is scoped to an owner. The UI's
// own link to one is a 404.
throw new FsError(
'not_navigable',
`"${directory.name}" is a folder someone shared with you. The file manager cannot open shared folders ` +
'(not even in the browser); ask for the files themselves to be shared, or find them in the owning course.',
);
}
if (!parent.ownerId) return { area: parent.area, ownerId: directory.id };
return { area: parent.area, ownerId: parent.ownerId, folderId: directory.id };
}
/** Splits a path into raw segments. Empty segments (`//`, trailing `/`) drop out. */
export function splitPath(path: string): string[] {
return path
.split('/')
.map((segment) => segment.trim())
.filter((segment) => segment.length > 0);
}
/** Joins names into a display path. Names keep any `/` they contain; see `resolve`. */
export function joinPath(parent: string, name: string): string {
return `${parent === '/' ? '' : parent}/${name}`;
}
function normalise(value: string): string {
return value.normalize('NFC').replace(/\s+/g, ' ').trim().toLocaleLowerCase('de');
}
export interface WalkEntry {
path: string;
/** The listed directory this entry came from; renderers group on it. */
parentPath: string;
depth: number;
parent: DirectoryRef;
directory?: { ref: DirectoryRef; name: string; id: string };
file?: FmFile;
}
export interface WalkResult {
entries: WalkEntry[];
/** Directories that were listed. */
visited: number;
/** Set when the budget ran out before the walk finished. */
truncated: boolean;
failures: { path: string; reason: string }[];
}
interface CachedListing {
at: number;
listing: Promise<FmListing>;
}
/** A listing is reused this long: long enough for ls→read, short enough to stay live. */
const LISTING_TTL_MS = 60_000;
/**
* The file manager as a tree of paths:
*
* / the four areas
* /my/… Persönliche Dateien
* /courses/<course>/… Kurs-Dateien
* /teams/<team>/… Team-Dateien
* /shared/… Geteilte Dateien (flat)
*
* Names are the file manager's own. Because course names contain `/` in real
* data ("LF07 - FIA24A/B - Sb/Ha"), a path is not split naively: resolution
* tries joining consecutive segments into one name and backtracks when a
* shorter reading leads nowhere. Any segment may also be an id instead of a
* name, which is always unambiguous and is what the listings print alongside.
*/
export class FileManager {
private readonly client: SchulcloudClient;
private readonly cache = new Map<string, CachedListing>();
constructor(client: SchulcloudClient) {
this.client = client;
}
/** Lists one directory. The root is synthetic and costs nothing. */
async list(ref: DirectoryRef): Promise<FmListing> {
const page = pageFor(ref);
if (!page) {
return { directories: FILE_AREAS.map((entry) => ({ id: entry.area, name: entry.area })), files: [] };
}
const cached = this.cache.get(page);
if (cached && Date.now() - cached.at < LISTING_TTL_MS) return cached.listing;
const listing = this.client.getFileManagerPage(page).then(parseFileListing);
this.cache.set(page, { at: Date.now(), listing });
// A failure must not be served from the cache for the next minute.
listing.catch(() => this.cache.delete(page));
return listing;
}
/** Resolves a path to a directory or a file. */
async resolve(path: string): Promise<FsNode> {
const segments = splitPath(path);
if (segments.length === 0) return { kind: 'directory', path: '/', ref: {}, name: '/' };
const first = segments[0] as string;
const area = FILE_AREAS.find((entry) => entry.aliases.includes(normalise(first)) || entry.area === first);
if (!area) {
throw new FsError(
'not_found',
`"${first}" is not a file area. The root holds ${FILE_AREAS.map((entry) => `/${entry.area} (${entry.label})`).join(', ')}.`,
);
}
const ref: DirectoryRef = { area: area.area };
const found = await this.descend(ref, `/${area.area}`, segments.slice(1));
return found;
}
private async descend(ref: DirectoryRef, path: string, rest: string[]): Promise<FsNode> {
if (rest.length === 0) return { kind: 'directory', path, ref, name: path.split('/').pop() || '/' };
const listing = await this.list(ref);
const readings = candidateReadings(listing, rest);
if (readings.length === 0) throw notFound(listing, rest[0] as string, path);
let lastError: FsError | undefined;
for (const reading of readings) {
if (reading.matches.length > 1) {
const options = reading.matches.map((match) => `"${match.entry.name}" (\`${match.entry.id}\`)`).join(', ');
throw new FsError(
'ambiguous',
`${path} holds more than one entry named "${reading.name}": ${options}. Use the id as that path segment instead.`,
);
}
const match = reading.matches[0] as { kind: 'directory'; entry: FmDirectory } | { kind: 'file'; entry: FmFile };
const remaining = rest.slice(reading.consumed);
const nextPath = joinPath(path, match.entry.name);
if (match.kind === 'file') {
if (remaining.length === 0) return { kind: 'file', path: nextPath, parent: ref, file: match.entry };
lastError = new FsError('not_a_directory', `${nextPath} is a file, not a folder.`);
continue;
}
try {
return await this.descend(childRef(ref, match.entry), nextPath, remaining);
} catch (error) {
// A shorter reading of a name containing "/" can lead nowhere while a
// longer one resolves; only give up once every reading has failed.
if (error instanceof FsError && error.code !== 'ambiguous' && error.code !== 'not_navigable') {
lastError = error;
continue;
}
throw error;
}
}
throw lastError ?? notFound(listing, rest[0] as string, path);
}
/**
* Walks a directory breadth-first, listing at most `maxDirectories` of them.
*
* Every listing is one page fetch, so the budget is the cost. A folder that
* cannot be read is recorded and skipped, never silently dropped.
*/
async walk(
start: { path: string; ref: DirectoryRef },
options: { maxDepth: number; maxDirectories: number; concurrency?: number },
): Promise<WalkResult> {
const entries: WalkEntry[] = [];
const failures: { path: string; reason: string }[] = [];
let visited = 0;
let truncated = false;
let frontier: { path: string; ref: DirectoryRef; depth: number }[] = [{ ...start, depth: 0 }];
while (frontier.length > 0) {
const next: typeof frontier = [];
const batch = frontier;
frontier = [];
const concurrency = Math.max(1, options.concurrency ?? 3);
for (let i = 0; i < batch.length; i += concurrency) {
const slice = batch.slice(i, i + concurrency);
await Promise.all(
slice.map(async (node) => {
// The root is synthetic and costs no request, so it does not count.
if (pageFor(node.ref)) {
if (visited >= options.maxDirectories) {
truncated = true;
return;
}
visited++;
}
let listing: FmListing;
try {
listing = await this.list(node.ref);
} catch (error) {
failures.push({ path: node.path, reason: error instanceof Error ? error.message : String(error) });
return;
}
// The root's areas keep their defined order; real folders sort by name.
const directories = pageFor(node.ref) ? sortByName(listing.directories) : listing.directories;
for (const directory of directories) {
const path = joinPath(node.path, directory.name);
let ref: DirectoryRef | undefined;
try {
ref = childRef(node.ref, directory);
} catch {
ref = undefined; // shared folders: listed, never opened
}
entries.push({
path,
parentPath: node.path,
depth: node.depth + 1,
parent: node.ref,
directory: { ref: ref ?? node.ref, name: directory.name, id: directory.id },
});
if (ref && node.depth + 1 < options.maxDepth) next.push({ path, ref, depth: node.depth + 1 });
}
for (const file of sortByName(listing.files)) {
entries.push({
path: joinPath(node.path, file.name),
parentPath: node.path,
depth: node.depth + 1,
parent: node.ref,
file,
});
}
}),
);
}
frontier = next;
}
// Unsorted across directories on purpose: sorting whole path strings
// interleaves a folder's children with a sibling that shares its prefix
// ("Sub/…" against "Sub - Kopie"). Renderers group on `parentPath`.
return { entries, visited, truncated, failures };
}
download(file: Pick<FmFile, 'id' | 'name'>): Promise<DownloadedFile> {
return this.client.downloadFileManagerFile(file.id, file.name);
}
}
type Match = { kind: 'directory'; entry: FmDirectory } | { kind: 'file'; entry: FmFile };
/**
* Every way the next path segments can name an entry, shortest first.
*
* `rest[0]`, then `rest[0]/rest[1]`, and so on — so a course called
* "LF07 - FIA24A/B - Sb/Ha" resolves even when typed plainly. An id segment
* matches too. Exact names are preferred over case-insensitive ones.
*/
function candidateReadings(listing: FmListing, rest: string[]): { name: string; consumed: number; matches: Match[] }[] {
const all: Match[] = [
...listing.directories.map((entry) => ({ kind: 'directory' as const, entry })),
...listing.files.map((entry) => ({ kind: 'file' as const, entry })),
];
const first = rest[0] as string;
if (/^[0-9a-f]{24}$/i.test(first)) {
const byId = all.filter((match) => match.entry.id.toLowerCase() === first.toLowerCase());
if (byId.length > 0) return [{ name: first, consumed: 1, matches: byId }];
}
const readings: { name: string; consumed: number; matches: Match[] }[] = [];
for (const strict of [true, false]) {
for (let take = 1; take <= rest.length; take++) {
const name = rest.slice(0, take).join('/');
const matches = all.filter((match) =>
strict ? match.entry.name.trim() === name : normalise(match.entry.name) === normalise(name),
);
if (matches.length > 0 && !readings.some((reading) => reading.consumed === take)) {
readings.push({ name, consumed: take, matches });
}
}
if (readings.length > 0) break;
}
return readings;
}
function notFound(listing: FmListing, segment: string, path: string): FsError {
const names = [...listing.directories.map((entry) => `${entry.name}/`), ...listing.files.map((entry) => entry.name)];
const needle = normalise(segment);
const close = names.filter((name) => normalise(name).includes(needle) || needle.includes(normalise(name).replace(/\/$/, '')));
const hint =
close.length > 0
? ` Did you mean: ${close.slice(0, 5).map((name) => `"${name}"`).join(', ')}?`
: names.length > 0
? ` It holds ${names.length} entr${names.length === 1 ? 'y' : 'ies'}; list it with fs_list.`
: ' It is empty.';
return new FsError('not_found', `No "${segment}" in ${path}.${hint}`);
}
function sortByName<T extends { name: string }>(items: T[]): T[] {
return [...items].sort((a, b) => compareNames(a.name, b.name));
}
/** Name order as a person expects it: German collation, "Blatt 2" before "Blatt 10". */
export function compareNames(a: string, b: string): number {
return a.localeCompare(b, 'de', { numeric: true, sensitivity: 'base' });
}
/**
* Case-insensitive substring; or, when the pattern uses * or ?, a glob that
* must match the whole name, as `find -name` does. Exported for testing.
*/
export function nameMatcher(pattern: string): (name: string) => boolean {
const needle = pattern.normalize('NFC').toLocaleLowerCase('de');
if (!/[*?]/.test(needle)) return (name) => name.normalize('NFC').toLocaleLowerCase('de').includes(needle);
const source = needle
.split('')
.map((char) => (char === '*' ? '.*' : char === '?' ? '.' : char.replace(/[.+^${}()|[\]\\]/g, '\\$&')))
.join('');
const regex = new RegExp(`^${source}$`, 'i');
return (name) => regex.test(name.normalize('NFC'));
}

View File

@@ -74,7 +74,7 @@ export function safeComponent(raw: string, fallback = 'untitled'): string {
* same card, which the API permits.
*/
export function mirrorPath(at: Breadcrumb, fileName: string, fileId: string): string {
const parts = [at.courseTitle, at.containerTitle, at.cardTitle]
const parts = [at.courseTitle, at.containerTitle, ...(at.folders ?? []), at.cardTitle]
.filter((part): part is string => Boolean(part && part.trim()))
.map((part) => safeComponent(part));

View File

@@ -2,6 +2,16 @@ import { createReadStream } from 'node:fs';
import { stat } from 'node:fs/promises';
import { Readable } from 'node:stream';
import express, { type Request, type Response, type Router } from 'express';
import { SchulcloudApiError } from '../core/client.ts';
import {
compareNames,
FileManagerMarkupError,
FsError,
nameMatcher,
type FmFile,
type FsErrorCode,
type WalkEntry,
} from '../core/legacy-files.ts';
import { resolveWithin } from '../core/paths.ts';
import type { Services } from '../services.ts';
@@ -64,6 +74,113 @@ export function createApiRouter(services: Services): Router {
}
});
// --- the file manager ("Dateien"), as a filesystem ----------------------
//
// Live, not from the index: these answer what the file manager holds now, and
// need no database. Paths are the same ones the MCP fs_* tools print.
router.get('/fs/list', async (req: Request, res: Response) => {
try {
const node = await services.files.resolve(stringParam(req.query.path) ?? '/');
if (node.kind === 'file') return res.json({ path: node.path, kind: 'file', file: node.file });
const listing = await services.files.list(node.ref);
return res.json({
path: node.path,
kind: 'directory',
area: node.ref.area ?? null,
directories: listing.directories.map((entry) => ({ ...entry, path: childPath(node.path, entry.name) })),
files: listing.files.map((entry) => ({ ...entry, path: childPath(node.path, entry.name) })),
});
} catch (error) {
return fsFail(res, error, 'fs list');
}
});
router.get('/fs/tree', async (req: Request, res: Response) => {
try {
const node = await services.files.resolve(stringParam(req.query.path) ?? '/');
if (node.kind === 'file') return res.json({ path: node.path, kind: 'file', file: node.file });
const result = await services.files.walk(node, {
maxDepth: boundedInt(req.query.depth, 3, 1, 12),
maxDirectories: boundedInt(req.query.maxFolders, 200, 1, 1000),
});
return res.json({
path: node.path,
kind: 'directory',
entries: result.entries.map(treeEntry),
visited: result.visited,
truncated: result.truncated,
failures: result.failures,
});
} catch (error) {
return fsFail(res, error, 'fs tree');
}
});
router.get('/fs/find', async (req: Request, res: Response) => {
const name = stringParam(req.query.name);
if (!name) return res.status(400).json({ error: 'bad_request', message: 'Give name.' });
try {
const node = await services.files.resolve(stringParam(req.query.path) ?? '/');
if (node.kind === 'file') return res.json({ path: node.path, kind: 'file', matches: [] });
const type = stringParam(req.query.type) ?? 'any';
const matches = nameMatcher(name);
const result = await services.files.walk(node, {
maxDepth: 12,
maxDirectories: boundedInt(req.query.maxFolders, 400, 1, 1000),
});
return res.json({
path: node.path,
kind: 'directory',
matches: result.entries
.filter((entry) => (type === 'file' ? entry.file : type === 'folder' ? entry.directory : true))
.filter((entry) => matches((entry.file ?? entry.directory)?.name ?? ''))
.sort((a, b) => compareNames(a.path, b.path))
.map(treeEntry),
visited: result.visited,
truncated: result.truncated,
failures: result.failures,
});
} catch (error) {
return fsFail(res, error, 'fs find');
}
});
router.get('/fs/file', async (req: Request, res: Response) => {
try {
const path = stringParam(req.query.path);
const id = stringParam(req.query.id);
let file: Pick<FmFile, 'id' | 'name'> & Partial<FmFile>;
if (path) {
const node = await services.files.resolve(path);
if (node.kind !== 'file') return res.status(400).json({ error: 'not_a_file', message: `${node.path} is a folder.` });
file = node.file;
} else if (id && /^[0-9a-f]{24}$/i.test(id)) {
file = { id, name: stringParam(req.query.name) ?? id };
} else {
return res.status(400).json({ error: 'bad_request', message: 'Give path, or id (and name).' });
}
if (file.blocked) {
return res.status(403).json({ error: 'blocked', message: 'The instance virus scanner blocked this file.' });
}
// Streamed straight through rather than buffered: the CLI uses this for
// whole folders, and videos routinely exceed any sensible in-memory cap.
const signed = await services.client.getFileManagerSignedUrl(file.id, file.name);
const upstream = await services.client.openSignedUrl(signed);
if (!upstream.body) return res.status(502).json({ error: 'upstream_failed', message: 'empty response' });
res.setHeader('Content-Type', file.mimeType || upstream.headers.get('content-type') || 'application/octet-stream');
res.setHeader('Content-Disposition', contentDisposition(file.name));
const length = upstream.headers.get('content-length');
if (length) res.setHeader('Content-Length', length);
res.setHeader('X-Schulcloud-Source', 'file-manager');
Readable.fromWeb(upstream.body as never).pipe(res);
} catch (error) {
return fsFail(res, error, 'fs file');
}
});
/**
* Streams one file. Served from the local mirror when present; otherwise
* proxied live, which is what keeps files too large to mirror reachable.
@@ -91,6 +208,8 @@ export function createApiRouter(services: Services): Router {
});
}
}
const known = await services.store.fileSource(fileId);
if (known?.source === 'file-manager') return await proxyFileManager(services, fileId, known, res);
return await proxyLive(services, fileId, res);
} catch (error) {
return fail(res, error, 'file');
@@ -101,6 +220,27 @@ export function createApiRouter(services: Services): Router {
}
/** Falls back to Schulcloud for anything not in the mirror, streaming through. */
/** Streams a file-manager file live, via its pre-signed URL; no credentials leave for the storage host. */
async function proxyFileManager(
services: Services,
fileId: string,
known: { name: string; mimeType: string; size: number },
res: Response,
): Promise<void> {
const signed = await services.client.getFileManagerSignedUrl(fileId, known.name);
const upstream = await services.client.openSignedUrl(signed);
if (!upstream.body) {
res.status(502).json({ error: 'upstream_failed', message: 'empty response' });
return;
}
res.setHeader('Content-Type', known.mimeType || 'application/octet-stream');
res.setHeader('Content-Disposition', contentDisposition(known.name));
const length = upstream.headers.get('content-length');
if (length) res.setHeader('Content-Length', length);
res.setHeader('X-Schulcloud-Source', 'live');
Readable.fromWeb(upstream.body as never).pipe(res);
}
async function proxyLive(services: Services, fileId: string, res: Response): Promise<void> {
const record = await services.client.getFileRecord(fileId);
if (record.securityCheckStatus === 'blocked') {
@@ -143,3 +283,64 @@ function fail(res: Response, error: unknown, what: string): void {
console.error(`[schulcloud-mcp] ${what} failed:`, error);
if (!res.headersSent) res.status(500).json({ error: 'internal_error' });
}
function stringParam(value: unknown): string | undefined {
const first = Array.isArray(value) ? value[0] : value;
return typeof first === 'string' && first.length > 0 ? first : undefined;
}
function boundedInt(value: unknown, fallback: number, min: number, max: number): number {
const parsed = Number.parseInt(stringParam(value) ?? '', 10);
return Number.isFinite(parsed) ? Math.min(max, Math.max(min, parsed)) : fallback;
}
function childPath(parent: string, name: string): string {
return `${parent === '/' ? '' : parent}/${name}`;
}
/** A walk entry as JSON; `name` travels separately because names may contain "/". */
function treeEntry(entry: WalkEntry) {
if (entry.directory) {
return { type: 'directory', path: entry.path, parentPath: entry.parentPath, depth: entry.depth, id: entry.directory.id, name: entry.directory.name };
}
const file = entry.file as FmFile;
return {
type: 'file',
path: entry.path,
parentPath: entry.parentPath,
depth: entry.depth,
id: file.id,
name: file.name,
size: file.size,
mimeType: file.mimeType ?? null,
blocked: file.blocked,
};
}
const FS_STATUS: Record<FsErrorCode, number> = {
not_found: 404,
ambiguous: 409,
not_a_directory: 400,
not_a_file: 400,
not_navigable: 422,
};
function fsFail(res: Response, error: unknown, what: string): void {
if (res.headersSent) return;
if (error instanceof FsError) {
res.status(FS_STATUS[error.code]).json({ error: error.code, message: error.message });
return;
}
if (error instanceof FileManagerMarkupError) {
res.status(502).json({ error: 'markup_changed', message: error.message });
return;
}
if (error instanceof SchulcloudApiError) {
// 401 upstream is the Pi's session, not the caller's token: say which.
const status = error.status === 401 ? 502 : error.status === 403 || error.status === 404 ? error.status : 502;
const message = error.status === 401 ? 'The Schulcloud session has expired on the server.' : error.message;
res.status(status).json({ error: 'upstream_failed', message });
return;
}
fail(res, error, what);
}

View File

@@ -117,6 +117,7 @@ export class Indexer {
includeLessonContents: true,
includeFiles: true,
includePersonalFiles: this.config.indexPersonalFiles,
includeFileManager: this.config.indexFileManager,
config: this.config,
});
@@ -192,7 +193,11 @@ export class Indexer {
}
try {
const downloaded = await this.client.downloadFile(file.record);
// The file manager's ids mean nothing to files-storage; route by store.
const downloaded =
file.source === 'file-manager'
? await this.client.downloadFileManagerFile(file.record.id, file.record.name)
: await this.client.downloadFile(file.record);
const relative = paths.get(entry.fileId);
if (!relative) return;
const absolute = resolveWithin(this.config.mirrorDir, relative);

View File

@@ -4,6 +4,7 @@ import { ServerContext } from '../context.ts';
import type { Services } from '../services.ts';
import { registerContentTools } from './tools/content.ts';
import { registerFileTools } from './tools/files.ts';
import { registerFilesystemTools } from './tools/filesystem.ts';
import { registerOverviewTools } from './tools/overview.ts';
import { registerRawTool } from './tools/raw.ts';
import { registerIndexTools } from './tools/index-tools.ts';
@@ -26,6 +27,11 @@ How the content is organised, and the usual path through it:
- **Tasks** ("Aufgaben") — homework. list_tasks across all courses, get_task for one.
- **Files** hang off boards, lessons and tasks. Every listing shows file ids; download_file fetches one and
extracts its text (PDF, Word, Excel, PowerPoint, OpenDocument) or returns an image inline.
- **The file manager ("Dateien")** is a separate store with a real folder tree, browsed with the fs_* tools:
/my (Persönliche Dateien), /courses/<course> (Kurs-Dateien), /teams/<team> (Team-Dateien) and /shared
(Geteilte Dateien). **Many teachers put their material only here**, so when a course page looks empty or the
worksheets are not on its boards, look in /courses/<course name>. fs_list and fs_tree browse, fs_find finds by
name, fs_read opens a file. list_files and download_file do not see these files.
- **Submissions** ("Abgaben") — what the user handed in. get_task shows that task's submission: the files,
the graded flag, the grade, what the user wrote, and the teacher's written feedback. A grade is a
percentage (0-100) or absent — there is no textual grade — and teachers often grade with the written
@@ -51,6 +57,7 @@ export function createServer(config: Config, services?: Services): { server: Mcp
registerContentTools(server, context);
registerRoomTools(server, context);
registerFileTools(server, context);
registerFilesystemTools(server, context);
registerSearchTool(server, context);
registerSubmissionTools(server, context);
registerIndexTools(server, context);

View File

@@ -7,6 +7,7 @@ import { dueLabel, formatDate, heading, htmlToText, joinSections, normalizeObjec
import { assembleBoard, type AssembledBoard, type AssembledElement } from '../../core/board.ts';
import { forEachLimited } from '../../core/crawl.ts';
import { fetchLessonPadText } from '../../core/etherpad.ts';
import type { FmListing } from '../../core/legacy-files.ts';
import { fetchLessonTaskLinks, withScrapedIds } from '../../core/lesson-page.ts';
import type {
CourseBoardResponse,
@@ -38,17 +39,21 @@ export function registerContentTools(server: McpServer, context: ServerContext):
},
async ({ courseId }) => {
try {
const [board, legacy] = await Promise.all([
const [board, legacy, courseFiles] = await Promise.all([
context.client.getCourseBoard(courseId),
// The v3 projection carries no description, teachers, members or
// timetable; /api/v1/courses still does. Optional on purpose — it
// is a legacy route, so its absence must cost detail, not the call.
context.client.getLegacyCourse(courseId).catch(() => undefined),
// The course's file-manager area is a different store from the page.
// Teachers who only upload files there leave the page itself empty,
// and reporting "empty" then sends the reader away from the material.
context.files.list({ area: 'courses', ownerId: courseId }).catch(() => undefined),
]);
const teachers = legacy
? await context.resolveNames([...(legacy.teacherIds ?? []), ...(legacy.substitutionIds ?? [])])
: { names: [], unresolved: 0 };
return text(formatCourseBoard(board, legacy, teachers));
return text(formatCourseBoard(board, legacy, teachers, courseFiles));
} catch (error) {
return toToolError(error, `read course ${courseId}`);
}
@@ -266,6 +271,7 @@ function formatCourseBoard(
board: CourseBoardResponse,
legacy?: LegacyCourse,
teachers: { names: string[]; unresolved: number } = { names: [], unresolved: 0 },
courseFiles?: FmListing,
): string {
const boards: string[] = [];
const lessons: string[] = [];
@@ -293,8 +299,18 @@ function formatCourseBoard(
formatCourseTimes(legacy?.times),
]);
const filesSection = formatCourseFiles(board.roomId, courseFiles);
if (boards.length + lessons.length + tasks.length === 0) {
return joinSections([heading(2, board.title), about, 'This course page is empty.']);
return joinSections([
heading(2, board.title),
`Course id: \`${board.roomId}\``,
about,
filesSection
? 'No boards, topics or tasks on the course page — the material is in the course files instead.'
: 'This course page is empty, and the course has no files in the file manager either.',
filesSection,
]);
}
return joinSections([
@@ -304,6 +320,25 @@ function formatCourseBoard(
boards.length > 0 && joinSections([heading(3, `Boards (${boards.length})`), boards.join('\n'), 'Read one with get_board.']),
lessons.length > 0 && joinSections([heading(3, `Topics (${lessons.length})`), lessons.join('\n'), 'Read one with get_lesson.']),
tasks.length > 0 && joinSections([heading(3, `Tasks (${tasks.length})`), tasks.join('\n'), 'Read one with get_task.']),
filesSection,
]);
}
/**
* The course's own file-manager area ("Kurs-Dateien"), when it holds anything.
*
* Only the top level is fetched — one page — so this says how much is there
* and where, rather than listing it; fs_tree does that.
*/
function formatCourseFiles(courseId: string, listing: FmListing | undefined): string | undefined {
if (!listing || listing.directories.length + listing.files.length === 0) return undefined;
const names = [...listing.directories.map((entry) => `${entry.name}/`), ...listing.files.map((entry) => entry.name)];
const shown = names.slice(0, 8).map((name) => `- ${name}`).join('\n');
return joinSections([
heading(3, 'Course files (Kurs-Dateien)'),
`${listing.directories.length} folder(s) and ${listing.files.length} file(s) at the top level, newest first:`,
shown + (names.length > 8 ? `\n- … and ${names.length - 8} more` : ''),
`See everything with fs_tree path "/courses/${courseId}", read one with fs_read.`,
]);
}

View File

@@ -2,6 +2,7 @@ import type { CallToolResult } from '@modelcontextprotocol/sdk/types.js';
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
import type { ServerContext } from '../../context.ts';
import type { DownloadedFile } from '../../core/client.ts';
import { extractContent, formatBytes } from '../../core/extract.ts';
import { formatDate, heading, joinSections } from '../../core/text.ts';
import { FILE_PARENT_TYPES, type FileParentType } from '../../core/types.ts';
@@ -16,9 +17,11 @@ export function registerFileTools(server: McpServer, context: ServerContext): vo
{
title: 'List files of an entity',
description:
'Files attached to one entity. Most of the time you do not need this — get_board, get_lesson and ' +
'get_task already list their own attachments. Reach for it to enumerate a course\'s own file area, ' +
'or a single board element\'s files (parentType "boardnodes", parentId = the element id).',
'Attachments on one entity in files-storage: a board element, a topic, a task, a submission. Most of the ' +
'time you do not need this — get_board, get_lesson and get_task already list their own attachments. ' +
'**Not for a course\'s files, personal files, team files or shared files**: those live in the file ' +
'manager ("Dateien"), a separate store this tool cannot see — it answers 0 for a course holding dozens of ' +
'worksheets. Use fs_list, fs_tree, fs_find and fs_read for them.',
inputSchema: {
parentType: z
.enum(FILE_PARENT_TYPES as [FileParentType, ...FileParentType[]])
@@ -61,9 +64,10 @@ export function registerFileTools(server: McpServer, context: ServerContext): vo
{
title: 'Download and read a file',
description:
'Fetches a file and returns its contents. PDFs, Word, Excel, PowerPoint and OpenDocument files are ' +
'extracted to text; images come back inline so you can look at them; anything else reports its type. ' +
'Pass raw=true to get base64 bytes instead of extracted text.',
'Fetches a board, topic or task attachment and returns its contents. PDFs, Word, Excel, PowerPoint and ' +
'OpenDocument files are extracted to text; images come back inline so you can look at them; anything ' +
'else reports its type. Pass raw=true to get base64 bytes instead of extracted text. For files from the ' +
'file manager (Persönliche Dateien, Kurs-Dateien, Team-Dateien, Geteilte Dateien) use fs_read instead.',
inputSchema: {
fileId: z.string().describe('File record id, from get_board, get_task, get_lesson or list_files.'),
raw: z
@@ -116,76 +120,99 @@ export function registerFileTools(server: McpServer, context: ServerContext): vo
.join('\n'),
].join('\n\n');
if (raw) {
return text(
joinSections([
header,
`Base64 (${file.bytes.length} bytes):`,
'```',
file.bytes.toString('base64'),
'```',
]),
);
}
const extraction = await extractContent(
file.bytes,
file.mimeType || record.mimeType,
record.name,
maxChars ?? context.config.maxExtractedChars,
);
if (extraction.kind === 'image' && extraction.image) {
const result: CallToolResult = {
content: [
{ type: 'text', text: joinSections([header, extraction.note]) },
{ type: 'image', data: extraction.image.base64, mimeType: extraction.image.mimeType },
],
};
return result;
}
if (extraction.kind === 'text') {
const body = extraction.text?.trim();
return text(
joinSections([
header,
extraction.note,
body ? joinSections([heading(3, 'Contents'), body]) : '_(the file contains no extractable text)_',
]),
);
}
// Nothing extractable — but files-storage may still be able to render
// the file as a picture. That is the whole answer for an image-only
// PDF: its pages *are* pictures, so a rasterised preview is readable
// where the bytes are not, and it needs no OCR on our side.
if (record.previewStatus === 'preview_possible') {
const preview = await context.client.getFilePreview(record, 500).catch(() => undefined);
if (preview && preview.mimeType.startsWith('image/')) {
const result: CallToolResult = {
content: [
{
type: 'text',
text: joinSections([
header,
// The note ends by suggesting raw bytes, which is no longer the
// best answer once a readable rendering is attached.
extraction.note.replace(' Use download_file with raw=true to get the bytes.', ''),
"Showing the instance's own rendered preview below, which is readable as a picture.",
]),
},
{ type: 'image', data: preview.bytes.toString('base64'), mimeType: preview.mimeType },
],
};
return result;
}
}
return text(joinSections([header, extraction.note]));
return await renderFileContent(context, header, file, {
name: record.name,
mimeType: record.mimeType,
raw,
maxChars,
// Nothing extractable — but files-storage may still be able to render
// the file as a picture. That is the whole answer for an image-only
// PDF: its pages *are* pictures, so a rasterised preview is readable
// where the bytes are not, and it needs no OCR on our side.
fallbackImage:
record.previewStatus === 'preview_possible'
? async () => {
const preview = await context.client.getFilePreview(record, 500).catch(() => undefined);
return preview && preview.mimeType.startsWith('image/') ? preview : undefined;
}
: undefined,
});
} catch (error) {
return toToolError(error, `download file ${fileId}`);
}
},
);
}
/**
* Renders a downloaded file for a tool result: base64 when asked for, an image
* inline, extracted text, or — for a format with no extractor — its note.
*
* Shared by download_file (files-storage) and fs_read (the file manager), which
* differ only in how the bytes were obtained and in what the header says.
* `fallbackImage` is download_file's preview route; the file manager has none.
*/
export async function renderFileContent(
context: ServerContext,
header: string,
file: DownloadedFile,
options: {
name: string;
mimeType?: string;
raw: boolean;
maxChars?: number;
fallbackImage?: () => Promise<DownloadedFile | undefined>;
},
): Promise<CallToolResult> {
if (options.raw) {
return text(joinSections([header, `Base64 (${file.bytes.length} bytes):`, '```', file.bytes.toString('base64'), '```']));
}
const extraction = await extractContent(
file.bytes,
file.mimeType && file.mimeType !== 'application/octet-stream' ? file.mimeType : (options.mimeType ?? file.mimeType),
options.name,
options.maxChars ?? context.config.maxExtractedChars,
);
if (extraction.kind === 'image' && extraction.image) {
return {
content: [
{ type: 'text', text: joinSections([header, extraction.note]) },
{ type: 'image', data: extraction.image.base64, mimeType: extraction.image.mimeType },
],
};
}
if (extraction.kind === 'text') {
const body = extraction.text?.trim();
return text(
joinSections([
header,
extraction.note,
body ? joinSections([heading(3, 'Contents'), body]) : '_(the file contains no extractable text)_',
]),
);
}
const image = await options.fallbackImage?.();
if (image) {
return {
content: [
{
type: 'text',
text: joinSections([
header,
// The note ends by suggesting raw bytes, which is no longer the
// best answer once a readable rendering is attached.
extraction.note.replace(/ Use \w+ with raw=true to get the bytes\./, ''),
"Showing the instance's own rendered preview below, which is readable as a picture.",
]),
},
{ type: 'image', data: image.bytes.toString('base64'), mimeType: image.mimeType },
],
};
}
return text(joinSections([header, extraction.note]));
}

401
src/mcp/tools/filesystem.ts Normal file
View File

@@ -0,0 +1,401 @@
import type { CallToolResult } from '@modelcontextprotocol/sdk/types.js';
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
import type { ServerContext } from '../../context.ts';
import { formatBytes } from '../../core/extract.ts';
import {
areaInfo,
compareNames,
FILE_AREAS,
FileManagerMarkupError,
FsError,
nameMatcher,
type DirectoryRef,
type FmFile,
type FsNode,
type WalkEntry,
} from '../../core/legacy-files.ts';
import { heading, joinSections } from '../../core/text.ts';
import { renderFileContent } from './files.ts';
import { failure, text, toToolError } from './result.ts';
const READ_ONLY = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true };
/**
* The "Dateien" file manager as filesystem tools: ls, tree, find, read.
*
* Deliberately separate from list_files / download_file, which read
* files-storage — board, topic and task attachments. The two stores do not
* overlap, and conflating them is how a course holding dozens of worksheets
* came to be reported as having 0 files.
*/
const AREA_NOTE =
'The file manager ("Dateien") is separate from course pages and holds four areas: ' +
'/my (Persönliche Dateien), /courses/<course name> (Kurs-Dateien), /teams/<team name> (Team-Dateien) and ' +
'/shared (Geteilte Dateien). Many teachers keep their material only in Kurs-Dateien, so a course whose page ' +
'looks empty often has its worksheets here.';
const PATH_NOTE =
'Paths use the names shown in listings, e.g. "/courses/FIA24B - LF2 (Rh)/Handlungssituation". Names may ' +
'contain "/" and still resolve; any segment may also be the id printed next to it, which is never ambiguous.';
export function registerFilesystemTools(server: McpServer, context: ServerContext): void {
server.registerTool(
'fs_list',
{
title: 'List a folder in the file manager',
description:
`Lists one folder of the Schulcloud file manager, like \`ls\`. ${AREA_NOTE} Start at "/" or ` +
'"/courses" to see what exists. Given a file path instead, shows that file\'s details. ' +
`${PATH_NOTE} Not for attachments on boards, topics or tasks — get_board, get_lesson and get_task list ` +
'those, and download_file reads them.',
inputSchema: {
path: z.string().default('/').describe('Folder to list, e.g. "/", "/courses", "/courses/<course>/<folder>".'),
},
annotations: READ_ONLY,
},
async ({ path }) => {
try {
const node = await context.files.resolve(path);
if (node.kind === 'file') return text(describeFile(node));
return text(await listDirectory(context, node));
} catch (error) {
return fsError(error, `list ${path}`);
}
},
);
server.registerTool(
'fs_tree',
{
title: 'Show a folder tree in the file manager',
description:
`Everything below a folder of the Schulcloud file manager, as an indented tree, like \`tree\`. ${AREA_NOTE} ` +
'Use it to get an overview of a course\'s files in one call — "/courses/<course>" — or of all course ' +
'files at a shallow depth. Each folder costs one page load, so the walk stops at `maxFolders` and says ' +
'so; narrow the path or lower the depth rather than raising the limit. To look for a name, fs_find is ' +
`cheaper. ${PATH_NOTE}`,
inputSchema: {
path: z.string().default('/').describe('Folder to start from.'),
depth: z.number().int().min(1).max(8).default(3).describe('How many levels below the folder to show.'),
maxFolders: z
.number()
.int()
.min(1)
.max(400)
.default(80)
.describe('Stop after listing this many folders.'),
},
annotations: READ_ONLY,
},
async ({ path, depth, maxFolders }) => {
try {
const node = await context.files.resolve(path);
if (node.kind === 'file') return text(describeFile(node));
const result = await context.files.walk(node, { maxDepth: depth, maxDirectories: maxFolders });
return text(renderTree(node, result.entries, { depth, maxFolders, ...result }));
} catch (error) {
return fsError(error, `walk ${path}`);
}
},
);
server.registerTool(
'fs_find',
{
title: 'Find files by name in the file manager',
description:
`Finds files and folders by name anywhere below a folder of the Schulcloud file manager, like \`find\`. ` +
`${AREA_NOTE} Without wildcards it matches any part of the name, case-insensitively. With "*" or "?" the ` +
'whole name must match, as with find -name — so "*.docx", or "*Erben*" for names containing Erben. Scope it with ' +
'`path` (e.g. "/courses/<course>") whenever you know the course: searching all of /courses walks every ' +
'folder of every course. This matches names only — to search inside documents use search, which ' +
`covers file-manager files once they are indexed. ${PATH_NOTE}`,
inputSchema: {
name: z
.string()
.min(1)
.describe('Part of the name ("Erbrecht"), or a whole-name pattern with * and ? ("*.docx", "*Erben*").'),
path: z.string().default('/').describe('Folder to search below. Default: every area.'),
type: z.enum(['any', 'file', 'folder']).default('any').describe('Only files, only folders, or both.'),
maxFolders: z
.number()
.int()
.min(1)
.max(600)
.default(250)
.describe('Stop after listing this many folders.'),
},
annotations: READ_ONLY,
},
async ({ name, path, type, maxFolders }) => {
try {
const node = await context.files.resolve(path);
if (node.kind === 'file') return text(describeFile(node));
const matcher = nameMatcher(name);
const result = await context.files.walk(node, { maxDepth: 12, maxDirectories: maxFolders });
const hits = result.entries
.filter((entry) => (type === 'file' ? entry.file : type === 'folder' ? entry.directory : true))
.filter((entry) => matcher((entry.file ?? entry.directory)?.name ?? ''))
.sort((a, b) => compareNames(a.path, b.path));
const scope = `${result.visited} folder(s) searched`;
const notes = [
result.truncated
? `_Stopped after ${maxFolders} folders, so there may be more matches. Narrow \`path\` to one course._`
: undefined,
failureNote(result.failures),
];
if (hits.length === 0) {
return text(joinSections([`No names matching "${name}" below ${node.path} (${scope}).`, ...notes]));
}
return text(
joinSections([
heading(2, `${hits.length} match(es) for "${name}" below ${node.path}`),
hits.map((entry) => entryLine(entry, { fullPath: true })).join('\n'),
`_${scope}._ Read a file with fs_read, open a folder with fs_list.`,
...notes,
]),
);
} catch (error) {
return fsError(error, `search ${path}`);
}
},
);
server.registerTool(
'fs_read',
{
title: 'Read a file from the file manager',
description:
`Fetches one file from the Schulcloud file manager and returns its contents, like \`cat\`. ${AREA_NOTE} ` +
'PDFs, Word, Excel, PowerPoint and OpenDocument files are extracted to text; images come back inline so ' +
'you can look at them; anything else reports its type. Pass raw=true for base64 bytes. Give the path ' +
'from a listing, or the file id and name. Not for board, topic or task attachments — use download_file ' +
`for those. ${PATH_NOTE}`,
inputSchema: {
path: z.string().optional().describe('File path, e.g. "/courses/<course>/<folder>/Arbeitsblatt.pdf".'),
fileId: z.string().optional().describe('File id from a listing, instead of a path.'),
name: z.string().optional().describe('The file name, when giving fileId; used to recognise the format.'),
raw: z.boolean().default(false).describe('Return base64-encoded bytes instead of extracted text.'),
maxChars: z
.number()
.int()
.min(500)
.max(500_000)
.optional()
.describe('Override the character limit on extracted text.'),
},
annotations: READ_ONLY,
},
async ({ path, fileId, name, raw, maxChars }) => {
try {
let file: Pick<FmFile, 'id' | 'name'> & Partial<FmFile>;
let where: string;
if (path) {
const node = await context.files.resolve(path);
if (node.kind !== 'file') {
return failure(`${node.path} is a folder, not a file. List it with fs_list, or use fs_tree.`);
}
file = node.file;
where = node.path;
} else if (fileId) {
if (!/^[0-9a-f]{24}$/i.test(fileId)) return failure(`"${fileId}" is not a file id.`);
file = { id: fileId, name: name?.trim() || fileId };
where = `file \`${fileId}\``;
} else {
return failure('Give either `path` or `fileId`.');
}
// The instance scans uploads; a file it rejected is not served.
if (file.blocked) {
return failure(`"${file.name}" was blocked by the instance's virus scanner and will not be downloaded.`);
}
const downloaded = await context.files.download(file);
const header = [
heading(2, file.name),
[
`- Path: ${where}`,
`- File id: \`${file.id}\``,
`- Type: ${file.mimeType ?? downloaded.mimeType}`,
`- Size: ${formatBytes(file.size ?? downloaded.bytes.length)}`,
downloaded.truncated
? `- **Download was capped at ${formatBytes(context.config.maxDownloadBytes)}; content is incomplete.**`
: undefined,
]
.filter(Boolean)
.join('\n'),
].join('\n\n');
return await renderFileContent(context, header, downloaded, {
name: file.name,
mimeType: file.mimeType,
raw,
maxChars,
});
} catch (error) {
return fsError(error, `read ${path ?? fileId}`);
}
},
);
}
async function listDirectory(context: ServerContext, node: Extract<FsNode, { kind: 'directory' }>): Promise<string> {
if (!node.ref.area) {
return joinSections([
heading(2, '/ — the file manager ("Dateien")'),
FILE_AREAS.map((entry) => `- **/${entry.area}/** — ${entry.label}: ${entry.summary}`).join('\n'),
'Open one with fs_list, e.g. path "/courses". A course\'s own files are under "/courses/<course name>".',
]);
}
const listing = await context.files.list(node.ref);
const area = areaInfo(node.ref.area);
const isOwnerList = (node.ref.area === 'courses' || node.ref.area === 'teams') && !node.ref.ownerId;
const directories = [...listing.directories].sort((a, b) => compareNames(a.name, b.name));
const files = [...listing.files].sort((a, b) => compareNames(a.name, b.name));
const title = heading(2, `${node.path}${area.label}`);
if (directories.length === 0 && files.length === 0) {
return joinSections([
title,
isOwnerList
? `No ${node.ref.area === 'courses' ? 'courses' : 'teams'} with a file area.`
: node.ref.area === 'shared'
? 'Nothing has been shared with you.'
: 'This folder is empty.',
]);
}
const lines = [
...directories.map((directory) => `- **${directory.name}/** (\`${directory.id}\`)`),
...files.map((file) => `- ${fileLine(file)}`),
];
const bytes = files.reduce((sum, file) => sum + file.size, 0);
const summary = isOwnerList
? `${directories.length} ${node.ref.area === 'courses' ? 'course' : 'team'}(s). Their files are inside; fs_tree with depth 2 shows which hold any.`
: `${directories.length} folder(s), ${files.length} file(s)${files.length ? `, ${formatBytes(bytes)}` : ''}.`;
return joinSections([
title,
lines.join('\n'),
summary,
node.ref.area === 'shared' && directories.length > 0
? '_Shared folders cannot be opened — the file manager has no route for them, not even in the browser._'
: undefined,
'Open a folder with fs_list (its name or id appended to this path), read a file with fs_read.',
]);
}
function describeFile(node: Extract<FsNode, { kind: 'file' }>): string {
return joinSections([
heading(2, node.file.name),
[
`- Path: ${node.path}`,
`- File id: \`${node.file.id}\``,
`- Type: ${node.file.mimeType ?? 'unknown'}`,
`- Size: ${formatBytes(node.file.size)}`,
node.file.blocked ? '- **Blocked by the instance virus scanner; it cannot be downloaded.**' : undefined,
]
.filter(Boolean)
.join('\n'),
node.file.blocked ? undefined : 'Read it with fs_read.',
]);
}
function fileLine(file: FmFile): string {
const type = file.mimeType ? `, ${file.mimeType}` : '';
const blocked = file.blocked ? ' **[blocked by virus scan]**' : '';
return `${file.name}${formatBytes(file.size)}${type} (\`${file.id}\`)${blocked}`;
}
function entryLine(entry: WalkEntry, options: { fullPath: boolean }): string {
const label = options.fullPath ? entry.path : (entry.file ?? entry.directory)?.name;
if (entry.directory) return `- **${label}/** (\`${entry.directory.id}\`)`;
if (entry.file) return `- ${fileLine({ ...entry.file, name: label ?? entry.file.name })}`;
return `- ${label}`;
}
/**
* An indented tree, built from each entry's parent rather than from sorting
* path strings — see `FileManager.walk` for why that distinction matters.
*/
function renderTree(
root: { path: string; ref: DirectoryRef },
entries: WalkEntry[],
info: { depth: number; maxFolders: number; visited: number; truncated: boolean; failures: { path: string; reason: string }[] },
): string {
const children = new Map<string, WalkEntry[]>();
for (const entry of entries) {
const list = children.get(entry.parentPath) ?? [];
list.push(entry);
children.set(entry.parentPath, list);
}
const lines: string[] = [];
let files = 0;
let folders = 0;
let bytes = 0;
const visit = (path: string, indent: string) => {
const kids = children.get(path) ?? [];
// The areas under "/" keep their own order (personal, courses, teams,
// shared); everything else sorts folders first, then by name.
if (path !== '/') {
kids.sort((a, b) => {
if (Boolean(a.directory) !== Boolean(b.directory)) return a.directory ? -1 : 1;
return compareNames(a.path, b.path);
});
}
for (const kid of kids) {
if (kid.directory) {
folders++;
// An area's "id" is its slug, not an id anything accepts; leave it out.
const id = /^[0-9a-f]{24}$/i.test(kid.directory.id) ? ` \`${kid.directory.id}\`` : '';
lines.push(`${indent}${kid.directory.name}/${id}`);
visit(kid.path, `${indent} `);
} else if (kid.file) {
files++;
bytes += kid.file.size;
const blocked = kid.file.blocked ? ' [blocked]' : '';
lines.push(`${indent}${kid.file.name} (${formatBytes(kid.file.size)}) \`${kid.file.id}\`${blocked}`);
}
}
};
visit(root.path, '');
const area = root.ref.area ? `${areaInfo(root.ref.area).label}` : '';
if (lines.length === 0) {
return joinSections([heading(2, `${root.path}${area}`), 'Nothing below this folder.', failureNote(info.failures)]);
}
return joinSections([
heading(2, `${root.path}${area}`),
['```', ...lines, '```'].join('\n'),
`${folders} folder(s), ${files} file(s), ${formatBytes(bytes)}${info.visited} folder(s) listed, ${info.depth} level(s) deep.`,
info.truncated
? `_Stopped after listing ${info.maxFolders} folders; the tree is incomplete. Start deeper, e.g. at one course._`
: undefined,
failureNote(info.failures),
'Read a file with fs_read (path = this folder plus the names above).',
]);
}
function failureNote(failures: { path: string; reason: string }[]): string | undefined {
if (failures.length === 0) return undefined;
const shown = failures.slice(0, 5).map((entry) => `${entry.path} (${entry.reason})`).join('; ');
return `_Could not list ${failures.length} folder(s): ${shown}${failures.length > 5 ? '; …' : ''}._`;
}
function fsError(error: unknown, action: string): CallToolResult {
if (error instanceof FsError) return failure(error.message);
if (error instanceof FileManagerMarkupError) {
return failure(
`Could not ${action}: the file manager page did not look like a file listing. Either the session is no ` +
'longer accepted (check whoami) or the page markup changed.',
);
}
return toToolError(error, action);
}

View File

@@ -1,6 +1,7 @@
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
import type { ServerContext } from '../../context.ts';
import { FILE_AREAS } from '../../core/legacy-files.ts';
import { crawl } from '../../core/crawl.ts';
import { searchSnapshot, type Hit } from '../../core/match.ts';
import { formatDate, heading, joinSections } from '../../core/text.ts';
@@ -144,14 +145,37 @@ function targetIdFor(hit: SearchResult): string {
return hit.nodeId;
}
/** Where to go next for a hit. File-manager files are read by path, not by download_file. */
function nextStep(hit: SearchResult): string {
if (hit.kind === 'file' && hit.meta?.source === 'file-manager') {
const fsPath = typeof hit.meta.fsPath === 'string' ? hit.meta.fsPath : undefined;
return fsPath
? `\`fs_read\` with path \`${fsPath}\``
: `\`fs_read\` with fileId \`${hit.nodeId}\` and name \`${hit.title}\``;
}
// A submission has no id of its own that any tool takes: get_task is
// reached through the *task*, so point at that rather than at the
// submission id, which would simply 404.
return `\`${TOOL_FOR[hit.kind] ?? 'api_get'}\` with id \`${targetIdFor(hit)}\``;
}
/** "Kurs-Dateien, <course>" or the area's own name, from the file's fs path. */
function fileManagerPlace(hit: SearchResult): string {
const area = typeof hit.meta?.fsPath === 'string' ? hit.meta.fsPath.split('/')[1] : undefined;
const known = FILE_AREAS.find((entry) => entry.area === area);
if (!known) return 'the file manager';
return known.area === 'courses' && hit.courseTitle ? `${known.label}, ${hit.courseTitle}` : known.label;
}
function formatIndexed(hit: SearchResult): string {
const where =
hit.kind === 'file' && hit.meta?.source === 'file-manager'
? `file in ${fileManagerPlace(hit)}`
: `${hit.kind} in ${hit.courseTitle || hit.path}`;
return [
`- **${hit.title}** — ${hit.kind} in ${hit.courseTitle || hit.path}`,
`- **${hit.title}** — ${where}`,
hit.snippet && hit.snippet !== hit.title ? ` ${hit.snippet}` : undefined,
// A submission has no id of its own that any tool takes: get_task is
// reached through the *task*, so point at that rather than at the
// submission id, which would simply 404.
`\`${TOOL_FOR[hit.kind] ?? 'api_get'}\` with id \`${targetIdFor(hit)}\``,
nextStep(hit),
]
.filter(Boolean)
.join('\n');

View File

@@ -1,5 +1,6 @@
import type { Config } from './config.ts';
import { SchulcloudClient } from './core/client.ts';
import { FileManager } from './core/legacy-files.ts';
import { Indexer } from './indexer/indexer.ts';
import { Store } from './store/store.ts';
@@ -14,12 +15,19 @@ import { Store } from './store/store.ts';
export interface Services {
config: Config;
client: SchulcloudClient;
/**
* The "Dateien" file manager. Process-wide so its short listing cache is
* shared: an `ls` in one MCP session and a `schulcloud fs get` from the CLI
* then cost one page fetch between them, not two.
*/
files: FileManager;
store: Store | undefined;
indexer: Indexer | undefined;
}
export async function createServices(config: Config): Promise<Services> {
const client = new SchulcloudClient(config);
const files = new FileManager(client);
const store = await Store.open(config.databaseUrl);
const indexer = store ? new Indexer(client, store, config) : undefined;
@@ -29,7 +37,7 @@ export async function createServices(config: Config): Promise<Services> {
'/files, /manifest and refresh_index are unavailable. Set DATABASE_URL to enable them.',
);
}
return { config, client, store, indexer };
return { config, client, files, store, indexer };
}
export async function closeServices(services: Services): Promise<void> {

View File

@@ -396,6 +396,31 @@ export class Store {
);
}
/**
* Where a file's bytes live, from the latest crawl.
*
* The file manager and files-storage use the same id shape but not the same
* ids, so a live fetch has to know which one to ask.
*/
async fileSource(
fileId: string,
): Promise<{ source: 'files-storage' | 'file-manager'; name: string; mimeType: string; size: number } | undefined> {
const crawlId = await this.latestCrawlId();
if (crawlId === undefined) return undefined;
const { rows } = await this.db.query<{ title: string; meta: Record<string, unknown> }>(
`SELECT title, meta FROM nodes WHERE crawl_id = $1 AND kind = 'file' AND node_id = $2`,
[crawlId, fileId],
);
const row = rows[0];
if (!row) return undefined;
return {
source: row.meta.source === 'file-manager' ? 'file-manager' : 'files-storage',
name: row.title,
mimeType: typeof row.meta.mimeType === 'string' ? row.meta.mimeType : 'application/octet-stream',
size: typeof row.meta.size === 'number' ? row.meta.size : 0,
};
}
async mirrorEntry(fileId: string): Promise<{ path: string; size: number; name: string; mimeType: string } | undefined> {
const { rows } = await this.db.query<{ mirror_path: string | null; mirror_size: string | null; name: string; mime_type: string | null }>(
`SELECT mirror_path, mirror_size, name, mime_type FROM file_texts WHERE file_id = $1`,
@@ -612,6 +637,8 @@ function fileNode(file: CrawledFile): StoredNode {
parentId: file.parentId,
securityCheckStatus: file.record.securityCheckStatus,
at: file.at,
source: file.source ?? 'files-storage',
...(file.fsPath ? { fsPath: file.fsPath } : {}),
},
// File records are less immutable than they look: `PATCH /file/rename/{id}`
// changes the name in place, keeping the id and the size, and teachers do

291
test/legacy-files.test.ts Normal file
View File

@@ -0,0 +1,291 @@
import assert from 'node:assert/strict';
import { describe, it } from 'node:test';
import { checkSignedUrl, type SchulcloudClient } from '../src/core/client.ts';
import {
childRef,
FileManager,
FileManagerMarkupError,
FsError,
pageFor,
parseFileListing,
splitPath,
} from '../src/core/legacy-files.ts';
/**
* Fixtures follow the legacy client's templates as served (files/files.hbs and
* files/files-grid.hbs, client 33.40): a folder is an `openfolder` button whose
* name sits unescaped after an icon; a file is a `card file` div carrying its
* name, size and id as data attributes, with the viewer type further inside.
*/
const folder = (id: string, name: string) => `
<div class="col-xs-12">
<button class="card card-block folder openfolder" data-testid="folder-open-button" data-folder-id="${id}" aria-label="
files.label.clickToOpenDir">
<strong class="card-title-directory">
<i class="fa fa-folder" data-testid="folder-icon" aria-hidden="true"></i> ${name}
</strong>
</button>
</div>`;
const file = (id: string, name: string, size: number, type = 'application/pdf', blocked = false) => `
<div class="col-sm-12" data-testid="files-section">
<div class="card file ${blocked ? 'btn-file-danger' : ''}" data-href="/files/file?file=${name}" data-file-name="${name}" data-file-size="${size}" data-file-id="${id}">
<div class="openFile">
<div class="card-block
${blocked ? '' : ' fileviewer'}"
${blocked ? '' : ` data-file-viewer-type="${type}"`}
${blocked ? '' : ` data-file-viewer-id="${id}"`}
tabindex="0" role="button">
<a class="col-sm-10 title" data-testid="file-title">${name}</a>
</div>
</div>
</div>
</div>`;
const page = (folders: string[], files: string[]) => `
<html><body><main id="main-content" class="container-fluid">
<div class="route-files">
${folders.length ? `<section class="directories"><div class="row">${folders.join('')}</div></section>` : ''}
<section class="files"><div class="row">${files.join('')}</div></section>
</div></main></body></html>`;
const id = (n: number) => n.toString(16).padStart(24, '0');
describe('parseFileListing', () => {
it('reads folders and files with their ids, sizes and types', () => {
const listing = parseFileListing(
page([folder(id(1), 'Handlungssituation')], [file(id(2), '13_AB_Wareneingang.pdf', 171941)]),
);
assert.deepEqual(listing.directories, [{ id: id(1), name: 'Handlungssituation' }]);
assert.deepEqual(listing.files, [
{ id: id(2), name: '13_AB_Wareneingang.pdf', size: 171941, mimeType: 'application/pdf', blocked: false },
]);
});
it('decodes entities in both folder and file names', () => {
const listing = parseFileListing(
page([folder(id(1), 'Lager &amp; Logistik')], [file(id(2), 'A&amp;B &quot;final&quot;.pdf', 10)]),
);
assert.equal(listing.directories[0]?.name, 'Lager & Logistik');
assert.equal(listing.files[0]?.name, 'A&B "final".pdf');
});
it('marks a file the virus scanner blocked, which also has no viewer type', () => {
const listing = parseFileListing(page([], [file(id(3), 'bad.exe', 5, undefined, true), file(id(4), 'ok.png', 7, 'image/png')]));
assert.equal(listing.files[0]?.blocked, true);
assert.equal(listing.files[0]?.mimeType, undefined);
// The next card's type must not leak into the blocked one.
assert.equal(listing.files[1]?.mimeType, 'image/png');
});
it('returns an empty listing for an empty folder', () => {
assert.deepEqual(parseFileListing(page([], [])), { directories: [], files: [] });
});
it('refuses a page that is not a file-manager listing rather than reporting 0 files', () => {
assert.throws(() => parseFileListing('<html><body>Anmelden</body></html>'), FileManagerMarkupError);
});
});
describe('pageFor and childRef', () => {
it('builds each listing route', () => {
assert.equal(pageFor({}), undefined);
assert.equal(pageFor({ area: 'my' }), '/files/my/');
assert.equal(pageFor({ area: 'my', folderId: id(1) }), `/files/my/${id(1)}`);
assert.equal(pageFor({ area: 'courses' }), '/files/courses/');
assert.equal(pageFor({ area: 'courses', ownerId: id(2) }), `/files/courses/${id(2)}`);
assert.equal(pageFor({ area: 'teams', ownerId: id(2), folderId: id(3) }), `/files/teams/${id(2)}/${id(3)}`);
assert.equal(pageFor({ area: 'shared' }), '/files/shared/');
});
it('addresses a nested folder by its own id, whatever its depth', () => {
const course = childRef({ area: 'courses' }, { id: id(2), name: 'Kurs' });
const top = childRef(course, { id: id(3), name: 'A' });
const deep = childRef(top, { id: id(4), name: 'B' });
assert.deepEqual(deep, { area: 'courses', ownerId: id(2), folderId: id(4) });
});
it('refuses to open a shared folder, which the file manager cannot do either', () => {
assert.throws(
() => childRef({ area: 'shared' }, { id: id(5), name: 'Ordner' }),
(error: unknown) => error instanceof FsError && error.code === 'not_navigable',
);
});
});
describe('splitPath', () => {
it('drops empty segments and surrounding spaces', () => {
assert.deepEqual(splitPath('//courses/ Kurs A /x.pdf/'), ['courses', 'Kurs A', 'x.pdf']);
assert.deepEqual(splitPath('/'), []);
});
});
/** A client that serves canned listing pages and records what was asked for. */
function fakeClient(pages: Record<string, string>) {
const requested: string[] = [];
const client = {
getFileManagerPage: async (path: string) => {
requested.push(path);
const html = pages[path];
if (html === undefined) throw new Error(`no fixture for ${path}`);
return html;
},
} as unknown as SchulcloudClient;
return { client, requested };
}
describe('FileManager.resolve', () => {
const course = id(10);
const slashCourse = id(11);
const sub = id(12);
const pdf = id(13);
const pages = {
'/files/courses/': page([folder(course, 'FIA24B - LF2 (Rh)'), folder(slashCourse, 'LF07 - FIA24A/B - Sb/Ha')], []),
[`/files/courses/${course}`]: page([folder(sub, 'Handlungssituation')], [file(pdf, 'Mahnwesen.pdf', 100)]),
[`/files/courses/${course}/${sub}`]: page([], [file(id(14), 'Lager.pdf', 50)]),
[`/files/courses/${slashCourse}`]: page([], [file(id(15), 'MQTT.docx', 70, 'application/vnd.openxmlformats-officedocument.wordprocessingml.document')]),
'/files/my/': page([], []),
};
it('resolves the root and each area without a request for the root', async () => {
const { client, requested } = fakeClient(pages);
const fs = new FileManager(client);
assert.deepEqual(await fs.resolve('/'), { kind: 'directory', path: '/', ref: {}, name: '/' });
const area = await fs.resolve('/Kurs-Dateien');
assert.equal(area.kind, 'directory');
assert.deepEqual(area.kind === 'directory' && area.ref, { area: 'courses' });
assert.deepEqual(requested, []);
});
it('walks names down to a file in a nested folder', async () => {
const { client } = fakeClient(pages);
const node = await new FileManager(client).resolve('/courses/FIA24B - LF2 (Rh)/Handlungssituation/Lager.pdf');
assert.equal(node.kind, 'file');
assert.equal(node.kind === 'file' && node.file.id, id(14));
assert.equal(node.path, '/courses/FIA24B - LF2 (Rh)/Handlungssituation/Lager.pdf');
});
it('resolves a course whose name contains slashes, typed plainly', async () => {
const { client } = fakeClient(pages);
const node = await new FileManager(client).resolve('/courses/LF07 - FIA24A/B - Sb/Ha/MQTT.docx');
assert.equal(node.kind === 'file' && node.file.id, id(15));
});
it('accepts ids as path segments', async () => {
const { client } = fakeClient(pages);
const node = await new FileManager(client).resolve(`/courses/${course}/${pdf}`);
assert.equal(node.kind === 'file' && node.file.name, 'Mahnwesen.pdf');
});
it('falls back to a case-insensitive match', async () => {
const { client } = fakeClient(pages);
const node = await new FileManager(client).resolve('/courses/fia24b - lf2 (rh)/handlungssituation');
assert.equal(node.kind, 'directory');
});
it('says what is there when a name is not found', async () => {
const { client } = fakeClient(pages);
await assert.rejects(
new FileManager(client).resolve('/courses/FIA24B - LF2 (Rh)/Handlung'),
(error: unknown) => error instanceof FsError && error.code === 'not_found' && /Handlungssituation/.test(error.message),
);
});
it('refuses to treat a file as a folder', async () => {
const { client } = fakeClient(pages);
await assert.rejects(
new FileManager(client).resolve('/courses/FIA24B - LF2 (Rh)/Mahnwesen.pdf/more'),
(error: unknown) => error instanceof FsError && error.code === 'not_a_directory',
);
});
it('reports an ambiguous name with the ids to use instead', async () => {
const twins = {
'/files/my/': page([folder(id(20), 'Kopie'), folder(id(21), 'Kopie')], []),
};
const { client } = fakeClient(twins);
await assert.rejects(
new FileManager(client).resolve('/my/Kopie'),
(error: unknown) =>
error instanceof FsError && error.code === 'ambiguous' && error.message.includes(id(20)) && error.message.includes(id(21)),
);
});
it('rejects an unknown area and names the real ones', async () => {
const { client } = fakeClient(pages);
await assert.rejects(
new FileManager(client).resolve('/Downloads/x'),
(error: unknown) => error instanceof FsError && /\/courses/.test(error.message),
);
});
it('caches a listing, so ls followed by read costs no second request', async () => {
const { client, requested } = fakeClient(pages);
const fs = new FileManager(client);
await fs.resolve('/courses/FIA24B - LF2 (Rh)/Mahnwesen.pdf');
await fs.resolve('/courses/FIA24B - LF2 (Rh)/Mahnwesen.pdf');
assert.equal(requested.filter((path) => path === '/files/courses/').length, 1);
});
});
describe('FileManager.walk', () => {
it('stays within its folder budget and says so', async () => {
const course = id(30);
const pages = {
'/files/courses/': page([folder(course, 'Kurs')], []),
[`/files/courses/${course}`]: page([folder(id(31), 'A'), folder(id(32), 'B')], []),
[`/files/courses/${course}/${id(31)}`]: page([], [file(id(33), 'a.pdf', 1)]),
[`/files/courses/${course}/${id(32)}`]: page([], [file(id(34), 'b.pdf', 1)]),
};
const { client } = fakeClient(pages);
const result = await new FileManager(client).walk(
{ path: '/courses', ref: { area: 'courses' } },
{ maxDepth: 5, maxDirectories: 2 },
);
assert.equal(result.visited, 2);
assert.equal(result.truncated, true);
});
it('records a folder it could not read instead of dropping it', async () => {
const course = id(40);
const pages = { '/files/courses/': page([folder(course, 'Kurs')], []) };
const { client } = fakeClient(pages);
const result = await new FileManager(client).walk(
{ path: '/courses', ref: { area: 'courses' } },
{ maxDepth: 3, maxDirectories: 10 },
);
assert.equal(result.failures.length, 1);
assert.equal(result.failures[0]?.path, '/courses/Kurs');
});
});
describe('the file-manager allowlist', () => {
// Some legacy GET routes write. The client must refuse them before any request.
const client = {
config: { baseUrl: 'https://example.org', jwt: 'x', requestTimeoutMs: 1000, maxDownloadBytes: 10 },
};
it('refuses the routes that write despite being GETs', async () => {
const { SchulcloudClient } = await import('../src/core/client.ts');
const real = new SchulcloudClient(client.config as never);
for (const path of ['/files/share/?file=' + id(1), `/files/file?file=${id(1)}&share=abc`, `/files/fileModel/${id(1)}/proxy`, '/files/search/?q=x', '/files/permittedDirectories/']) {
await assert.rejects(real.getFileManagerPage(path), /refusing file-manager path/, path);
}
});
});
describe('checkSignedUrl', () => {
it('accepts an https storage url from an https instance', () => {
assert.equal(checkSignedUrl('https://s3.example.com/bucket/key?X-Amz-Signature=1', 'https://schulcloud.example').host, 's3.example.com');
});
it('refuses plaintext from an https instance, and credentials in the url', () => {
assert.throws(() => checkSignedUrl('http://10.0.0.1/x', 'https://schulcloud.example'), /refusing a http:/);
assert.throws(() => checkSignedUrl('https://user:pw@s3.example.com/x', 'https://schulcloud.example'), /credentials/);
assert.throws(() => checkSignedUrl('file:///etc/passwd', 'http://localhost:4400'), /refusing a file:/);
});
it('allows http when the instance itself is http, as the local one is', () => {
assert.equal(checkSignedUrl('http://localhost:9900/x', 'http://localhost:4400').protocol, 'http:');
});
});