Read submitted text and teacher feedback from the homework page

You were right that this data never reaches the browser as an API call. The legacy front end calls the Feathers API server-side for submission.comment, submission.grade and submission.gradeComment and renders them into GET /homework/{taskId}. That Feathers API is not exposed publicly — /api/v1/* 404s — so the rendered page is the only way to reach these fields from outside. core/homework-page.ts parses it, hooked on the data-testid attributes the project's own e2e tests use rather than incidental markup. The page authenticates by jwt *cookie*; an Authorization header is ignored and redirects to the identity provider. Every field is optional and parse failures return undefined, so a markup change degrades to "not found" and cannot break get_task. The wording distinguishes the two: absent feedback is reported as not found, never as none given. Measured on one course: 4 of 7 graded submissions carry feedback no API call can return — "vollständig und nachvollziehbar", "Feedback siehe Zettel", and so on. This exposed a bug in a shared utility: htmlToText decoded only six entities, so any named entity passed through raw. German content makes that routine — "vollständig" would have reached the model verbatim from boards and task descriptions too, not just here. It now decodes named, decimal and hex references in one pass, so &auml; stays literal instead of decoding twice, and leaves unknown names alone rather than mangling them. Also fixes a documented-recovery bug found while restoring the session: `docker compose restart` does not re-read env_file, so it silently kept serving the dead token. `up -d` is correct and the docs said the wrong thing. 78 tests, 38/38 smoke; verified end to end through Claude Code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 13:25:35 +02:00
parent ac08e17b48
commit a8badc2fd1
10 changed files with 355 additions and 28 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -127,6 +127,11 @@ These cost real time to discover; `docs/API.md` has the full list with evidence.
  no fetch-by-id, and the payload has no submitted text, grade comment or
  graded-at — `/api/v1`, which had them, is not served here. Don't imply absent
  feedback means none was given.
+- **Submitted text and grade comments are scraped, not fetched.** No API
+  exposes them; the legacy page `GET /homework/{taskId}` renders them, and it
+  authenticates by `jwt` **cookie**, not bearer. `core/homework-page.ts` parses
+  it on `data-testid` hooks and every field is optional — a markup change must
+  degrade to "not found", never break `get_task`.
 - **files-storage listing ignores the `parentType` path segment** — filter on
  each record's own `parentType`, or submission files get reported as grading
  files.
--- a/docs/API.md
+++ b/docs/API.md
@@ -129,6 +129,17 @@ not served on this instance (404 across the board), so they are simply
 unavailable. Submitted *files* are reachable through files-storage with
 `parentType: 'submissions'`.

+**Submitted text and written feedback exist only in the rendered web page.**
+The legacy front end calls the Feathers API server-side for
+`submission.comment`, `submission.grade` and `submission.gradeComment` and
+renders them into `GET /homework/{taskId}`. That Feathers API is not exposed
+publicly (`/api/v1/*` 404s), so the page is the only way to reach these from
+outside. `core/homework-page.ts` parses it, hooked on the `data-testid`
+attributes the project's own e2e tests use. Note the page authenticates with the
+`jwt` **cookie** — an `Authorization` header is ignored and redirects to the
+identity provider. Measured: 4 of 7 graded submissions in one course carried
+feedback no API call can return.
+
 **files-storage ignores `parentType` when listing.** Asking for
 `.../gradings/{submissionId}` returns the files parented to that id whatever
 their type — the records come back saying `parentType: "submissions"`. The path
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@@ -70,7 +70,31 @@ boards, cards, files and tasks. **Impossible today at any speed** — the API ha
 changed-since filter anywhere. Arguably the most useful item on this list for a
 student, and nearly free once the sync job exists.

-## 4. Still open
+## 4. Remaining gaps, measured
+
+Scanned the instance's 98 GET routes against what the tools cover. Almost
+everything student-facing is now covered; what is left, with live checks:
+
+| Area | State on this account |
+|---|---|
+| `GET /groups/class` | **3 real classes** — the only uncovered endpoint with data |
+| `GET /rooms` | empty — the feature is unused here |
+| `GET /media-boards/me` | exists, no content |
+| `GET /course-info` | 403 — teacher/admin only |
+| `GET /alert` | empty |
+| `tools`, `oauth2`, `school`, `registrations`, `systems`, `user-login-migrations` | admin/auth plumbing, not student-facing |
+
+Genuinely unavailable, not merely uncovered:
+
+- **Collaborative text editor contents** — the endpoint returns an editor URL,
+  never the document text.
+- **Calendar** — a separate service, absent from the v3 document.
+- **Image-only PDFs** — 22 of 255 indexed files are scans with no text layer, so
+  their contents cannot be searched without OCR.
+- **Numeric grades** — the API's `grade` was null on every graded submission
+  here, so that path stays unverified against real data.
+
+## 5. Still open

 - **Video/audio transcription** — this account has 5 MP4s and a WebM that are
  currently just "here is a file you cannot read".
--- a/src/core/homework-page.ts
+++ b/src/core/homework-page.ts
@@ -0,0 +1,120 @@
+import type { Config } from '../config.ts';
+import { decodeEntities, htmlToText } from './text.ts';
+
+/**
+ * Reads submission detail off the legacy web UI's homework page.
+ *
+ * This is scraping, and it is deliberate. The v3 API's only submission route
+ * returns `{id, submitters, isSubmitted, isGraded, grade,
+ * submittingCourseGroupName}` — no submitted text, no written feedback. Those
+ * fields exist, but only the legacy Feathers API has them, and that API is not
+ * exposed publicly on this instance (`/api/v1/*` 404s). The legacy front end
+ * calls it server-side and renders the result, so the rendered page is the only
+ * place this data can be reached from outside.
+ *
+ * Measured on a real course: 4 of 7 graded submissions carried teacher feedback
+ * that no API call can retrieve.
+ *
+ * Two things make this less fragile than scraping usually is:
+ *  - the hooks are `data-testid` attributes, which exist for the project's own
+ *    end-to-end tests and so are not incidental markup;
+ *  - nothing depends on it. Every field is optional, parse failures return
+ *    undefined, and the API-derived facts are unaffected.
+ *
+ * Authentication is the `jwt` **cookie**, not a bearer header: the front end
+ * ignores `Authorization` and redirects to the identity provider.
+ */
+
+export interface SubmissionDetail {
+	/** The student's own typed answer, when they wrote one. */
+	submittedText?: string;
+	/** The teacher's written feedback. */
+	gradeComment?: string;
+	/** Percentage grade, when one was given. */
+	gradePercent?: number;
+	/** Files the teacher returned with the grading. */
+	gradingFiles: { id: string; name: string }[];
+	/** Files the student handed in — cross-check for the files-storage listing. */
+	submittedFiles: { id: string; name: string }[];
+}
+
+export async function fetchSubmissionDetail(
+	config: Config,
+	taskId: string,
+): Promise<SubmissionDetail | undefined> {
+	const url = `${config.baseUrl}/homework/${encodeURIComponent(taskId)}`;
+	let html: string;
+	try {
+		const response = await fetch(url, {
+			headers: { Cookie: `jwt=${config.jwt}`, Accept: 'text/html' },
+			signal: AbortSignal.timeout(config.requestTimeoutMs),
+			redirect: 'follow',
+		});
+		// A redirect away from the instance means the cookie was not accepted;
+		// there is nothing to parse and nothing worth reporting as an error.
+		if (!response.ok || !new URL(response.url).hostname.endsWith(new URL(config.baseUrl).hostname)) {
+			return undefined;
+		}
+		html = await response.text();
+	} catch {
+		return undefined;
+	}
+
+	return parseHomeworkPage(html);
+}
+
+/** Exported for testing: the parsing is pure and deserves fixtures, not a network. */
+export function parseHomeworkPage(html: string): SubmissionDetail | undefined {
+	const submission = section(html, 'submission');
+	const feedback = section(html, 'feedback');
+	if (submission === undefined && feedback === undefined) return undefined;
+
+	const detail: SubmissionDetail = { gradingFiles: [], submittedFiles: [] };
+
+	// The student's own text: a textarea while the submission is still editable,
+	// a plain div once it is not.
+	const typed =
+		/data-testid="submission-text"[^>]*>([\s\S]*?)<\/textarea>/.exec(html)?.[1] ??
+		(submission ? /<div class="comment"[^>]*>([\s\S]*?)<\/div>/.exec(submission)?.[1] : undefined);
+	const typedText = clean(typed);
+	if (typedText) detail.submittedText = typedText;
+
+	const comment = clean(/data-testid="feedback-comment"[^>]*>([\s\S]*?)<\/div>/.exec(html)?.[1]);
+	if (comment) detail.gradeComment = comment;
+
+	// Rendered by the template as "you have solved X%" in the feedback tab only.
+	if (feedback) {
+		const percent = /(\d{1,3})\s*%/.exec(stripTags(feedback));
+		if (percent) detail.gradePercent = Number(percent[1]);
+	}
+
+	detail.submittedFiles = fileCards(submission);
+	detail.gradingFiles = fileCards(feedback);
+	return detail;
+}
+
+/** The tab bodies are server-rendered `<section id="…">` blocks. */
+function section(html: string, id: string): string | undefined {
+	return new RegExp(`<section id="${id}"[^>]*>([\\s\\S]*?)</section>`).exec(html)?.[1];
+}
+
+/** File cards carry their id and name as data attributes. */
+function fileCards(markup: string | undefined): { id: string; name: string }[] {
+	if (!markup) return [];
+	const found: { id: string; name: string }[] = [];
+	const pattern = /data-file-name="([^"]*)"[^>]*data-file-size="[^"]*"[^>]*data-file-id="([0-9a-f]{24})"/g;
+	for (const match of markup.matchAll(pattern)) {
+		found.push({ name: decodeEntities(match[1] ?? ''), id: match[2] ?? '' });
+	}
+	return found;
+}
+
+function clean(markup: string | undefined): string | undefined {
+	if (!markup) return undefined;
+	const text = htmlToText(markup).trim();
+	return text.length > 0 ? text : undefined;
+}
+
+function stripTags(markup: string): string {
+	return markup.replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ');
+}
--- a/src/core/text.ts
+++ b/src/core/text.ts
@@ -7,10 +7,56 @@
 * Mongo ids — is dropped.
 */

+/**
+ * Named HTML entities worth decoding.
+ *
+ * Not the full HTML5 table — that is ~2000 entries for a handful this content
+ * actually uses. German umlauts are the ones that matter: the legacy web pages
+ * emit `&auml;` rather than UTF-8, so without these a grade comment comes back
+ * as "vollst&auml;ndig".
+ */
+const NAMED_ENTITIES: Record<string, string> = {
+	amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ',
+	auml: 'ä', ouml: 'ö', uuml: 'ü', Auml: 'Ä', Ouml: 'Ö', Uuml: 'Ü', szlig: 'ß',
+	aacute: 'á', agrave: 'à', acirc: 'â', eacute: 'é', egrave: 'è', ecirc: 'ê',
+	iacute: 'í', igrave: 'ì', oacute: 'ó', ograve: 'ò', ocirc: 'ô',
+	uacute: 'ú', ugrave: 'ù', ccedil: 'ç', ntilde: 'ñ',
+	euro: '€', pound: '£', deg: '°', middot: '·', bull: '•', hellip: '…',
+	ndash: '–', mdash: '—', laquo: '«', raquo: '»',
+	lsquo: '\u2018', rsquo: '\u2019', ldquo: '\u201c', rdquo: '\u201d', bdquo: '\u201e',
+	times: '×', divide: '÷', plusmn: '±', copy: '©', reg: '®', trade: '™', shy: '',
+};
+
+/**
+ * Decodes HTML entities: named, decimal and hexadecimal.
+ *
+ * `&amp;` is handled by the same pass as everything else rather than last,
+ * so `&amp;auml;` stays the literal text `&auml;` instead of being decoded
+ * twice into `ä`.
+ */
+export function decodeEntities(value: string): string {
+	return value.replace(/&(#x?[0-9a-fA-F]+|[a-zA-Z][a-zA-Z0-9]{1,31});/g, (whole, body: string) => {
+		if (body.startsWith('#')) {
+			const code = body[1] === 'x' || body[1] === 'X'
+				? Number.parseInt(body.slice(2), 16)
+				: Number.parseInt(body.slice(1), 10);
+			if (!Number.isFinite(code) || code <= 0 || code > 0x10ffff) return whole;
+			try {
+				return String.fromCodePoint(code);
+			} catch {
+				return whole;
+			}
+		}
+		// Unknown names are left alone: mangling them loses information.
+		return NAMED_ENTITIES[body] ?? whole;
+	});
+}
+
 /** Collapses Schulcloud's CKEditor HTML into plain text, keeping link targets. */
 export function htmlToText(html: string | undefined | null): string {
 	if (!html) return '';
-	return html
+	return decodeEntities(
+		html
 			.replace(/<br\s*\/?>/gi, '\n')
 			.replace(/<\/(p|div|h[1-6]|li|tr)>/gi, '\n')
 			.replace(/<li[^>]*>/gi, '- ')
@@ -20,13 +66,8 @@ export function htmlToText(html: string | undefined | null): string {
 				if (!text) return href;
 				return text === href ? href : `${text} (${href})`;
 			})
-		.replace(/<[^>]+>/g, '')
-		.replace(/&nbsp;/g, ' ')
-		.replace(/&lt;/g, '<')
-		.replace(/&gt;/g, '>')
-		.replace(/&quot;/g, '"')
-		.replace(/&#39;|&apos;/g, "'")
-		.replace(/&amp;/g, '&')
+			.replace(/<[^>]+>/g, ''),
+	)
 		.replace(/[ \t]+\n/g, '\n')
 		.replace(/\n{3,}/g, '\n\n')
 		.trim();
--- a/src/mcp/server.ts
+++ b/src/mcp/server.ts
@@ -25,10 +25,12 @@ How the content is organised, and the usual path through it:
  - **Tasks** ("Aufgaben") — homework. list_tasks across all courses, get_task for one.
 - **Files** hang off boards, lessons and tasks. Every listing shows file ids; download_file fetches one and
  extracts its text (PDF, Word, Excel, PowerPoint, OpenDocument) or returns an image inline.
- **Submissions** ("Abgaben") — what the user handed in. get_task shows the submission for that task;
-  list_submissions surveys them. Only files, the graded flag and a numeric grade are available: the API
-  exposes no submitted text and no written feedback, so say so rather than implying none was given.
-  On a teacher account these tools report other people's submissions too.
+- **Submissions** ("Abgaben") — what the user handed in. get_task shows that task's submission: the files,
+  the graded flag, the grade, what the user wrote, and the teacher's written feedback. list_submissions
+  surveys them across tasks ("what is still ungraded?"). The written parts are read from the web page
+  because no API exposes them, so they can be missing even when feedback exists — if none is shown for a
+  graded submission, say it was not found rather than that none was given. On a teacher account these
+  tools report other people's submissions too.

 When the user names a topic rather than a course, use search — the API has no search endpoint, so it walks the
 courses and matches client-side, which takes a few seconds but covers board text and file names.
--- a/src/mcp/tools/content.ts
+++ b/src/mcp/tools/content.ts
@@ -98,7 +98,8 @@ export function registerContentTools(server: McpServer, context: ServerContext):
 			title: 'Get task',
 			description:
 				'Full detail for one task: description, due date, attached files, and **what the account handed ' +
-				'in** — submission id, graded state, grade, and the submitted files ready for download_file. ' +
+				'in** — submission id, graded state, grade, the submitted files ready for download_file, what ' +
+				'the user wrote, and the teacher\'s written feedback. ' +
 				'The API has no single-task endpoint, so this locates the task through the task lists and course ' +
 				'pages — pass courseId when you know it, which makes the lookup immediate instead of a scan.',
 			inputSchema: {
--- a/src/mcp/tools/submissions.ts
+++ b/src/mcp/tools/submissions.ts
@@ -2,6 +2,7 @@ import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
 import { z } from 'zod';
 import type { ServerContext } from '../../context.ts';
 import { forEachLimited } from '../../core/crawl.ts';
+import { fetchSubmissionDetail } from '../../core/homework-page.ts';
 import { dueLabel, heading, joinSections } from '../../core/text.ts';
 import type { FileRecord, SubmissionStatus, TaskContent } from '../../core/types.ts';
 import { formatFileLine } from './content.ts';
@@ -92,6 +93,11 @@ export function registerSubmissionTools(server: McpServer, context: ServerContex
 	);
 }

+/** True when at least one of these submissions is marked graded. */
+function anyGraded(statuses: SubmissionStatus[]): boolean {
+	return statuses.some((status) => status.isGraded);
+}
+
 function formatRow({ task, status }: { task: TaskContent; status: SubmissionStatus }): string {
 	const state = status.isSubmitted ? 'submitted' : 'not submitted';
 	const graded = status.isGraded
@@ -144,9 +150,13 @@ export async function describeSubmission(
 	context: ServerContext,
 	taskId: string,
 ): Promise<string | undefined> {
-	const [me, statuses] = await Promise.all([
+	const [me, statuses, detail] = await Promise.all([
 		context.me(),
 		context.client.listSubmissionStatuses(taskId).catch(() => [] as SubmissionStatus[]),
+		// Submitted text and written feedback exist only in the rendered web page;
+		// see core/homework-page.ts. Optional by construction — a failure here
+		// costs detail, not the whole answer.
+		fetchSubmissionDetail(context.config, taskId).catch(() => undefined),
 	]);
 	if (statuses.length === 0) return undefined;

@@ -156,9 +166,12 @@ export async function describeSubmission(
 	const parts: string[] = [];
 	for (const status of relevant) {
 		const files = await loadSubmissionFiles(context, status.id);
+		const percent = detail?.gradePercent;
 		const graded = status.isGraded
 			? status.grade !== null && status.grade !== undefined
 				? `graded **${status.grade}**`
+				: percent !== undefined
+					? `graded **${percent}%**`
 					: 'graded (no numeric grade recorded)'
 			: 'not graded yet';

@@ -180,12 +193,26 @@ export async function describeSubmission(
 		);
 	}

+	// Only the rendered page carries these, and only for the caller's own work.
+	const written = [
+		detail?.submittedText
+			? joinSections([heading(4, 'What you wrote'), detail.submittedText])
+			: undefined,
+		detail?.gradeComment
+			? joinSections([heading(4, "Teacher's feedback"), detail.gradeComment])
+			: undefined,
+	].filter(Boolean) as string[];
+
 	return joinSections([
 		heading(3, 'Your submission'),
 		parts.join('\n\n'),
-		'Read any of these with download_file. The API exposes no submitted text or written ' +
-			'feedback — only files, the graded flag and a numeric grade — so a comment-only ' +
-			'response from the teacher will not appear here.',
+		...written,
+		'Read any attachment with download_file.' +
+			(written.length === 0 && anyGraded(relevant)
+				? ' No written feedback was found for this submission. It is read from the web page rather ' +
+					'than an API, so treat this as "not found", not as "none was given" — the teacher may ' +
+					'have responded on paper or in person.'
+				: ''),
 	]);
 }

--- a/test/homework-page.test.ts
+++ b/test/homework-page.test.ts
@@ -0,0 +1,73 @@
+import assert from 'node:assert/strict';
+import { describe, it } from 'node:test';
+import { parseHomeworkPage } from '../src/core/homework-page.ts';
+
+/**
+ * Fixtures mirror the legacy client's templates (feedback.hbs, submission.hbs)
+ * as actually served — including the parts that are absent for a student, which
+ * is what makes every field optional.
+ */
+const page = (parts: { submission?: string; feedback?: string }) => `
+<html><body>
+<nav class="nav tab-links"><a href="#activetabid=extended">Details</a></nav>
+<section id="extended" class="tab-content">keine Beschreibung vorhanden</section>
+<section id="submission" class="tab-content">${parts.submission ?? ''}</section>
+${parts.feedback === undefined ? '' : `<section id="feedback" class="tab-content">${parts.feedback}</section>`}
+</body></html>`;
+
+const fileCard = (id: string, name: string, size = 1234) =>
+	`<div class="card file " data-file-name="${name}" data-file-size="${size}" data-file-id="${id}"></div>`;
+
+describe('parseHomeworkPage', () => {
+	it("reads the teacher's written feedback, which no API exposes", () => {
+		const html = page({
+			feedback: `<div class="comment ckcontent" data-testid="feedback-comment"><p>vollst&auml;ndig und nachvollziehbar</p></div>`,
+		});
+		assert.equal(parseHomeworkPage(html)?.gradeComment, 'vollständig und nachvollziehbar');
+	});
+
+	it('reads a percentage grade from the feedback tab', () => {
+		const html = page({ feedback: '<p>Du hast 85% erreicht</p>' });
+		assert.equal(parseHomeworkPage(html)?.gradePercent, 85);
+	});
+
+	it('reads the student\'s typed answer from the editable textarea', () => {
+		const html = page({
+			submission: `<textarea data-testid="submission-text" name="comment">   <p>Meine L&ouml;sung</p>  </textarea>`,
+		});
+		assert.equal(parseHomeworkPage(html)?.submittedText, 'Meine Lösung');
+	});
+
+	it('reads the typed answer from the read-only form once submission closed', () => {
+		const html = page({ submission: `<div class="comment"><p>Abgegebener Text</p></div>` });
+		assert.equal(parseHomeworkPage(html)?.submittedText, 'Abgegebener Text');
+	});
+
+	it('separates submitted files from files the teacher returned', () => {
+		const html = page({
+			submission: fileCard('a'.repeat(24), 'meine-abgabe.pdf'),
+			feedback: fileCard('b'.repeat(24), 'korrektur.pdf'),
+		});
+		const detail = parseHomeworkPage(html);
+		assert.deepEqual(detail?.submittedFiles, [{ id: 'a'.repeat(24), name: 'meine-abgabe.pdf' }]);
+		assert.deepEqual(detail?.gradingFiles, [{ id: 'b'.repeat(24), name: 'korrektur.pdf' }]);
+	});
+
+	it('returns a result with nothing set when the page carries no submission', () => {
+		// The common case for an ungraded task: sections present but empty.
+		const detail = parseHomeworkPage(page({}));
+		assert.ok(detail);
+		assert.equal(detail.submittedText, undefined);
+		assert.equal(detail.gradeComment, undefined);
+		assert.deepEqual(detail.gradingFiles, []);
+	});
+
+	it('gives up rather than guessing when the markup is not a homework page', () => {
+		assert.equal(parseHomeworkPage('<html><body>Anmelden</body></html>'), undefined);
+	});
+
+	it('decodes entities in file names', () => {
+		const html = page({ submission: fileCard('c'.repeat(24), 'A&amp;B &quot;final&quot;.pdf') });
+		assert.equal(parseHomeworkPage(html)?.submittedFiles[0]?.name, 'A&B "final".pdf');
+	});
+});
--- a/test/render.test.ts
+++ b/test/render.test.ts
@@ -71,3 +71,26 @@ describe('normalizeObjectId', () => {
 		assert.equal(normalizeObjectId({}), undefined);
 	});
 });
+
+describe('decodeEntities', () => {
+	it('decodes German umlauts, which the legacy pages emit as named entities', () => {
+		assert.equal(htmlToText('<p>vollst&auml;ndig und nachvollziehbar</p>'), 'vollständig und nachvollziehbar');
+		assert.equal(htmlToText('<p>Gr&ouml;&szlig;e, &Uuml;bung</p>'), 'Größe, Übung');
+	});
+
+	it('decodes decimal and hex numeric references', () => {
+		assert.equal(htmlToText('<p>&#8364; &#x20AC; &#228;</p>'), '€ € ä');
+	});
+
+	it('does not double-decode: &amp;auml; stays literal text', () => {
+		assert.equal(htmlToText('<p>&amp;auml;</p>'), '&auml;');
+	});
+
+	it('leaves unknown entities alone rather than mangling them', () => {
+		assert.equal(htmlToText('<p>&notarealentity; &amp;</p>'), '&notarealentity; &');
+	});
+
+	it('ignores out-of-range numeric references', () => {
+		assert.equal(htmlToText('<p>&#1114112;</p>'), '&#1114112;');
+	});
+});