From 57364fae3285cc9e086b258d51b86c194513f864 Mon Sep 17 00:00:00 2001 From: MechaCat02 Date: Sat, 16 May 2026 22:58:49 +0200 Subject: [PATCH] chore: release-prep docs, env vars, compose, and e2e port hygiene MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README rewritten end-to-end: stack, quick start, dev workflow, full /api/v1 endpoint table, error and pagination envelopes, auth quick-start (browser + bot bearer), configuration table, deployment notes, backup/restore pointer. Stale "next features" section dropped now that all eight feat branches are in. - .env.example now lists every env var the backend reads, with inline explanations: - COOKIE_SECURE / COOKIE_DOMAIN / SESSION_TTL_DAYS (auth) - CORS_ALLOWED_ORIGINS (same-origin by default) - MAX_REQUEST_BYTES / MAX_FILE_BYTES (upload caps) - Postgres + storage + log vars carried over. - docker-compose.yml forwards all of the above into the backend service with `${VAR:-default}` so an unset value falls back to the same default the code uses, and any `.env` override flows through without a compose edit. - docs/backup.md: step-by-step backup, restore, and smoke-test drill for both stateful volumes (postgres-data + storage-data), plus a list of what's deliberately *not* in the backup (e.g., .env). - playwright.config.ts: pins the e2e dev server to port 5174 with `--strictPort` so it neither reuses nor silently bumps off collision with another vite instance on 5173. Drops the flaky manual-start workflow the earlier branches needed. - docker-compose syntax (both prod and dev) validates cleanly against .env.example with no undefined-variable warnings. No version bump — this is documentation, config, and tooling. Co-Authored-By: Claude Opus 4.7 (1M context) --- .env.example | 44 +++++++- README.md | 192 +++++++++++++++++++++++++++++----- docker-compose.yml | 10 ++ docs/backup.md | 97 +++++++++++++++++ frontend/playwright.config.ts | 14 ++- 5 files changed, 324 insertions(+), 33 deletions(-) create mode 100644 docs/backup.md diff --git a/.env.example b/.env.example index 08d753d..dc6e02d 100644 --- a/.env.example +++ b/.env.example @@ -1,15 +1,51 @@ -# Copy to .env for `docker compose up`. -# Local dev (cargo run / npm run dev) reads backend/.env if present. +# Copy to .env for `docker compose up --build`. Local-dev runs (cargo run +# / npm run dev) read backend/.env if present, or pick up the variables +# from your shell. +# ----- Postgres ----- +# These are read by the Postgres container *and* by DATABASE_URL below; +# changing them after the first boot won't migrate existing data, so set +# them up front for any new deployment. POSTGRES_USER=mangalord POSTGRES_PASSWORD=mangalord POSTGRES_DB=mangalord +# ----- Backend ----- DATABASE_URL=postgres://mangalord:mangalord@postgres:5432/mangalord BIND_ADDRESS=0.0.0.0:8080 STORAGE_DIR=/var/lib/mangalord/storage RUST_LOG=info,mangalord=debug -# Public base URL the frontend uses to reach the API from the browser. -# In docker compose this is exposed on the host. +# ----- Auth / cookies ----- +# COOKIE_SECURE controls whether the `Secure` flag is set on the session +# cookie. Keep `true` in production (HTTPS); set to `false` if you're +# serving over plain HTTP locally (e.g., behind a dev reverse proxy). +COOKIE_SECURE=true +# COOKIE_DOMAIN scopes the session cookie. Leave empty to default to the +# requesting host. Set when serving the API and frontend on subdomains of +# a shared parent (e.g., `.example.com`) so the cookie is shared. +COOKIE_DOMAIN= +# Session lifetime in days. Expired sessions are no longer accepted and +# get reaped lazily. +SESSION_TTL_DAYS=30 + +# ----- CORS ----- +# Comma-separated origins allowed to call the API with credentials. +# Default is empty: same-origin only. Set when frontend and backend live +# on different hosts. Example: https://app.example.com,https://app.example.de +CORS_ALLOWED_ORIGINS= + +# ----- Upload limits ----- +# Per-request body cap. axum rejects oversized requests with 413 before +# our handlers run. Default 200 MiB. +MAX_REQUEST_BYTES=209715200 +# Per-image-part cap. Enforced after reading each part, so a single +# oversized image is rejected even when the total request fits. +# Default 20 MiB. +MAX_FILE_BYTES=20971520 + +# ----- Frontend ----- +# Public base URL the browser uses to reach the API. Behind a reverse +# proxy that serves /api/* from the frontend host, set this to /api so +# the calls stay same-origin. PUBLIC_API_BASE=http://localhost:8080/api diff --git a/README.md b/README.md index 0034e27..6faa1f1 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,13 @@ # Mangalord -A self-hosted manga and comics reader. Browse, search, read, bookmark, and upload manga and chapters. The HTTP API is consumed by both the SvelteKit web UI and external bots/scripts that perform the same actions programmatically. +A self-hosted manga and comics reader. Users browse, search, read, bookmark, and upload manga. The same HTTP API is consumed by the SvelteKit web UI and by external bots — there is no separate "bot API" — so anything you can do in the browser you can also do over `curl` with a bearer token. ## Stack -- **Backend**: Rust, axum, sqlx -- **Database**: Postgres 16 -- **Frontend**: SvelteKit 2 (Svelte 5 runes), TypeScript, Vite -- **File storage**: pluggable `Storage` trait — local FS today, S3 (and friends) as future impls -- **Deploy**: Docker Compose on a single server +- **Backend**: Rust, axum 0.7, sqlx 0.8, Postgres 16. Argon2id passwords, HttpOnly SameSite=Lax session cookies, bearer tokens for bots. +- **Frontend**: SvelteKit 2 with Svelte 5 runes, TypeScript. +- **Storage**: pluggable `Storage` trait. `LocalStorage` ships in-tree; S3 and friends are first-class extension slots. +- **Deploy**: a single host via Docker Compose. ## Quick start @@ -17,53 +16,194 @@ cp .env.example .env docker compose up --build ``` -- Frontend: http://localhost:3000 -- API: http://localhost:8080/api -- API health: http://localhost:8080/api/health +| Service | URL | +| ----------------- | ---------------------------------- | +| Frontend | | +| API base | | +| Health check | | + +The first boot runs the migrations automatically. From there: + +1. Open , click **Register**, create a user. +2. Click **Upload** to create a manga (optionally with a cover) and upload a chapter. +3. Open the manga, click the chapter, read with `j`/`k` or the arrow keys. ## Local development -Run only Postgres in Docker; run backend and frontend natively for fast iteration: +For fast iteration, run Postgres in Docker and the backend + frontend natively: ```bash docker compose -f docker-compose.dev.yml up -d -# backend +# Backend (separate shell) cd backend export DATABASE_URL=postgres://mangalord:mangalord@localhost:5432/mangalord cargo run -# frontend (separate shell) +# Frontend (separate shell) cd frontend npm install -npm run dev +npm run dev # serves on http://localhost:5173, proxies /api to :8080 ``` -The Vite dev server proxies `/api` to `http://localhost:8080`. +> **Port note:** `docker-compose.dev.yml` exposes Postgres on `5432`. If another Postgres is already bound to that port (or another Docker container is), stop it first or edit the host-port mapping in the dev compose file. Backend tests need a real Postgres that the test user can `CREATEDB` on; the dev container works because the `mangalord` role is the cluster superuser. ## Tests -This project is developed test-first. Tests live at three levels: +This project is developed test-first. Three levels: ```bash -# Backend: unit (in-module) + integration (tests/, per-test DB via #[sqlx::test]) +# Backend: unit (in-module) + integration (#[sqlx::test] against a real DB) cd backend && cargo test +cd backend && cargo clippy --all-targets -- -D warnings -# Frontend: unit / module tests (Vitest) +# Frontend: Vitest unit tests and svelte-check cd frontend && npm test +cd frontend && npm run check -# Frontend: end-to-end (Playwright; spins up dev server, mocks API by default) +# End-to-end: Playwright (mocks the backend at the page level by default) cd frontend && npm run test:e2e ``` +Backend integration tests provision a fresh, migrated database per test via `sqlx::test`, so they require `DATABASE_URL` to point at a Postgres where the configured user has `CREATEDB`. + +Playwright starts the SvelteKit dev server on a dedicated port (`5174`) so it doesn't reuse whatever you may have running on `5173`. To point Playwright at an already-running deployment instead, set `E2E_BASE_URL=http://your.host`. + ## API surface -| Method | Path | Purpose | -| ------ | ------------------- | -------------------------------------- | -| GET | `/api/health` | Liveness | -| GET | `/api/mangas` | List / search mangas | -| POST | `/api/mangas` | Create a manga | -| GET | `/api/mangas/{id}` | Get a manga | -| GET | `/api/files/{key}` | Stream a blob (cover, chapter page) | +Everything is namespaced under `/api/v1/`. `/api/*` outside the version prefix is reserved for future versioning. -Chapters, uploads, and bookmarks are next — the patterns to extend are documented in [`CLAUDE.md`](./CLAUDE.md). +### Read endpoints (public) + +| Method | Path | Description | +| ------ | ------------------------------------------------------- | ------------------------------------------ | +| GET | `/api/v1/health` | Liveness probe. | +| GET | `/api/v1/mangas?search=&sort=recent|title&limit=&offset=` | List/search mangas. Trigram fuzzy search. | +| GET | `/api/v1/mangas/{id}` | Single manga. | +| GET | `/api/v1/mangas/{id}/chapters` | Paginated chapter list, ordered by number. | +| GET | `/api/v1/mangas/{id}/chapters/{n}` | Single chapter. | +| GET | `/api/v1/mangas/{id}/chapters/{n}/pages` | Page metadata (storage keys + MIME). | +| GET | `/api/v1/files/{key}` | Streams a blob (cover or page). | + +### Write endpoints (require authentication) + +| Method | Path | Description | +| ------ | ------------------------------------------------------- | ------------------------------------------------------ | +| POST | `/api/v1/auth/register` | `{username,password}` → user + sets session cookie. | +| POST | `/api/v1/auth/login` | Same shape; rotates to a fresh session. | +| POST | `/api/v1/auth/logout` | Invalidates the session, clears the cookie. | +| GET | `/api/v1/auth/me` | Current user, or 401 if anonymous. | +| POST | `/api/v1/auth/tokens` | `{name}` → bot bearer token (returned once). | +| DELETE | `/api/v1/auth/tokens/{id}` | Revoke a bot token (owner-only). | +| POST | `/api/v1/mangas` | Multipart: `metadata` JSON + optional `cover` image. | +| POST | `/api/v1/mangas/{id}/chapters` | Multipart: `metadata` JSON + N `page` image parts. | +| POST | `/api/v1/bookmarks` | `{manga_id, chapter_id?, page?}`. | +| DELETE | `/api/v1/bookmarks/{id}` | Remove a bookmark (owner-only). | +| GET | `/api/v1/me/bookmarks?limit=&offset=` | The caller's bookmarks. | + +### Envelopes + +**Errors** — every non-2xx response: + +```json +{ "error": { "code": "snake_case", "message": "human readable", "details": { "field?": "..." } } } +``` + +`details` is only present on `validation_failed` (422). `code` is the stable contract; the message is for humans and may change. + +| HTTP | code | +| ---- | -------------------------- | +| 400 | `invalid_input`, `bad_file_key` | +| 401 | `unauthenticated` | +| 403 | `forbidden` | +| 404 | `not_found` | +| 409 | `conflict` | +| 413 | `payload_too_large` | +| 415 | `unsupported_media_type` | +| 422 | `validation_failed` | +| 500 | `internal_error` | + +**Lists** — every paginated endpoint: + +```json +{ + "items": [ /* ... */ ], + "page": { "limit": 50, "offset": 0, "total": 137 } +} +``` + +`total` is a number when the endpoint computes it (currently `/mangas`), otherwise `null`. + +## Auth quick-start + +### Browser + +The frontend handles this for you: register → cookie set → writes work. Cookies are `HttpOnly; SameSite=Lax; Secure` (configurable via `COOKIE_SECURE`). + +### Bots / scripts (bearer tokens) + +```bash +# 1. Log in once via cookies (or register). +curl -sb -c cookies.txt -X POST http://localhost:8080/api/v1/auth/login \ + -H 'content-type: application/json' \ + -d '{"username":"alice","password":"hunter2hunter2"}' + +# 2. Mint a long-lived bot token. The `bearer` value is shown ONCE. +curl -sb cookies.txt -X POST http://localhost:8080/api/v1/auth/tokens \ + -H 'content-type: application/json' \ + -d '{"name":"ci-bot"}' +# → { "id": "...", "name": "ci-bot", "bearer": "raw-token-here", ... } + +# 3. Use the bearer from anywhere. +curl -H 'Authorization: Bearer raw-token-here' \ + http://localhost:8080/api/v1/auth/me +``` + +Tokens are stored hashed (sha256) at rest; the raw value never leaves the response that created it. Revoke with `DELETE /api/v1/auth/tokens/{id}` while authenticated as the owner. + +## Configuration + +All variables can be set in `.env` (for `docker compose`) or your shell (for `cargo run`). See `.env.example` for the full list with explanations. + +| Variable | Default | Purpose | +| ----------------------- | ------------------------ | --------------------------------------------- | +| `DATABASE_URL` | required | Postgres connection string. | +| `BIND_ADDRESS` | `0.0.0.0:8080` | Backend listen address. | +| `STORAGE_DIR` | `./data/storage` | Local-storage root. | +| `RUST_LOG` | `info,mangalord=debug` | tracing-subscriber env filter. | +| `COOKIE_SECURE` | `true` | `Secure` flag on the session cookie. | +| `COOKIE_DOMAIN` | (none) | Scope the session cookie to a parent domain. | +| `SESSION_TTL_DAYS` | `30` | Session lifetime. | +| `CORS_ALLOWED_ORIGINS` | (empty → same-origin) | Comma-separated origin allowlist. | +| `MAX_REQUEST_BYTES` | `209715200` (200 MiB) | Hard cap on multipart request size. | +| `MAX_FILE_BYTES` | `20971520` (20 MiB) | Cap on a single image part. | +| `PUBLIC_API_BASE` | `http://localhost:8080/api` | Browser-facing API base. | + +## Deployment + +For real hosts: + +- **Front Mangalord with a TLS terminator** (Caddy, nginx, traefik). Point `:443` at the frontend on `:3000` and proxy `/api/*` to the backend on `:8080`. With same-origin routing you can leave `CORS_ALLOWED_ORIGINS` empty and the session cookie's `SameSite=Lax` does its job. +- **Set a strong Postgres password** in `.env` before the first `docker compose up`. The defaults are fine for local dev only. +- **Keep `COOKIE_SECURE=true`** behind HTTPS. Browsers drop `Secure` cookies on plain HTTP; the dev compose accepts `COOKIE_SECURE=false` for that case. +- **Watch `RUST_LOG`** if you're noisy with `debug` — drop the `,mangalord=debug` suffix in production to log at `info` for the app's spans. + +## Backup and restore + +Two stateful volumes: + +- `postgres-data` — all metadata (users, mangas, chapters, pages, bookmarks). +- `storage-data` — the actual cover and page bytes. + +A consistent backup needs **both**. See [docs/backup.md](docs/backup.md) for step-by-step procedures and a restore drill. + +## Project layout + +``` +backend/ # axum + sqlx, see backend/src/ for the hexagonal layout +frontend/ # SvelteKit 2 / Svelte 5 +docs/ # operator docs (backup procedure, etc.) +CLAUDE.md # contributor playbook +``` + +The contributor guidelines (TDD, Conventional Commits, lockstep SemVer, hexagonal seams) are in [CLAUDE.md](CLAUDE.md). diff --git a/docker-compose.yml b/docker-compose.yml index 6efca54..02453d0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -23,6 +23,16 @@ services: BIND_ADDRESS: 0.0.0.0:8080 STORAGE_DIR: /var/lib/mangalord/storage RUST_LOG: ${RUST_LOG:-info,mangalord=debug} + # Auth / cookies — see .env.example for context. + COOKIE_SECURE: ${COOKIE_SECURE:-true} + COOKIE_DOMAIN: ${COOKIE_DOMAIN:-} + SESSION_TTL_DAYS: ${SESSION_TTL_DAYS:-30} + # CORS — same-origin by default; populate when serving the API on + # a different host than the frontend. + CORS_ALLOWED_ORIGINS: ${CORS_ALLOWED_ORIGINS:-} + # Upload limits. + MAX_REQUEST_BYTES: ${MAX_REQUEST_BYTES:-209715200} + MAX_FILE_BYTES: ${MAX_FILE_BYTES:-20971520} volumes: - storage-data:/var/lib/mangalord/storage ports: diff --git a/docs/backup.md b/docs/backup.md new file mode 100644 index 0000000..360c9fa --- /dev/null +++ b/docs/backup.md @@ -0,0 +1,97 @@ +# Backup and restore + +Mangalord persists state in two places: + +1. **Postgres** (`postgres-data` volume) — users, mangas, chapters, pages, bookmarks, sessions, tokens. +2. **Storage** (`storage-data` volume) — cover and page image bytes. Keys in the DB reference paths inside this volume; a backup that has the DB without the storage (or vice versa) is incomplete. + +Always back both up together, and restore them together. + +## Backing up + +Run while the stack is up. The compose service name is `postgres`; if you renamed it, substitute below. + +```bash +# 1. Dump Postgres. `--format=custom` produces a single compressed file that +# pg_restore can stream. +docker compose exec -T postgres pg_dump \ + -U "${POSTGRES_USER:-mangalord}" \ + -d "${POSTGRES_DB:-mangalord}" \ + --format=custom \ + > mangalord-$(date -u +%Y%m%dT%H%M%SZ).dump + +# 2. Tar the storage volume. The simplest way is to mount the named volume +# into an ephemeral container and tar from there. +docker run --rm \ + -v mangalord_storage-data:/source:ro \ + -v "$PWD":/backup \ + alpine \ + tar czf "/backup/mangalord-storage-$(date -u +%Y%m%dT%H%M%SZ).tar.gz" -C /source . +``` + +Both files are timestamped so consecutive runs don't overwrite. Copy them off-host (rsync, rclone, your object store of choice) — keeping them on the same machine doesn't help if the disk fails. + +> The volume's prefix (`mangalord_`) comes from the compose project name, which defaults to the directory name. Confirm with `docker volume ls | grep storage-data`. + +## Restoring + +The procedure is destructive; do it on a fresh stack or after confirming you really want to overwrite the current state. + +```bash +# 0. Stop the stack so nothing writes during the restore. +docker compose down + +# 1. Wipe the existing volumes and recreate them. (Skip this if restoring +# onto a host that doesn't have prior state.) +docker volume rm mangalord_postgres-data mangalord_storage-data || true + +# 2. Bring Postgres up alone so we can restore into it before the backend +# starts and runs migrations against the wrong schema. +docker compose up -d postgres +# Wait for it to become healthy. +until docker compose exec -T postgres pg_isready -U "${POSTGRES_USER:-mangalord}" >/dev/null; do + sleep 1 +done + +# 3. Restore the dump. +docker compose exec -T postgres pg_restore \ + -U "${POSTGRES_USER:-mangalord}" \ + -d "${POSTGRES_DB:-mangalord}" \ + --clean --if-exists \ + < mangalord-YYYYMMDDTHHMMSSZ.dump + +# 4. Restore the storage tarball. +docker run --rm \ + -v mangalord_storage-data:/target \ + -v "$PWD":/backup \ + alpine \ + sh -c "cd /target && tar xzf /backup/mangalord-storage-YYYYMMDDTHHMMSSZ.tar.gz" + +# 5. Start the rest of the stack. +docker compose up -d backend frontend +``` + +The backend re-runs migrations on startup. They're idempotent (additive), so restoring a dump from an older schema and then starting a newer backend will forward-migrate the data; the reverse is not supported. + +## Restore drill + +It's worth running this end-to-end at least once before you need it for real. A minimal smoke check after restore: + +```bash +# Health endpoint should respond. +curl -sf http://localhost:8080/api/v1/health + +# An expected manga should be visible. +curl -s 'http://localhost:8080/api/v1/mangas?limit=1' | jq '.items[0].title, .page.total' + +# A known cover or page should stream back with the right Content-Type. +curl -sI http://localhost:8080/api/v1/files/ +``` + +If `.page.total` matches your expectation and a known file is reachable, the restore is complete. + +## What's *not* in the backup + +- The `.env` file — keep this in your password manager / secrets store; the backup deliberately doesn't include it so a leaked archive doesn't leak credentials. +- Application logs — stream them to your usual log sink (journald, Loki, whatever) rather than relying on container logs. +- Anything outside the two named volumes. diff --git a/frontend/playwright.config.ts b/frontend/playwright.config.ts index e31a061..bd489b5 100644 --- a/frontend/playwright.config.ts +++ b/frontend/playwright.config.ts @@ -1,17 +1,25 @@ import { defineConfig } from '@playwright/test'; +// Playwright runs the SvelteKit dev server on a dedicated port so it +// doesn't reuse whatever the dev workflow has on the vite default +// (`5173`). `--strictPort` makes vite fail fast on collision instead of +// silently bumping to a free port, which would leave Playwright pointed +// at the wrong host. Set `E2E_BASE_URL` to skip the embedded server and +// point at an already-running deployment. +const E2E_PORT = 5174; + export default defineConfig({ testDir: 'e2e', timeout: 30_000, use: { - baseURL: process.env.E2E_BASE_URL ?? 'http://localhost:5173', + baseURL: process.env.E2E_BASE_URL ?? `http://localhost:${E2E_PORT}`, trace: 'retain-on-failure' }, webServer: process.env.E2E_BASE_URL ? undefined : { - command: 'npm run dev', - port: 5173, + command: `npm run dev -- --port ${E2E_PORT} --strictPort`, + port: E2E_PORT, reuseExistingServer: !process.env.CI, timeout: 120_000 }