diff --git a/README.md b/README.md index 7c31d6f..a9b4e57 100644 --- a/README.md +++ b/README.md @@ -1,37 +1,43 @@ # Hankan · 한칸 A Korean reading tutor for manhwa. One codebase, two shells: a web app and an -Android app. **Works entirely offline — there is no server.** +Android app. **It works entirely offline; a server is optional.** -Ported from a single-file Claude artifact. The curriculum, the tutor prompt and -the five logic modules came finished and tested; this repo is the application -built around them. +Ported from a single-file Claude artifact, and migrated to its reworked 16 Sep +export. The curriculum, the tutor prompt, the logic modules and the checks +that measure them came finished and tested; this repo is the application built +around them. --- ## What is here ``` -data/ lib/ prompt/ validate.mjs verbatim from the export bundle — do not edit -shared/ band table + phonological ladder, shared by app and build +data/ lib/ prompt/ fixtures/ verbatim from the export bundle — do not edit +validate.mjs audit-gate.mjs the bundle's own checks, and run-checks.sh to run both +shared/ band table, phonological ladder, lemma ids, sync protocol types/ TypeScript declarations for lib/ and shared/ tools/dict/ the dictionary build pipeline app/ Vite + React + TypeScript, and the Capacitor shell - src/db/ one storage interface, two SQLite drivers - src/domain/ the gate, the lexicon, SRS, the tutor clients - src/sync/ push/pull against the Pi - src/ui/ six tabs, the review overlay, the 한글 keyboard + src/db/ one storage interface, two SQLite drivers, the write rules + src/domain/ the gate, the resolver, the turn, the learner model, SRS + src/sync/ protocol 2: hydrate, compare-and-swap, resolve + src/ui/shell/ routes over history, the nav, layers, popovers + src/ui/tutor/ the lesson: conversation, exercises, answer mode, word lookup + src/ui/tabs/ 오늘, 설정, 단어, 문장, 문법, 활용 연습, 한글, 읽기 연습 + src/ui/review/ the review screen public/dict/ generated dictionary — committed, shipped server/ sync + the tutor endpoint (optional) tools/icons/ the 한칸 mark, generated -test/ lib goldens, driver conformance, domain, sync, SSE +test/ lib goldens, drivers, domain, the gate audit, sync, UI history ``` -`data/`, `lib/`, `prompt/` and `validate.mjs` are byte-identical to the export -and are excluded from lint and formatting. `lib/conjugation.js` encodes the seven -Korean irregular classes and is the reason no runtime morphological analyser is -needed; `lib/hangul.js` implements 두벌식 composition. Types are supplied -alongside in `types/`, so neither file had to be touched. +`data/`, `lib/`, `prompt/`, `fixtures/`, `validate.mjs` and `audit-gate.mjs` +are byte-identical to the export and excluded from lint. Where the artifact's +inline code and `lib/` disagree, the port follows `lib/`, because `lib/` is +what `audit-gate.mjs` measures. Where `lib/` is looser than PORT.md, the +stricter rule sits at the call site and `lib/` stays untouched. Types for +`lib/` live alongside in `types/`. ## Running it @@ -42,7 +48,7 @@ npm run dev # http://localhost:5173 ``` ```bash -npm run check # validate.mjs, typecheck, tests, roadmap assertion +npm run check # validate.mjs, the gate audit, typecheck, tests, roadmap assertion npm run build # production build + service worker ``` @@ -58,122 +64,172 @@ npm run cap:sync cd app/android && ./gradlew assembleDebug ``` -`npx cap sync` copies `app/dist/` — including the dictionary — into the APK's -assets, so the phone build is as offline as the web one. +`npx cap sync` copies `app/dist/`, dictionary included, into the APK's assets, +so the phone build is as offline as the web one. ## How it works -### Storage — one interface, two drivers +### Storage: one interface, two drivers `app/src/db/` exposes a single `Db` interface. `sqlite.web.ts` runs `@sqlite.org/sqlite-wasm` in a dedicated Worker over the **OPFS SAHPool VFS** (the plain OPFS VFS needs COOP/COEP headers, which neither a static host nor the Capacitor webview reliably provides). `sqlite.native.ts` uses the Capacitor -SQLite plugin. Both apply the same migration array from `migrations.ts`. +SQLite plugin. Both apply the same migrations from `migrations.ts`. **Seeded state never carries a write timestamp.** The artifact had a sync bug where a fresh device stamped its own empty defaults as newer than the server's -real history and clobbered it. Three layers stop that from being expressible: -every syncable table declares `updated_at INTEGER NOT NULL DEFAULT 0`, so -forgetting the column is the safe failure; `db/writes.ts` splits every mutation -into `seedX()` (never stamps) and `editX()` (always stamps) and is the only file -allowed to read the clock; an ESLint rule enforces that, and the conformance -suite asserts a freshly seeded database has no non-zero `updated_at` anywhere. +real history and clobbered it. `db/writes.ts` splits every mutation into +`seedX()` (never stamps, never dirty) and `editX()` (stamps, marks the row +dirty, bumps its revision), and it is the only file allowed to read the clock. +An ESLint rule enforces that, and the conformance suite asserts a freshly seeded +database has nothing to push. ### The dictionary `npm run dict:build` merges a dictionary source, a frequency list, the curated -data in `data/`, and a hand-written grammar lexicon, then runs +data in `data/` and a hand-written grammar lexicon, then runs `lib/conjugation.js → surfaceForms()` over every verb and adjective to fill the -`surface` table. Lookup of a conjugated form is therefore an index hit, not an -analysis — **no runtime morphological analyser ships**. +`surface` table. Looking up a conjugated form is an index hit, not an analysis. +**No runtime morphological analyser ships.** + +**A card names its word, not a build position.** A lemma's id is a hash of its +headword and part of speech (`shared/lemma-id.mjs`), so rebuilding the +dictionary can never quietly move a card onto a different word. The band files +carry no ids at all, and a changed build reloads its rows without touching +cards. Every word the roadmap introduces is exactly one card, tagged with its +unit, including the sixteen forms (봐, 갔어, 먹었어 …) no dictionary lists. Sources are chosen automatically: KRDICT if a download has been vendored (it has curated learner glosses and a graded 초급/중급/고급 level), otherwise the kaikki.org Korean extract. KRDICT's download page is a JavaScript form behind -anti-bot protection, so it cannot be fetched by a script — see the comment at -the top of `tools/dict/fetch.mjs`. +anti-bot protection, so a script cannot fetch it. The comment at the top of +`tools/dict/fetch.mjs` explains. Frequency needs care: a subtitle frequency list holds *surface* forms while a dictionary holds lemmas, and a naive join on the headword gives every verb a -frequency of roughly zero. `tools/dict/freq-forms.mjs` inverts the join — -expanding each lemma into the forms it plausibly takes and summing — which +frequency of roughly zero. `tools/dict/freq-forms.mjs` inverts the join, +expanding each lemma into the forms it plausibly takes and summing. That recovers 하다 from 118 to 89,041. Output is one gzipped row dump per band, committed under `app/public/dict/`. -They are static assets in the bundle and in the APK's assets, so no server is -involved. Attribution is in [NOTICE.md](NOTICE.md); share-alike attaches to the +Attribution is in [NOTICE.md](NOTICE.md); share-alike attaches to the dictionary data, not to this code. -### The gate +### The gate: what the tutor may use -`lib/gate.js` computes what the tutor may teach from the curriculum plus +`lib/gate.js` computes what the tutor may teach from the curriculum and progress, and `renderGate()` renders it into `{{GATE}}` in -`prompt/tutor-system.md`, which ships unchanged. `buildGate()` takes a -`vocabQuery` hook; `app/src/domain/gate.ts` fills it with a frequency-band query -so vocabulary grows as units are finished. Three refinements sit inside that -hook, all of them narrowing: +`prompt/tutor-system.md`, which ships unchanged. -1. words a not-yet-finished unit is the first to introduce are excluded, so a - band ceiling cannot smuggle 3.4's material into 2.1; -2. during Phase 1 the results are filtered by the same phonological ladder - `validate.mjs` checks, so a band cannot hand the learner a 겹받침 at 1.4; -3. the list is capped at 800 by frequency, because `renderGate()` inlines it - into the prompt. The cap is strictly more restrictive than the band. +**One resolver.** "Which dictionary word is this form?" is asked when the +learner taps a word and when the gate checks the tutor. `lib/lexicon.js` +answers both, with the SQLite dictionary behind it only where it has no route +(`domain/resolver.ts`). The order was measured on the bundle's 54 real tutor +messages: the dictionary alone flags 4 and 2 (too weak; 마셔 becomes 마시다 +again), `lib/lexicon.js` first flags **7 and 2**, the shipped gate's numbers. +`test/domain/gate-audit.test.ts` reproduces the audit through the app's own +database, loader and resolver. -Confidence is clamped to `MAX_DELTA_PER_TURN` per turn. The artifact wrote the -model's number straight into the advancement gate, so one hallucinated -`::progress 95` could skip a unit. +**What is allowed:** the words of finished units, the current unit's words and +revisits, every word he has met (any card out of "new"), and, from Phase 2, the +frequency band. The words the tutor is *told* it may use and the words it is +*checked* against are one set. -### The tutor +**The client enforces; the prompt only explains.** Every reply is scanned before +he sees it (`scanTask`, `proseIsKorean`). A refused draft is never stored, +shown or applied: the tutor is asked again and told exactly why, twice at most. +After that the reply is shown with its words flagged, and the next turn names +them. -`app/src/domain/stub-tutor.ts` stands in for the model and implements the exact -contract the real endpoint will (`onText` receives cumulative text; an aborted -turn keeps what it streamed). It rotates all four exercise types, builds its -`::words` from the current unit's real vocabulary, and climbs `::progress` -gradually, so every render path — including the 85% advancement banner — is -reachable with no server. Swapping in the Pi's SSE endpoint later touches only -that file. +### The turn: what marking may change + +The turn lives in `domain/turn.ts`, where it can be tested without React. + +- **Progress is earned.** `::progress` is ignored until the unit has an answer. + A rise is capped at +25 per message and a fall is honoured in full. The next + unit is offered only at 85% with three answers behind it, and in a 다지기 + review only once nothing on its checklist is open. `advanceUnit()` enforces + this too, not only the banner. +- **Recall evidence.** `::result` feeds a per-word record kept apart from the + schedule. A word is *learned* after three correct answers in separate rounds, + at least five rounds apart, none after a lookup. The schedule takes at most + one good grade a day from marking; in the artifact, five good rounds in one + afternoon made a word "secure" by interval alone. +- **Phase reviews.** The client holds the 다지기 checklist, each unit's rule + and every word the phase introduced (132 items for Phase 1), worked in + batches of ten. `::confirmed` ticks a rule on the tutor's word but a word + only on evidence. +- **The letter-level check.** The model cannot see jamo, so a recall answer + carries the app's own comparison (`letterCheck`) for the tutor to quote. +- **The prompt keeps its cache.** The per-round sections (the practice set, + the checklist, retry notes) follow the shipped prompt as a second, uncached + system block, so the unchanged prefix still hits Anthropic's prompt cache and + a local server's KV cache. + +### The screen + +Mobile-first, after the reworked artifact. Five destinations: 선생님, 복습 +(review), 오늘, 단어, 학습. They sit in a bottom bar under 600px, a 76px rail +to 840px and a labelled rail beyond. The page never scrolls; each route owns +one scroller. + +- **Routes are history** (`ui/shell/history.ts`). The back gesture leaves a + sub-page, and the word sheet, a popover and the review screen are layers + whose entries Back closes first. The rules are tested against a history + whose traversals land late, as a browser's do. +- **The lesson** is one column: roadmap strip, conversation, quick replies, + composer. The word list is docked from 840px and a sheet with three detents + below it, taking its height from the shell so nothing being answered is + covered. +- **Answer mode.** Below 840px, focusing an exercise's answer strips the + screen back to the question, the field and an answer bar. The 한글 keyboard + follows whichever field was touched last and comes up by itself for a + recall exercise. +- **Any word can be looked up**, in three tiers: an underline on every Korean + word showing what it is to him, a popover beside it (and English → 한국어 on + an exercise's English), and the full word list. A chip glosses on a long + press or with 힌트. Every lookup counts as a lookup in the answer. + +### The stand-in tutor + +`app/src/domain/stub-tutor.ts` implements the same contract as the real +endpoint and exercises every path offline: all five exercise types, `::words` +from the current unit, `::result`, `::confirmed`, and progress reported only +after answers. The app is complete with no server. ## Checks | Command | What it guards | |---|---| -| `node validate.mjs` | the curriculum. **PASS — 0 blocking, 0 advisory** | -| `npm test` | lib golden tests, driver conformance, the seeded-timestamp rule, the gate | -| `npm run dict:assert` | **371/371 roadmap words resolve** (was 167 unglossable) | -| `npm run typecheck` · `npm run lint` | types, and the clock guard on `src/db/` | +| `sh run-checks.sh` | `validate.mjs` (the curriculum, **PASS — 0 blocking, 0 advisory**) and `audit-gate.mjs` (the word gate against 54 real tutor messages, **7 of 41 and 2 of 13 flagged**) | +| `npm test` | lib goldens, driver conformance, the seed rule, the gate audit through the port, the turn, sync against a real Postgres, UI history | +| `npm run dict:assert` | **371/371 roadmap words resolve**, each as exactly one card | +| `npm run typecheck` · `npm run lint` | types, hooks, and the clock guard on `src/db/` | -CI runs them in that order, `validate.mjs` first. +CI runs `run-checks.sh` first, before anything is installed. The sync tests need +Postgres and a running server; `.github/workflows/ci.yml` starts both, and +[server/README.md](server/README.md#running-it-locally) shows how locally. -## Sync and the tutor — optional, and genuinely optional +## Sync and the tutor: optional The app is complete without a server: its own SQLite, the shipped dictionary, -and a local stand-in tutor. Pointing it at a Pi adds two things — the real -선생님, and syncing between devices. Everything degrades to the offline -behaviour when the server is unreachable, and a sync failure is recorded in -settings rather than surfaced as an interruption. +and the stand-in tutor. Pointing it at a Pi adds two things, the real 선생님 +and syncing between devices. When the server is unreachable, everything falls +back to offline behaviour, and a failed send puts the message back in the box. -Setup, the Caddy config, and how sync resolves conflicts: [server/README.md](server/README.md). +Sync is built around three rules, each bought with lost work in the artifact: -Three details worth knowing here: +1. **Hydrate before pushing.** A device pulls everything first. A week-old + laptop cannot overwrite a week of work on the phone. +2. **A counter, not a clock.** A row is written only if the device last saw + the server's current version of it. Clock skew decides nothing. +3. **No silent shrinking.** When two copies disagree, the one that holds more + wins: more reviews, more answers, the further unit, the union of notes. A + deliberate deletion (a reset, a cleared lesson, a forgotten card) is + declared and obeyed. -- **Seeded rows carry `updated_at = 0`**, so a fresh device can neither push - its empty defaults nor win a conflict with them. The artifact's clobbering - bug is unrepresentable rather than merely avoided, and - `test/sync/roundtrip.test.ts` asserts it against a real Postgres. -- **An allowlist decides what leaves the device.** `meta` mixes preferences - with per-install bookkeeping; `dict.loadedBands` crossing the wire would - tell a phone it holds rows it never downloaded. -- **The system prompt is cached.** It is ~12k characters of gate, identical - for as long as the learner stays in one unit, so every turn after the first - reads it at a fraction of the input price. -- **The model is a config line, not a code change.** `HANKAN_TUTOR_BACKEND` - selects the Claude API, any OpenAI-compatible endpoint (LM Studio, Ollama, - llama.cpp, vLLM, OpenRouter), or a keyless echo backend for proving a - deployment. Keeping the system prompt first and unchanged is what makes - Anthropic's prefix cache work — and, it turns out, a local server's KV - cache too: 1,563ms to first token cold, 324ms warm. +Setup, the Caddy config and the protocol: [server/README.md](server/README.md). ## Not in this pass @@ -181,8 +237,9 @@ Android beyond the existing Capacitor scaffold, and Play Store packaging. ## Known limitation -`lib/blocks.js` `parse()` does not close a `::gloss` block on its `=` line, so a -multi-sentence gloss collapses into one run-on line. Since `lib/` ships -unchanged, `app/src/domain/gloss.ts` splits the block and parses each sentence -separately. The behaviour is pinned in `test/lib/blocks.test.ts`; if the library -is ever revised, the one-line fix is `cur = null` and both workarounds go. +`lib/blocks.js` `parse()` builds a message's prose by deleting each block with a +pattern that also eats the `::` opening the block after it. It also runs a block +on to the next `::`, so when a gloss block is missing its closing `::` (local +models forget it routinely) the paragraph after it is read as gloss rows. Since +`lib/` ships unchanged, `app/src/domain/gloss.ts` reads the prose and the gloss +blocks from the raw text; the header of that file has the details. diff --git a/server/README.md b/server/README.md index 64579a3..92cb19e 100644 --- a/server/README.md +++ b/server/README.md @@ -8,13 +8,16 @@ entirely offline against its own SQLite and a local stand-in tutor. Adding a server turns on two things: the real 선생님, and syncing between devices. ``` -POST /api/tutor {system, history, message} → SSE token stream -GET /api/sync ?cursor=N → rows newer than the cursor -POST /api/sync {rows} → upsert, last-write-wins -GET /health → no auth, for the healthcheck +POST /api/tutor {system, systemTail, history, message} → SSE token stream +GET /api/sync ?cursor=N → {epoch, rows, cursor, more}: a page of rows after N +POST /api/sync {rows} → {epoch, applied, conflicts}: compare-and-swap writes +GET /health → no auth, for the healthcheck ``` -Everything under `/api` requires `Authorization: Bearer $HANKAN_TOKEN`. +Everything under `/api` requires `Authorization: Bearer $HANKAN_TOKEN`, and +`/api/sync` also requires `x-hankan-protocol: 2`. An older app gets +**426 Upgrade Required** rather than a write the server would misread. See +[How sync works](#how-sync-works). ### CORS — required for the phone @@ -42,8 +45,11 @@ CREATE DATABASE hankan OWNER hankan; SQL ``` -The schema applies itself on boot — `server/sql/001-schema.sql` is idempotent, -so there is no migration step to run by hand. +The schema applies itself on boot. `server/sql/001-schema.sql` and +`002-protocol-2.sql` are idempotent, so there is no migration step to run by +hand. The first boot on protocol 2 drops any protocol-1 rows once; every +device then hydrates and offers its own database back, which was always the +source of truth. ### 2. Configure @@ -98,30 +104,62 @@ heartbeat every 15s, which defeat most intermediary caching and idle timeouts. ### 4. Connect the app -In the app: 오늘 → 서버 → the URL and the token. It syncs on connect, when the -tab regains focus, and every five minutes. +In the app: 오늘 → ⚙ 설정 → Server and sync → the URL and the token. It syncs +on connect, when the app regains focus, and every five minutes. ## How sync works -Row-level, last-write-wins on `updated_at`, cursor-based on a server-assigned -`change_seq`. One user, so the loser of a conflict is at worst one SRS grade. +Protocol 2 (`shared/sync-protocol.mjs`, `app/src/sync/`, `server/src/db.ts`). +It is row-level, and every row carries three columns on the device: `base_seq`, +the server version it last agreed with; `dirty`, set by every edit; and `rev`, +which tells a push that lands after a further edit not to clear it. -Rows are stored generically — primary key as text, body as JSONB — because the -server never reads inside a row. It stores and orders them; the client -interprets them. That keeps the two schemas from having to move in lockstep. +The artifact lost a week of work to its sync, and each of the three rules +below closes one way that happened. -Three things are load-bearing: +**1. Hydrate before pushing.** A device pulls every page before it may push +anything. A laptop last opened a week ago comes back, adopts the week of work +the phone did, and only then offers its own edits. Changing the server, or the +server's *epoch* (a random id replaced whenever its copy is thrown away), +forces a full hydration again. -- **`change_seq` advances on every update, via a trigger.** A row edited after - a client last pulled would otherwise sit below that client's cursor and - never be delivered. Putting it in a trigger means no write path can forget. -- **The pull cursor advances only as rows are applied**, never from the push - response. The server's newest `change_seq` includes rows this device has not - seen; adopting it would skip them permanently, and nothing would ever ask - for that range again. -- **Deletes travel as tombstones.** A deleted row leaves nothing to compare - timestamps against, so without one the other device pushes its still-live - copy back and the row silently returns. +**2. A counter, not a clock.** The server keeps a `change_seq` per row and +writes a pushed row only if the push's `base` equals it: compare-and-swap. +Anything else comes back as a conflict, with the server's current copy. +`updated_at` is kept for display and decides nothing, so a device with its +clock an hour out cannot win by it. Pushes are serialised and pulls take a +shared advisory lock, so a pull can never skip a `change_seq` that a +concurrent push is about to commit. + +**3. No silent shrinking.** A conflict goes to `app/src/sync/resolve.ts`, +where the copy that holds more wins: a card with more reviews behind it, +evidence with more observed, progress merged (done if either finished it, +the higher answer count), the further unit on the roadmap, grammar notes and +flags unioned, the better reading-drill round. A tie goes to the server, so +every device lands on one copy. + +Shrinking is always declared, never inferred: + +- **Deliberate deletes** travel as tombstones, and a delete is obeyed against a + concurrent edit: a card forgotten on one device stays forgotten. +- **Resets** (clearing the lesson, resetting the roadmap or everything) raise + a marker, `reset.`, a counter every device compares with the last one + it applied. A device that had not heard of the reset empties the same + things, its own offline work included, because a reset it did not know about + still wins. +- **Trimming the transcript is local.** A device keeps its last turns on + screen, but the trim is never a deletion, and a turn not yet pushed is never + trimmed. The artifact's trim tombstoned the other device's turns. + +Also fixed on the way: keys are JSON arrays, so a key containing a space +(몇 명) no longer stops sync permanently; chat ids are UUIDv7, so two devices +writing offline cannot collide; and the study log and peek tallies are kept per +device and summed, so two devices' reviews of the same day both count. + +Rows are stored generically on the server, the primary key as text and the +body as JSONB, because the server never reads inside a row. It stores, orders +and versions them; the client interprets them. That keeps the two schemas from +having to move in lockstep. ### What never syncs @@ -133,13 +171,15 @@ every row the desktop has, and the word rail would then fail to find words it believes are present. `server.token`, `sync.*` and `schema_version` are excluded for related reasons. -### The bug this schema is shaped around +### The bug this is shaped around The original artifact stamped a fresh device's empty defaults as newer than -the server's real history, and clobbered it. Here seeded and defaulted rows -carry `updated_at = 0`, so they can never be dirty and can never win a -conflict. It is not avoided, it is unrepresentable — -`test/sync/roundtrip.test.ts` asserts it against a real Postgres. +the server's real history, and clobbered it. Here a seeded row is never dirty, +so it has nothing to push, and a device hydrates before it pushes at all. The +week-old-laptop case, clock skew, offline chat on two devices, resets against +devices that missed them, a forget racing a review, keys with spaces, and a +server that lost its data are each a scenario in `test/sync/roundtrip.test.ts`, +run against a real Postgres. ## The tutor @@ -147,10 +187,14 @@ conflict. It is not avoided, it is unrepresentable — owns, and the new message; it holds nothing between requests, so a dropped connection costs one turn rather than the conversation. -The system prompt is passed as a cached block. It is ~12k characters of gate -and is byte-identical for as long as the learner stays in one unit — many -turns — so every turn after the first reads the prefix at a fraction of the -input price. This is the single biggest cost lever in the design. +The system prompt comes in two parts. `system` is the shipped prompt with the +gate filled in, ~12k characters that stay byte-identical for as long as the +learner stays in one unit. It is sent as a cached block, so every turn after +the first reads it at a fraction of the input price, the single biggest cost +lever in the design. `systemTail` holds what changes every round (the +practice set, the 다지기 checklist, the reason a draft was refused) and follows +as a second, uncached block, so it never breaks the cached prefix. The +OpenAI-compatible backend joins the two, prefix first. ### Choosing a model