fabi/Hankan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sync): protocol 2 — hydration, a counter not a clock, no silent shrinking

Browse Source 
The reworked bundle's PORT.md makes row-level last-write-wins conditional
on three gates, each learned by losing real data. The port's sync broke
all three, and had four more ways to lose or stall work. Both ends change,
so this is one protocol version, refused by the other side if mismatched.

Gate 1, hydration. A device now pulls every page the server holds before
it may push anything; it used to push first. The 14 Sep laptop — a
week-old copy re-stamped at boot and pushed over a week of phone work —
is now a test, and the phone's week survives it. Boot writes nothing
syncable either: the lesson opens with the app's own words (the artifact's
seeded turn) and waits for Start, instead of stamping a reply and a
progress edit before a server can even be configured.

Gate 2, a counter. Every row remembers the change_seq it last agreed with
(base_seq). The server applies a write only if that still matches —
compare-and-swap under an advisory lock — and otherwise returns its copy
as a conflict. No clock is compared anywhere: a device an hour fast used
to win every conflict for an hour, and a slow one's newer edit was
silently dropped with HTTP 200. dirty and rev replace the timestamp
watermark, which lost edits whenever a clock moved backwards.

Gate 3, no silent shrinking. A conflict is settled by what each copy
holds (sync/resolve.ts): more reviews, more evidence, a finished unit, the
further roadmap position, the union of learned grammar. A deliberate
shrink is explicit: a reset or a cleared lesson raises a marker every
device obeys, including its own unsynced edits, so a reset is not undone
by a device that had not heard of it. Trimming the transcript is local
and tombstones nothing — it used to delete the other device's turns.

Also fixed on the way:
  · keys travel as JSON arrays — a space in 몇 명 used to stop every
    device's pull at that row, permanently;
  · pulls take a shared lock against pushes, so a change_seq committed
    out of order can no longer be skipped;
  · study_log and peek are per device and summed, so two devices' reviews
    of one day both count;
  · each user's data has an epoch; a server that lost it is detected,
    and the device re-hydrates and offers its data back;
  · a protocol-1 client is refused with 426 rather than half-understood.

Migration 9 adds the columns, re-keys the counters and tombstones; the
server drops protocol-1 rows once (none were deployed). Verified: 14
two-device scenarios against a real Postgres, and two browser profiles
syncing a lesson through the UI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-09-16 20:28:35 +02:00
parent bf9b5950da
commit 9ab5aba6b3
21 changed files with 1692 additions and 579 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
.github/workflows/ci.yml vendored
View File

@@ -32,9 +32,10 @@ jobs:
- name: Typecheck - name: Typecheck
run: npm run typecheck run: npm run typecheck
# The sync round-trip needs a real Postgres and a running server, so a # The sync round-trip needs a real Postgres and a running server: a fake
# fake would not exercise the change_seq trigger, the last-write-wins # would not exercise the compare-and-swap push, the advisory locks that
# upsert, or cursor paging — which is where sync actually goes wrong. # keep a pull from skipping a change_seq, or cursor paging — which is
# where sync actually goes wrong.
- name: Start Postgres - name: Start Postgres
run: | run: |
docker run -d --name hankan-pg \ docker run -d --name hankan-pg \

117
app/src/db/migrations.ts
View File

@@ -260,8 +260,125 @@ export const MIGRATIONS: Migration[] = [
`, `,
run: remodelRoadAndChat, run: remodelRoadAndChat,
}, },
{
id: 9,
name: "sync protocol 2 — base_seq and dirty on every row, counters per device",
sql: /* sql */ `
-- base_seq: the server change_seq this row last agreed with.
-- dirty: a local edit not yet on the server.
-- rev: bumped by every local edit, so a push can tell whether
-- the row changed again while it was in flight.
-- All default to 0: a seeded row is never dirty and never travels.
ALTER TABLE card ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE card ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE card ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE progress ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE progress ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE progress ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE chat ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE chat ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE chat ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE meta ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE meta ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE meta ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE custom_word ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE custom_word ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE custom_word ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE evidence ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE evidence ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE evidence ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE confusion ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE confusion ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE confusion ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE phase_ledger ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE phase_ledger ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE phase_ledger ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE tombstone ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE tombstone ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE tombstone ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
-- Counters per device, summed on read — two devices adding to the same
-- day each own their row, so neither can erase the other's reviews.
CREATE TABLE study_log_v9 (
day INTEGER NOT NULL,
device TEXT NOT NULL,
reviews INTEGER NOT NULL DEFAULT 0,
correct INTEGER NOT NULL DEFAULT 0,
drills INTEGER NOT NULL DEFAULT 0,
updated_at INTEGER NOT NULL DEFAULT 0,
base_seq INTEGER NOT NULL DEFAULT 0,
dirty INTEGER NOT NULL DEFAULT 0,
rev INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (day, device)
) WITHOUT ROWID;
INSERT INTO study_log_v9 (day, device, reviews, correct, drills, updated_at)
SELECT day, (SELECT v FROM meta WHERE k = 'sync.device'), reviews, correct, drills, updated_at
FROM study_log;
DROP TABLE study_log;
ALTER TABLE study_log_v9 RENAME TO study_log;
CREATE TABLE peek_v9 (
form TEXT NOT NULL,
device TEXT NOT NULL,
count INTEGER NOT NULL DEFAULT 0,
updated_at INTEGER NOT NULL DEFAULT 0,
base_seq INTEGER NOT NULL DEFAULT 0,
dirty INTEGER NOT NULL DEFAULT 0,
rev INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (form, device)
) WITHOUT ROWID;
INSERT INTO peek_v9 (form, device, count, updated_at)
SELECT form, (SELECT v FROM meta WHERE k = 'sync.device'), count, updated_at FROM peek;
DROP TABLE peek;
ALTER TABLE peek_v9 RENAME TO peek;
CREATE INDEX IF NOT EXISTS card_dirty ON card(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS progress_dirty ON progress(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS chat_dirty ON chat(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS meta_dirty ON meta(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS custom_word_dirty ON custom_word(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS evidence_dirty ON evidence(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS confusion_dirty ON confusion(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS phase_ledger_dirty ON phase_ledger(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS tombstone_dirty ON tombstone(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS study_log_dirty ON study_log(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS peek_dirty ON peek(dirty) WHERE dirty = 1;
-- Protocol 1's position means nothing to protocol 2. With no epoch
-- recorded, the next sync re-hydrates: it pulls everything, then
-- offers every stamped row back to the server.
DELETE FROM meta WHERE k IN ('sync.cursor', 'sync.pushedAt');
`,
run: jsonTombstoneKeys,
},
]; ];
/**
* Migration 9's data step: tombstone keys in protocol 2's form, a JSON array
* of the key's values. Protocol 1 stored the bare value, or values joined by
* a space — which a key containing a space could not survive.
*/
async function jsonTombstoneKeys(db: Db): Promise<void> {
const device = (await db.get<{ v: string }>("SELECT v FROM meta WHERE k = 'sync.device'"))?.v ?? "";
const graves = await db.all<{ tbl: string; pk: string }>("SELECT tbl, pk FROM tombstone");
for (const g of graves) {
if (g.pk.startsWith("[")) continue; // written in the new form already
const key =
g.tbl === "card"
? [Number(g.pk)]
: g.tbl === "study_log"
? [Number(g.pk), device]
: g.tbl === "peek"
? [g.pk, device]
: [g.pk];
await db.run("UPDATE OR REPLACE tombstone SET pk = ? WHERE tbl = ? AND pk = ?", [
JSON.stringify(key),
g.tbl,
g.pk,
]);
}
}
/** /**
* Migration 8's data step. * Migration 8's data step.
* *

303
app/src/db/writes.ts
View File

@@ -7,27 +7,36 @@
real history, and last-write-wins duly clobbered months of progress with real history, and last-write-wins duly clobbered months of progress with
an empty seed. an empty seed.
The rule that prevents it: SEEDED OR DEFAULTED STATE NEVER CARRIES A WRITE The rule that prevents it: SEEDED OR DEFAULTED STATE IS NEVER A WRITE.
TIMESTAMP. Only a genuine user edit stamps the clock. Only a genuine user edit stamps the clock and marks its row dirty.
Three layers enforce it: 1. the DDL — `updated_at`, `dirty` and `rev` all default to 0 on every
syncable table, so forgetting them is the safe failure: an INSERT
1. the DDL — `updated_at INTEGER NOT NULL DEFAULT 0` on every syncable that omits them is invisible to sync (see migrations.ts);
table, so forgetting the column is the safe failure, not the unsafe 2. this file — seedX() helpers never mention them; editX() helpers
one (see migrations.ts); always stamp updated_at, set dirty = 1 and bump rev;
2. this file — seedX() helpers never mention updated_at; editX() helpers
always set it from now();
3. lint + test — Date.now() is banned everywhere under src/db/ except 3. lint + test — Date.now() is banned everywhere under src/db/ except
here, and test/db/conformance.ts asserts a freshly seeded database here, and test/db/conformance.ts asserts a freshly seeded database
has no non-zero updated_at anywhere. has nothing stamped and nothing dirty.
Sync is out of scope for this pass. The point is that when it lands, the ════ WHAT SYNC NEEDS FROM A WRITE ════
schema already cannot express the bug. */
dirty = 1 queues the row for the server; rev tells the sync client
whether the row changed again while its push was in flight. base_seq is
the sync client's alone — no write here touches it, except that a delete
carries the deleted row's base_seq into its tombstone, which is what lets
the server compare-and-swap the delete like any other write. */
import type { Db, Params } from "./types.js"; import type { Db, Params } from "./types.js";
import type { Card, Grade } from "@lib/srs.js"; import type { Card, Grade } from "@lib/srs.js";
import { lemmaId } from "@shared/lemma-id.mjs"; import { lemmaId } from "@shared/lemma-id.mjs";
import { uuidv7 } from "./ids.js"; import {
RESET_SCOPES,
SYNC_TABLES,
encodePk,
type ResetScope as MarkerScope,
} from "@shared/sync-protocol.mjs";
import { randomId, uuidv7 } from "./ids.js";
/* ───────────────────────────────────────────────────────────────────── /* ─────────────────────────────────────────────────────────────────────
The clock. This is the ONLY place in src/db/ that reads it. The clock. This is the ONLY place in src/db/ that reads it.
@@ -36,25 +45,54 @@ import { uuidv7 } from "./ids.js";
/** Wall-clock milliseconds, for stamping a genuine user edit. */ /** Wall-clock milliseconds, for stamping a genuine user edit. */
export const now = (): number => Date.now(); export const now = (): number => Date.now();
/* ─────────────────────────────────────────────────────────────────────
Shared helpers
───────────────────────────────────────────────────────────────────── */
/** This install's id — the owner of its per-device counter rows. */
const devices = new WeakMap<Db, string>();
export async function deviceId(db: Db): Promise<string> {
const known = devices.get(db);
if (known) return known;
let id = (await db.get<{ v: string }>("SELECT v FROM meta WHERE k = 'sync.device'"))?.v;
if (!id) {
id = randomId();
// Describes this install, not the learner: never stamped, never synced.
await db.run("INSERT OR IGNORE INTO meta (k, v, updated_at) VALUES ('sync.device', ?, 0)", [id]);
}
devices.set(db, id);
return id;
}
/** /**
* Record that a row was deleted. * Delete rows AND record that they were deleted, in the caller's transaction.
* *
* A delete leaves nothing for last-write-wins to compare against, so without * A delete leaves nothing behind to compare, so without a tombstone another
* this the other device would push its still-live copy back and the row * device would push its live copy straight back. The tombstone carries the
* would silently return. Always written inside the same transaction as the * row's base_seq: the server applies the delete only if nothing has changed
* delete it describes. * the row since this device last agreed with it.
*/ */
async function tombstone(db: Db, tbl: string, pk: string | number): Promise<void> { async function remove(db: Db, tbl: string, where: string, params: Params = []): Promise<void> {
await db.run( const spec = SYNC_TABLES[tbl]!;
`INSERT INTO tombstone (tbl, pk, updated_at) VALUES (?, ?, ?) const rows = await db.all<Record<string, string | number>>(
ON CONFLICT(tbl, pk) DO UPDATE SET updated_at = excluded.updated_at`, `SELECT ${spec.pk.join(", ")}, base_seq FROM ${tbl} WHERE ${where}`,
[tbl, String(pk), now()], params,
); );
const t = now();
for (const r of rows) {
await db.run(
`INSERT INTO tombstone (tbl, pk, updated_at, base_seq, dirty, rev) VALUES (?, ?, ?, ?, 1, 1)
ON CONFLICT(tbl, pk) DO UPDATE SET updated_at = excluded.updated_at,
base_seq = excluded.base_seq, dirty = 1, rev = tombstone.rev + 1`,
[tbl, encodePk(tbl, r), t, Number(r.base_seq ?? 0)],
);
}
await db.run(`DELETE FROM ${tbl} WHERE ${where}`, params);
} }
/* ═════════════════════════════════════════════════════════════════════ /* ═════════════════════════════════════════════════════════════════════
SEED WRITES — defaults, first-run state, anything the user did not do. SEED WRITES — defaults, first-run state, anything the user did not do.
None of these may set updated_at; the column default of 0 is the point. None of these may set updated_at, dirty or rev; the defaults are the point.
═════════════════════════════════════════════════════════════════════ */ ═════════════════════════════════════════════════════════════════════ */
/** First-run roadmap position. Unit 1.1 is where everyone starts. */ /** First-run roadmap position. Unit 1.1 is where everyone starts. */
@@ -90,28 +128,26 @@ export async function seedChatTurn(db: Db, role: string, body: string, at: numbe
} }
/* ═════════════════════════════════════════════════════════════════════ /* ═════════════════════════════════════════════════════════════════════
EDIT WRITES — a real action by the learner. Every one stamps the clock. EDIT WRITES — a real action by the learner. Every one stamps the clock
and marks its row for the server.
═════════════════════════════════════════════════════════════════════ */ ═════════════════════════════════════════════════════════════════════ */
/** Answering a card in the review overlay. */ /** Answering a card in the review overlay. */
export async function editCard(db: Db, lemmaId: number, card: Card): Promise<void> { export async function editCard(db: Db, lemmaId: number, card: Card): Promise<void> {
await db.run( await db.run(
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at) `INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at, dirty, rev)
VALUES (?, ?, ?, ?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 1, 1)
ON CONFLICT(lemma_id) DO UPDATE SET ON CONFLICT(lemma_id) DO UPDATE SET
state = excluded.state, ease = excluded.ease, interval = excluded.interval, state = excluded.state, ease = excluded.ease, interval = excluded.interval,
due = excluded.due, reps = excluded.reps, lapses = excluded.lapses, due = excluded.due, reps = excluded.reps, lapses = excluded.lapses,
updated_at = excluded.updated_at`, updated_at = excluded.updated_at, dirty = 1, rev = card.rev + 1`,
[lemmaId, card.state, card.ease, card.interval, card.due, card.reps, card.lapses, now()], [lemmaId, card.state, card.ease, card.interval, card.due, card.reps, card.lapses, now()],
); );
} }
/** Forgetting a card back to new, from the vocabulary tab. */ /** Forgetting a card back to new, from the vocabulary tab. */
export async function editCardReset(db: Db, lemmaId: number): Promise<void> { export async function editCardReset(db: Db, lemmaId: number): Promise<void> {
await db.tx(async (tx) => { await db.tx((tx) => remove(tx, "card", "lemma_id = ?", [lemmaId]));
await tx.run("DELETE FROM card WHERE lemma_id = ?", [lemmaId]);
await tombstone(tx, "card", lemmaId);
});
} }
/** The tutor's ::progress read, or the learner's "not yet". */ /** The tutor's ::progress read, or the learner's "not yet". */
@@ -121,10 +157,9 @@ export async function editUnitConfidence(
confidence: number, confidence: number,
): Promise<void> { ): Promise<void> {
await db.run( await db.run(
`INSERT INTO progress (unit_id, confidence, updated_at) `INSERT INTO progress (unit_id, confidence, updated_at, dirty, rev) VALUES (?, ?, ?, 1, 1)
VALUES (?, ?, ?)
ON CONFLICT(unit_id) DO UPDATE SET confidence = excluded.confidence, ON CONFLICT(unit_id) DO UPDATE SET confidence = excluded.confidence,
updated_at = excluded.updated_at`, updated_at = excluded.updated_at, dirty = 1, rev = progress.rev + 1`,
[unitId, Math.max(0, Math.min(100, Math.round(confidence))), now()], [unitId, Math.max(0, Math.min(100, Math.round(confidence))), now()],
); );
} }
@@ -132,8 +167,9 @@ export async function editUnitConfidence(
/** A unit finished. Nothing un-finishes one short of a reset. */ /** A unit finished. Nothing un-finishes one short of a reset. */
export async function editUnitDone(db: Db, unitId: string): Promise<void> { export async function editUnitDone(db: Db, unitId: string): Promise<void> {
await db.run( await db.run(
`INSERT INTO progress (unit_id, done, updated_at) VALUES (?, 1, ?) `INSERT INTO progress (unit_id, done, updated_at, dirty, rev) VALUES (?, 1, ?, 1, 1)
ON CONFLICT(unit_id) DO UPDATE SET done = 1, updated_at = excluded.updated_at`, ON CONFLICT(unit_id) DO UPDATE SET done = 1,
updated_at = excluded.updated_at, dirty = 1, rev = progress.rev + 1`,
[unitId, now()], [unitId, now()],
); );
} }
@@ -146,8 +182,9 @@ export async function editCurrentUnit(db: Db, unitId: string): Promise<void> {
/** A preference the learner changed. */ /** A preference the learner changed. */
export async function editMeta(db: Db, k: string, v: string): Promise<void> { export async function editMeta(db: Db, k: string, v: string): Promise<void> {
await db.run( await db.run(
`INSERT INTO meta (k, v, updated_at) VALUES (?, ?, ?) `INSERT INTO meta (k, v, updated_at, dirty, rev) VALUES (?, ?, ?, 1, 1)
ON CONFLICT(k) DO UPDATE SET v = excluded.v, updated_at = excluded.updated_at`, ON CONFLICT(k) DO UPDATE SET v = excluded.v,
updated_at = excluded.updated_at, dirty = 1, rev = meta.rev + 1`,
[k, v, now()], [k, v, now()],
); );
} }
@@ -156,29 +193,47 @@ export async function editMeta(db: Db, k: string, v: string): Promise<void> {
export async function editChatTurn(db: Db, role: string, body: string): Promise<void> { export async function editChatTurn(db: Db, role: string, body: string): Promise<void> {
const t = now(); const t = now();
await db.run( await db.run(
"INSERT INTO chat (id, role, body, created_at, updated_at) VALUES (?, ?, ?, ?, ?)", "INSERT INTO chat (id, role, body, created_at, updated_at, dirty, rev) VALUES (?, ?, ?, ?, ?, 1, 1)",
[uuidv7(t), role, body, t, t], [uuidv7(t), role, body, t, t],
); );
} }
/**
* Raise a reset marker: `reset.<scope>` counts deliberate shrinks, and every
* device that sees it rise empties the same things. The device raising it
* records that it has already done so, so its own marker coming back from
* the server does not wipe what it has written since.
*/
async function raiseMarker(db: Db, scope: MarkerScope): Promise<void> {
const key = `reset.${scope}`;
const at = Number((await db.get<{ v: string }>("SELECT v FROM meta WHERE k = ?", [key]))?.v ?? 0);
const next = String((Number.isFinite(at) ? at : 0) + 1);
await editMeta(db, key, next);
await db.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES (?, ?, 0)", [`sync.applied.${key}`, next]);
}
/** Wipe the transcript. Progress and cards are untouched. */ /** Wipe the transcript. Progress and cards are untouched. */
export async function editChatClear(db: Db): Promise<void> { export async function editChatClear(db: Db): Promise<void> {
await db.tx(async (tx) => { await db.tx(async (tx) => {
const rows = await tx.all<{ id: string }>("SELECT id FROM chat"); await remove(tx, "chat", "1 = 1");
await tx.run("DELETE FROM chat"); await raiseMarker(tx, "chat");
for (const r of rows) await tombstone(tx, "chat", r.id);
}); });
} }
/** Trim the transcript. The artifact kept the last 26 turns. */ /**
export async function editChatTrim(db: Db, keep: number): Promise<void> { * Keep the transcript to its last `keep` turns — on THIS device.
await db.tx(async (tx) => { *
const kept = "SELECT id FROM chat ORDER BY created_at DESC, id DESC LIMIT ?"; * Not a deletion: nothing is tombstoned, and the server keeps the history.
const doomed = await tx.all<{ id: string }>(`SELECT id FROM chat WHERE id NOT IN (${kept})`, [keep]); * A trim used to tombstone every turn outside this device's newest 26, which
if (!doomed.length) return; * on a device that had just pulled another's turns deleted those turns
await tx.run(`DELETE FROM chat WHERE id NOT IN (${kept})`, [keep]); * everywhere. A turn not yet pushed is never trimmed.
for (const r of doomed) await tombstone(tx, "chat", r.id); */
}); export async function pruneChat(db: Db, keep: number): Promise<void> {
await db.run(
`DELETE FROM chat WHERE dirty = 0 AND id NOT IN
(SELECT id FROM chat ORDER BY created_at DESC, id DESC LIMIT ?)`,
[keep],
);
} }
/** Study happened today: reviews, correct answers, drill answers. */ /** Study happened today: reviews, correct answers, drill answers. */
@@ -189,24 +244,24 @@ export async function editStudyLog(
): Promise<void> { ): Promise<void> {
const { reviews = 0, correct = 0, drills = 0 } = delta; const { reviews = 0, correct = 0, drills = 0 } = delta;
await db.run( await db.run(
`INSERT INTO study_log (day, reviews, correct, drills, updated_at) `INSERT INTO study_log (day, device, reviews, correct, drills, updated_at, dirty, rev)
VALUES (?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?, ?, 1, 1)
ON CONFLICT(day) DO UPDATE SET ON CONFLICT(day, device) DO UPDATE SET
reviews = study_log.reviews + excluded.reviews, reviews = study_log.reviews + excluded.reviews,
correct = study_log.correct + excluded.correct, correct = study_log.correct + excluded.correct,
drills = study_log.drills + excluded.drills, drills = study_log.drills + excluded.drills,
updated_at = excluded.updated_at`, updated_at = excluded.updated_at, dirty = 1, rev = study_log.rev + 1`,
[day, reviews, correct, drills, now()], [day, await deviceId(db), reviews, correct, drills, now()],
); );
} }
/** He looked a word up in the rail. Feeds the persistent underline. */ /** He looked a word up in the rail. Feeds the persistent underline. */
export async function editPeek(db: Db, form: string): Promise<void> { export async function editPeek(db: Db, form: string): Promise<void> {
await db.run( await db.run(
`INSERT INTO peek (form, count, updated_at) VALUES (?, 1, ?) `INSERT INTO peek (form, device, count, updated_at, dirty, rev) VALUES (?, ?, 1, ?, 1, 1)
ON CONFLICT(form) DO UPDATE SET count = peek.count + 1, ON CONFLICT(form, device) DO UPDATE SET count = peek.count + 1,
updated_at = excluded.updated_at`, updated_at = excluded.updated_at, dirty = 1, rev = peek.rev + 1`,
[form, now()], [form, await deviceId(db), now()],
); );
} }
@@ -235,6 +290,23 @@ export interface AddedWord {
created: boolean; created: boolean;
} }
/** The derived rows that make a custom word look like any other word. */
export async function deriveCustomLemma(db: Db, word: CustomWord): Promise<number> {
const id = lemmaId(word.headword, word.pos);
await db.run(
`INSERT OR REPLACE INTO lemma (id, headword, pos, freq_rank, level, gloss_en, gloss_ko,
unit_band, source)
VALUES (?, ?, ?, NULL, NULL, ?, '', 0, 'custom')`,
[id, word.headword, word.pos, word.gloss],
);
await db.run("INSERT OR REPLACE INTO surface (form, lemma_id, analysis) VALUES (?, ?, ?)", [
word.headword,
id,
"headword, custom",
]);
return id;
}
/** /**
* Add a word of the learner's own. * Add a word of the learner's own.
* *
@@ -254,44 +326,31 @@ export async function editAddCustomWord(db: Db, word: CustomWord): Promise<Added
[headword, word.pos], [headword, word.pos],
); );
const t = now();
const newCardFor = (id: number) =>
tx.run(
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at, dirty, rev)
VALUES (?, 0, 2.5, 0, 0, 0, 0, ?, 1, 1)
ON CONFLICT(lemma_id) DO NOTHING`,
[id, t],
);
if (existing) { if (existing) {
// Already known — just make sure it is studiable. Its gloss stays the // Already known — just make sure it is studiable. Its gloss stays the
// dictionary's; overwriting curated content from a text field would be // dictionary's; overwriting curated content from a text field would be
// a poor trade. // a poor trade.
await tx.run( await newCardFor(existing.id);
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at)
VALUES (?, 0, 2.5, 0, 0, 0, 0, ?)
ON CONFLICT(lemma_id) DO NOTHING`,
[existing.id, now()],
);
return { lemmaId: existing.id, created: false }; return { lemmaId: existing.id, created: false };
} }
const id = lemmaId(headword, word.pos);
const t = now();
await tx.run( await tx.run(
`INSERT INTO custom_word (headword, pos, gloss, updated_at) VALUES (?, ?, ?, ?) `INSERT INTO custom_word (headword, pos, gloss, updated_at, dirty, rev) VALUES (?, ?, ?, ?, 1, 1)
ON CONFLICT(headword, pos) DO UPDATE SET gloss = excluded.gloss, updated_at = excluded.updated_at`, ON CONFLICT(headword, pos) DO UPDATE SET gloss = excluded.gloss,
updated_at = excluded.updated_at, dirty = 1, rev = custom_word.rev + 1`,
[headword, word.pos, gloss, t], [headword, word.pos, gloss, t],
); );
await tx.run( const id = await deriveCustomLemma(tx, { headword, gloss, pos: word.pos });
`INSERT OR REPLACE INTO lemma (id, headword, pos, freq_rank, level, gloss_en, gloss_ko, await newCardFor(id);
unit_band, source)
VALUES (?, ?, ?, NULL, NULL, ?, '', 0, 'custom')`,
[id, headword, word.pos, gloss],
);
await tx.run("INSERT OR REPLACE INTO surface (form, lemma_id, analysis) VALUES (?, ?, ?)", [
headword,
id,
"headword, custom",
]);
await tx.run(
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at)
VALUES (?, 0, 2.5, 0, 0, 0, 0, ?)
ON CONFLICT(lemma_id) DO NOTHING`,
[id, t],
);
return { lemmaId: id, created: true }; return { lemmaId: id, created: true };
}); });
} }
@@ -304,12 +363,10 @@ export async function editRemoveCustomWord(db: Db, lemmaId: number): Promise<voi
); );
if (!word) return; // never touch shipped dictionary rows if (!word) return; // never touch shipped dictionary rows
await tx.run("DELETE FROM card WHERE lemma_id = ?", [lemmaId]); await remove(tx, "card", "lemma_id = ?", [lemmaId]);
await remove(tx, "custom_word", "headword = ? AND pos = ?", [word.headword, word.pos]);
await tx.run("DELETE FROM surface WHERE lemma_id = ?", [lemmaId]); await tx.run("DELETE FROM surface WHERE lemma_id = ?", [lemmaId]);
await tx.run("DELETE FROM lemma WHERE id = ?", [lemmaId]); await tx.run("DELETE FROM lemma WHERE id = ?", [lemmaId]);
await tx.run("DELETE FROM custom_word WHERE headword = ? AND pos = ?", [word.headword, word.pos]);
await tombstone(tx, "card", lemmaId);
await tombstone(tx, "custom_word", JSON.stringify([word.headword, word.pos]));
}); });
} }
@@ -317,6 +374,10 @@ export async function editRemoveCustomWord(db: Db, lemmaId: number): Promise<voi
RESET — deliberately destructive, so it is spelled out here rather than RESET — deliberately destructive, so it is spelled out here rather than
assembled ad hoc at a call site. assembled ad hoc at a call site.
Every row is tombstoned, and the scope's marker is raised so a device
holding rows this one never saw empties them too — a reset is the one
shrink that must win over work it has not heard of.
Neither scope touches `lemma` or `surface` for shipped words: the Neither scope touches `lemma` or `surface` for shipped words: the
dictionary is reference data, rebuildable from the assets, and wiping it dictionary is reference data, rebuildable from the assets, and wiping it
would leave the app unable to gloss anything until the bands reloaded. would leave the app unable to gloss anything until the bands reloaded.
@@ -324,64 +385,20 @@ export async function editRemoveCustomWord(db: Db, lemmaId: number): Promise<voi
export type ResetScope = "roadmap" | "everything"; export type ResetScope = "roadmap" | "everything";
/** Meta keys holding the roadmap: where he is, what he has done recently. */ /** The meta keys a reset scope empties, as a WHERE clause. */
const ROAD_META = "k LIKE 'road.%'"; const metaWhere = (scope: MarkerScope): string =>
RESET_SCOPES[scope].meta.map((p) => `k LIKE '${p}%'`).join(" OR ") || "0";
/** Meta keys holding the learner himself: preferences, notes, counters. */
const LEARNER_META = "k LIKE 'prefs.%' OR k LIKE 'grammar.%' OR k LIKE 'trainer.%' OR k LIKE 'learner.%'";
export async function editReset(db: Db, scope: ResetScope): Promise<void> { export async function editReset(db: Db, scope: ResetScope): Promise<void> {
await db.tx(async (tx) => { await db.tx(async (tx) => {
// Tombstone before deleting, while the keys are still readable — a reset for (const tbl of RESET_SCOPES[scope].tables) await remove(tx, tbl, "1 = 1");
// must propagate, or the next pull restores everything it just erased. await remove(tx, "meta", metaWhere(scope));
for (const r of await tx.all<{ unit_id: string }>("SELECT unit_id FROM progress"))
await tombstone(tx, "progress", r.unit_id);
for (const r of await tx.all<{ id: string }>("SELECT id FROM chat"))
await tombstone(tx, "chat", r.id);
for (const r of await tx.all<{ phase: number; kind: string; item: string }>(
"SELECT phase, kind, item FROM phase_ledger",
))
await tombstone(tx, "phase_ledger", JSON.stringify([r.phase, r.kind, r.item]));
for (const r of await tx.all<{ k: string }>(`SELECT k FROM meta WHERE ${ROAD_META}`))
await tombstone(tx, "meta", r.k);
await tx.run("DELETE FROM progress");
await tx.run("DELETE FROM chat");
await tx.run("DELETE FROM phase_ledger");
await tx.run(`DELETE FROM meta WHERE ${ROAD_META}`);
if (scope === "everything") { if (scope === "everything") {
for (const r of await tx.all<{ lemma_id: number }>("SELECT lemma_id FROM card"))
await tombstone(tx, "card", r.lemma_id);
for (const r of await tx.all<{ day: number }>("SELECT day FROM study_log"))
await tombstone(tx, "study_log", r.day);
for (const r of await tx.all<{ form: string }>("SELECT form FROM peek"))
await tombstone(tx, "peek", r.form);
for (const r of await tx.all<{ headword: string; pos: string }>(
"SELECT headword, pos FROM custom_word",
))
await tombstone(tx, "custom_word", JSON.stringify([r.headword, r.pos]));
for (const r of await tx.all<{ word: string }>("SELECT word FROM evidence"))
await tombstone(tx, "evidence", r.word);
for (const r of await tx.all<{ word: string }>("SELECT word FROM confusion"))
await tombstone(tx, "confusion", r.word);
for (const r of await tx.all<{ k: string }>(`SELECT k FROM meta WHERE ${LEARNER_META}`))
await tombstone(tx, "meta", r.k);
await tx.run("DELETE FROM card");
await tx.run("DELETE FROM study_log");
await tx.run("DELETE FROM peek");
await tx.run("DELETE FROM custom_word");
await tx.run("DELETE FROM evidence");
await tx.run("DELETE FROM confusion");
await tx.run("DELETE FROM surface WHERE lemma_id IN (SELECT id FROM lemma WHERE source = 'custom')"); await tx.run("DELETE FROM surface WHERE lemma_id IN (SELECT id FROM lemma WHERE source = 'custom')");
await tx.run("DELETE FROM lemma WHERE source = 'custom'"); await tx.run("DELETE FROM lemma WHERE source = 'custom'");
// Preferences, grammar flags and notes, the trainer score and the
// round counter. Device bookkeeping (schema_version, dict.*, sync.*)
// is left alone — it describes this install, not the learner.
await tx.run(`DELETE FROM meta WHERE ${LEARNER_META}`);
} }
await raiseMarker(tx, scope);
}); });
} }

8
app/src/domain/cards.ts
View File

@@ -177,7 +177,13 @@ export interface DayRow {
} }
export async function studyLog(db: Db, sinceDay: number): Promise<DayRow[]> { export async function studyLog(db: Db, sinceDay: number): Promise<DayRow[]> {
return db.all<DayRow>("SELECT * FROM study_log WHERE day >= ? ORDER BY day", [sinceDay]); // One row per device per day — summed, so a phone's reviews and a
// laptop's both count and neither can overwrite the other's.
return db.all<DayRow>(
`SELECT day, SUM(reviews) AS reviews, SUM(correct) AS correct, SUM(drills) AS drills
FROM study_log WHERE day >= ? GROUP BY day ORDER BY day`,
[sinceDay],
);
} }
/** Consecutive days with any activity, counting back from today. */ /** Consecutive days with any activity, counting back from today. */

12
app/src/state/store.tsx
View File

@@ -30,7 +30,8 @@ import {
writeServerConfig, writeServerConfig,
type ServerConfig, type ServerConfig,
} from "../domain/server-config.js"; } from "../domain/server-config.js";
import { SYNC_PAUSED, trySync, type SyncResult } from "../sync/client.js"; import { forgetServerPosition, trySync, type SyncResult } from "../sync/client.js";
import { unitIndex } from "../domain/gate.js";
import type { ProgressState } from "@lib/gate.js"; import type { ProgressState } from "@lib/gate.js";
import type { FocusMode } from "../domain/gate.js"; import type { FocusMode } from "../domain/gate.js";
@@ -232,6 +233,9 @@ export function StoreProvider({
if (!store) return; if (!store) return;
if (cfg) await writeServerConfig(store.db, cfg); if (cfg) await writeServerConfig(store.db, cfg);
else await clearServerConfig(store.db); else await clearServerConfig(store.db);
// A different server is a different conversation: hydrate from its
// first page before anything here may be pushed to it.
await forgetServerPosition(store.db);
setServerState(cfg); setServerState(cfg);
}, },
[store], [store],
@@ -243,13 +247,9 @@ export function StoreProvider({
const running = useRef(false); const running = useRef(false);
const syncNow = useCallback(async () => { const syncNow = useCallback(async () => {
if (!store || !server || running.current) return; if (!store || !server || running.current) return;
if (SYNC_PAUSED) {
setSyncState({ at: Date.now(), result: null, error: SYNC_PAUSED, running: false });
return;
}
running.current = true; running.current = true;
setSyncState((s) => ({ ...s, running: true })); setSyncState((s) => ({ ...s, running: true }));
const result = await trySync(store.db, server); const result = await trySync(store.db, server, { unitIndex });
running.current = false; running.current = false;
setSyncState({ setSyncState({
at: Date.now(), at: Date.now(),

532
app/src/sync/client.ts
View File

@@ -1,39 +1,34 @@
/* The sync loop: push what changed here, pull what changed there, apply. /* The sync loop, protocol 2: pull everything, then push what changed here.
Offline-first is not negotiable. Sync runs only when a server is Offline-first is not negotiable. Sync runs only when a server is
configured, and every failure is non-fatal — the app must stay completely configured, and every failure is non-fatal — the app must stay completely
usable with the Pi unreachable, which is the acceptance criterion the usable with the Pi unreachable. Nothing in here may throw into the UI.
whole port was built around. Nothing in here may throw into the UI. */
The order is the first gate. A device may not push until it has pulled
every page the server holds for this epoch: the week-old laptop that
pushed first is exactly how the artifact lost a week of the phone's work.
See shared/sync-protocol.mjs for the other two gates. */
import type { Db } from "../db/types.js"; import type { Db } from "../db/types.js";
import { deriveCustomLemma } from "../db/writes.js";
import { import {
PAGE_SIZE, PAGE_SIZE,
PROTOCOL,
PROTOCOL_HEADER,
RESET_SCOPES,
SYNC_TABLES, SYNC_TABLES,
SYNC_TABLE_NAMES, SYNC_TABLE_NAMES,
incomingWins, decodePk,
encodePk,
isSyncableMetaKey, isSyncableMetaKey,
resetCovers,
type PullResponse,
type PushResponse,
type ServerRow,
type WireRow,
} from "@shared/sync-protocol.mjs"; } from "@shared/sync-protocol.mjs";
import { lemmaId } from "@shared/lemma-id.mjs";
/** A row on the wire: its table, its columns, and when it last changed. */ import { resolve, type Data, type ResolveContext } from "./resolve.js";
export interface SyncRow {
tbl: string;
data: Record<string, string | number | null>;
updated_at: number;
/** Present and true when this row is a delete. */
deleted?: boolean;
}
export interface PushRequest {
rows: SyncRow[];
}
export interface PushResponse {
cursor: number;
}
export interface PullResponse {
rows: SyncRow[];
cursor: number;
more: boolean;
}
export interface SyncConfig { export interface SyncConfig {
baseUrl: string; baseUrl: string;
@@ -43,209 +38,405 @@ export interface SyncConfig {
export interface SyncResult { export interface SyncResult {
pushed: number; pushed: number;
pulled: number; pulled: number;
conflicts: number;
cursor: number; cursor: number;
/** The server was new to this device, or had been reset: everything was
pulled from the start and every stamped row offered back. */
rehydrated: boolean;
} }
/** const EPOCH = "sync.epoch";
* Why sync is not running, or null when it is. const CURSOR = "sync.cursor";
* const HYDRATED = "sync.hydrated";
* Cards are keyed by stable lemma ids now, while the server still holds rows
* written under the old positional ids. Exchanging them would plant cards
* under ids that name no word, on both sides. Everything stays local — which
* this app is built to do — until the sync protocol that replaces this one
* ships with its own server schema.
*/
export const SYNC_PAUSED: string | null =
"Sync is paused while the storage format changes. Everything is kept on this device";
const CURSOR_KEY = "sync.cursor"; /** Rounds of push-and-settle before giving up until the next sync. */
const PUSHED_KEY = "sync.pushedAt"; const MAX_PUSH_ROUNDS = 5;
/* ── local bookkeeping ──────────────────────────────────────────────── /* ── local bookkeeping ────────────────────────────────────────────────
These are device state, not user data. They are written with Device state, not user data: written unstamped and never dirty, and
updated_at = 0 and excluded from the syncable-meta allowlist, so they `sync.*` is outside the syncable-meta allowlist besides. */
never travel. */
async function readNum(db: Db, key: string): Promise<number> { async function readMeta(db: Db, k: string): Promise<string | undefined> {
const row = await db.get<{ v: string }>("SELECT v FROM meta WHERE k = ?", [key]); return (await db.get<{ v: string }>("SELECT v FROM meta WHERE k = ?", [k]))?.v;
const n = row ? Number(row.v) : 0;
return Number.isFinite(n) ? n : 0;
} }
async function writeNum(db: Db, key: string, value: number): Promise<void> { async function writeMeta(db: Db, k: string, v: string | number): Promise<void> {
await db.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES (?, ?, 0)", [ await db.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES (?, ?, 0)", [k, String(v)]);
key,
String(value),
]);
} }
/* ── collect ─────────────────────────────────────────────────────────── */ /** Forget this device's place with the server, so the next sync starts over. */
export async function forgetServerPosition(db: Db): Promise<void> {
await db.run("DELETE FROM meta WHERE k IN (?, ?, ?)", [EPOCH, CURSOR, HYDRATED]);
}
/** Everything edited on this device since the last successful push. */ /** Has this device pulled everything its server holds? */
export async function collectDirty(db: Db, pushedAt: number): Promise<SyncRow[]> { export async function isHydrated(db: Db): Promise<boolean> {
const out: SyncRow[] = []; return (await readMeta(db, HYDRATED)) === "1";
}
for (const tbl of SYNC_TABLE_NAMES) { /* ── rows ─────────────────────────────────────────────────────────────── */
const spec = SYNC_TABLES[tbl]!;
const cols = [...spec.cols, "updated_at"].join(", "); type LocalRow = Data & { base_seq: number; dirty: number; rev: number; updated_at: number };
const rows = await db.all<Record<string, string | number>>(
`SELECT ${cols} FROM ${tbl} WHERE updated_at > ? ORDER BY updated_at LIMIT ?`, interface Local {
[pushedAt, PAGE_SIZE], live: LocalRow | undefined;
tomb: { base_seq: number; dirty: number; rev: number } | undefined;
}
const whereOf = (tbl: string) => SYNC_TABLES[tbl]!.pk.map((c) => `${c} = ?`).join(" AND ");
async function readLocal(db: Db, tbl: string, pk: string): Promise<Local> {
const live = await db.get<LocalRow>(`SELECT * FROM ${tbl} WHERE ${whereOf(tbl)}`, decodePk(pk));
const tomb = await db.get<{ base_seq: number; dirty: number; rev: number }>(
"SELECT base_seq, dirty, rev FROM tombstone WHERE tbl = ? AND pk = ?",
[tbl, pk],
); );
return { live, tomb };
}
for (const row of rows) { const pick = (tbl: string, row: Data): Data =>
// The allowlist is applied at the point of collection, so a Object.fromEntries(SYNC_TABLES[tbl]!.cols.map((c) => [c, row[c] ?? null]));
// device-local meta key cannot leave this machine even by accident.
if (tbl === "meta" && !isSyncableMetaKey(String(row.k))) continue; /** Write a row as the server holds it: clean, and agreed at `seq`. */
const { updated_at, ...data } = row; async function adopt(db: Db, row: ServerRow): Promise<void> {
out.push({ tbl, data, updated_at: Number(updated_at) }); const spec = SYNC_TABLES[row.tbl]!;
const key = decodePk(row.pk);
if (row.deleted) {
const before = await db.get<Data>(`SELECT * FROM ${row.tbl} WHERE ${whereOf(row.tbl)}`, key);
await db.run(`DELETE FROM ${row.tbl} WHERE ${whereOf(row.tbl)}`, key);
// A clean tombstone remembers the server's seq for this key, so a later
// re-creation here pushes against the right base.
await db.run(
`INSERT INTO tombstone (tbl, pk, updated_at, base_seq, dirty, rev) VALUES (?, ?, ?, ?, 0, 0)
ON CONFLICT(tbl, pk) DO UPDATE SET base_seq = excluded.base_seq, dirty = 0`,
[row.tbl, row.pk, row.updated_at ?? 0, row.seq],
);
if (row.tbl === "custom_word" && before) {
const id = lemmaId(String(before.headword), String(before.pos));
await db.run("DELETE FROM surface WHERE lemma_id = ? AND lemma_id IN (SELECT id FROM lemma WHERE source = 'custom')", [id]);
await db.run("DELETE FROM lemma WHERE id = ? AND source = 'custom'", [id]);
}
return;
}
const data = pick(row.tbl, row.data ?? {});
const cols = spec.cols;
await db.run(
`INSERT INTO ${row.tbl} (${cols.join(", ")}, updated_at, base_seq, dirty)
VALUES (${cols.map(() => "?").join(", ")}, ?, ?, 0)
ON CONFLICT(${spec.pk.join(", ")}) DO UPDATE SET
${cols.map((c) => `${c} = excluded.${c}`).join(", ")},
updated_at = excluded.updated_at, base_seq = excluded.base_seq, dirty = 0`,
[...cols.map((c) => data[c] ?? null), row.updated_at ?? 0, row.seq],
);
await db.run("DELETE FROM tombstone WHERE tbl = ? AND pk = ?", [row.tbl, row.pk]);
if (row.tbl === "custom_word") {
await deriveCustomLemma(db, {
headword: String(data.headword),
pos: String(data.pos),
gloss: String(data.gloss ?? ""),
});
}
if (row.tbl === "meta" && String(data.k).startsWith("reset.")) {
await obeyMarker(db, String(data.k), String(data.v));
} }
} }
const graves = await db.all<{ tbl: string; pk: string; updated_at: number }>( /** Ours stands: agree with the server's seq, so the next push overwrites it. */
"SELECT tbl, pk, updated_at FROM tombstone WHERE updated_at > ? ORDER BY updated_at LIMIT ?", async function keep(db: Db, tbl: string, pk: string, local: Local, seq: number): Promise<void> {
[pushedAt, PAGE_SIZE], if (local.live) {
await db.run(`UPDATE ${tbl} SET base_seq = ?, dirty = 1 WHERE ${whereOf(tbl)}`, [seq, ...decodePk(pk)]);
} else {
await db.run("UPDATE tombstone SET base_seq = ?, dirty = 1 WHERE tbl = ? AND pk = ?", [seq, tbl, pk]);
}
}
/** A combination of both: written here, agreed at `seq`, and pushed. */
async function merge(db: Db, tbl: string, pk: string, seq: number, data: Data): Promise<void> {
const cols = SYNC_TABLES[tbl]!.cols;
await db.run(
`UPDATE ${tbl} SET ${cols.map((c) => `${c} = ?`).join(", ")},
base_seq = ?, dirty = 1, rev = rev + 1 WHERE ${whereOf(tbl)}`,
[...cols.map((c) => data[c] ?? null), seq, ...decodePk(pk)],
);
}
/** Settle one disagreement between this device and the server. */
async function settle(db: Db, remote: ServerRow, local: Local, ctx: ResolveContext): Promise<void> {
// The server has no row at all: nothing to disagree with.
if (remote.seq === 0) {
if (local.live) await keep(db, remote.tbl, remote.pk, local, 0);
else await db.run("DELETE FROM tombstone WHERE tbl = ? AND pk = ?", [remote.tbl, remote.pk]);
return;
}
const localSide = local.live
? { deleted: false, data: local.live as Data }
: { deleted: true, data: null };
const outcome = resolve(remote.tbl, localSide, { deleted: remote.deleted, data: remote.data }, ctx);
if (outcome.kind === "adopt") await adopt(db, remote);
else if (outcome.kind === "keep") await keep(db, remote.tbl, remote.pk, local, remote.seq);
else await merge(db, remote.tbl, remote.pk, remote.seq, outcome.data);
}
/* ── deliberate shrinks ───────────────────────────────────────────────── */
/**
* Empty what a reset marker covers, once per rise of its counter.
*
* Unsynced edits here go too: a reset this device had not heard of still
* wins over work done on the data it reset. Rows written after the reset
* arrive later in the pull — higher change_seq — and survive it.
*/
async function obeyMarker(db: Db, key: string, value: string): Promise<void> {
const scope = key.slice("reset.".length) as keyof typeof RESET_SCOPES;
const spec = RESET_SCOPES[scope];
if (!spec) return;
const seen = Number((await readMeta(db, `sync.applied.${key}`)) ?? 0);
const n = Number(value);
if (!(n > seen)) return;
for (const tbl of spec.tables) {
await db.run(`DELETE FROM ${tbl}`);
await db.run("DELETE FROM tombstone WHERE tbl = ?", [tbl]);
}
for (const prefix of spec.meta) {
await db.run("DELETE FROM meta WHERE k LIKE ?", [`${prefix}%`]);
}
if (spec.tables.includes("custom_word")) {
await db.run("DELETE FROM surface WHERE lemma_id IN (SELECT id FROM lemma WHERE source = 'custom')");
await db.run("DELETE FROM lemma WHERE source = 'custom'");
}
await writeMeta(db, `sync.applied.${key}`, n);
}
/** Resets raised here that the server has not heard of yet. */
async function pendingResets(db: Db): Promise<string[]> {
const rows = await db.all<{ k: string }>("SELECT k FROM meta WHERE k LIKE 'reset.%' AND dirty = 1");
return rows.map((r) => r.k.slice("reset.".length));
}
/* ── pull ─────────────────────────────────────────────────────────────── */
async function applyIncoming(
db: Db,
row: ServerRow,
pending: string[],
ctx: ResolveContext,
): Promise<"applied" | "settled" | "skipped"> {
const spec = SYNC_TABLES[row.tbl];
if (!spec) return "skipped"; // a newer server; ignore rather than fail
const key = decodePk(row.pk);
const probe: Data = row.data ?? Object.fromEntries(spec.pk.map((c, i) => [c, key[i] ?? null]));
if (row.tbl === "meta" && !isSyncableMetaKey(String(probe.k ?? key[0]))) return "skipped";
// A reset raised here, not yet pushed, covers this row: the server's copy
// predates it and is about to be emptied everywhere anyway.
if (pending.some((scope) => resetCovers(scope, row.tbl, probe))) return "skipped";
const local = await readLocal(db, row.tbl, row.pk);
const mine = local.live ?? local.tomb;
if (mine && mine.base_seq === row.seq) return "skipped"; // our own write, echoed back
if (!mine || !mine.dirty) {
await adopt(db, row);
return "applied";
}
await settle(db, row, local, ctx);
return "settled";
}
/* ── push ─────────────────────────────────────────────────────────────── */
interface Outgoing {
wire: WireRow;
rev: number;
/** A live row re-created over a tombstone this device had applied. */
overTomb: boolean;
}
const keyOf = (tbl: string, pk: string) => JSON.stringify([tbl, pk]);
/** Up to one page of dirty rows: deletes and reset markers first, so the
server orders a reset before anything written after it. */
async function collectDirty(db: Db): Promise<Outgoing[]> {
const out: Outgoing[] = [];
const graves = await db.all<{ tbl: string; pk: string; base_seq: number; rev: number; updated_at: number }>(
"SELECT tbl, pk, base_seq, rev, updated_at FROM tombstone WHERE dirty = 1 LIMIT ?",
[PAGE_SIZE],
); );
for (const g of graves) { for (const g of graves) {
out.push({ tbl: g.tbl, data: { pk: g.pk }, updated_at: Number(g.updated_at), deleted: true }); if (!SYNC_TABLES[g.tbl]) continue;
out.push({
wire: { tbl: g.tbl, pk: g.pk, base: g.base_seq, deleted: true, data: null, updated_at: g.updated_at },
rev: g.rev,
overTomb: false,
});
} }
const ordered = ["meta", ...SYNC_TABLE_NAMES.filter((t) => t !== "meta")];
for (const tbl of ordered) {
if (out.length >= PAGE_SIZE) break;
const rows = await db.all<LocalRow>(
`SELECT * FROM ${tbl} WHERE dirty = 1 ${tbl === "meta" ? "ORDER BY k LIKE 'reset.%' DESC" : ""} LIMIT ?`,
[PAGE_SIZE - out.length],
);
for (const row of rows) {
if (tbl === "meta" && !isSyncableMetaKey(String(row.k))) {
// Device bookkeeping that was edited: it never travels, so it is
// never really dirty.
await db.run("UPDATE meta SET dirty = 0 WHERE k = ?", [String(row.k)]);
continue;
}
const pk = encodePk(tbl, row);
let base = row.base_seq;
let overTomb = false;
if (base === 0) {
const tomb = await db.get<{ base_seq: number }>(
"SELECT base_seq FROM tombstone WHERE tbl = ? AND pk = ? AND dirty = 0",
[tbl, pk],
);
if (tomb) {
base = tomb.base_seq;
overTomb = true;
}
}
out.push({
wire: { tbl, pk, base, deleted: false, data: pick(tbl, row), updated_at: row.updated_at },
rev: row.rev,
overTomb,
});
}
}
return out; return out;
} }
/* ── apply ───────────────────────────────────────────────────────────── */ async function acknowledge(db: Db, sent: Outgoing[], applied: PushResponse["applied"]): Promise<void> {
const byKey = new Map(sent.map((s) => [keyOf(s.wire.tbl, s.wire.pk), s]));
/** Apply one incoming row, last-write-wins against what is already here. */ for (const a of applied) {
async function applyRow(db: Db, row: SyncRow): Promise<boolean> { const s = byKey.get(keyOf(a.tbl, a.pk));
const spec = SYNC_TABLES[row.tbl]; if (!s) continue;
if (!spec) return false; // unknown table — a newer server; ignore rather than fail if (s.wire.deleted) {
await db.run(
if (row.deleted) { `UPDATE tombstone SET base_seq = ?, dirty = CASE WHEN rev = ? THEN 0 ELSE dirty END
const pk = String(row.data.pk); WHERE tbl = ? AND pk = ?`,
const where = spec.pk.map((c) => `${c} = ?`).join(" AND "); [a.seq, s.rev, a.tbl, a.pk],
const parts = pk.split(" ");
const local = await db.get<{ updated_at: number }>(
`SELECT updated_at FROM ${row.tbl} WHERE ${where}`,
parts,
); );
if (local && !incomingWins(row.updated_at, local.updated_at)) return false; } else {
// Clean only if nothing changed the row while the push was in flight.
await db.tx(async (tx) => { await db.run(
await tx.run(`DELETE FROM ${row.tbl} WHERE ${where}`, parts); `UPDATE ${a.tbl} SET base_seq = ?, dirty = CASE WHEN rev = ? THEN 0 ELSE dirty END
await tx.run( WHERE ${whereOf(a.tbl)}`,
`INSERT INTO tombstone (tbl, pk, updated_at) VALUES (?, ?, ?) [a.seq, s.rev, ...decodePk(a.pk)],
ON CONFLICT(tbl, pk) DO UPDATE SET updated_at = excluded.updated_at`,
[row.tbl, pk, row.updated_at],
); );
}); if (s.overTomb) await db.run("DELETE FROM tombstone WHERE tbl = ? AND pk = ?", [a.tbl, a.pk]);
return true; }
}
} }
if (row.tbl === "meta" && !isSyncableMetaKey(String(row.data.k))) return false; /* ── transport ────────────────────────────────────────────────────────── */
const where = spec.pk.map((c) => `${c} = ?`).join(" AND "); async function call<T>(cfg: SyncConfig, path: string, init: RequestInit, signal?: AbortSignal): Promise<T> {
const keys = spec.pk.map((c) => row.data[c] as string | number);
const local = await db.get<{ updated_at: number }>(
`SELECT updated_at FROM ${row.tbl} WHERE ${where}`,
keys,
);
if (local && !incomingWins(row.updated_at, local.updated_at)) return false;
const cols = [...spec.cols, "updated_at"];
const holes = cols.map(() => "?").join(", ");
const values = [...spec.cols.map((c) => row.data[c] ?? null), row.updated_at];
await db.run(`INSERT OR REPLACE INTO ${row.tbl} (${cols.join(", ")}) VALUES (${holes})`, values);
return true;
}
/* ── transport ───────────────────────────────────────────────────────── */
async function call<T>(
cfg: SyncConfig,
path: string,
init: RequestInit,
signal?: AbortSignal,
): Promise<T> {
const res = await fetch(`${cfg.baseUrl.replace(/\/$/, "")}${path}`, { const res = await fetch(`${cfg.baseUrl.replace(/\/$/, "")}${path}`, {
...init, ...init,
signal, signal,
headers: { headers: {
"content-type": "application/json", "content-type": "application/json",
authorization: `Bearer ${cfg.token}`, authorization: `Bearer ${cfg.token}`,
[PROTOCOL_HEADER]: String(PROTOCOL),
...(init.headers ?? {}), ...(init.headers ?? {}),
}, },
}); });
if (res.status === 426) throw new Error("the server speaks a different sync protocol — update it");
if (!res.ok) throw new Error(`sync ${path}: HTTP ${res.status}`); if (!res.ok) throw new Error(`sync ${path}: HTTP ${res.status}`);
return (await res.json()) as T; return (await res.json()) as T;
} }
/* ── the loop ────────────────────────────────────────────────────────── */ /* ── the loop ─────────────────────────────────────────────────────────── */
/** Re-agree with a server from nothing: every stamped row is offered again. */
async function rehydrate(db: Db, epoch: string): Promise<void> {
await db.tx(async (tx) => {
for (const tbl of [...SYNC_TABLE_NAMES, "tombstone"]) {
await tx.run(
`UPDATE ${tbl} SET base_seq = 0, dirty = CASE WHEN updated_at > 0 THEN 1 ELSE 0 END, rev = rev + 1`,
);
}
await writeMeta(tx, EPOCH, epoch);
await writeMeta(tx, CURSOR, 0);
await writeMeta(tx, HYDRATED, 0);
});
}
/** /**
* One full exchange. Push first so local edits are never lost to an * One full exchange: pull every page, then push until nothing is dirty.
* incoming row that would have overwritten them; then pull, paging until *
* the server has nothing newer. * `ctx` lets a conflict ask the roadmap which of two units is further on.
*/ */
export async function syncOnce( export async function syncOnce(
db: Db, db: Db,
cfg: SyncConfig, cfg: SyncConfig,
ctx: ResolveContext,
signal?: AbortSignal, signal?: AbortSignal,
): Promise<SyncResult> { ): Promise<SyncResult> {
// Anything edited after this instant stays dirty, whatever the pull let epoch = (await readMeta(db, EPOCH)) ?? "";
// brings back. That is what keeps an edit made *during* a sync from being let cursor = Number((await readMeta(db, CURSOR)) ?? 0) || 0;
// marked clean by a remote row that happens to carry a later timestamp. let rehydrated = false;
const startedAt = Date.now();
const pushedAt = await readNum(db, PUSHED_KEY);
const dirty = await collectDirty(db, pushedAt);
let watermark = pushedAt;
if (dirty.length) {
const body: PushRequest = { rows: dirty };
await call<PushResponse>(
cfg,
"/api/sync",
{ method: "POST", body: JSON.stringify(body) },
signal,
);
// Only advance once the server has them. A failed push throws before
// this line and the rows stay dirty, which is the safe direction.
watermark = dirty.reduce((m, r) => Math.max(m, r.updated_at), watermark);
await writeNum(db, PUSHED_KEY, watermark);
}
/* The pull cursor advances ONLY as rows are applied.
It must never be taken from the push response: the server's newest
change_seq includes rows this device has not seen, and adopting it
would skip them permanently — a silent, unrecoverable data loss, since
nothing would ever ask for that range again. */
let cursor = await readNum(db, CURSOR_KEY);
let pulled = 0; let pulled = 0;
let conflicts = 0;
/* Gate 1: hydrate. Nothing leaves this device until every page is in. */
for (;;) { for (;;) {
const page = await call<PullResponse>( const page = await call<PullResponse>(cfg, `/api/sync?cursor=${cursor}`, { method: "GET" }, signal);
cfg, if (page.epoch !== epoch) {
`/api/sync?cursor=${cursor}`, if (rehydrated) throw new Error("the server was reset during this sync");
{ method: "GET" }, await rehydrate(db, page.epoch);
signal, epoch = page.epoch;
); cursor = 0;
rehydrated = true;
for (const row of page.rows) { continue;
if (await applyRow(db, row)) pulled++; }
// A row we just received is not a local edit. Marking it clean stops const pending = await pendingResets(db);
// the next sync pushing it straight back. Bounded by startedAt so a for (const row of page.rows) {
// concurrent local edit is never swept up with it. const r = await db.tx((tx) => applyIncoming(tx, row, pending, ctx));
if (row.updated_at <= startedAt) watermark = Math.max(watermark, row.updated_at); if (r === "applied") pulled++;
if (r === "settled") conflicts++;
} }
cursor = page.cursor; cursor = page.cursor;
await writeNum(db, CURSOR_KEY, cursor); await writeMeta(db, CURSOR, cursor);
if (!page.more) break; if (!page.more) break;
} }
await writeMeta(db, HYDRATED, 1);
if (watermark > pushedAt) await writeNum(db, PUSHED_KEY, watermark); /* Then push. A conflict is settled locally and, if ours stands, pushed
again against the server's current seq. */
let pushed = 0;
for (let round = 0; round < MAX_PUSH_ROUNDS; round++) {
const batch = await collectDirty(db);
if (!batch.length) break;
const res = await call<PushResponse>(
cfg,
"/api/sync",
{ method: "POST", body: JSON.stringify({ rows: batch.map((b) => b.wire) }) },
signal,
);
if (res.epoch !== epoch) {
// Reset between our pull and our push: start over next time.
await forgetServerPosition(db);
break;
}
await acknowledge(db, batch, res.applied);
pushed += res.applied.length;
for (const c of res.conflicts) {
await db.tx(async (tx) => settle(tx, c, await readLocal(tx, c.tbl, c.pk), ctx));
conflicts++;
}
}
return { pushed: dirty.length, pulled, cursor }; return { pushed, pulled, conflicts, cursor, rehydrated };
} }
/** /**
@@ -258,11 +449,12 @@ export async function syncOnce(
export async function trySync( export async function trySync(
db: Db, db: Db,
cfg: SyncConfig | null, cfg: SyncConfig | null,
ctx: ResolveContext,
signal?: AbortSignal, signal?: AbortSignal,
): Promise<SyncResult | null> { ): Promise<SyncResult | null> {
if (!cfg?.baseUrl || !cfg.token || SYNC_PAUSED) return null; if (!cfg?.baseUrl || !cfg.token) return null;
try { try {
return await syncOnce(db, cfg, signal); return await syncOnce(db, cfg, ctx, signal);
} catch (err) { } catch (err) {
console.warn("[sync] skipped:", err instanceof Error ? err.message : err); console.warn("[sync] skipped:", err instanceof Error ? err.message : err);
return null; return null;

131
app/src/sync/resolve.ts Normal file
View File

@@ -0,0 +1,131 @@
/* Gate 3: what to do when two copies of one row disagree.
A conflict means the server's copy changed after this device last agreed
with it, while this device changed its own copy too. The counter says
that much and no more: it cannot say which change is "right".
The artifact's answer, which PORT.md adopts: THE COPY THAT HOLDS MORE
WINS. For one learner whose worst conflict is a duplicated grade, "more
reviews, more evidence, more finished units" is a better tiebreak than
"newer" — it converges, and it never trades real work for an empty
default. The one exception is a deliberate delete, which is obeyed.
Three outcomes:
adopt — take the server's copy;
keep — keep ours, and push it back over the server's;
merge — write a combination, and push that. */
export type Value = string | number | null;
export type Data = Record<string, Value>;
export interface Side {
deleted: boolean;
data: Data | null;
}
export type Resolution = { kind: "adopt" } | { kind: "keep" } | { kind: "merge"; data: Data };
export interface ResolveContext {
/** A unit's position on the roadmap, or -1. */
unitIndex: (unitId: string) => number;
}
const ADOPT: Resolution = { kind: "adopt" };
const KEEP: Resolution = { kind: "keep" };
const num = (v: Value | undefined): number => {
const n = Number(v ?? 0);
return Number.isFinite(n) ? n : 0;
};
/** More wins; a tie goes to the server, so every device lands on one copy. */
const heavier = (local: number, remote: number): Resolution => (local > remote ? KEEP : ADOPT);
/** Pick adopt/keep when a merge came out equal to one side. */
function settle(local: Data, remote: Data, merged: Data): Resolution {
const same = (a: Data) => Object.keys(merged).every((k) => String(merged[k]) === String(a[k]));
if (same(remote)) return ADOPT;
if (same(local)) return KEEP;
return { kind: "merge", data: merged };
}
function parseObject(v: Value | undefined): Record<string, unknown> | null {
try {
const o = JSON.parse(String(v));
return o && typeof o === "object" && !Array.isArray(o) ? (o as Record<string, unknown>) : null;
} catch {
return null;
}
}
function resolveMeta(local: Data, remote: Data, ctx: ResolveContext): Resolution {
const k = String(remote.k ?? local.k ?? "");
if (k === "road.unit") {
return heavier(ctx.unitIndex(String(local.v)), ctx.unitIndex(String(remote.v)));
}
if (k === "learner.round" || k.startsWith("reset.")) {
return heavier(num(local.v), num(remote.v));
}
if (k === "trainer.conjugation") {
return heavier(num(parseObject(local.v)?.n as Value), num(parseObject(remote.v)?.n as Value));
}
if (k === "grammar.learned" || k === "grammar.notes") {
const a = parseObject(local.v);
const b = parseObject(remote.v);
if (!a || !b) return ADOPT;
// Union. Where both have a key, the server's value for a flag; for a
// note, the longer one — a note is only ever grown by its owner.
const out: Record<string, unknown> = { ...a, ...b };
if (k === "grammar.notes") {
for (const key of Object.keys(a)) {
if (key in b && String(a[key] ?? "").length > String(b[key] ?? "").length) out[key] = a[key];
}
}
return settle(local, remote, { ...remote, v: JSON.stringify(out) });
}
// Preferences, recent task types: the server's copy.
return ADOPT;
}
export function resolve(tbl: string, local: Side, remote: Side, ctx: ResolveContext): Resolution {
if (local.deleted && remote.deleted) return ADOPT;
// A deliberate delete is obeyed, whichever side made it.
if (remote.deleted) return ADOPT;
if (local.deleted) return KEEP;
const a = local.data ?? {};
const b = remote.data ?? {};
switch (tbl) {
case "card":
return heavier(num(a.reps) + num(a.lapses), num(b.reps) + num(b.lapses));
case "evidence":
return heavier(
num(a.ok) + num(a.wrong) + num(a.lookups),
num(b.ok) + num(b.wrong) + num(b.lookups),
);
case "progress": {
const more = num(a.answers) > num(b.answers) ? a : b;
return settle(a, b, {
...b,
done: Math.max(num(a.done), num(b.done)),
answers: Math.max(num(a.answers), num(b.answers)),
confidence: more.confidence ?? 0,
note: more.note ?? "",
});
}
case "meta":
return resolveMeta(a, b, ctx);
// Keyed so that the same key means the same thing on every device:
// chat turns and per-device counters never genuinely conflict, and for
// a custom word's gloss, a confusion or a ledger tick the server's copy
// is as good as ours.
default:
return ADOPT;
}
}

66
app/src/ui/tutor/TutorTab.tsx
View File

@@ -15,14 +15,16 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react"; import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { useStore } from "../../state/store.js"; import { useStore } from "../../state/store.js";
import { editChatClear, editChatTrim, editChatTurn, seedChatTurn } from "../../db/writes.js"; import { editChatClear, editChatTurn, pruneChat, seedChatTurn } from "../../db/writes.js";
import { parseMessage } from "../../domain/gloss.js"; import { parseMessage } from "../../domain/gloss.js";
import { import {
assemblePrompt, assemblePrompt,
gateFor, gateFor,
FOCUS_MODES, FOCUS_MODES,
UNITS,
VOCAB_CAP, VOCAB_CAP,
type BandQuery, type BandQuery,
type FlatUnit,
} from "../../domain/gate.js"; } from "../../domain/gate.js";
import type { FocusMode } from "../../domain/gate.js"; import type { FocusMode } from "../../domain/gate.js";
import { applyProgressReport, currentUnit } from "../../domain/progress.js"; import { applyProgressReport, currentUnit } from "../../domain/progress.js";
@@ -88,6 +90,24 @@ async function readBandWords(db: Db, band: number, ceiling: number): Promise<str
return rows.map((r) => r.headword); return rows.map((r) => r.headword);
} }
/* ── the opening ─────────────────────────────────────────────────── */
/** The app's own first message — the artifact's wording. */
function openingTurn(unit: FlatUnit): string {
const first = unit.id === UNITS[0]!.id;
return [
"반가워요. 저는 선생님이에요 — your reading tutor.",
"",
first
? "We're starting at the beginning, and we'll go one small step at a time. Nothing will turn up in an exercise that I haven't taught you first — if it does, tell me and I'll drop it."
: "Picking up where you left off. Nothing will turn up in an exercise that I haven't taught you first — if it does, tell me and I'll drop it.",
"",
`The plan is six phases, ${UNITS.length} units, ending with you reading a manhwa page at speed. **Roadmap** above shows the whole thing; you can jump anywhere in it whenever you like.`,
"",
`Hit **시작 · Start unit** below, and we'll pick up ${unit.id} ${unit.ko} — ${unit.name}.`,
].join("\n");
}
/* ── the tab ─────────────────────────────────────────────────────── */ /* ── the tab ─────────────────────────────────────────────────────── */
export function TutorTab() { export function TutorTab() {
@@ -216,6 +236,11 @@ export function TutorTab() {
focus: prefs.focus, focus: prefs.focus,
}); });
const history = turns.map((t) => ({ role: t.role, content: t.body })); const history = turns.map((t) => ({ role: t.role, content: t.body }));
// The transcript opens with the app's own assistant turn, and a
// conversation has to open with the learner — the artifact's fix.
if (history[0]?.role === "assistant") {
history.unshift({ role: "user", content: "Let's continue the lesson." });
}
try { try {
const result = await sample( const result = await sample(
@@ -232,7 +257,7 @@ export function TutorTab() {
remounted it once the read returned — the reply visibly remounted it once the read returned — the reply visibly
disappeared and the log jumped by its height each time. */ disappeared and the log jumped by its height each time. */
await editChatTurn(db, "assistant", result.text); await editChatTurn(db, "assistant", result.text);
await editChatTrim(db, KEEP_TURNS); await pruneChat(db, KEEP_TURNS);
const committed = await readTurns(); const committed = await readTurns();
/* All three in one batch. Leaving `busy` set until the finally /* All three in one batch. Leaving `busy` set until the finally
block meant one render where the reply was committed but the block meant one render where the reply was committed but the
@@ -288,7 +313,14 @@ export function TutorTab() {
/* Open the lesson if there is no transcript yet — EXACTLY ONCE. /* Open the lesson if there is no transcript yet — EXACTLY ONCE.
The guard is claimed synchronously, before the first await: setting it The guard is claimed synchronously, before the first await: setting it
after one would let every concurrent run past it, which is precisely how after one would let every concurrent run past it, which is precisely how
this managed to seed the opening turn three times over. */ this managed to seed the opening turn three times over.
The opening is written by the app, not asked of the model, and it is a
seed: unstamped, never dirty, never synced. Booting used to send "Start
unit" at once, so a fresh install wrote a stamped reply and a progress
edit before anyone could configure a server — boot-time writes that
looked like the newest work in the world. Now nothing is written until
the learner presses Start. */
const opened = useRef(false); const opened = useRef(false);
useEffect(() => { useEffect(() => {
if (opened.current) return; if (opened.current) return;
@@ -296,12 +328,10 @@ export function TutorTab() {
void (async () => { void (async () => {
if ((await loadTurns()) > 0) return; // a transcript already exists if ((await loadTurns()) > 0) return; // a transcript already exists
// The opening turn is app-supplied, not a user edit: unstamped. await seedChatTurn(db, "assistant", openingTurn(unit), 0);
await seedChatTurn(db, "user", `Start unit ${unit.id}.`, 0);
await loadTurns(); await loadTurns();
await sendRef.current(`Start unit ${unit.id}.`, { record: false });
})(); })();
// Mount only: `send` and `unit.id` are read through the ref / at mount. // Mount only: `unit` is read at mount.
// eslint-disable-next-line react-hooks/exhaustive-deps // eslint-disable-next-line react-hooks/exhaustive-deps
}, []); }, []);
@@ -313,9 +343,12 @@ export function TutorTab() {
await editChatClear(db); await editChatClear(db);
setRevealed(new Set()); setRevealed(new Set());
setRecent([]); setRecent([]);
await seedChatTurn(db, "assistant", openingTurn(unit), 0);
await loadTurns(); await loadTurns();
await sendRef.current(`Start unit ${unit.id}.`, { record: true }); }, [db, loadTurns, unit]);
}, [db, loadTurns, unit.id]);
/** Nothing asked yet: the opening is all there is, and Start begins it. */
const notStarted = !turns.some((t) => t.role === "user");
/* ── the rail ── */ /* ── the rail ── */
@@ -536,6 +569,21 @@ export function TutorTab() {
</div> </div>
{notStarted && !busy && (
<div className="chat-start">
<button
className="btn primary"
onClick={() =>
void send(
`Let's start unit ${unit.id} ${unit.ko} (${unit.name}). Give me the full introduction, then a first exercise.`,
)
}
>
시작 · Start unit
</button>
</div>
)}
{error && <div className="callout warn chat-error">{error}</div>} {error && <div className="callout warn chat-error">{error}</div>}
<div className="chat-foot" ref={foot}> <div className="chat-foot" ref={foot}>

7
app/src/ui/tutor/tutor.css
View File

@@ -189,3 +189,10 @@
border-top: 1px solid var(--line); border-top: 1px solid var(--line);
margin: 10px 0; margin: 10px 0;
} }
/* Start: the only action on a transcript that is just the app's opening. */
.chat-start {
display: flex;
justify-content: center;
padding: 10px 0 4px;
}

32
server/sql/002-protocol-2.sql Normal file
View File

@@ -0,0 +1,32 @@
-- Sync protocol 2.
--
-- Rows are now written only by compare-and-swap on change_seq (see
-- server/src/db.ts), keyed by a JSON array of the key's values, and every
-- user's data belongs to an epoch: a random id that changes whenever the
-- server's copy is thrown away, so a client can tell it is talking to a
-- server that no longer holds what it pushed and must hydrate from scratch.
CREATE TABLE IF NOT EXISTS sync_meta (
k TEXT PRIMARY KEY,
v TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS sync_epoch (
user_id TEXT PRIMARY KEY,
epoch TEXT NOT NULL
);
-- Protocol 1 rows cannot be read as protocol 2: their keys were joined with
-- a space and every one was ordered by a client's clock. They are dropped
-- once, and each user's epoch with them — so every client re-hydrates and
-- offers its own database back, which is the copy that was always the
-- source of truth. No deployment held protocol 1 data when this shipped.
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM sync_meta WHERE k = 'protocol' AND v = '2') THEN
DELETE FROM sync_row;
DELETE FROM sync_epoch;
INSERT INTO sync_meta (k, v) VALUES ('protocol', '2')
ON CONFLICT (k) DO UPDATE SET v = EXCLUDED.v;
END IF;
END $$;

159
server/src/db.ts
View File

@@ -1,47 +1,89 @@
/* Postgres access, and the two queries sync is made of. */ /* Postgres access, and the two operations sync is made of.
Protocol 2 (shared/sync-protocol.mjs). The server's whole job is order:
it assigns change_seq, it refuses a write made against a stale copy, and
it never lets a pull see half of a push. It does not read inside a row —
it stores and orders them; the client interprets them. */
import { readFile } from "node:fs/promises"; import { readFile } from "node:fs/promises";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import pg from "pg"; import pg from "pg";
export interface SyncRow { /* The wire shapes, as in types/shared/sync-protocol.mjs.d.ts. Restated here
because the server's type check reads the .mjs itself, not its
declarations. */
type Value = string | number | null;
/** A row a client pushes. `base` is the change_seq it last agreed with. */
export interface WireRow {
tbl: string; tbl: string;
data: Record<string, string | number | null>; pk: string;
updated_at: number; base: number;
deleted?: boolean; deleted?: boolean;
data?: Record<string, Value> | null;
updated_at?: number;
}
/** A row as the server holds it. */
export interface ServerRow {
tbl: string;
pk: string;
seq: number;
deleted: boolean;
data: Record<string, Value> | null;
updated_at?: number;
} }
export interface Store { export interface Store {
pull(userId: string, cursor: number, limit: number): Promise<{ rows: SyncRow[]; cursor: number; more: boolean }>; pull(userId: string, cursor: number, limit: number): Promise<{ epoch: string; rows: ServerRow[]; cursor: number; more: boolean }>;
push(userId: string, rows: SyncRow[]): Promise<number>; push(userId: string, rows: WireRow[]): Promise<{ epoch: string; applied: { tbl: string; pk: string; seq: number }[]; conflicts: ServerRow[] }>;
/** Test-only; the route that calls it exists only under HANKAN_TEST_MODE. */ /** Test-only; the route that calls it exists only under HANKAN_TEST_MODE. */
reset(userId: string): Promise<void>; reset(userId: string): Promise<void>;
close(): Promise<void>; close(): Promise<void>;
} }
/** type Query = { query: pg.Pool["query"] };
* The row's primary key as a single string.
* /** This user's epoch, created on first contact. */
* The client sends `{pk}` for a delete and the full row otherwise, so a async function epochOf(db: Query, userId: string): Promise<string> {
* delete already carries its key and a live row needs one derived from the await db.query(
* table's key columns. Both sides use the same rule, from the shared "INSERT INTO sync_epoch (user_id, epoch) VALUES ($1, gen_random_uuid()::text) ON CONFLICT (user_id) DO NOTHING",
* protocol module. [userId],
*/ );
function keyOf(row: SyncRow, pkCols: string[]): string { const { rows } = await db.query("SELECT epoch FROM sync_epoch WHERE user_id = $1", [userId]);
if (row.deleted && row.data.pk != null) return String(row.data.pk); return String(rows[0].epoch);
return pkCols.map((c) => String(row.data[c])).join(" ");
} }
const asServerRow = (r: Record<string, unknown>): ServerRow => ({
tbl: r.tbl as string,
pk: r.pk as string,
seq: Number(r.change_seq),
deleted: Boolean(r.deleted),
data: r.deleted ? null : (r.data as ServerRow["data"]),
updated_at: Number(r.updated_at),
});
export async function openStore(connectionString: string, pkFor: (tbl: string) => string[] | null): Promise<Store> { export async function openStore(connectionString: string, pkFor: (tbl: string) => string[] | null): Promise<Store> {
const pool = new pg.Pool({ connectionString, max: 4 }); const pool = new pg.Pool({ connectionString, max: 4 });
const schema = await readFile(fileURLToPath(new URL("../sql/001-schema.sql", import.meta.url)), "utf8"); for (const file of ["001-schema.sql", "002-protocol-2.sql"]) {
await pool.query(schema); await pool.query(await readFile(fileURLToPath(new URL(`../sql/${file}`, import.meta.url)), "utf8"));
}
return { return {
async pull(userId, cursor, limit) { async pull(userId, cursor, limit) {
const client = await pool.connect();
try {
await client.query("BEGIN");
/* Shared, against push's exclusive lock on the same key. Without it a
push that took seq 10 but committed after another took seq 11 could
be passed over: a pull that saw 11 moves the cursor beyond 10, and
nothing would ever ask for 10 again. */
await client.query("SELECT pg_advisory_xact_lock_shared(hashtext($1))", [userId]);
const epoch = await epochOf(client, userId);
// One extra row tells us whether another page exists without a count. // One extra row tells us whether another page exists without a count.
const { rows } = await pool.query( const { rows } = await client.query(
`SELECT tbl, pk, data, updated_at, deleted, change_seq `SELECT tbl, pk, data, updated_at, deleted, change_seq
FROM sync_row FROM sync_row
WHERE user_id = $1 AND change_seq > $2 WHERE user_id = $1 AND change_seq > $2
@@ -49,51 +91,73 @@ export async function openStore(connectionString: string, pkFor: (tbl: string) =
LIMIT $3`, LIMIT $3`,
[userId, cursor, limit + 1], [userId, cursor, limit + 1],
); );
await client.query("COMMIT");
const more = rows.length > limit; const more = rows.length > limit;
const page = more ? rows.slice(0, limit) : rows; const page = (more ? rows.slice(0, limit) : rows).map(asServerRow);
return { epoch, rows: page, cursor: page.length ? page[page.length - 1]!.seq : cursor, more };
return { } catch (err) {
rows: page.map((r) => ({ await client.query("ROLLBACK");
tbl: r.tbl as string, throw err;
data: r.deleted ? { pk: r.pk as string } : (r.data as Record<string, string | number | null>), } finally {
updated_at: Number(r.updated_at), client.release();
...(r.deleted ? { deleted: true } : {}), }
})),
cursor: page.length ? Number(page[page.length - 1].change_seq) : cursor,
more,
};
}, },
async push(userId, rows) { async push(userId, rows) {
const client = await pool.connect(); const client = await pool.connect();
try { try {
await client.query("BEGIN"); await client.query("BEGIN");
// One push at a time per user: compare-and-swap needs a stable read.
await client.query("SELECT pg_advisory_xact_lock(hashtext($1))", [userId]);
const epoch = await epochOf(client, userId);
const applied: { tbl: string; pk: string; seq: number }[] = [];
const conflicts: ServerRow[] = [];
for (const row of rows) { for (const row of rows) {
const pkCols = pkFor(row.tbl); if (!pkFor(row.tbl) || typeof row.pk !== "string") continue; // a table this server does not know
if (!pkCols) continue; // a table this server does not know about const base = Number(row.base) || 0;
// Last-write-wins, resolved in the database so two devices pushing const { rows: current } = await client.query(
// at once cannot interleave a read and a write around it. `SELECT tbl, pk, data, updated_at, deleted, change_seq FROM sync_row
await client.query( WHERE user_id = $1 AND tbl = $2 AND pk = $3 FOR UPDATE`,
[userId, row.tbl, row.pk],
);
const cur = current[0] as Record<string, unknown> | undefined;
/* Gate 2. The write stands only if the sender last agreed with the
row the server holds now — or neither has ever seen it. Anything
else goes back as a conflict, with the server's copy, and the
client settles it. No clock is consulted anywhere. */
const agreed = cur ? Number(cur.change_seq) === base : base === 0;
if (!agreed) {
conflicts.push(
cur ? asServerRow(cur) : { tbl: row.tbl, pk: row.pk, seq: 0, deleted: true, data: null },
);
continue;
}
const { rows: written } = await client.query(
`INSERT INTO sync_row (user_id, tbl, pk, data, updated_at, deleted) `INSERT INTO sync_row (user_id, tbl, pk, data, updated_at, deleted)
VALUES ($1, $2, $3, $4, $5, $6) VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (user_id, tbl, pk) DO UPDATE ON CONFLICT (user_id, tbl, pk) DO UPDATE
SET data = EXCLUDED.data, SET data = EXCLUDED.data, updated_at = EXCLUDED.updated_at, deleted = EXCLUDED.deleted
updated_at = EXCLUDED.updated_at, RETURNING change_seq`,
deleted = EXCLUDED.deleted [
WHERE EXCLUDED.updated_at > sync_row.updated_at`, userId,
[userId, row.tbl, keyOf(row, pkCols), JSON.stringify(row.data), row.updated_at, row.deleted ?? false], row.tbl,
row.pk,
JSON.stringify(row.deleted ? {} : (row.data ?? {})),
Number(row.updated_at) || 0,
Boolean(row.deleted),
],
); );
applied.push({ tbl: row.tbl, pk: row.pk, seq: Number(written[0].change_seq) });
} }
const { rows: top } = await client.query(
"SELECT COALESCE(max(change_seq), 0) AS c FROM sync_row WHERE user_id = $1",
[userId],
);
await client.query("COMMIT"); await client.query("COMMIT");
return Number(top[0].c); return { epoch, applied, conflicts };
} catch (err) { } catch (err) {
await client.query("ROLLBACK"); await client.query("ROLLBACK");
throw err; throw err;
@@ -104,6 +168,7 @@ export async function openStore(connectionString: string, pkFor: (tbl: string) =
async reset(userId) { async reset(userId) {
await pool.query("DELETE FROM sync_row WHERE user_id = $1", [userId]); await pool.query("DELETE FROM sync_row WHERE user_id = $1", [userId]);
await pool.query("DELETE FROM sync_epoch WHERE user_id = $1", [userId]);
}, },
async close() { async close() {

23
server/src/main.ts
View File

@@ -7,8 +7,8 @@
import { serve } from "@hono/node-server"; import { serve } from "@hono/node-server";
import { Hono } from "hono"; import { Hono } from "hono";
import { pkFor, PAGE_SIZE } from "../../shared/sync-protocol.mjs"; import { pkFor, PAGE_SIZE, PROTOCOL, PROTOCOL_HEADER } from "../../shared/sync-protocol.mjs";
import { openStore, type SyncRow } from "./db.ts"; import { openStore, type WireRow } from "./db.ts";
import { tutorRoute } from "./tutor.ts"; import { tutorRoute } from "./tutor.ts";
const PORT = Number(process.env.PORT ?? 8787); const PORT = Number(process.env.PORT ?? 8787);
@@ -71,7 +71,7 @@ app.use("*", async (c, next) => {
return c.body(null, 204, { return c.body(null, 204, {
"Access-Control-Allow-Origin": allow, "Access-Control-Allow-Origin": allow,
"Access-Control-Allow-Methods": "GET, POST, OPTIONS", "Access-Control-Allow-Methods": "GET, POST, OPTIONS",
"Access-Control-Allow-Headers": "authorization, content-type", "Access-Control-Allow-Headers": `authorization, content-type, ${PROTOCOL_HEADER}`,
"Access-Control-Max-Age": "86400", "Access-Control-Max-Age": "86400",
Vary: "Origin", Vary: "Origin",
}); });
@@ -102,6 +102,17 @@ app.get("/health", (c) => c.json({ ok: true }));
/* ── sync ────────────────────────────────────────────────────────────── */ /* ── sync ────────────────────────────────────────────────────────────── */
/* Refuse any client not speaking this protocol. A client on protocol 1 is
exactly the device the artifact lost data to — an old build that pushes
before it pulls and trusts its own clock — so it gets a clear 426 rather
than a half-understood write. */
app.use("/api/sync", async (c, next) => {
if (c.req.header(PROTOCOL_HEADER) !== String(PROTOCOL)) {
return c.json({ error: "this server speaks sync protocol 2 — update the app", protocol: PROTOCOL }, 426);
}
await next();
});
app.get("/api/sync", async (c) => { app.get("/api/sync", async (c) => {
const cursor = Number(c.req.query("cursor") ?? 0); const cursor = Number(c.req.query("cursor") ?? 0);
if (!Number.isFinite(cursor) || cursor < 0) return c.json({ error: "bad cursor" }, 400); if (!Number.isFinite(cursor) || cursor < 0) return c.json({ error: "bad cursor" }, 400);
@@ -109,10 +120,10 @@ app.get("/api/sync", async (c) => {
}); });
app.post("/api/sync", async (c) => { app.post("/api/sync", async (c) => {
const body = (await c.req.json()) as { rows?: SyncRow[] }; const body = (await c.req.json()) as { rows?: WireRow[] };
const rows = Array.isArray(body.rows) ? body.rows : []; const rows = Array.isArray(body.rows) ? body.rows : [];
if (rows.length > PAGE_SIZE * 4) return c.json({ error: "too many rows" }, 413); if (rows.length > PAGE_SIZE) return c.json({ error: "too many rows" }, 413);
return c.json({ cursor: await store.push(USER_ID, rows) }); return c.json(await store.push(USER_ID, rows));
}); });
/* Wipes this user's rows. Exists only when HANKAN_TEST_MODE is set, so it /* Wipes this user's rows. Exists only when HANKAN_TEST_MODE is set, so it

142
shared/sync-protocol.mjs
View File

@@ -1,18 +1,41 @@
/* The sync wire format, defined once and imported by both sides. /* The sync wire format, defined once and imported by both sides.
Row-level, last-write-wins on `updated_at`, cursor-based on a Protocol 2. Row-level and cursor-based on a server-assigned change_seq,
server-assigned `change_seq`. One user, so the loser of a conflict is at as PORT.md specifies — and ordered by that counter, never by a clock.
worst one SRS grade — LWW is adequate, and a merge policy would be
over-engineering.
THE RULE THAT MATTERS: seeded and defaulted rows carry updated_at = 0. ════ THE THREE GATES ════════════════════════════════════════════════
The artifact's sync bug was a fresh device stamping its own empty state
as newer than the server's real history and clobbering it. Here a seed
row can never be dirty (0 is never greater than any watermark) and can
never win a conflict (0 is never greater than any timestamp). The bug is
not avoided, it is unrepresentable. */
/** Tables that sync, and the columns forming each primary key. */ The artifact lost a week of a real user's work without these, and the
port's first sync broke all three. In short (lib/sync.js has the long
version, applied there to whole documents):
1. HYDRATION. A device pulls everything the server holds before it may
push anything. A laptop with a week-old copy used to boot, re-stamp
that copy through a migration, and push it over the phone's week.
2. A COUNTER, NOT A CLOCK. Every row remembers the change_seq it last
agreed with the server on (base_seq). The server applies a write only
if that is still the row's current change_seq — compare-and-swap —
and otherwise hands the current row back as a conflict. A phone clock
four minutes fast used to decide which of two grades survived.
3. NO SILENT SHRINKING. A conflict is settled by what each copy HOLDS,
not by which is newer (see app/src/sync/resolve.ts). Emptying things
on purpose is explicit: a reset or a cleared lesson writes a marker
every other device obeys, so the shrink is declared, never raced.
Seeded and defaulted rows are never dirty, so they never leave the
device: a fresh install cannot claim its empty defaults are news. */
export const PROTOCOL = 2;
/** Header the client sends; the server refuses any other protocol. */
export const PROTOCOL_HEADER = "x-hankan-protocol";
/** Maximum rows in one push or pull page. */
export const PAGE_SIZE = 500;
/** Tables that sync: their primary-key columns and the columns that travel. */
export const SYNC_TABLES = { export const SYNC_TABLES = {
card: { card: {
pk: ["lemma_id"], pk: ["lemma_id"],
@@ -21,8 +44,31 @@ export const SYNC_TABLES = {
progress: { pk: ["unit_id"], cols: ["unit_id", "done", "confidence", "answers", "note"] }, progress: { pk: ["unit_id"], cols: ["unit_id", "done", "confidence", "answers", "note"] },
chat: { pk: ["id"], cols: ["id", "role", "body", "created_at"] }, chat: { pk: ["id"], cols: ["id", "role", "body", "created_at"] },
meta: { pk: ["k"], cols: ["k", "v"] }, meta: { pk: ["k"], cols: ["k", "v"] },
study_log: { pk: ["day"], cols: ["day", "reviews", "correct", "drills"] }, /* Counters, kept per device and summed on read. Two devices each adding
peek: { pk: ["form"], cols: ["form", "count"] }, a review to the same day would otherwise both write "reviews = n+1",
and whichever landed second would erase the other's review. A device
only ever writes its own row, so these never conflict at all. */
study_log: { pk: ["day", "device"], cols: ["day", "device", "reviews", "correct", "drills"] },
peek: { pk: ["form", "device"], cols: ["form", "device", "count"] },
custom_word: { pk: ["headword", "pos"], cols: ["headword", "pos", "gloss"] },
evidence: {
pk: ["word"],
cols: [
"word",
"ok",
"wrong",
"lookups",
"streak",
"first_round",
"last_round",
"last_seen",
"rounds",
"first_ok_round",
"last_ok_round",
],
},
confusion: { pk: ["word"], cols: ["word", "mistook"] },
phase_ledger: { pk: ["phase", "kind", "item"], cols: ["phase", "kind", "item", "confirmed"] },
}; };
export const SYNC_TABLE_NAMES = Object.keys(SYNC_TABLES); export const SYNC_TABLE_NAMES = Object.keys(SYNC_TABLES);
@@ -31,39 +77,67 @@ export const SYNC_TABLE_NAMES = Object.keys(SYNC_TABLES);
export const pkFor = (tbl) => export const pkFor = (tbl) =>
Object.prototype.hasOwnProperty.call(SYNC_TABLES, tbl) ? SYNC_TABLES[tbl].pk : null; Object.prototype.hasOwnProperty.call(SYNC_TABLES, tbl) ? SYNC_TABLES[tbl].pk : null;
/**
* A row's key on the wire: its primary-key values as a JSON array.
*
* Protocol 1 joined them with a space and split them back on one, so a key
* containing a space — the curriculum's 몇 명 and 신경 쓰다 can be peek
* forms — came back as two values and the statement binding them threw.
* The pull stopped at that row, on every device, permanently.
*/
export const encodePk = (tbl, row) => JSON.stringify(SYNC_TABLES[tbl].pk.map((c) => row[c]));
export const decodePk = (pk) => JSON.parse(pk);
/** /**
* `meta` mixes the learner's data with bookkeeping that describes one * `meta` mixes the learner's data with bookkeeping that describes one
* install. Only the former may cross the wire. * install. Only the former may cross the wire.
* *
* `dict.loadedBands` is the dangerous one: replicating it would tell a * `dict.*` is the dangerous one: replicating the loaded-band record would
* phone that had loaded bands 0-2 that it holds every row the desktop has, * tell a phone it holds rows it never downloaded. `schema_version` would be
* and the word rail would then fail to find words it believes are present. * worse — a device could be told it has run a migration it has not.
* `schema_version` would be worse — a device could be told it has run a * `server.*` holds this device's endpoint and bearer token, and `sync.*`
* migration it has not. `server.*` holds this device's endpoint and bearer * is this device's own position in the conversation with the server.
* token: syncing a token through the endpoint it authenticates would be
* circular, and a base URL is network-specific.
*/ */
const SYNCABLE_META_EXACT = new Set(["grammar.learned", "grammar.notes", "trainer.conjugation"]); const SYNCABLE_META_EXACT = new Set(["grammar.learned", "grammar.notes", "trainer.conjugation"]);
const SYNCABLE_META_PREFIX = ["prefs."]; const SYNCABLE_META_PREFIX = ["prefs.", "road.", "learner.", "reset."];
export function isSyncableMetaKey(key) { export function isSyncableMetaKey(key) {
if (SYNCABLE_META_EXACT.has(key)) return true; if (SYNCABLE_META_EXACT.has(key)) return true;
return SYNCABLE_META_PREFIX.some((p) => key.startsWith(p)); return SYNCABLE_META_PREFIX.some((p) => key.startsWith(p));
} }
/** Stable string key for a row, used to match it across devices. */
export const rowKey = (table, row) => SYNC_TABLES[table].pk.map((c) => String(row[c])).join(" ");
/** Rows a device may send: dirty since its last successful push. */
export const isDirty = (row, pushedAt) => Number(row.updated_at) > Number(pushedAt);
/** /**
* Last-write-wins. Strictly greater, so equal timestamps leave the local row * What each deliberate shrink empties. A marker meta key, `reset.<scope>`,
* alone — a tie means both sides already agree, or the clocks are close * holds a counter; a device that sees it rise empties the same things
* enough that flapping would be worse than either outcome. * locally — including its own unsynced edits, because a reset it had not
* yet heard of still wins over work done on the old data.
*/ */
export const incomingWins = (incomingUpdatedAt, localUpdatedAt) => export const RESET_SCOPES = {
Number(incomingUpdatedAt) > Number(localUpdatedAt ?? -1); chat: { tables: ["chat"], meta: [] },
roadmap: { tables: ["progress", "chat", "phase_ledger"], meta: ["road."] },
everything: {
tables: [
"progress",
"chat",
"phase_ledger",
"card",
"study_log",
"peek",
"custom_word",
"evidence",
"confusion",
],
meta: ["road.", "prefs.", "grammar.", "trainer.", "learner."],
},
};
/** Maximum rows in one push or pull page. */ /** Does a reset of this scope empty this row? */
export const PAGE_SIZE = 500; export function resetCovers(scope, tbl, row) {
const s = RESET_SCOPES[scope];
if (!s) return false;
if (tbl === "meta") {
const k = String(row.k ?? "");
return !k.startsWith("reset.") && s.meta.some((p) => k.startsWith(p));
}
return s.tables.includes(tbl);
}

39
test/db/conformance.ts
View File

@@ -14,6 +14,7 @@ import {
seedMeta, seedMeta,
seedProgress, seedProgress,
editCard, editCard,
editCardReset,
editCurrentUnit, editCurrentUnit,
editMeta, editMeta,
editPeek, editPeek,
@@ -201,6 +202,44 @@ export function conformanceSuite(name: string, open: () => Promise<Db>): void {
newer than the server's real history. newer than the server's real history.
═══════════════════════════════════════════════════════════════ */ ═══════════════════════════════════════════════════════════════ */
describe("seeded state carries no write timestamp", () => { describe("seeded state carries no write timestamp", () => {
it("leaves nothing dirty after a full seed, so nothing seeded ever travels", async () => {
await seedProgress(db, "1.1");
await seedMeta(db, "prefs.newPerDay", "10");
await db.run(
"INSERT INTO lemma (id, headword, pos, gloss_en, source) VALUES (1,'밥','noun','rice','curated')",
);
await seedCard(db, 1, newCard());
await seedChatTurn(db, "assistant", "안녕! 시작하자.", 0);
for (const table of SYNCABLE) {
const row = await db.get<{ n: number }>(`SELECT count(*) AS n FROM ${table} WHERE dirty != 0`);
expect(row?.n, `${table} has a dirty seed row`).toBe(0);
}
});
it("marks every genuine edit dirty and bumps its revision each time", async () => {
await db.run(
"INSERT INTO lemma (id, headword, pos, gloss_en, source) VALUES (3,'책','noun','book','curated')",
);
await editCard(db, 3, newCard());
await editCard(db, 3, grade(newCard(), GOOD, 20_000));
expect(await db.get("SELECT dirty, rev FROM card WHERE lemma_id = 3")).toEqual({ dirty: 1, rev: 2 });
});
it("carries a deleted row's base_seq into its tombstone", async () => {
await db.run(
"INSERT INTO lemma (id, headword, pos, gloss_en, source) VALUES (4,'꽃','noun','flower','curated')",
);
await editCard(db, 4, newCard());
await db.run("UPDATE card SET base_seq = 77, dirty = 0 WHERE lemma_id = 4");
await editCardReset(db, 4);
expect(await db.get("SELECT pk, base_seq, dirty FROM tombstone WHERE tbl = 'card'")).toEqual({
pk: "[4]",
base_seq: 77,
dirty: 1,
});
});
it("leaves updated_at at 0 across every syncable table after a full seed", async () => { it("leaves updated_at at 0 across every syncable table after a full seed", async () => {
await seedProgress(db, "1.1"); await seedProgress(db, "1.1");
await seedMeta(db, "focus", "auto"); await seedMeta(db, "focus", "auto");

3
test/db/migration-6.test.ts
View File

@@ -57,7 +57,8 @@ describe("migration 6", () => {
const graves = await db.all<{ pk: string; updated_at: number }>( const graves = await db.all<{ pk: string; updated_at: number }>(
"SELECT pk, updated_at FROM tombstone WHERE tbl = 'card'", "SELECT pk, updated_at FROM tombstone WHERE tbl = 'card'",
); );
expect(graves).toEqual([{ pk: String(lemmaId("책", "noun")), updated_at: 333 }]); // As JSON, once migration 9 has written keys in protocol 2's form.
expect(graves).toEqual([{ pk: JSON.stringify([lemmaId("책", "noun")]), updated_at: 333 }]);
}); });
it("turns a custom lemma into a custom_word, stamped when its card was made", async () => { it("turns a custom lemma into a custom_word, stamped when its card was made", async () => {

3
test/db/migration-8.test.ts
View File

@@ -60,7 +60,8 @@ describe("migration 8", () => {
it("renames a chat tombstone the same way", async () => { it("renames a chat tombstone the same way", async () => {
const device = (await db.get<{ v: string }>("SELECT v FROM meta WHERE k = 'sync.device'"))!.v; const device = (await db.get<{ v: string }>("SELECT v FROM meta WHERE k = 'sync.device'"))!.v;
expect(await db.all("SELECT pk, updated_at FROM tombstone WHERE tbl = 'chat'")).toEqual([ expect(await db.all("SELECT pk, updated_at FROM tombstone WHERE tbl = 'chat'")).toEqual([
{ pk: `legacy:${device}:0000000009`, updated_at: 400 }, // As JSON, once migration 9 has written keys in protocol 2's form.
{ pk: JSON.stringify([`legacy:${device}:0000000009`]), updated_at: 400 },
]); ]);
}); });

71
test/db/migration-9.test.ts Normal file
View File

@@ -0,0 +1,71 @@
/* Migration 9 — sync protocol 2's columns, per-device counters, JSON keys.
A database as migration 8 left it, carrying protocol 1's bookkeeping and
a tombstone of every shape protocol 1 wrote. */
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { SqliteWasmDb } from "@app/db/sqlite-wasm-core.js";
import { migrate } from "@app/db/migrate.js";
import type { Db } from "@app/db/types.js";
let db: Db;
let device: string;
beforeEach(async () => {
db = await SqliteWasmDb.open({ memory: true });
await migrate(db, 8);
device = (await db.get<{ v: string }>("SELECT v FROM meta WHERE k = 'sync.device'"))!.v;
await db.exec(`
INSERT INTO study_log (day, reviews, correct, drills, updated_at) VALUES (20000, 5, 4, 1, 700);
INSERT INTO peek (form, count, updated_at) VALUES ('몇 명', 3, 800);
INSERT INTO card (lemma_id, state, reps, updated_at) VALUES (55855054575946, 1, 2, 900);
INSERT INTO meta (k, v, updated_at) VALUES ('sync.cursor', '41', 0), ('sync.pushedAt', '900', 0);
INSERT INTO tombstone (tbl, pk, updated_at) VALUES
('card', '1434356355562999', 10),
('study_log', '19999', 11),
('peek', '몇 명', 12),
('meta', 'prefs.goal', 13),
('custom_word', '["던전","noun"]', 14);
`);
await migrate(db);
});
afterEach(async () => {
await db.close();
});
describe("migration 9", () => {
it("gives every syncable row base_seq, dirty and rev, all zero", async () => {
expect(await db.get("SELECT base_seq, dirty, rev, updated_at FROM card")).toEqual({
base_seq: 0,
dirty: 0,
rev: 0,
updated_at: 900,
});
});
it("hands this install's counters to this install", async () => {
expect(await db.get("SELECT day, device, reviews, correct, drills, updated_at FROM study_log")).toEqual({
day: 20000,
device,
reviews: 5,
correct: 4,
drills: 1,
updated_at: 700,
});
expect(await db.get("SELECT form, device, count FROM peek")).toEqual({ form: "몇 명", device, count: 3 });
});
it("writes every tombstone key as a JSON array, a space included", async () => {
const graves = await db.all<{ tbl: string; pk: string }>("SELECT tbl, pk FROM tombstone ORDER BY updated_at");
expect(graves).toEqual([
{ tbl: "card", pk: "[1434356355562999]" },
{ tbl: "study_log", pk: JSON.stringify([19999, device]) },
{ tbl: "peek", pk: JSON.stringify(["몇 명", device]) },
{ tbl: "meta", pk: '["prefs.goal"]' },
{ tbl: "custom_word", pk: '["던전","noun"]' },
]);
});
it("drops protocol 1's position, so the next sync hydrates from scratch", async () => {
expect(await db.all("SELECT k FROM meta WHERE k IN ('sync.cursor', 'sync.pushedAt')")).toEqual([]);
});
});

7
test/server/http.test.ts
View File

@@ -27,13 +27,16 @@ run("CORS", () => {
headers: { headers: {
origin: PHONE_ORIGIN, origin: PHONE_ORIGIN,
"access-control-request-method": "POST", "access-control-request-method": "POST",
"access-control-request-headers": "authorization,content-type", "access-control-request-headers": "authorization,content-type,x-hankan-protocol",
}, },
}); });
expect(res.status).not.toBe(401); expect(res.status).not.toBe(401);
expect(res.status).toBeLessThan(300); expect(res.status).toBeLessThan(300);
expect(res.headers.get("access-control-allow-origin")).toBe(PHONE_ORIGIN); expect(res.headers.get("access-control-allow-origin")).toBe(PHONE_ORIGIN);
expect(res.headers.get("access-control-allow-headers")).toMatch(/authorization/i); expect(res.headers.get("access-control-allow-headers")).toMatch(/authorization/i);
// Without this the phone's browser refuses to send the protocol header,
// and every sync from it is answered 426.
expect(res.headers.get("access-control-allow-headers")).toMatch(/x-hankan-protocol/i);
}); });
it("allows the Authorization header, without which the token cannot be sent", async () => { it("allows the Authorization header, without which the token cannot be sent", async () => {
@@ -46,7 +49,7 @@ run("CORS", () => {
it("puts the allow-origin header on the real response too", async () => { it("puts the allow-origin header on the real response too", async () => {
const res = await fetch(`${BASE}/api/sync?cursor=0`, { const res = await fetch(`${BASE}/api/sync?cursor=0`, {
headers: { origin: PHONE_ORIGIN, authorization: `Bearer ${TOKEN}` }, headers: { origin: PHONE_ORIGIN, authorization: `Bearer ${TOKEN}`, "x-hankan-protocol": "2" },
}); });
expect(res.status).toBe(200); expect(res.status).toBe(200);
expect(res.headers.get("access-control-allow-origin")).toBe(PHONE_ORIGIN); expect(res.headers.get("access-control-allow-origin")).toBe(PHONE_ORIGIN);

118
test/sync/resolve.test.ts Normal file
View File

@@ -0,0 +1,118 @@
/* Gate 3 — what happens when two copies of one row disagree.
The rule is PORT.md's: the copy that holds more wins, a tie goes to the
server so every device lands on one copy, and a deliberate delete is
obeyed. These pin each table's reading of "more". */
import { describe, it, expect } from "vitest";
import { resolve, type Resolution, type Side } from "@app/sync/resolve.js";
/** The merged meta value, parsed. */
const mergedValue = (r: Resolution): unknown => {
if (r.kind !== "merge") throw new Error(`expected a merge, got ${r.kind}`);
return JSON.parse(String(r.data.v));
};
const ctx = { unitIndex: (id: string) => ["1.1", "1.2", "1.3", "1.10", "2.1"].indexOf(id) };
const live = (data: Record<string, string | number | null>): Side => ({ deleted: false, data });
const gone: Side = { deleted: true, data: null };
describe("deletes", () => {
it("obeys a delete from the server over an edit made here", () => {
expect(resolve("card", live({ reps: 9 }), gone, ctx)).toEqual({ kind: "adopt" });
});
it("keeps a delete made here over an edit on the server", () => {
expect(resolve("card", gone, live({ reps: 9 }), ctx)).toEqual({ kind: "keep" });
});
});
describe("cards and evidence — more work wins", () => {
it("keeps the card with more reviews behind it", () => {
expect(resolve("card", live({ reps: 6, lapses: 1 }), live({ reps: 5, lapses: 0 }), ctx)).toEqual({ kind: "keep" });
expect(resolve("card", live({ reps: 2, lapses: 0 }), live({ reps: 5, lapses: 0 }), ctx)).toEqual({ kind: "adopt" });
});
it("gives a tie to the server", () => {
expect(resolve("card", live({ reps: 5, lapses: 0 }), live({ reps: 5, lapses: 0 }), ctx)).toEqual({ kind: "adopt" });
});
it("weighs evidence by everything observed, lookups included", () => {
expect(
resolve("evidence", live({ ok: 2, wrong: 1, lookups: 2 }), live({ ok: 3, wrong: 0, lookups: 0 }), ctx),
).toEqual({ kind: "keep" });
});
});
describe("progress — merged, never un-finished", () => {
it("keeps a unit finished if either copy finished it", () => {
const r = resolve(
"progress",
live({ unit_id: "1.1", done: 1, confidence: 60, answers: 4, note: "mine" }),
live({ unit_id: "1.1", done: 0, confidence: 90, answers: 7, note: "theirs" }),
ctx,
);
expect(r).toEqual({
kind: "merge",
data: { unit_id: "1.1", done: 1, confidence: 90, answers: 7, note: "theirs" },
});
});
it("takes confidence from the copy with more answers behind it", () => {
expect(
resolve(
"progress",
live({ unit_id: "1.1", done: 0, confidence: 95, answers: 9, note: "a" }),
live({ unit_id: "1.1", done: 0, confidence: 40, answers: 3, note: "b" }),
ctx,
),
).toEqual({ kind: "keep" });
});
});
describe("meta", () => {
it("puts him on the further unit, by roadmap order and not by string", () => {
// "1.10" sorts before "1.2" as text; on the roadmap it comes after.
expect(resolve("meta", live({ k: "road.unit", v: "1.10" }), live({ k: "road.unit", v: "1.2" }), ctx)).toEqual({
kind: "keep",
});
});
it("never moves a counter backwards", () => {
expect(resolve("meta", live({ k: "learner.round", v: "12" }), live({ k: "learner.round", v: "9" }), ctx)).toEqual({
kind: "keep",
});
expect(resolve("meta", live({ k: "reset.chat", v: "1" }), live({ k: "reset.chat", v: "2" }), ctx)).toEqual({
kind: "adopt",
});
});
it("unites grammar flags learned on two devices", () => {
const r = resolve(
"meta",
live({ k: "grammar.learned", v: JSON.stringify({ a: true }) }),
live({ k: "grammar.learned", v: JSON.stringify({ b: true }) }),
ctx,
);
expect(mergedValue(r)).toEqual({ a: true, b: true });
});
it("keeps the longer of two versions of one note", () => {
const r = resolve(
"meta",
live({ k: "grammar.notes", v: JSON.stringify({ p1: "a long careful note" }) }),
live({ k: "grammar.notes", v: JSON.stringify({ p1: "short", p2: "other" }) }),
ctx,
);
expect(mergedValue(r)).toEqual({
p1: "a long careful note",
p2: "other",
});
});
it("takes the server's copy of a preference", () => {
expect(resolve("meta", live({ k: "prefs.goal", v: "40" }), live({ k: "prefs.goal", v: "20" }), ctx)).toEqual({
kind: "adopt",
});
});
});

418
test/sync/roundtrip.test.ts
View File

@@ -1,21 +1,21 @@
/* Two devices, one server. /* Two devices, one server — protocol 2, and the three gates.
Runs against a real Postgres and a real server — a fake would not exercise Runs against a real Postgres and a real server: a fake would not exercise
the trigger that advances change_seq, the LWW clause in the upsert, or the compare-and-swap, the advisory locks or the change_seq trigger, which
the cursor paging, which is where sync actually goes wrong. is where sync actually goes wrong.
Skipped unless HANKAN_TEST_SERVER is set, so the suite still runs on a Skipped unless HANKAN_TEST_SERVER is set, so the suite still runs on a
machine without Docker: machine without Docker:
docker run -d --name hankan-pg-test -e POSTGRES_PASSWORD=test \ docker run -d --rm --name hankan-pg-test -e POSTGRES_PASSWORD=test \
-e POSTGRES_DB=hankan -p 55432:5432 postgres:16-alpine -e POSTGRES_DB=hankan -p 127.0.0.1:55433:5432 postgres:16-alpine
DATABASE_URL=postgres://postgres:test@localhost:55432/hankan \ DATABASE_URL=postgres://postgres:test@127.0.0.1:55433/hankan \
HANKAN_TOKEN=test-token PORT=8788 \ HANKAN_TOKEN=test-token HANKAN_TEST_MODE=1 HANKAN_TUTOR_BACKEND=echo \
node --experimental-strip-types server/src/main.ts PORT=8788 node --experimental-strip-types server/src/main.ts
HANKAN_TEST_SERVER=http://localhost:8788 npm test HANKAN_TEST_SERVER=http://localhost:8788 npm test
*/ */
import { describe, it, expect, beforeAll, beforeEach } from "vitest"; import { describe, it, expect, beforeAll, beforeEach, afterEach, vi } from "vitest";
import { SqliteWasmDb } from "@app/db/sqlite-wasm-core.js"; import { SqliteWasmDb } from "@app/db/sqlite-wasm-core.js";
import { migrate } from "@app/db/migrate.js"; import { migrate } from "@app/db/migrate.js";
@@ -23,190 +23,326 @@ import type { Db } from "@app/db/types.js";
import { import {
editCard, editCard,
editCardReset, editCardReset,
editChatClear,
editChatTurn,
editMeta, editMeta,
editPeek,
editReset,
editStudyLog,
editUnitConfidence, editUnitConfidence,
editUnitDone,
seedCard, seedCard,
seedMeta, seedMeta,
seedProgress, seedProgress,
} from "@app/db/writes.js"; } from "@app/db/writes.js";
import { syncOnce, type SyncConfig } from "@app/sync/client.js"; import { syncOnce, type SyncConfig } from "@app/sync/client.js";
import { unitIndex } from "@app/domain/gate.js";
import { newCard } from "@lib/srs.js"; import { newCard } from "@lib/srs.js";
import { lemmaId } from "@shared/lemma-id.mjs";
import { PROTOCOL_HEADER } from "@shared/sync-protocol.mjs";
const BASE = process.env.HANKAN_TEST_SERVER; const BASE = process.env.HANKAN_TEST_SERVER;
const TOKEN = process.env.HANKAN_TEST_TOKEN ?? "test-token"; const TOKEN = process.env.HANKAN_TEST_TOKEN ?? "test-token";
const suite = BASE ? describe : describe.skip; const suite = BASE ? describe : describe.skip;
/** A fresh device: its own database, migrated, with the dictionary rows it const RICE = lemmaId("밥", "noun");
needs to hang cards off. */ const SCHOOL = lemmaId("학교", "noun");
const WATER = lemmaId("물", "noun");
/** A fresh device: its own database, migrated, with lemmas to hang cards on. */
async function device(): Promise<Db> { async function device(): Promise<Db> {
const db = await SqliteWasmDb.open({ memory: true }); const db = await SqliteWasmDb.open({ memory: true });
await migrate(db); await migrate(db);
await db.exec(` await db.run(
INSERT INTO lemma (id, headword, pos, gloss_en, source) VALUES `INSERT INTO lemma (id, headword, pos, gloss_en, source) VALUES
(1, '밥', 'noun', 'rice', 'curated'), (?, '밥', 'noun', 'rice', 'curated'), (?, '학교', 'noun', 'school', 'curated'),
(2, '학교', 'noun', 'school', 'curated'), (?, '물', 'noun', 'water', 'curated')`,
(3, '물', 'noun', 'water', 'curated'); [RICE, SCHOOL, WATER],
`); );
return db; return db;
} }
/** Wipe the server between tests so each starts from an empty history. */ const one = async <T>(db: Db, sql: string, params: (string | number)[] = []) => db.get<T>(sql, params);
async function resetServer(cfg: SyncConfig): Promise<void> { const turns = async (db: Db) =>
await fetch(`${cfg.baseUrl}/api/test/reset`, { (await db.all<{ body: string }>("SELECT body FROM chat ORDER BY created_at, id")).map((t) => t.body);
method: "POST",
headers: { authorization: `Bearer ${cfg.token}` },
});
}
suite("sync — two devices, one server", () => { suite("sync — protocol 2, two devices, one server", () => {
let cfg: SyncConfig; let cfg: SyncConfig;
const sync = (db: Db) => syncOnce(db, cfg, { unitIndex });
const open: Db[] = [];
const make = async () => {
const db = await device();
open.push(db);
return db;
};
beforeAll(() => { beforeAll(() => {
cfg = { baseUrl: BASE!, token: TOKEN }; cfg = { baseUrl: BASE!, token: TOKEN };
}); });
beforeEach(async () => { beforeEach(async () => {
await resetServer(cfg); await fetch(`${cfg.baseUrl}/api/test/reset`, {
method: "POST",
headers: { authorization: `Bearer ${cfg.token}` },
});
});
afterEach(async () => {
vi.restoreAllMocks();
while (open.length) await open.pop()!.close();
}); });
it("carries an edit from one device to the other", async () => { it("carries an edit from one device to the other", async () => {
const a = await device(); const a = await make();
const b = await device(); const b = await make();
await editUnitConfidence(a, "1.1", 64); await editUnitConfidence(a, "1.1", 64);
await syncOnce(a, cfg); await sync(a);
await syncOnce(b, cfg); await sync(b);
expect(await one(b, "SELECT confidence FROM progress WHERE unit_id = '1.1'")).toEqual({ confidence: 64 });
const row = await b.get<{ confidence: number }>(
"SELECT confidence FROM progress WHERE unit_id = '1.1'",
);
expect(row?.confidence).toBe(64);
await a.close();
await b.close();
}); });
/* THE ARTIFACT'S BUG, as an executable test. /* The artifact's first bug: a fresh device's defaults clobbering real
progress. Seeds are never dirty, so they never leave the device. */
A fresh device stamped its own empty defaults as newer than the it("never lets a fresh device's seeded state overwrite real progress", async () => {
server's real history and clobbered it. Here the seed rows carry const a = await make();
updated_at = 0, so they are never dirty and never win — the device
converges onto the server's data instead of destroying it. */
it("a fresh device's seeded state never overwrites real progress", async () => {
const a = await device();
await editUnitConfidence(a, "1.1", 88); await editUnitConfidence(a, "1.1", 88);
await editCard(a, 1, { ...newCard(), state: 2, interval: 21, due: 20_050, reps: 9 }); await editMeta(a, "road.unit", "1.3");
await syncOnce(a, cfg); await editCard(a, RICE, { ...newCard(), state: 2, interval: 21, due: 20_050, reps: 9 });
await sync(a);
// A brand-new device does exactly what boot does: seeds defaults. const fresh = await make();
const fresh = await device();
await seedProgress(fresh, "1.1"); await seedProgress(fresh, "1.1");
await seedMeta(fresh, "prefs.newPerDay", "10"); await seedMeta(fresh, "prefs.newPerDay", "10");
await seedCard(fresh, 1, newCard()); await seedCard(fresh, RICE, newCard());
const result = await sync(fresh);
expect(result.pushed, "a seed is not news").toBe(0);
const seeded = await fresh.get<{ n: number }>( expect(await one(fresh, "SELECT v FROM meta WHERE k = 'road.unit'")).toEqual({ v: "1.3" });
"SELECT count(*) AS n FROM progress WHERE updated_at != 0", expect(await one(fresh, "SELECT interval FROM card WHERE lemma_id = ?", [RICE])).toEqual({ interval: 21 });
); await sync(a);
expect(seeded?.n, "seed rows must carry no write timestamp").toBe(0); expect(await one(a, "SELECT confidence FROM progress WHERE unit_id = '1.1'")).toEqual({ confidence: 88 });
await syncOnce(fresh, cfg);
// The fresh device adopts the real data…
const onFresh = await fresh.get<{ confidence: number }>(
"SELECT confidence FROM progress WHERE unit_id = '1.1'",
);
expect(onFresh?.confidence).toBe(88);
// …and the original device still has it after a round trip.
await syncOnce(a, cfg);
const onA = await a.get<{ confidence: number }>(
"SELECT confidence FROM progress WHERE unit_id = '1.1'",
);
expect(onA?.confidence, "the server's real history survived").toBe(88);
const card = await a.get<{ interval: number }>("SELECT interval FROM card WHERE lemma_id = 1");
expect(card?.interval).toBe(21);
await a.close();
await fresh.close();
}); });
it("last write wins on a genuine conflict", async () => { /* 14 Sep. A laptop with a week-old copy booted, re-stamped that copy, and
const a = await device(); pushed it over a week of phone work. Gate 1: it pulls first, so the
const b = await device(); phone's week arrives before anything leaves the laptop — and the laptop's
stale re-stamp meets the week as a conflict, which the week wins. */
it("hydrates before it pushes: a week-old laptop cannot overwrite a week of work", async () => {
const laptop = await make();
const phone = await make();
await editUnitConfidence(laptop, "1.1", 40);
await sync(laptop);
await sync(phone);
await editUnitConfidence(a, "2.1", 30); // A week on the phone.
await syncOnce(a, cfg); await editUnitDone(phone, "1.1");
await syncOnce(b, cfg); await editUnitConfidence(phone, "1.1", 91);
await editMeta(phone, "road.unit", "1.2");
await editChatTurn(phone, "user", "a week of lessons");
await editCard(phone, WATER, { ...newCard(), state: 2, interval: 8, due: 20_010, reps: 6 });
await sync(phone);
// b edits later, so b wins. // The laptop boots and a stale write stamps the old data as new.
await new Promise((r) => setTimeout(r, 5)); await editUnitConfidence(laptop, "1.1", 40);
await editUnitConfidence(b, "2.1", 70); await editMeta(laptop, "road.unit", "1.1");
await syncOnce(b, cfg); await sync(laptop);
await syncOnce(a, cfg); await sync(phone);
const onA = await a.get<{ confidence: number }>( for (const d of [phone, laptop]) {
"SELECT confidence FROM progress WHERE unit_id = '2.1'", expect(await one(d, "SELECT done, confidence FROM progress WHERE unit_id = '1.1'")).toEqual({
); done: 1,
expect(onA?.confidence).toBe(70); confidence: 91,
});
await a.close(); expect(await one(d, "SELECT v FROM meta WHERE k = 'road.unit'")).toEqual({ v: "1.2" });
await b.close(); expect(await turns(d)).toEqual(["a week of lessons"]);
expect(await one(d, "SELECT reps FROM card WHERE lemma_id = ?", [WATER])).toEqual({ reps: 6 });
}
}); });
it("propagates a delete through a tombstone", async () => { /* Gate 2. Protocol 1 compared timestamps, so a device whose clock ran an
const a = await device(); hour fast won every conflict for an hour. The counter does not care. */
const b = await device(); it("orders edits by the server's counter, not by either device's clock", async () => {
const fast = await make();
const slow = await make();
const real = Date.now();
await editCard(a, 2, { ...newCard(), state: 2, interval: 10 }); vi.spyOn(Date, "now").mockReturnValue(real + 60 * 60_000);
await syncOnce(a, cfg); await editMeta(fast, "prefs.goal", "40");
await syncOnce(b, cfg); vi.restoreAllMocks();
expect(await b.get("SELECT lemma_id FROM card WHERE lemma_id = 2")).toBeDefined(); await sync(fast);
await sync(slow);
// Without a tombstone the delete is invisible and b would push the card // Later in fact, earlier by its own clock.
// back on its next turn. vi.spyOn(Date, "now").mockReturnValue(real);
await new Promise((r) => setTimeout(r, 5)); await editMeta(slow, "prefs.goal", "25");
await editCardReset(a, 2); vi.restoreAllMocks();
await syncOnce(a, cfg); await sync(slow);
await syncOnce(b, cfg); await sync(fast);
expect(await b.get("SELECT lemma_id FROM card WHERE lemma_id = 2")).toBeUndefined(); expect(await one(fast, "SELECT v FROM meta WHERE k = 'prefs.goal'")).toEqual({ v: "25" });
expect(await one(slow, "SELECT v FROM meta WHERE k = 'prefs.goal'")).toEqual({ v: "25" });
// And it stays deleted after b syncs again — no resurrection. });
await syncOnce(b, cfg);
await syncOnce(a, cfg); it("keeps both devices' offline turns — ids that cannot collide", async () => {
expect(await a.get("SELECT lemma_id FROM card WHERE lemma_id = 2")).toBeUndefined(); const a = await make();
const b = await make();
await a.close(); await sync(a);
await b.close(); await sync(b);
await editChatTurn(a, "user", "a1");
await editChatTurn(a, "assistant", "a2");
await editChatTurn(b, "user", "b1");
await editChatTurn(b, "assistant", "b2");
await sync(a);
await sync(b);
await sync(a);
for (const d of [a, b]) expect((await turns(d)).sort()).toEqual(["a1", "a2", "b1", "b2"]);
});
/* Gate 3. A cleared lesson is a deliberate shrink: the other device obeys
it, even holding more turns than the device that cleared. */
it("keeps a cleared lesson cleared against a device holding a longer transcript", async () => {
const a = await make();
const b = await make();
for (const t of ["one", "two", "three"]) await editChatTurn(a, "user", t);
await sync(a);
await sync(b);
await editChatTurn(b, "user", "offline four");
await editChatTurn(b, "user", "offline five");
await editChatClear(a);
await editChatTurn(a, "user", "a new lesson");
await sync(a);
await sync(b);
await sync(a);
expect(await turns(b)).toEqual(["a new lesson"]);
expect(await turns(a)).toEqual(["a new lesson"]);
});
it("honours a full reset on a device that had not heard of it, its own offline work included", async () => {
const a = await make();
const b = await make();
await editCard(a, RICE, { ...newCard(), state: 1, reps: 1 });
await editUnitConfidence(a, "1.1", 50);
await sync(a);
await sync(b);
await editCard(b, SCHOOL, { ...newCard(), state: 1, reps: 1 }); // a card a never saw
await editUnitConfidence(b, "1.2", 30);
await editReset(a, "everything");
await sync(a);
await sync(b);
await sync(a);
for (const d of [a, b]) {
expect(await one(d, "SELECT count(*) AS n FROM card")).toEqual({ n: 0 });
expect(await one(d, "SELECT count(*) AS n FROM progress")).toEqual({ n: 0 });
}
});
it("keeps a forgotten card forgotten against a review made before hearing of it", async () => {
const a = await make();
const b = await make();
await editCard(a, WATER, { ...newCard(), state: 2, interval: 10, reps: 3 });
await sync(a);
await sync(b);
await editCardReset(a, WATER);
await sync(a);
await editCard(b, WATER, { ...newCard(), state: 2, interval: 20, reps: 4 });
await sync(b);
await sync(a);
for (const d of [a, b]) expect(await one(d, "SELECT lemma_id FROM card WHERE lemma_id = ?", [WATER])).toBeUndefined();
});
it("re-creates a card forgotten elsewhere, once this device has heard of the forgetting", async () => {
const a = await make();
const b = await make();
await editCard(a, RICE, { ...newCard(), state: 1, reps: 1 });
await sync(a);
await sync(b);
await editCardReset(a, RICE);
await sync(a);
await sync(b); // b hears of it
await editCard(b, RICE, { ...newCard(), state: 1, reps: 1 }); // and studies it again
await sync(b);
await sync(a);
expect(await one(a, "SELECT reps FROM card WHERE lemma_id = ?", [RICE])).toEqual({ reps: 1 });
});
/* Protocol 1 joined a key's values with a space and split them back on
one; 몇 명 came back as two values and stopped every pull at that row. */
it("carries and deletes a row whose key contains a space", async () => {
const a = await make();
const b = await make();
await editPeek(a, "몇 명");
await editPeek(a, "몇 명");
await sync(a);
await sync(b);
expect(await one(b, "SELECT SUM(count) AS n FROM peek WHERE form = '몇 명'")).toEqual({ n: 2 });
await editReset(a, "everything");
await sync(a);
await sync(b);
expect(await one(b, "SELECT count(*) AS n FROM peek")).toEqual({ n: 0 });
});
it("counts two devices' reviews of the same day, rather than keeping one", async () => {
const a = await make();
const b = await make();
await editStudyLog(a, 20_000, { reviews: 2 });
await editStudyLog(b, 20_000, { reviews: 3 });
await sync(a);
await sync(b);
await sync(a);
for (const d of [a, b]) {
expect(await one(d, "SELECT SUM(reviews) AS n FROM study_log WHERE day = 20000")).toEqual({ n: 5 });
}
}); });
/* Device-local bookkeeping must not travel. dict.loadedBands is the
dangerous one: it would tell a device it holds rows it never
downloaded, and the word rail would then miss words it believes are
present. */
it("never syncs device-local meta", async () => { it("never syncs device-local meta", async () => {
const a = await device(); const a = await make();
const b = await device(); const b = await make();
await editMeta(a, "prefs.goal", "40"); await editMeta(a, "prefs.goal", "40");
// These are written the way the app writes them — unstamped bookkeeping. await editMeta(a, "server.token", "secret");
await a.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES ('dict.loadedBands','[0,1,2,3,4,5]',?)", [ await a.run("INSERT OR REPLACE INTO meta (k, v, updated_at, dirty) VALUES ('dict.loadedBands', '[0,1,2]', 1, 1)");
Date.now(), await sync(a);
]); await sync(b);
await a.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES ('server.token','secret',?)", [
Date.now(),
]);
await syncOnce(a, cfg); expect(await one(b, "SELECT v FROM meta WHERE k = 'prefs.goal'")).toEqual({ v: "40" });
await syncOnce(b, cfg); expect(await one(b, "SELECT v FROM meta WHERE k = 'server.token'")).toBeUndefined();
expect(await one(b, "SELECT v FROM meta WHERE k = 'dict.loadedBands'")).toBeUndefined();
});
expect((await b.get<{ v: string }>("SELECT v FROM meta WHERE k='prefs.goal'"))?.v).toBe("40"); it("re-hydrates when the server has lost what it held, and offers its data back", async () => {
expect(await b.get("SELECT v FROM meta WHERE k='dict.loadedBands'")).toBeUndefined(); const a = await make();
expect(await b.get("SELECT v FROM meta WHERE k='server.token'")).toBeUndefined(); await editUnitConfidence(a, "2.1", 70);
await sync(a);
await a.close(); await fetch(`${cfg.baseUrl}/api/test/reset`, {
await b.close(); method: "POST",
headers: { authorization: `Bearer ${cfg.token}` },
});
const again = await sync(a);
expect(again.rehydrated).toBe(true);
expect(again.pushed).toBeGreaterThan(0);
const b = await make();
await sync(b);
expect(await one(b, "SELECT confidence FROM progress WHERE unit_id = '2.1'")).toEqual({ confidence: 70 });
});
it("refuses a client that does not speak protocol 2", async () => {
const res = await fetch(`${cfg.baseUrl}/api/sync?cursor=0`, {
headers: { authorization: `Bearer ${cfg.token}` },
});
expect(res.status).toBe(426);
const ok = await fetch(`${cfg.baseUrl}/api/sync?cursor=0`, {
headers: { authorization: `Bearer ${cfg.token}`, [PROTOCOL_HEADER]: "2" },
});
expect(ok.status).toBe(200);
}); });
}); });

53
types/shared/sync-protocol.mjs.d.ts vendored
View File

@@ -1,17 +1,60 @@
/* Declarations for shared/sync-protocol.mjs. */ /* Declarations for shared/sync-protocol.mjs — protocol 2. */
export interface SyncTableSpec { export interface SyncTableSpec {
pk: string[]; pk: string[];
cols: string[]; cols: string[];
} }
export const PROTOCOL: 2;
export const PROTOCOL_HEADER: string;
export const PAGE_SIZE: number;
export const SYNC_TABLES: Record<string, SyncTableSpec>; export const SYNC_TABLES: Record<string, SyncTableSpec>;
export const SYNC_TABLE_NAMES: string[]; export const SYNC_TABLE_NAMES: string[];
export const PAGE_SIZE: number;
export function pkFor(tbl: string): string[] | null; export function pkFor(tbl: string): string[] | null;
/** A row's primary-key values as a JSON array — the key on the wire. */
export function encodePk(tbl: string, row: Record<string, unknown>): string;
export function decodePk(pk: string): (string | number)[];
export function isSyncableMetaKey(key: string): boolean; export function isSyncableMetaKey(key: string): boolean;
export function rowKey(table: string, row: Record<string, unknown>): string;
export function isDirty(row: { updated_at: number }, pushedAt: number): boolean; export type ResetScope = "chat" | "roadmap" | "everything";
export function incomingWins(incomingUpdatedAt: number, localUpdatedAt: number | null): boolean; export const RESET_SCOPES: Record<ResetScope, { tables: string[]; meta: string[] }>;
export function resetCovers(scope: string, tbl: string, row: Record<string, unknown>): boolean;
/** A row on the wire. `base` is the change_seq the sender last agreed with. */
export interface WireRow {
tbl: string;
pk: string;
base: number;
deleted?: boolean;
data?: Record<string, string | number | null> | null;
updated_at?: number;
}
/** A row as the server holds it: `seq` is its current change_seq. */
export interface ServerRow {
tbl: string;
pk: string;
seq: number;
deleted: boolean;
data: Record<string, string | number | null> | null;
updated_at?: number;
}
export interface PullResponse {
epoch: string;
rows: ServerRow[];
cursor: number;
more: boolean;
}
export interface PushResponse {
epoch: string;
applied: { tbl: string; pk: string; seq: number }[];
/** Rows the push could not overwrite: the server's current version. A
row absent on the server comes back with seq 0 and deleted. */
conflicts: ServerRow[];
}