fabi/Hankan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sync): protocol 2 — hydration, a counter not a clock, no silent shrinking

Browse Source 
The reworked bundle's PORT.md makes row-level last-write-wins conditional
on three gates, each learned by losing real data. The port's sync broke
all three, and had four more ways to lose or stall work. Both ends change,
so this is one protocol version, refused by the other side if mismatched.

Gate 1, hydration. A device now pulls every page the server holds before
it may push anything; it used to push first. The 14 Sep laptop — a
week-old copy re-stamped at boot and pushed over a week of phone work —
is now a test, and the phone's week survives it. Boot writes nothing
syncable either: the lesson opens with the app's own words (the artifact's
seeded turn) and waits for Start, instead of stamping a reply and a
progress edit before a server can even be configured.

Gate 2, a counter. Every row remembers the change_seq it last agreed with
(base_seq). The server applies a write only if that still matches —
compare-and-swap under an advisory lock — and otherwise returns its copy
as a conflict. No clock is compared anywhere: a device an hour fast used
to win every conflict for an hour, and a slow one's newer edit was
silently dropped with HTTP 200. dirty and rev replace the timestamp
watermark, which lost edits whenever a clock moved backwards.

Gate 3, no silent shrinking. A conflict is settled by what each copy
holds (sync/resolve.ts): more reviews, more evidence, a finished unit, the
further roadmap position, the union of learned grammar. A deliberate
shrink is explicit: a reset or a cleared lesson raises a marker every
device obeys, including its own unsynced edits, so a reset is not undone
by a device that had not heard of it. Trimming the transcript is local
and tombstones nothing — it used to delete the other device's turns.

Also fixed on the way:
  · keys travel as JSON arrays — a space in 몇 명 used to stop every
    device's pull at that row, permanently;
  · pulls take a shared lock against pushes, so a change_seq committed
    out of order can no longer be skipped;
  · study_log and peek are per device and summed, so two devices' reviews
    of one day both count;
  · each user's data has an epoch; a server that lost it is detected,
    and the device re-hydrates and offers its data back;
  · a protocol-1 client is refused with 426 rather than half-understood.

Migration 9 adds the columns, re-keys the counters and tombstones; the
server drops protocol-1 rows once (none were deployed). Verified: 14
two-device scenarios against a real Postgres, and two browser profiles
syncing a lesson through the UI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-09-16 20:28:35 +02:00
parent bf9b5950da
commit 9ab5aba6b3
21 changed files with 1692 additions and 579 deletions
Download Patch File Download Diff File Expand all files Collapse all files

142
shared/sync-protocol.mjs
View File

@@ -1,18 +1,41 @@
/* The sync wire format, defined once and imported by both sides.
Row-level, last-write-wins on `updated_at`, cursor-based on a
server-assigned `change_seq`. One user, so the loser of a conflict is at
worst one SRS grade — LWW is adequate, and a merge policy would be
over-engineering.
Protocol 2. Row-level and cursor-based on a server-assigned change_seq,
as PORT.md specifies — and ordered by that counter, never by a clock.
THE RULE THAT MATTERS: seeded and defaulted rows carry updated_at = 0.
The artifact's sync bug was a fresh device stamping its own empty state
as newer than the server's real history and clobbering it. Here a seed
row can never be dirty (0 is never greater than any watermark) and can
never win a conflict (0 is never greater than any timestamp). The bug is
not avoided, it is unrepresentable. */
════ THE THREE GATES ════════════════════════════════════════════════
/** Tables that sync, and the columns forming each primary key. */
The artifact lost a week of a real user's work without these, and the
port's first sync broke all three. In short (lib/sync.js has the long
version, applied there to whole documents):
1. HYDRATION. A device pulls everything the server holds before it may
push anything. A laptop with a week-old copy used to boot, re-stamp
that copy through a migration, and push it over the phone's week.
2. A COUNTER, NOT A CLOCK. Every row remembers the change_seq it last
agreed with the server on (base_seq). The server applies a write only
if that is still the row's current change_seq — compare-and-swap —
and otherwise hands the current row back as a conflict. A phone clock
four minutes fast used to decide which of two grades survived.
3. NO SILENT SHRINKING. A conflict is settled by what each copy HOLDS,
not by which is newer (see app/src/sync/resolve.ts). Emptying things
on purpose is explicit: a reset or a cleared lesson writes a marker
every other device obeys, so the shrink is declared, never raced.
Seeded and defaulted rows are never dirty, so they never leave the
device: a fresh install cannot claim its empty defaults are news. */
export const PROTOCOL = 2;
/** Header the client sends; the server refuses any other protocol. */
export const PROTOCOL_HEADER = "x-hankan-protocol";
/** Maximum rows in one push or pull page. */
export const PAGE_SIZE = 500;
/** Tables that sync: their primary-key columns and the columns that travel. */
export const SYNC_TABLES = {
card: {
pk: ["lemma_id"],
@@ -21,8 +44,31 @@ export const SYNC_TABLES = {
progress: { pk: ["unit_id"], cols: ["unit_id", "done", "confidence", "answers", "note"] },
chat: { pk: ["id"], cols: ["id", "role", "body", "created_at"] },
meta: { pk: ["k"], cols: ["k", "v"] },
study_log: { pk: ["day"], cols: ["day", "reviews", "correct", "drills"] },
peek: { pk: ["form"], cols: ["form", "count"] },
/* Counters, kept per device and summed on read. Two devices each adding
a review to the same day would otherwise both write "reviews = n+1",
and whichever landed second would erase the other's review. A device
only ever writes its own row, so these never conflict at all. */
study_log: { pk: ["day", "device"], cols: ["day", "device", "reviews", "correct", "drills"] },
peek: { pk: ["form", "device"], cols: ["form", "device", "count"] },
custom_word: { pk: ["headword", "pos"], cols: ["headword", "pos", "gloss"] },
evidence: {
pk: ["word"],
cols: [
"word",
"ok",
"wrong",
"lookups",
"streak",
"first_round",
"last_round",
"last_seen",
"rounds",
"first_ok_round",
"last_ok_round",
],
},
confusion: { pk: ["word"], cols: ["word", "mistook"] },
phase_ledger: { pk: ["phase", "kind", "item"], cols: ["phase", "kind", "item", "confirmed"] },
};
export const SYNC_TABLE_NAMES = Object.keys(SYNC_TABLES);
@@ -31,39 +77,67 @@ export const SYNC_TABLE_NAMES = Object.keys(SYNC_TABLES);
export const pkFor = (tbl) =>
Object.prototype.hasOwnProperty.call(SYNC_TABLES, tbl) ? SYNC_TABLES[tbl].pk : null;
/**
* A row's key on the wire: its primary-key values as a JSON array.
*
* Protocol 1 joined them with a space and split them back on one, so a key
* containing a space — the curriculum's 몇 명 and 신경 쓰다 can be peek
* forms — came back as two values and the statement binding them threw.
* The pull stopped at that row, on every device, permanently.
*/
export const encodePk = (tbl, row) => JSON.stringify(SYNC_TABLES[tbl].pk.map((c) => row[c]));
export const decodePk = (pk) => JSON.parse(pk);
/**
* `meta` mixes the learner's data with bookkeeping that describes one
* install. Only the former may cross the wire.
*
* `dict.loadedBands` is the dangerous one: replicating it would tell a
* phone that had loaded bands 0-2 that it holds every row the desktop has,
* and the word rail would then fail to find words it believes are present.
* `schema_version` would be worse — a device could be told it has run a
* migration it has not. `server.*` holds this device's endpoint and bearer
* token: syncing a token through the endpoint it authenticates would be
* circular, and a base URL is network-specific.
* `dict.*` is the dangerous one: replicating the loaded-band record would
* tell a phone it holds rows it never downloaded. `schema_version` would be
* worse — a device could be told it has run a migration it has not.
* `server.*` holds this device's endpoint and bearer token, and `sync.*`
* is this device's own position in the conversation with the server.
*/
const SYNCABLE_META_EXACT = new Set(["grammar.learned", "grammar.notes", "trainer.conjugation"]);
const SYNCABLE_META_PREFIX = ["prefs."];
const SYNCABLE_META_PREFIX = ["prefs.", "road.", "learner.", "reset."];
export function isSyncableMetaKey(key) {
if (SYNCABLE_META_EXACT.has(key)) return true;
return SYNCABLE_META_PREFIX.some((p) => key.startsWith(p));
}
/** Stable string key for a row, used to match it across devices. */
export const rowKey = (table, row) => SYNC_TABLES[table].pk.map((c) => String(row[c])).join(" ");
/** Rows a device may send: dirty since its last successful push. */
export const isDirty = (row, pushedAt) => Number(row.updated_at) > Number(pushedAt);
/**
* Last-write-wins. Strictly greater, so equal timestamps leave the local row
* alone — a tie means both sides already agree, or the clocks are close
* enough that flapping would be worse than either outcome.
* What each deliberate shrink empties. A marker meta key, `reset.<scope>`,
* holds a counter; a device that sees it rise empties the same things
* locally — including its own unsynced edits, because a reset it had not
* yet heard of still wins over work done on the old data.
*/
export const incomingWins = (incomingUpdatedAt, localUpdatedAt) =>
Number(incomingUpdatedAt) > Number(localUpdatedAt ?? -1);
export const RESET_SCOPES = {
chat: { tables: ["chat"], meta: [] },
roadmap: { tables: ["progress", "chat", "phase_ledger"], meta: ["road."] },
everything: {
tables: [
"progress",
"chat",
"phase_ledger",
"card",
"study_log",
"peek",
"custom_word",
"evidence",
"confusion",
],
meta: ["road.", "prefs.", "grammar.", "trainer.", "learner."],
},
};
/** Maximum rows in one push or pull page. */
export const PAGE_SIZE = 500;
/** Does a reset of this scope empty this row? */
export function resetCovers(scope, tbl, row) {
const s = RESET_SCOPES[scope];
if (!s) return false;
if (tbl === "meta") {
const k = String(row.k ?? "");
return !k.startsWith("reset.") && s.meta.some((p) => k.startsWith(p));
}
return s.tables.includes(tbl);
}