fabi/Hankan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sync): protocol 2 — hydration, a counter not a clock, no silent shrinking

Browse Source 
The reworked bundle's PORT.md makes row-level last-write-wins conditional
on three gates, each learned by losing real data. The port's sync broke
all three, and had four more ways to lose or stall work. Both ends change,
so this is one protocol version, refused by the other side if mismatched.

Gate 1, hydration. A device now pulls every page the server holds before
it may push anything; it used to push first. The 14 Sep laptop — a
week-old copy re-stamped at boot and pushed over a week of phone work —
is now a test, and the phone's week survives it. Boot writes nothing
syncable either: the lesson opens with the app's own words (the artifact's
seeded turn) and waits for Start, instead of stamping a reply and a
progress edit before a server can even be configured.

Gate 2, a counter. Every row remembers the change_seq it last agreed with
(base_seq). The server applies a write only if that still matches —
compare-and-swap under an advisory lock — and otherwise returns its copy
as a conflict. No clock is compared anywhere: a device an hour fast used
to win every conflict for an hour, and a slow one's newer edit was
silently dropped with HTTP 200. dirty and rev replace the timestamp
watermark, which lost edits whenever a clock moved backwards.

Gate 3, no silent shrinking. A conflict is settled by what each copy
holds (sync/resolve.ts): more reviews, more evidence, a finished unit, the
further roadmap position, the union of learned grammar. A deliberate
shrink is explicit: a reset or a cleared lesson raises a marker every
device obeys, including its own unsynced edits, so a reset is not undone
by a device that had not heard of it. Trimming the transcript is local
and tombstones nothing — it used to delete the other device's turns.

Also fixed on the way:
  · keys travel as JSON arrays — a space in 몇 명 used to stop every
    device's pull at that row, permanently;
  · pulls take a shared lock against pushes, so a change_seq committed
    out of order can no longer be skipped;
  · study_log and peek are per device and summed, so two devices' reviews
    of one day both count;
  · each user's data has an epoch; a server that lost it is detected,
    and the device re-hydrates and offers its data back;
  · a protocol-1 client is refused with 426 rather than half-understood.

Migration 9 adds the columns, re-keys the counters and tombstones; the
server drops protocol-1 rows once (none were deployed). Verified: 14
two-device scenarios against a real Postgres, and two browser profiles
syncing a lesson through the UI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-09-16 20:28:35 +02:00
parent bf9b5950da
commit 9ab5aba6b3
21 changed files with 1692 additions and 579 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
server/sql/002-protocol-2.sql Normal file
View File

@@ -0,0 +1,32 @@
-- Sync protocol 2.
--
-- Rows are now written only by compare-and-swap on change_seq (see
-- server/src/db.ts), keyed by a JSON array of the key's values, and every
-- user's data belongs to an epoch: a random id that changes whenever the
-- server's copy is thrown away, so a client can tell it is talking to a
-- server that no longer holds what it pushed and must hydrate from scratch.
CREATE TABLE IF NOT EXISTS sync_meta (
k TEXT PRIMARY KEY,
v TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS sync_epoch (
user_id TEXT PRIMARY KEY,
epoch TEXT NOT NULL
);
-- Protocol 1 rows cannot be read as protocol 2: their keys were joined with
-- a space and every one was ordered by a client's clock. They are dropped
-- once, and each user's epoch with them — so every client re-hydrates and
-- offers its own database back, which is the copy that was always the
-- source of truth. No deployment held protocol 1 data when this shipped.
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM sync_meta WHERE k = 'protocol' AND v = '2') THEN
DELETE FROM sync_row;
DELETE FROM sync_epoch;
INSERT INTO sync_meta (k, v) VALUES ('protocol', '2')
ON CONFLICT (k) DO UPDATE SET v = EXCLUDED.v;
END IF;
END $$;

177
server/src/db.ts
View File

@@ -1,99 +1,163 @@
/* Postgres access, and the two queries sync is made of. */
/* Postgres access, and the two operations sync is made of.
Protocol 2 (shared/sync-protocol.mjs). The server's whole job is order:
it assigns change_seq, it refuses a write made against a stale copy, and
it never lets a pull see half of a push. It does not read inside a row —
it stores and orders them; the client interprets them. */
import { readFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
import pg from "pg";
export interface SyncRow {
/* The wire shapes, as in types/shared/sync-protocol.mjs.d.ts. Restated here
because the server's type check reads the .mjs itself, not its
declarations. */
type Value = string | number | null;
/** A row a client pushes. `base` is the change_seq it last agreed with. */
export interface WireRow {
tbl: string;
data: Record<string, string | number | null>;
updated_at: number;
pk: string;
base: number;
deleted?: boolean;
data?: Record<string, Value> | null;
updated_at?: number;
}
/** A row as the server holds it. */
export interface ServerRow {
tbl: string;
pk: string;
seq: number;
deleted: boolean;
data: Record<string, Value> | null;
updated_at?: number;
}
export interface Store {
pull(userId: string, cursor: number, limit: number): Promise<{ rows: SyncRow[]; cursor: number; more: boolean }>;
push(userId: string, rows: SyncRow[]): Promise<number>;
pull(userId: string, cursor: number, limit: number): Promise<{ epoch: string; rows: ServerRow[]; cursor: number; more: boolean }>;
push(userId: string, rows: WireRow[]): Promise<{ epoch: string; applied: { tbl: string; pk: string; seq: number }[]; conflicts: ServerRow[] }>;
/** Test-only; the route that calls it exists only under HANKAN_TEST_MODE. */
reset(userId: string): Promise<void>;
close(): Promise<void>;
}
/**
* The row's primary key as a single string.
*
* The client sends `{pk}` for a delete and the full row otherwise, so a
* delete already carries its key and a live row needs one derived from the
* table's key columns. Both sides use the same rule, from the shared
* protocol module.
*/
function keyOf(row: SyncRow, pkCols: string[]): string {
if (row.deleted && row.data.pk != null) return String(row.data.pk);
return pkCols.map((c) => String(row.data[c])).join(" ");
type Query = { query: pg.Pool["query"] };
/** This user's epoch, created on first contact. */
async function epochOf(db: Query, userId: string): Promise<string> {
await db.query(
"INSERT INTO sync_epoch (user_id, epoch) VALUES ($1, gen_random_uuid()::text) ON CONFLICT (user_id) DO NOTHING",
[userId],
);
const { rows } = await db.query("SELECT epoch FROM sync_epoch WHERE user_id = $1", [userId]);
return String(rows[0].epoch);
}
const asServerRow = (r: Record<string, unknown>): ServerRow => ({
tbl: r.tbl as string,
pk: r.pk as string,
seq: Number(r.change_seq),
deleted: Boolean(r.deleted),
data: r.deleted ? null : (r.data as ServerRow["data"]),
updated_at: Number(r.updated_at),
});
export async function openStore(connectionString: string, pkFor: (tbl: string) => string[] | null): Promise<Store> {
const pool = new pg.Pool({ connectionString, max: 4 });
const schema = await readFile(fileURLToPath(new URL("../sql/001-schema.sql", import.meta.url)), "utf8");
await pool.query(schema);
for (const file of ["001-schema.sql", "002-protocol-2.sql"]) {
await pool.query(await readFile(fileURLToPath(new URL(`../sql/${file}`, import.meta.url)), "utf8"));
}
return {
async pull(userId, cursor, limit) {
// One extra row tells us whether another page exists without a count.
const { rows } = await pool.query(
`SELECT tbl, pk, data, updated_at, deleted, change_seq
FROM sync_row
WHERE user_id = $1 AND change_seq > $2
ORDER BY change_seq
LIMIT $3`,
[userId, cursor, limit + 1],
);
const client = await pool.connect();
try {
await client.query("BEGIN");
/* Shared, against push's exclusive lock on the same key. Without it a
push that took seq 10 but committed after another took seq 11 could
be passed over: a pull that saw 11 moves the cursor beyond 10, and
nothing would ever ask for 10 again. */
await client.query("SELECT pg_advisory_xact_lock_shared(hashtext($1))", [userId]);
const epoch = await epochOf(client, userId);
// One extra row tells us whether another page exists without a count.
const { rows } = await client.query(
`SELECT tbl, pk, data, updated_at, deleted, change_seq
FROM sync_row
WHERE user_id = $1 AND change_seq > $2
ORDER BY change_seq
LIMIT $3`,
[userId, cursor, limit + 1],
);
await client.query("COMMIT");
const more = rows.length > limit;
const page = more ? rows.slice(0, limit) : rows;
return {
rows: page.map((r) => ({
tbl: r.tbl as string,
data: r.deleted ? { pk: r.pk as string } : (r.data as Record<string, string | number | null>),
updated_at: Number(r.updated_at),
...(r.deleted ? { deleted: true } : {}),
})),
cursor: page.length ? Number(page[page.length - 1].change_seq) : cursor,
more,
};
const more = rows.length > limit;
const page = (more ? rows.slice(0, limit) : rows).map(asServerRow);
return { epoch, rows: page, cursor: page.length ? page[page.length - 1]!.seq : cursor, more };
} catch (err) {
await client.query("ROLLBACK");
throw err;
} finally {
client.release();
}
},
async push(userId, rows) {
const client = await pool.connect();
try {
await client.query("BEGIN");
// One push at a time per user: compare-and-swap needs a stable read.
await client.query("SELECT pg_advisory_xact_lock(hashtext($1))", [userId]);
const epoch = await epochOf(client, userId);
const applied: { tbl: string; pk: string; seq: number }[] = [];
const conflicts: ServerRow[] = [];
for (const row of rows) {
const pkCols = pkFor(row.tbl);
if (!pkCols) continue; // a table this server does not know about
if (!pkFor(row.tbl) || typeof row.pk !== "string") continue; // a table this server does not know
const base = Number(row.base) || 0;
// Last-write-wins, resolved in the database so two devices pushing
// at once cannot interleave a read and a write around it.
await client.query(
const { rows: current } = await client.query(
`SELECT tbl, pk, data, updated_at, deleted, change_seq FROM sync_row
WHERE user_id = $1 AND tbl = $2 AND pk = $3 FOR UPDATE`,
[userId, row.tbl, row.pk],
);
const cur = current[0] as Record<string, unknown> | undefined;
/* Gate 2. The write stands only if the sender last agreed with the
row the server holds now — or neither has ever seen it. Anything
else goes back as a conflict, with the server's copy, and the
client settles it. No clock is consulted anywhere. */
const agreed = cur ? Number(cur.change_seq) === base : base === 0;
if (!agreed) {
conflicts.push(
cur ? asServerRow(cur) : { tbl: row.tbl, pk: row.pk, seq: 0, deleted: true, data: null },
);
continue;
}
const { rows: written } = await client.query(
`INSERT INTO sync_row (user_id, tbl, pk, data, updated_at, deleted)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (user_id, tbl, pk) DO UPDATE
SET data = EXCLUDED.data,
updated_at = EXCLUDED.updated_at,
deleted = EXCLUDED.deleted
WHERE EXCLUDED.updated_at > sync_row.updated_at`,
[userId, row.tbl, keyOf(row, pkCols), JSON.stringify(row.data), row.updated_at, row.deleted ?? false],
SET data = EXCLUDED.data, updated_at = EXCLUDED.updated_at, deleted = EXCLUDED.deleted
RETURNING change_seq`,
[
userId,
row.tbl,
row.pk,
JSON.stringify(row.deleted ? {} : (row.data ?? {})),
Number(row.updated_at) || 0,
Boolean(row.deleted),
],
);
applied.push({ tbl: row.tbl, pk: row.pk, seq: Number(written[0].change_seq) });
}
const { rows: top } = await client.query(
"SELECT COALESCE(max(change_seq), 0) AS c FROM sync_row WHERE user_id = $1",
[userId],
);
await client.query("COMMIT");
return Number(top[0].c);
return { epoch, applied, conflicts };
} catch (err) {
await client.query("ROLLBACK");
throw err;
@@ -104,6 +168,7 @@ export async function openStore(connectionString: string, pkFor: (tbl: string) =
async reset(userId) {
await pool.query("DELETE FROM sync_row WHERE user_id = $1", [userId]);
await pool.query("DELETE FROM sync_epoch WHERE user_id = $1", [userId]);
},
async close() {

23
server/src/main.ts
View File

@@ -7,8 +7,8 @@
import { serve } from "@hono/node-server";
import { Hono } from "hono";
import { pkFor, PAGE_SIZE } from "../../shared/sync-protocol.mjs";
import { openStore, type SyncRow } from "./db.ts";
import { pkFor, PAGE_SIZE, PROTOCOL, PROTOCOL_HEADER } from "../../shared/sync-protocol.mjs";
import { openStore, type WireRow } from "./db.ts";
import { tutorRoute } from "./tutor.ts";
const PORT = Number(process.env.PORT ?? 8787);
@@ -71,7 +71,7 @@ app.use("*", async (c, next) => {
return c.body(null, 204, {
"Access-Control-Allow-Origin": allow,
"Access-Control-Allow-Methods": "GET, POST, OPTIONS",
"Access-Control-Allow-Headers": "authorization, content-type",
"Access-Control-Allow-Headers": `authorization, content-type, ${PROTOCOL_HEADER}`,
"Access-Control-Max-Age": "86400",
Vary: "Origin",
});
@@ -102,6 +102,17 @@ app.get("/health", (c) => c.json({ ok: true }));
/* ── sync ────────────────────────────────────────────────────────────── */
/* Refuse any client not speaking this protocol. A client on protocol 1 is
exactly the device the artifact lost data to — an old build that pushes
before it pulls and trusts its own clock — so it gets a clear 426 rather
than a half-understood write. */
app.use("/api/sync", async (c, next) => {
if (c.req.header(PROTOCOL_HEADER) !== String(PROTOCOL)) {
return c.json({ error: "this server speaks sync protocol 2 — update the app", protocol: PROTOCOL }, 426);
}
await next();
});
app.get("/api/sync", async (c) => {
const cursor = Number(c.req.query("cursor") ?? 0);
if (!Number.isFinite(cursor) || cursor < 0) return c.json({ error: "bad cursor" }, 400);
@@ -109,10 +120,10 @@ app.get("/api/sync", async (c) => {
});
app.post("/api/sync", async (c) => {
const body = (await c.req.json()) as { rows?: SyncRow[] };
const body = (await c.req.json()) as { rows?: WireRow[] };
const rows = Array.isArray(body.rows) ? body.rows : [];
if (rows.length > PAGE_SIZE * 4) return c.json({ error: "too many rows" }, 413);
return c.json({ cursor: await store.push(USER_ID, rows) });
if (rows.length > PAGE_SIZE) return c.json({ error: "too many rows" }, 413);
return c.json(await store.push(USER_ID, rows));
});
/* Wipes this user's rows. Exists only when HANKAN_TEST_MODE is set, so it