fabi/Hankan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sync): protocol 2 — hydration, a counter not a clock, no silent shrinking

Browse Source 
The reworked bundle's PORT.md makes row-level last-write-wins conditional
on three gates, each learned by losing real data. The port's sync broke
all three, and had four more ways to lose or stall work. Both ends change,
so this is one protocol version, refused by the other side if mismatched.

Gate 1, hydration. A device now pulls every page the server holds before
it may push anything; it used to push first. The 14 Sep laptop — a
week-old copy re-stamped at boot and pushed over a week of phone work —
is now a test, and the phone's week survives it. Boot writes nothing
syncable either: the lesson opens with the app's own words (the artifact's
seeded turn) and waits for Start, instead of stamping a reply and a
progress edit before a server can even be configured.

Gate 2, a counter. Every row remembers the change_seq it last agreed with
(base_seq). The server applies a write only if that still matches —
compare-and-swap under an advisory lock — and otherwise returns its copy
as a conflict. No clock is compared anywhere: a device an hour fast used
to win every conflict for an hour, and a slow one's newer edit was
silently dropped with HTTP 200. dirty and rev replace the timestamp
watermark, which lost edits whenever a clock moved backwards.

Gate 3, no silent shrinking. A conflict is settled by what each copy
holds (sync/resolve.ts): more reviews, more evidence, a finished unit, the
further roadmap position, the union of learned grammar. A deliberate
shrink is explicit: a reset or a cleared lesson raises a marker every
device obeys, including its own unsynced edits, so a reset is not undone
by a device that had not heard of it. Trimming the transcript is local
and tombstones nothing — it used to delete the other device's turns.

Also fixed on the way:
  · keys travel as JSON arrays — a space in 몇 명 used to stop every
    device's pull at that row, permanently;
  · pulls take a shared lock against pushes, so a change_seq committed
    out of order can no longer be skipped;
  · study_log and peek are per device and summed, so two devices' reviews
    of one day both count;
  · each user's data has an epoch; a server that lost it is detected,
    and the device re-hydrates and offers its data back;
  · a protocol-1 client is refused with 426 rather than half-understood.

Migration 9 adds the columns, re-keys the counters and tombstones; the
server drops protocol-1 rows once (none were deployed). Verified: 14
two-device scenarios against a real Postgres, and two browser profiles
syncing a lesson through the UI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-09-16 20:28:35 +02:00
parent bf9b5950da
commit 9ab5aba6b3
21 changed files with 1692 additions and 579 deletions
Download Patch File Download Diff File Expand all files Collapse all files

117
app/src/db/migrations.ts
View File

@@ -260,8 +260,125 @@ export const MIGRATIONS: Migration[] = [
`,
run: remodelRoadAndChat,
},
{
id: 9,
name: "sync protocol 2 — base_seq and dirty on every row, counters per device",
sql: /* sql */ `
-- base_seq: the server change_seq this row last agreed with.
-- dirty: a local edit not yet on the server.
-- rev: bumped by every local edit, so a push can tell whether
-- the row changed again while it was in flight.
-- All default to 0: a seeded row is never dirty and never travels.
ALTER TABLE card ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE card ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE card ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE progress ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE progress ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE progress ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE chat ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE chat ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE chat ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE meta ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE meta ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE meta ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE custom_word ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE custom_word ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE custom_word ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE evidence ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE evidence ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE evidence ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE confusion ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE confusion ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE confusion ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE phase_ledger ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE phase_ledger ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE phase_ledger ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
ALTER TABLE tombstone ADD COLUMN base_seq INTEGER NOT NULL DEFAULT 0;
ALTER TABLE tombstone ADD COLUMN dirty INTEGER NOT NULL DEFAULT 0;
ALTER TABLE tombstone ADD COLUMN rev INTEGER NOT NULL DEFAULT 0;
-- Counters per device, summed on read — two devices adding to the same
-- day each own their row, so neither can erase the other's reviews.
CREATE TABLE study_log_v9 (
day INTEGER NOT NULL,
device TEXT NOT NULL,
reviews INTEGER NOT NULL DEFAULT 0,
correct INTEGER NOT NULL DEFAULT 0,
drills INTEGER NOT NULL DEFAULT 0,
updated_at INTEGER NOT NULL DEFAULT 0,
base_seq INTEGER NOT NULL DEFAULT 0,
dirty INTEGER NOT NULL DEFAULT 0,
rev INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (day, device)
) WITHOUT ROWID;
INSERT INTO study_log_v9 (day, device, reviews, correct, drills, updated_at)
SELECT day, (SELECT v FROM meta WHERE k = 'sync.device'), reviews, correct, drills, updated_at
FROM study_log;
DROP TABLE study_log;
ALTER TABLE study_log_v9 RENAME TO study_log;
CREATE TABLE peek_v9 (
form TEXT NOT NULL,
device TEXT NOT NULL,
count INTEGER NOT NULL DEFAULT 0,
updated_at INTEGER NOT NULL DEFAULT 0,
base_seq INTEGER NOT NULL DEFAULT 0,
dirty INTEGER NOT NULL DEFAULT 0,
rev INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (form, device)
) WITHOUT ROWID;
INSERT INTO peek_v9 (form, device, count, updated_at)
SELECT form, (SELECT v FROM meta WHERE k = 'sync.device'), count, updated_at FROM peek;
DROP TABLE peek;
ALTER TABLE peek_v9 RENAME TO peek;
CREATE INDEX IF NOT EXISTS card_dirty ON card(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS progress_dirty ON progress(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS chat_dirty ON chat(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS meta_dirty ON meta(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS custom_word_dirty ON custom_word(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS evidence_dirty ON evidence(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS confusion_dirty ON confusion(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS phase_ledger_dirty ON phase_ledger(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS tombstone_dirty ON tombstone(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS study_log_dirty ON study_log(dirty) WHERE dirty = 1;
CREATE INDEX IF NOT EXISTS peek_dirty ON peek(dirty) WHERE dirty = 1;
-- Protocol 1's position means nothing to protocol 2. With no epoch
-- recorded, the next sync re-hydrates: it pulls everything, then
-- offers every stamped row back to the server.
DELETE FROM meta WHERE k IN ('sync.cursor', 'sync.pushedAt');
`,
run: jsonTombstoneKeys,
},
];
/**
* Migration 9's data step: tombstone keys in protocol 2's form, a JSON array
* of the key's values. Protocol 1 stored the bare value, or values joined by
* a space — which a key containing a space could not survive.
*/
async function jsonTombstoneKeys(db: Db): Promise<void> {
const device = (await db.get<{ v: string }>("SELECT v FROM meta WHERE k = 'sync.device'"))?.v ?? "";
const graves = await db.all<{ tbl: string; pk: string }>("SELECT tbl, pk FROM tombstone");
for (const g of graves) {
if (g.pk.startsWith("[")) continue; // written in the new form already
const key =
g.tbl === "card"
? [Number(g.pk)]
: g.tbl === "study_log"
? [Number(g.pk), device]
: g.tbl === "peek"
? [g.pk, device]
: [g.pk];
await db.run("UPDATE OR REPLACE tombstone SET pk = ? WHERE tbl = ? AND pk = ?", [
JSON.stringify(key),
g.tbl,
g.pk,
]);
}
}
/**
* Migration 8's data step.
*

303
app/src/db/writes.ts
View File

@@ -7,27 +7,36 @@
real history, and last-write-wins duly clobbered months of progress with
an empty seed.
The rule that prevents it: SEEDED OR DEFAULTED STATE NEVER CARRIES A WRITE
TIMESTAMP. Only a genuine user edit stamps the clock.
The rule that prevents it: SEEDED OR DEFAULTED STATE IS NEVER A WRITE.
Only a genuine user edit stamps the clock and marks its row dirty.
Three layers enforce it:
1. the DDL — `updated_at INTEGER NOT NULL DEFAULT 0` on every syncable
table, so forgetting the column is the safe failure, not the unsafe
one (see migrations.ts);
2. this file — seedX() helpers never mention updated_at; editX() helpers
always set it from now();
1. the DDL — `updated_at`, `dirty` and `rev` all default to 0 on every
syncable table, so forgetting them is the safe failure: an INSERT
that omits them is invisible to sync (see migrations.ts);
2. this file — seedX() helpers never mention them; editX() helpers
always stamp updated_at, set dirty = 1 and bump rev;
3. lint + test — Date.now() is banned everywhere under src/db/ except
here, and test/db/conformance.ts asserts a freshly seeded database
has no non-zero updated_at anywhere.
has nothing stamped and nothing dirty.
Sync is out of scope for this pass. The point is that when it lands, the
schema already cannot express the bug. */
════ WHAT SYNC NEEDS FROM A WRITE ════
dirty = 1 queues the row for the server; rev tells the sync client
whether the row changed again while its push was in flight. base_seq is
the sync client's alone — no write here touches it, except that a delete
carries the deleted row's base_seq into its tombstone, which is what lets
the server compare-and-swap the delete like any other write. */
import type { Db, Params } from "./types.js";
import type { Card, Grade } from "@lib/srs.js";
import { lemmaId } from "@shared/lemma-id.mjs";
import { uuidv7 } from "./ids.js";
import {
RESET_SCOPES,
SYNC_TABLES,
encodePk,
type ResetScope as MarkerScope,
} from "@shared/sync-protocol.mjs";
import { randomId, uuidv7 } from "./ids.js";
/* ─────────────────────────────────────────────────────────────────────
The clock. This is the ONLY place in src/db/ that reads it.
@@ -36,25 +45,54 @@ import { uuidv7 } from "./ids.js";
/** Wall-clock milliseconds, for stamping a genuine user edit. */
export const now = (): number => Date.now();
/* ─────────────────────────────────────────────────────────────────────
Shared helpers
───────────────────────────────────────────────────────────────────── */
/** This install's id — the owner of its per-device counter rows. */
const devices = new WeakMap<Db, string>();
export async function deviceId(db: Db): Promise<string> {
const known = devices.get(db);
if (known) return known;
let id = (await db.get<{ v: string }>("SELECT v FROM meta WHERE k = 'sync.device'"))?.v;
if (!id) {
id = randomId();
// Describes this install, not the learner: never stamped, never synced.
await db.run("INSERT OR IGNORE INTO meta (k, v, updated_at) VALUES ('sync.device', ?, 0)", [id]);
}
devices.set(db, id);
return id;
}
/**
* Record that a row was deleted.
* Delete rows AND record that they were deleted, in the caller's transaction.
*
* A delete leaves nothing for last-write-wins to compare against, so without
* this the other device would push its still-live copy back and the row
* would silently return. Always written inside the same transaction as the
* delete it describes.
* A delete leaves nothing behind to compare, so without a tombstone another
* device would push its live copy straight back. The tombstone carries the
* row's base_seq: the server applies the delete only if nothing has changed
* the row since this device last agreed with it.
*/
async function tombstone(db: Db, tbl: string, pk: string | number): Promise<void> {
await db.run(
`INSERT INTO tombstone (tbl, pk, updated_at) VALUES (?, ?, ?)
ON CONFLICT(tbl, pk) DO UPDATE SET updated_at = excluded.updated_at`,
[tbl, String(pk), now()],
async function remove(db: Db, tbl: string, where: string, params: Params = []): Promise<void> {
const spec = SYNC_TABLES[tbl]!;
const rows = await db.all<Record<string, string | number>>(
`SELECT ${spec.pk.join(", ")}, base_seq FROM ${tbl} WHERE ${where}`,
params,
);
const t = now();
for (const r of rows) {
await db.run(
`INSERT INTO tombstone (tbl, pk, updated_at, base_seq, dirty, rev) VALUES (?, ?, ?, ?, 1, 1)
ON CONFLICT(tbl, pk) DO UPDATE SET updated_at = excluded.updated_at,
base_seq = excluded.base_seq, dirty = 1, rev = tombstone.rev + 1`,
[tbl, encodePk(tbl, r), t, Number(r.base_seq ?? 0)],
);
}
await db.run(`DELETE FROM ${tbl} WHERE ${where}`, params);
}
/* ═════════════════════════════════════════════════════════════════════
SEED WRITES — defaults, first-run state, anything the user did not do.
None of these may set updated_at; the column default of 0 is the point.
None of these may set updated_at, dirty or rev; the defaults are the point.
═════════════════════════════════════════════════════════════════════ */
/** First-run roadmap position. Unit 1.1 is where everyone starts. */
@@ -90,28 +128,26 @@ export async function seedChatTurn(db: Db, role: string, body: string, at: numbe
}
/* ═════════════════════════════════════════════════════════════════════
EDIT WRITES — a real action by the learner. Every one stamps the clock.
EDIT WRITES — a real action by the learner. Every one stamps the clock
and marks its row for the server.
═════════════════════════════════════════════════════════════════════ */
/** Answering a card in the review overlay. */
export async function editCard(db: Db, lemmaId: number, card: Card): Promise<void> {
await db.run(
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at, dirty, rev)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 1, 1)
ON CONFLICT(lemma_id) DO UPDATE SET
state = excluded.state, ease = excluded.ease, interval = excluded.interval,
due = excluded.due, reps = excluded.reps, lapses = excluded.lapses,
updated_at = excluded.updated_at`,
updated_at = excluded.updated_at, dirty = 1, rev = card.rev + 1`,
[lemmaId, card.state, card.ease, card.interval, card.due, card.reps, card.lapses, now()],
);
}
/** Forgetting a card back to new, from the vocabulary tab. */
export async function editCardReset(db: Db, lemmaId: number): Promise<void> {
await db.tx(async (tx) => {
await tx.run("DELETE FROM card WHERE lemma_id = ?", [lemmaId]);
await tombstone(tx, "card", lemmaId);
});
await db.tx((tx) => remove(tx, "card", "lemma_id = ?", [lemmaId]));
}
/** The tutor's ::progress read, or the learner's "not yet". */
@@ -121,10 +157,9 @@ export async function editUnitConfidence(
confidence: number,
): Promise<void> {
await db.run(
`INSERT INTO progress (unit_id, confidence, updated_at)
VALUES (?, ?, ?)
`INSERT INTO progress (unit_id, confidence, updated_at, dirty, rev) VALUES (?, ?, ?, 1, 1)
ON CONFLICT(unit_id) DO UPDATE SET confidence = excluded.confidence,
updated_at = excluded.updated_at`,
updated_at = excluded.updated_at, dirty = 1, rev = progress.rev + 1`,
[unitId, Math.max(0, Math.min(100, Math.round(confidence))), now()],
);
}
@@ -132,8 +167,9 @@ export async function editUnitConfidence(
/** A unit finished. Nothing un-finishes one short of a reset. */
export async function editUnitDone(db: Db, unitId: string): Promise<void> {
await db.run(
`INSERT INTO progress (unit_id, done, updated_at) VALUES (?, 1, ?)
ON CONFLICT(unit_id) DO UPDATE SET done = 1, updated_at = excluded.updated_at`,
`INSERT INTO progress (unit_id, done, updated_at, dirty, rev) VALUES (?, 1, ?, 1, 1)
ON CONFLICT(unit_id) DO UPDATE SET done = 1,
updated_at = excluded.updated_at, dirty = 1, rev = progress.rev + 1`,
[unitId, now()],
);
}
@@ -146,8 +182,9 @@ export async function editCurrentUnit(db: Db, unitId: string): Promise<void> {
/** A preference the learner changed. */
export async function editMeta(db: Db, k: string, v: string): Promise<void> {
await db.run(
`INSERT INTO meta (k, v, updated_at) VALUES (?, ?, ?)
ON CONFLICT(k) DO UPDATE SET v = excluded.v, updated_at = excluded.updated_at`,
`INSERT INTO meta (k, v, updated_at, dirty, rev) VALUES (?, ?, ?, 1, 1)
ON CONFLICT(k) DO UPDATE SET v = excluded.v,
updated_at = excluded.updated_at, dirty = 1, rev = meta.rev + 1`,
[k, v, now()],
);
}
@@ -156,29 +193,47 @@ export async function editMeta(db: Db, k: string, v: string): Promise<void> {
export async function editChatTurn(db: Db, role: string, body: string): Promise<void> {
const t = now();
await db.run(
"INSERT INTO chat (id, role, body, created_at, updated_at) VALUES (?, ?, ?, ?, ?)",
"INSERT INTO chat (id, role, body, created_at, updated_at, dirty, rev) VALUES (?, ?, ?, ?, ?, 1, 1)",
[uuidv7(t), role, body, t, t],
);
}
/**
* Raise a reset marker: `reset.<scope>` counts deliberate shrinks, and every
* device that sees it rise empties the same things. The device raising it
* records that it has already done so, so its own marker coming back from
* the server does not wipe what it has written since.
*/
async function raiseMarker(db: Db, scope: MarkerScope): Promise<void> {
const key = `reset.${scope}`;
const at = Number((await db.get<{ v: string }>("SELECT v FROM meta WHERE k = ?", [key]))?.v ?? 0);
const next = String((Number.isFinite(at) ? at : 0) + 1);
await editMeta(db, key, next);
await db.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES (?, ?, 0)", [`sync.applied.${key}`, next]);
}
/** Wipe the transcript. Progress and cards are untouched. */
export async function editChatClear(db: Db): Promise<void> {
await db.tx(async (tx) => {
const rows = await tx.all<{ id: string }>("SELECT id FROM chat");
await tx.run("DELETE FROM chat");
for (const r of rows) await tombstone(tx, "chat", r.id);
await remove(tx, "chat", "1 = 1");
await raiseMarker(tx, "chat");
});
}
/** Trim the transcript. The artifact kept the last 26 turns. */
export async function editChatTrim(db: Db, keep: number): Promise<void> {
await db.tx(async (tx) => {
const kept = "SELECT id FROM chat ORDER BY created_at DESC, id DESC LIMIT ?";
const doomed = await tx.all<{ id: string }>(`SELECT id FROM chat WHERE id NOT IN (${kept})`, [keep]);
if (!doomed.length) return;
await tx.run(`DELETE FROM chat WHERE id NOT IN (${kept})`, [keep]);
for (const r of doomed) await tombstone(tx, "chat", r.id);
});
/**
* Keep the transcript to its last `keep` turns — on THIS device.
*
* Not a deletion: nothing is tombstoned, and the server keeps the history.
* A trim used to tombstone every turn outside this device's newest 26, which
* on a device that had just pulled another's turns deleted those turns
* everywhere. A turn not yet pushed is never trimmed.
*/
export async function pruneChat(db: Db, keep: number): Promise<void> {
await db.run(
`DELETE FROM chat WHERE dirty = 0 AND id NOT IN
(SELECT id FROM chat ORDER BY created_at DESC, id DESC LIMIT ?)`,
[keep],
);
}
/** Study happened today: reviews, correct answers, drill answers. */
@@ -189,24 +244,24 @@ export async function editStudyLog(
): Promise<void> {
const { reviews = 0, correct = 0, drills = 0 } = delta;
await db.run(
`INSERT INTO study_log (day, reviews, correct, drills, updated_at)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(day) DO UPDATE SET
`INSERT INTO study_log (day, device, reviews, correct, drills, updated_at, dirty, rev)
VALUES (?, ?, ?, ?, ?, ?, 1, 1)
ON CONFLICT(day, device) DO UPDATE SET
reviews = study_log.reviews + excluded.reviews,
correct = study_log.correct + excluded.correct,
drills = study_log.drills + excluded.drills,
updated_at = excluded.updated_at`,
[day, reviews, correct, drills, now()],
updated_at = excluded.updated_at, dirty = 1, rev = study_log.rev + 1`,
[day, await deviceId(db), reviews, correct, drills, now()],
);
}
/** He looked a word up in the rail. Feeds the persistent underline. */
export async function editPeek(db: Db, form: string): Promise<void> {
await db.run(
`INSERT INTO peek (form, count, updated_at) VALUES (?, 1, ?)
ON CONFLICT(form) DO UPDATE SET count = peek.count + 1,
updated_at = excluded.updated_at`,
[form, now()],
`INSERT INTO peek (form, device, count, updated_at, dirty, rev) VALUES (?, ?, 1, ?, 1, 1)
ON CONFLICT(form, device) DO UPDATE SET count = peek.count + 1,
updated_at = excluded.updated_at, dirty = 1, rev = peek.rev + 1`,
[form, await deviceId(db), now()],
);
}
@@ -235,6 +290,23 @@ export interface AddedWord {
created: boolean;
}
/** The derived rows that make a custom word look like any other word. */
export async function deriveCustomLemma(db: Db, word: CustomWord): Promise<number> {
const id = lemmaId(word.headword, word.pos);
await db.run(
`INSERT OR REPLACE INTO lemma (id, headword, pos, freq_rank, level, gloss_en, gloss_ko,
unit_band, source)
VALUES (?, ?, ?, NULL, NULL, ?, '', 0, 'custom')`,
[id, word.headword, word.pos, word.gloss],
);
await db.run("INSERT OR REPLACE INTO surface (form, lemma_id, analysis) VALUES (?, ?, ?)", [
word.headword,
id,
"headword, custom",
]);
return id;
}
/**
* Add a word of the learner's own.
*
@@ -254,44 +326,31 @@ export async function editAddCustomWord(db: Db, word: CustomWord): Promise<Added
[headword, word.pos],
);
const t = now();
const newCardFor = (id: number) =>
tx.run(
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at, dirty, rev)
VALUES (?, 0, 2.5, 0, 0, 0, 0, ?, 1, 1)
ON CONFLICT(lemma_id) DO NOTHING`,
[id, t],
);
if (existing) {
// Already known — just make sure it is studiable. Its gloss stays the
// dictionary's; overwriting curated content from a text field would be
// a poor trade.
await tx.run(
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at)
VALUES (?, 0, 2.5, 0, 0, 0, 0, ?)
ON CONFLICT(lemma_id) DO NOTHING`,
[existing.id, now()],
);
await newCardFor(existing.id);
return { lemmaId: existing.id, created: false };
}
const id = lemmaId(headword, word.pos);
const t = now();
await tx.run(
`INSERT INTO custom_word (headword, pos, gloss, updated_at) VALUES (?, ?, ?, ?)
ON CONFLICT(headword, pos) DO UPDATE SET gloss = excluded.gloss, updated_at = excluded.updated_at`,
`INSERT INTO custom_word (headword, pos, gloss, updated_at, dirty, rev) VALUES (?, ?, ?, ?, 1, 1)
ON CONFLICT(headword, pos) DO UPDATE SET gloss = excluded.gloss,
updated_at = excluded.updated_at, dirty = 1, rev = custom_word.rev + 1`,
[headword, word.pos, gloss, t],
);
await tx.run(
`INSERT OR REPLACE INTO lemma (id, headword, pos, freq_rank, level, gloss_en, gloss_ko,
unit_band, source)
VALUES (?, ?, ?, NULL, NULL, ?, '', 0, 'custom')`,
[id, headword, word.pos, gloss],
);
await tx.run("INSERT OR REPLACE INTO surface (form, lemma_id, analysis) VALUES (?, ?, ?)", [
headword,
id,
"headword, custom",
]);
await tx.run(
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at)
VALUES (?, 0, 2.5, 0, 0, 0, 0, ?)
ON CONFLICT(lemma_id) DO NOTHING`,
[id, t],
);
const id = await deriveCustomLemma(tx, { headword, gloss, pos: word.pos });
await newCardFor(id);
return { lemmaId: id, created: true };
});
}
@@ -304,12 +363,10 @@ export async function editRemoveCustomWord(db: Db, lemmaId: number): Promise<voi
);
if (!word) return; // never touch shipped dictionary rows
await tx.run("DELETE FROM card WHERE lemma_id = ?", [lemmaId]);
await remove(tx, "card", "lemma_id = ?", [lemmaId]);
await remove(tx, "custom_word", "headword = ? AND pos = ?", [word.headword, word.pos]);
await tx.run("DELETE FROM surface WHERE lemma_id = ?", [lemmaId]);
await tx.run("DELETE FROM lemma WHERE id = ?", [lemmaId]);
await tx.run("DELETE FROM custom_word WHERE headword = ? AND pos = ?", [word.headword, word.pos]);
await tombstone(tx, "card", lemmaId);
await tombstone(tx, "custom_word", JSON.stringify([word.headword, word.pos]));
});
}
@@ -317,6 +374,10 @@ export async function editRemoveCustomWord(db: Db, lemmaId: number): Promise<voi
RESET — deliberately destructive, so it is spelled out here rather than
assembled ad hoc at a call site.
Every row is tombstoned, and the scope's marker is raised so a device
holding rows this one never saw empties them too — a reset is the one
shrink that must win over work it has not heard of.
Neither scope touches `lemma` or `surface` for shipped words: the
dictionary is reference data, rebuildable from the assets, and wiping it
would leave the app unable to gloss anything until the bands reloaded.
@@ -324,64 +385,20 @@ export async function editRemoveCustomWord(db: Db, lemmaId: number): Promise<voi
export type ResetScope = "roadmap" | "everything";
/** Meta keys holding the roadmap: where he is, what he has done recently. */
const ROAD_META = "k LIKE 'road.%'";
/** Meta keys holding the learner himself: preferences, notes, counters. */
const LEARNER_META = "k LIKE 'prefs.%' OR k LIKE 'grammar.%' OR k LIKE 'trainer.%' OR k LIKE 'learner.%'";
/** The meta keys a reset scope empties, as a WHERE clause. */
const metaWhere = (scope: MarkerScope): string =>
RESET_SCOPES[scope].meta.map((p) => `k LIKE '${p}%'`).join(" OR ") || "0";
export async function editReset(db: Db, scope: ResetScope): Promise<void> {
await db.tx(async (tx) => {
// Tombstone before deleting, while the keys are still readable — a reset
// must propagate, or the next pull restores everything it just erased.
for (const r of await tx.all<{ unit_id: string }>("SELECT unit_id FROM progress"))
await tombstone(tx, "progress", r.unit_id);
for (const r of await tx.all<{ id: string }>("SELECT id FROM chat"))
await tombstone(tx, "chat", r.id);
for (const r of await tx.all<{ phase: number; kind: string; item: string }>(
"SELECT phase, kind, item FROM phase_ledger",
))
await tombstone(tx, "phase_ledger", JSON.stringify([r.phase, r.kind, r.item]));
for (const r of await tx.all<{ k: string }>(`SELECT k FROM meta WHERE ${ROAD_META}`))
await tombstone(tx, "meta", r.k);
await tx.run("DELETE FROM progress");
await tx.run("DELETE FROM chat");
await tx.run("DELETE FROM phase_ledger");
await tx.run(`DELETE FROM meta WHERE ${ROAD_META}`);
for (const tbl of RESET_SCOPES[scope].tables) await remove(tx, tbl, "1 = 1");
await remove(tx, "meta", metaWhere(scope));
if (scope === "everything") {
for (const r of await tx.all<{ lemma_id: number }>("SELECT lemma_id FROM card"))
await tombstone(tx, "card", r.lemma_id);
for (const r of await tx.all<{ day: number }>("SELECT day FROM study_log"))
await tombstone(tx, "study_log", r.day);
for (const r of await tx.all<{ form: string }>("SELECT form FROM peek"))
await tombstone(tx, "peek", r.form);
for (const r of await tx.all<{ headword: string; pos: string }>(
"SELECT headword, pos FROM custom_word",
))
await tombstone(tx, "custom_word", JSON.stringify([r.headword, r.pos]));
for (const r of await tx.all<{ word: string }>("SELECT word FROM evidence"))
await tombstone(tx, "evidence", r.word);
for (const r of await tx.all<{ word: string }>("SELECT word FROM confusion"))
await tombstone(tx, "confusion", r.word);
for (const r of await tx.all<{ k: string }>(`SELECT k FROM meta WHERE ${LEARNER_META}`))
await tombstone(tx, "meta", r.k);
await tx.run("DELETE FROM card");
await tx.run("DELETE FROM study_log");
await tx.run("DELETE FROM peek");
await tx.run("DELETE FROM custom_word");
await tx.run("DELETE FROM evidence");
await tx.run("DELETE FROM confusion");
await tx.run("DELETE FROM surface WHERE lemma_id IN (SELECT id FROM lemma WHERE source = 'custom')");
await tx.run("DELETE FROM lemma WHERE source = 'custom'");
// Preferences, grammar flags and notes, the trainer score and the
// round counter. Device bookkeeping (schema_version, dict.*, sync.*)
// is left alone — it describes this install, not the learner.
await tx.run(`DELETE FROM meta WHERE ${LEARNER_META}`);
}
await raiseMarker(tx, scope);
});
}

8
app/src/domain/cards.ts
View File

@@ -177,7 +177,13 @@ export interface DayRow {
}
export async function studyLog(db: Db, sinceDay: number): Promise<DayRow[]> {
return db.all<DayRow>("SELECT * FROM study_log WHERE day >= ? ORDER BY day", [sinceDay]);
// One row per device per day — summed, so a phone's reviews and a
// laptop's both count and neither can overwrite the other's.
return db.all<DayRow>(
`SELECT day, SUM(reviews) AS reviews, SUM(correct) AS correct, SUM(drills) AS drills
FROM study_log WHERE day >= ? GROUP BY day ORDER BY day`,
[sinceDay],
);
}
/** Consecutive days with any activity, counting back from today. */

12
app/src/state/store.tsx
View File

@@ -30,7 +30,8 @@ import {
writeServerConfig,
type ServerConfig,
} from "../domain/server-config.js";
import { SYNC_PAUSED, trySync, type SyncResult } from "../sync/client.js";
import { forgetServerPosition, trySync, type SyncResult } from "../sync/client.js";
import { unitIndex } from "../domain/gate.js";
import type { ProgressState } from "@lib/gate.js";
import type { FocusMode } from "../domain/gate.js";
@@ -232,6 +233,9 @@ export function StoreProvider({
if (!store) return;
if (cfg) await writeServerConfig(store.db, cfg);
else await clearServerConfig(store.db);
// A different server is a different conversation: hydrate from its
// first page before anything here may be pushed to it.
await forgetServerPosition(store.db);
setServerState(cfg);
},
[store],
@@ -243,13 +247,9 @@ export function StoreProvider({
const running = useRef(false);
const syncNow = useCallback(async () => {
if (!store || !server || running.current) return;
if (SYNC_PAUSED) {
setSyncState({ at: Date.now(), result: null, error: SYNC_PAUSED, running: false });
return;
}
running.current = true;
setSyncState((s) => ({ ...s, running: true }));
const result = await trySync(store.db, server);
const result = await trySync(store.db, server, { unitIndex });
running.current = false;
setSyncState({
at: Date.now(),

534
app/src/sync/client.ts
View File

@@ -1,39 +1,34 @@
/* The sync loop: push what changed here, pull what changed there, apply.
/* The sync loop, protocol 2: pull everything, then push what changed here.
Offline-first is not negotiable. Sync runs only when a server is
configured, and every failure is non-fatal — the app must stay completely
usable with the Pi unreachable, which is the acceptance criterion the
whole port was built around. Nothing in here may throw into the UI. */
usable with the Pi unreachable. Nothing in here may throw into the UI.
The order is the first gate. A device may not push until it has pulled
every page the server holds for this epoch: the week-old laptop that
pushed first is exactly how the artifact lost a week of the phone's work.
See shared/sync-protocol.mjs for the other two gates. */
import type { Db } from "../db/types.js";
import { deriveCustomLemma } from "../db/writes.js";
import {
PAGE_SIZE,
PROTOCOL,
PROTOCOL_HEADER,
RESET_SCOPES,
SYNC_TABLES,
SYNC_TABLE_NAMES,
incomingWins,
decodePk,
encodePk,
isSyncableMetaKey,
resetCovers,
type PullResponse,
type PushResponse,
type ServerRow,
type WireRow,
} from "@shared/sync-protocol.mjs";
/** A row on the wire: its table, its columns, and when it last changed. */
export interface SyncRow {
tbl: string;
data: Record<string, string | number | null>;
updated_at: number;
/** Present and true when this row is a delete. */
deleted?: boolean;
}
export interface PushRequest {
rows: SyncRow[];
}
export interface PushResponse {
cursor: number;
}
export interface PullResponse {
rows: SyncRow[];
cursor: number;
more: boolean;
}
import { lemmaId } from "@shared/lemma-id.mjs";
import { resolve, type Data, type ResolveContext } from "./resolve.js";
export interface SyncConfig {
baseUrl: string;
@@ -43,209 +38,405 @@ export interface SyncConfig {
export interface SyncResult {
pushed: number;
pulled: number;
conflicts: number;
cursor: number;
/** The server was new to this device, or had been reset: everything was
pulled from the start and every stamped row offered back. */
rehydrated: boolean;
}
/**
* Why sync is not running, or null when it is.
*
* Cards are keyed by stable lemma ids now, while the server still holds rows
* written under the old positional ids. Exchanging them would plant cards
* under ids that name no word, on both sides. Everything stays local — which
* this app is built to do — until the sync protocol that replaces this one
* ships with its own server schema.
*/
export const SYNC_PAUSED: string | null =
"Sync is paused while the storage format changes. Everything is kept on this device";
const EPOCH = "sync.epoch";
const CURSOR = "sync.cursor";
const HYDRATED = "sync.hydrated";
const CURSOR_KEY = "sync.cursor";
const PUSHED_KEY = "sync.pushedAt";
/** Rounds of push-and-settle before giving up until the next sync. */
const MAX_PUSH_ROUNDS = 5;
/* ── local bookkeeping ────────────────────────────────────────────────
These are device state, not user data. They are written with
updated_at = 0 and excluded from the syncable-meta allowlist, so they
never travel. */
Device state, not user data: written unstamped and never dirty, and
`sync.*` is outside the syncable-meta allowlist besides. */
async function readNum(db: Db, key: string): Promise<number> {
const row = await db.get<{ v: string }>("SELECT v FROM meta WHERE k = ?", [key]);
const n = row ? Number(row.v) : 0;
return Number.isFinite(n) ? n : 0;
async function readMeta(db: Db, k: string): Promise<string | undefined> {
return (await db.get<{ v: string }>("SELECT v FROM meta WHERE k = ?", [k]))?.v;
}
async function writeNum(db: Db, key: string, value: number): Promise<void> {
await db.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES (?, ?, 0)", [
key,
String(value),
]);
async function writeMeta(db: Db, k: string, v: string | number): Promise<void> {
await db.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES (?, ?, 0)", [k, String(v)]);
}
/* ── collect ─────────────────────────────────────────────────────────── */
/** Forget this device's place with the server, so the next sync starts over. */
export async function forgetServerPosition(db: Db): Promise<void> {
await db.run("DELETE FROM meta WHERE k IN (?, ?, ?)", [EPOCH, CURSOR, HYDRATED]);
}
/** Everything edited on this device since the last successful push. */
export async function collectDirty(db: Db, pushedAt: number): Promise<SyncRow[]> {
const out: SyncRow[] = [];
/** Has this device pulled everything its server holds? */
export async function isHydrated(db: Db): Promise<boolean> {
return (await readMeta(db, HYDRATED)) === "1";
}
for (const tbl of SYNC_TABLE_NAMES) {
const spec = SYNC_TABLES[tbl]!;
const cols = [...spec.cols, "updated_at"].join(", ");
const rows = await db.all<Record<string, string | number>>(
`SELECT ${cols} FROM ${tbl} WHERE updated_at > ? ORDER BY updated_at LIMIT ?`,
[pushedAt, PAGE_SIZE],
/* ── rows ─────────────────────────────────────────────────────────────── */
type LocalRow = Data & { base_seq: number; dirty: number; rev: number; updated_at: number };
interface Local {
live: LocalRow | undefined;
tomb: { base_seq: number; dirty: number; rev: number } | undefined;
}
const whereOf = (tbl: string) => SYNC_TABLES[tbl]!.pk.map((c) => `${c} = ?`).join(" AND ");
async function readLocal(db: Db, tbl: string, pk: string): Promise<Local> {
const live = await db.get<LocalRow>(`SELECT * FROM ${tbl} WHERE ${whereOf(tbl)}`, decodePk(pk));
const tomb = await db.get<{ base_seq: number; dirty: number; rev: number }>(
"SELECT base_seq, dirty, rev FROM tombstone WHERE tbl = ? AND pk = ?",
[tbl, pk],
);
return { live, tomb };
}
const pick = (tbl: string, row: Data): Data =>
Object.fromEntries(SYNC_TABLES[tbl]!.cols.map((c) => [c, row[c] ?? null]));
/** Write a row as the server holds it: clean, and agreed at `seq`. */
async function adopt(db: Db, row: ServerRow): Promise<void> {
const spec = SYNC_TABLES[row.tbl]!;
const key = decodePk(row.pk);
if (row.deleted) {
const before = await db.get<Data>(`SELECT * FROM ${row.tbl} WHERE ${whereOf(row.tbl)}`, key);
await db.run(`DELETE FROM ${row.tbl} WHERE ${whereOf(row.tbl)}`, key);
// A clean tombstone remembers the server's seq for this key, so a later
// re-creation here pushes against the right base.
await db.run(
`INSERT INTO tombstone (tbl, pk, updated_at, base_seq, dirty, rev) VALUES (?, ?, ?, ?, 0, 0)
ON CONFLICT(tbl, pk) DO UPDATE SET base_seq = excluded.base_seq, dirty = 0`,
[row.tbl, row.pk, row.updated_at ?? 0, row.seq],
);
for (const row of rows) {
// The allowlist is applied at the point of collection, so a
// device-local meta key cannot leave this machine even by accident.
if (tbl === "meta" && !isSyncableMetaKey(String(row.k))) continue;
const { updated_at, ...data } = row;
out.push({ tbl, data, updated_at: Number(updated_at) });
if (row.tbl === "custom_word" && before) {
const id = lemmaId(String(before.headword), String(before.pos));
await db.run("DELETE FROM surface WHERE lemma_id = ? AND lemma_id IN (SELECT id FROM lemma WHERE source = 'custom')", [id]);
await db.run("DELETE FROM lemma WHERE id = ? AND source = 'custom'", [id]);
}
return;
}
const graves = await db.all<{ tbl: string; pk: string; updated_at: number }>(
"SELECT tbl, pk, updated_at FROM tombstone WHERE updated_at > ? ORDER BY updated_at LIMIT ?",
[pushedAt, PAGE_SIZE],
const data = pick(row.tbl, row.data ?? {});
const cols = spec.cols;
await db.run(
`INSERT INTO ${row.tbl} (${cols.join(", ")}, updated_at, base_seq, dirty)
VALUES (${cols.map(() => "?").join(", ")}, ?, ?, 0)
ON CONFLICT(${spec.pk.join(", ")}) DO UPDATE SET
${cols.map((c) => `${c} = excluded.${c}`).join(", ")},
updated_at = excluded.updated_at, base_seq = excluded.base_seq, dirty = 0`,
[...cols.map((c) => data[c] ?? null), row.updated_at ?? 0, row.seq],
);
await db.run("DELETE FROM tombstone WHERE tbl = ? AND pk = ?", [row.tbl, row.pk]);
if (row.tbl === "custom_word") {
await deriveCustomLemma(db, {
headword: String(data.headword),
pos: String(data.pos),
gloss: String(data.gloss ?? ""),
});
}
if (row.tbl === "meta" && String(data.k).startsWith("reset.")) {
await obeyMarker(db, String(data.k), String(data.v));
}
}
/** Ours stands: agree with the server's seq, so the next push overwrites it. */
async function keep(db: Db, tbl: string, pk: string, local: Local, seq: number): Promise<void> {
if (local.live) {
await db.run(`UPDATE ${tbl} SET base_seq = ?, dirty = 1 WHERE ${whereOf(tbl)}`, [seq, ...decodePk(pk)]);
} else {
await db.run("UPDATE tombstone SET base_seq = ?, dirty = 1 WHERE tbl = ? AND pk = ?", [seq, tbl, pk]);
}
}
/** A combination of both: written here, agreed at `seq`, and pushed. */
async function merge(db: Db, tbl: string, pk: string, seq: number, data: Data): Promise<void> {
const cols = SYNC_TABLES[tbl]!.cols;
await db.run(
`UPDATE ${tbl} SET ${cols.map((c) => `${c} = ?`).join(", ")},
base_seq = ?, dirty = 1, rev = rev + 1 WHERE ${whereOf(tbl)}`,
[...cols.map((c) => data[c] ?? null), seq, ...decodePk(pk)],
);
}
/** Settle one disagreement between this device and the server. */
async function settle(db: Db, remote: ServerRow, local: Local, ctx: ResolveContext): Promise<void> {
// The server has no row at all: nothing to disagree with.
if (remote.seq === 0) {
if (local.live) await keep(db, remote.tbl, remote.pk, local, 0);
else await db.run("DELETE FROM tombstone WHERE tbl = ? AND pk = ?", [remote.tbl, remote.pk]);
return;
}
const localSide = local.live
? { deleted: false, data: local.live as Data }
: { deleted: true, data: null };
const outcome = resolve(remote.tbl, localSide, { deleted: remote.deleted, data: remote.data }, ctx);
if (outcome.kind === "adopt") await adopt(db, remote);
else if (outcome.kind === "keep") await keep(db, remote.tbl, remote.pk, local, remote.seq);
else await merge(db, remote.tbl, remote.pk, remote.seq, outcome.data);
}
/* ── deliberate shrinks ───────────────────────────────────────────────── */
/**
* Empty what a reset marker covers, once per rise of its counter.
*
* Unsynced edits here go too: a reset this device had not heard of still
* wins over work done on the data it reset. Rows written after the reset
* arrive later in the pull — higher change_seq — and survive it.
*/
async function obeyMarker(db: Db, key: string, value: string): Promise<void> {
const scope = key.slice("reset.".length) as keyof typeof RESET_SCOPES;
const spec = RESET_SCOPES[scope];
if (!spec) return;
const seen = Number((await readMeta(db, `sync.applied.${key}`)) ?? 0);
const n = Number(value);
if (!(n > seen)) return;
for (const tbl of spec.tables) {
await db.run(`DELETE FROM ${tbl}`);
await db.run("DELETE FROM tombstone WHERE tbl = ?", [tbl]);
}
for (const prefix of spec.meta) {
await db.run("DELETE FROM meta WHERE k LIKE ?", [`${prefix}%`]);
}
if (spec.tables.includes("custom_word")) {
await db.run("DELETE FROM surface WHERE lemma_id IN (SELECT id FROM lemma WHERE source = 'custom')");
await db.run("DELETE FROM lemma WHERE source = 'custom'");
}
await writeMeta(db, `sync.applied.${key}`, n);
}
/** Resets raised here that the server has not heard of yet. */
async function pendingResets(db: Db): Promise<string[]> {
const rows = await db.all<{ k: string }>("SELECT k FROM meta WHERE k LIKE 'reset.%' AND dirty = 1");
return rows.map((r) => r.k.slice("reset.".length));
}
/* ── pull ─────────────────────────────────────────────────────────────── */
async function applyIncoming(
db: Db,
row: ServerRow,
pending: string[],
ctx: ResolveContext,
): Promise<"applied" | "settled" | "skipped"> {
const spec = SYNC_TABLES[row.tbl];
if (!spec) return "skipped"; // a newer server; ignore rather than fail
const key = decodePk(row.pk);
const probe: Data = row.data ?? Object.fromEntries(spec.pk.map((c, i) => [c, key[i] ?? null]));
if (row.tbl === "meta" && !isSyncableMetaKey(String(probe.k ?? key[0]))) return "skipped";
// A reset raised here, not yet pushed, covers this row: the server's copy
// predates it and is about to be emptied everywhere anyway.
if (pending.some((scope) => resetCovers(scope, row.tbl, probe))) return "skipped";
const local = await readLocal(db, row.tbl, row.pk);
const mine = local.live ?? local.tomb;
if (mine && mine.base_seq === row.seq) return "skipped"; // our own write, echoed back
if (!mine || !mine.dirty) {
await adopt(db, row);
return "applied";
}
await settle(db, row, local, ctx);
return "settled";
}
/* ── push ─────────────────────────────────────────────────────────────── */
interface Outgoing {
wire: WireRow;
rev: number;
/** A live row re-created over a tombstone this device had applied. */
overTomb: boolean;
}
const keyOf = (tbl: string, pk: string) => JSON.stringify([tbl, pk]);
/** Up to one page of dirty rows: deletes and reset markers first, so the
server orders a reset before anything written after it. */
async function collectDirty(db: Db): Promise<Outgoing[]> {
const out: Outgoing[] = [];
const graves = await db.all<{ tbl: string; pk: string; base_seq: number; rev: number; updated_at: number }>(
"SELECT tbl, pk, base_seq, rev, updated_at FROM tombstone WHERE dirty = 1 LIMIT ?",
[PAGE_SIZE],
);
for (const g of graves) {
out.push({ tbl: g.tbl, data: { pk: g.pk }, updated_at: Number(g.updated_at), deleted: true });
if (!SYNC_TABLES[g.tbl]) continue;
out.push({
wire: { tbl: g.tbl, pk: g.pk, base: g.base_seq, deleted: true, data: null, updated_at: g.updated_at },
rev: g.rev,
overTomb: false,
});
}
const ordered = ["meta", ...SYNC_TABLE_NAMES.filter((t) => t !== "meta")];
for (const tbl of ordered) {
if (out.length >= PAGE_SIZE) break;
const rows = await db.all<LocalRow>(
`SELECT * FROM ${tbl} WHERE dirty = 1 ${tbl === "meta" ? "ORDER BY k LIKE 'reset.%' DESC" : ""} LIMIT ?`,
[PAGE_SIZE - out.length],
);
for (const row of rows) {
if (tbl === "meta" && !isSyncableMetaKey(String(row.k))) {
// Device bookkeeping that was edited: it never travels, so it is
// never really dirty.
await db.run("UPDATE meta SET dirty = 0 WHERE k = ?", [String(row.k)]);
continue;
}
const pk = encodePk(tbl, row);
let base = row.base_seq;
let overTomb = false;
if (base === 0) {
const tomb = await db.get<{ base_seq: number }>(
"SELECT base_seq FROM tombstone WHERE tbl = ? AND pk = ? AND dirty = 0",
[tbl, pk],
);
if (tomb) {
base = tomb.base_seq;
overTomb = true;
}
}
out.push({
wire: { tbl, pk, base, deleted: false, data: pick(tbl, row), updated_at: row.updated_at },
rev: row.rev,
overTomb,
});
}
}
return out;
}
/* ── apply ───────────────────────────────────────────────────────────── */
/** Apply one incoming row, last-write-wins against what is already here. */
async function applyRow(db: Db, row: SyncRow): Promise<boolean> {
const spec = SYNC_TABLES[row.tbl];
if (!spec) return false; // unknown table — a newer server; ignore rather than fail
if (row.deleted) {
const pk = String(row.data.pk);
const where = spec.pk.map((c) => `${c} = ?`).join(" AND ");
const parts = pk.split(" ");
const local = await db.get<{ updated_at: number }>(
`SELECT updated_at FROM ${row.tbl} WHERE ${where}`,
parts,
);
if (local && !incomingWins(row.updated_at, local.updated_at)) return false;
await db.tx(async (tx) => {
await tx.run(`DELETE FROM ${row.tbl} WHERE ${where}`, parts);
await tx.run(
`INSERT INTO tombstone (tbl, pk, updated_at) VALUES (?, ?, ?)
ON CONFLICT(tbl, pk) DO UPDATE SET updated_at = excluded.updated_at`,
[row.tbl, pk, row.updated_at],
async function acknowledge(db: Db, sent: Outgoing[], applied: PushResponse["applied"]): Promise<void> {
const byKey = new Map(sent.map((s) => [keyOf(s.wire.tbl, s.wire.pk), s]));
for (const a of applied) {
const s = byKey.get(keyOf(a.tbl, a.pk));
if (!s) continue;
if (s.wire.deleted) {
await db.run(
`UPDATE tombstone SET base_seq = ?, dirty = CASE WHEN rev = ? THEN 0 ELSE dirty END
WHERE tbl = ? AND pk = ?`,
[a.seq, s.rev, a.tbl, a.pk],
);
});
return true;
} else {
// Clean only if nothing changed the row while the push was in flight.
await db.run(
`UPDATE ${a.tbl} SET base_seq = ?, dirty = CASE WHEN rev = ? THEN 0 ELSE dirty END
WHERE ${whereOf(a.tbl)}`,
[a.seq, s.rev, ...decodePk(a.pk)],
);
if (s.overTomb) await db.run("DELETE FROM tombstone WHERE tbl = ? AND pk = ?", [a.tbl, a.pk]);
}
}
if (row.tbl === "meta" && !isSyncableMetaKey(String(row.data.k))) return false;
const where = spec.pk.map((c) => `${c} = ?`).join(" AND ");
const keys = spec.pk.map((c) => row.data[c] as string | number);
const local = await db.get<{ updated_at: number }>(
`SELECT updated_at FROM ${row.tbl} WHERE ${where}`,
keys,
);
if (local && !incomingWins(row.updated_at, local.updated_at)) return false;
const cols = [...spec.cols, "updated_at"];
const holes = cols.map(() => "?").join(", ");
const values = [...spec.cols.map((c) => row.data[c] ?? null), row.updated_at];
await db.run(`INSERT OR REPLACE INTO ${row.tbl} (${cols.join(", ")}) VALUES (${holes})`, values);
return true;
}
/* ── transport ───────────────────────────────────────────────────────── */
/* ── transport ────────────────────────────────────────────────────────── */
async function call<T>(
cfg: SyncConfig,
path: string,
init: RequestInit,
signal?: AbortSignal,
): Promise<T> {
async function call<T>(cfg: SyncConfig, path: string, init: RequestInit, signal?: AbortSignal): Promise<T> {
const res = await fetch(`${cfg.baseUrl.replace(/\/$/, "")}${path}`, {
...init,
signal,
headers: {
"content-type": "application/json",
authorization: `Bearer ${cfg.token}`,
[PROTOCOL_HEADER]: String(PROTOCOL),
...(init.headers ?? {}),
},
});
if (res.status === 426) throw new Error("the server speaks a different sync protocol — update it");
if (!res.ok) throw new Error(`sync ${path}: HTTP ${res.status}`);
return (await res.json()) as T;
}
/* ── the loop ────────────────────────────────────────────────────────── */
/* ── the loop ─────────────────────────────────────────────────────────── */
/** Re-agree with a server from nothing: every stamped row is offered again. */
async function rehydrate(db: Db, epoch: string): Promise<void> {
await db.tx(async (tx) => {
for (const tbl of [...SYNC_TABLE_NAMES, "tombstone"]) {
await tx.run(
`UPDATE ${tbl} SET base_seq = 0, dirty = CASE WHEN updated_at > 0 THEN 1 ELSE 0 END, rev = rev + 1`,
);
}
await writeMeta(tx, EPOCH, epoch);
await writeMeta(tx, CURSOR, 0);
await writeMeta(tx, HYDRATED, 0);
});
}
/**
* One full exchange. Push first so local edits are never lost to an
* incoming row that would have overwritten them; then pull, paging until
* the server has nothing newer.
* One full exchange: pull every page, then push until nothing is dirty.
*
* `ctx` lets a conflict ask the roadmap which of two units is further on.
*/
export async function syncOnce(
db: Db,
cfg: SyncConfig,
ctx: ResolveContext,
signal?: AbortSignal,
): Promise<SyncResult> {
// Anything edited after this instant stays dirty, whatever the pull
// brings back. That is what keeps an edit made *during* a sync from being
// marked clean by a remote row that happens to carry a later timestamp.
const startedAt = Date.now();
const pushedAt = await readNum(db, PUSHED_KEY);
const dirty = await collectDirty(db, pushedAt);
let watermark = pushedAt;
if (dirty.length) {
const body: PushRequest = { rows: dirty };
await call<PushResponse>(
cfg,
"/api/sync",
{ method: "POST", body: JSON.stringify(body) },
signal,
);
// Only advance once the server has them. A failed push throws before
// this line and the rows stay dirty, which is the safe direction.
watermark = dirty.reduce((m, r) => Math.max(m, r.updated_at), watermark);
await writeNum(db, PUSHED_KEY, watermark);
}
/* The pull cursor advances ONLY as rows are applied.
It must never be taken from the push response: the server's newest
change_seq includes rows this device has not seen, and adopting it
would skip them permanently — a silent, unrecoverable data loss, since
nothing would ever ask for that range again. */
let cursor = await readNum(db, CURSOR_KEY);
let epoch = (await readMeta(db, EPOCH)) ?? "";
let cursor = Number((await readMeta(db, CURSOR)) ?? 0) || 0;
let rehydrated = false;
let pulled = 0;
let conflicts = 0;
/* Gate 1: hydrate. Nothing leaves this device until every page is in. */
for (;;) {
const page = await call<PullResponse>(
cfg,
`/api/sync?cursor=${cursor}`,
{ method: "GET" },
signal,
);
for (const row of page.rows) {
if (await applyRow(db, row)) pulled++;
// A row we just received is not a local edit. Marking it clean stops
// the next sync pushing it straight back. Bounded by startedAt so a
// concurrent local edit is never swept up with it.
if (row.updated_at <= startedAt) watermark = Math.max(watermark, row.updated_at);
const page = await call<PullResponse>(cfg, `/api/sync?cursor=${cursor}`, { method: "GET" }, signal);
if (page.epoch !== epoch) {
if (rehydrated) throw new Error("the server was reset during this sync");
await rehydrate(db, page.epoch);
epoch = page.epoch;
cursor = 0;
rehydrated = true;
continue;
}
const pending = await pendingResets(db);
for (const row of page.rows) {
const r = await db.tx((tx) => applyIncoming(tx, row, pending, ctx));
if (r === "applied") pulled++;
if (r === "settled") conflicts++;
}
cursor = page.cursor;
await writeNum(db, CURSOR_KEY, cursor);
await writeMeta(db, CURSOR, cursor);
if (!page.more) break;
}
await writeMeta(db, HYDRATED, 1);
if (watermark > pushedAt) await writeNum(db, PUSHED_KEY, watermark);
/* Then push. A conflict is settled locally and, if ours stands, pushed
again against the server's current seq. */
let pushed = 0;
for (let round = 0; round < MAX_PUSH_ROUNDS; round++) {
const batch = await collectDirty(db);
if (!batch.length) break;
const res = await call<PushResponse>(
cfg,
"/api/sync",
{ method: "POST", body: JSON.stringify({ rows: batch.map((b) => b.wire) }) },
signal,
);
if (res.epoch !== epoch) {
// Reset between our pull and our push: start over next time.
await forgetServerPosition(db);
break;
}
await acknowledge(db, batch, res.applied);
pushed += res.applied.length;
for (const c of res.conflicts) {
await db.tx(async (tx) => settle(tx, c, await readLocal(tx, c.tbl, c.pk), ctx));
conflicts++;
}
}
return { pushed: dirty.length, pulled, cursor };
return { pushed, pulled, conflicts, cursor, rehydrated };
}
/**
@@ -258,11 +449,12 @@ export async function syncOnce(
export async function trySync(
db: Db,
cfg: SyncConfig | null,
ctx: ResolveContext,
signal?: AbortSignal,
): Promise<SyncResult | null> {
if (!cfg?.baseUrl || !cfg.token || SYNC_PAUSED) return null;
if (!cfg?.baseUrl || !cfg.token) return null;
try {
return await syncOnce(db, cfg, signal);
return await syncOnce(db, cfg, ctx, signal);
} catch (err) {
console.warn("[sync] skipped:", err instanceof Error ? err.message : err);
return null;

131
app/src/sync/resolve.ts Normal file
View File

@@ -0,0 +1,131 @@
/* Gate 3: what to do when two copies of one row disagree.
A conflict means the server's copy changed after this device last agreed
with it, while this device changed its own copy too. The counter says
that much and no more: it cannot say which change is "right".
The artifact's answer, which PORT.md adopts: THE COPY THAT HOLDS MORE
WINS. For one learner whose worst conflict is a duplicated grade, "more
reviews, more evidence, more finished units" is a better tiebreak than
"newer" — it converges, and it never trades real work for an empty
default. The one exception is a deliberate delete, which is obeyed.
Three outcomes:
adopt — take the server's copy;
keep — keep ours, and push it back over the server's;
merge — write a combination, and push that. */
export type Value = string | number | null;
export type Data = Record<string, Value>;
export interface Side {
deleted: boolean;
data: Data | null;
}
export type Resolution = { kind: "adopt" } | { kind: "keep" } | { kind: "merge"; data: Data };
export interface ResolveContext {
/** A unit's position on the roadmap, or -1. */
unitIndex: (unitId: string) => number;
}
const ADOPT: Resolution = { kind: "adopt" };
const KEEP: Resolution = { kind: "keep" };
const num = (v: Value | undefined): number => {
const n = Number(v ?? 0);
return Number.isFinite(n) ? n : 0;
};
/** More wins; a tie goes to the server, so every device lands on one copy. */
const heavier = (local: number, remote: number): Resolution => (local > remote ? KEEP : ADOPT);
/** Pick adopt/keep when a merge came out equal to one side. */
function settle(local: Data, remote: Data, merged: Data): Resolution {
const same = (a: Data) => Object.keys(merged).every((k) => String(merged[k]) === String(a[k]));
if (same(remote)) return ADOPT;
if (same(local)) return KEEP;
return { kind: "merge", data: merged };
}
function parseObject(v: Value | undefined): Record<string, unknown> | null {
try {
const o = JSON.parse(String(v));
return o && typeof o === "object" && !Array.isArray(o) ? (o as Record<string, unknown>) : null;
} catch {
return null;
}
}
function resolveMeta(local: Data, remote: Data, ctx: ResolveContext): Resolution {
const k = String(remote.k ?? local.k ?? "");
if (k === "road.unit") {
return heavier(ctx.unitIndex(String(local.v)), ctx.unitIndex(String(remote.v)));
}
if (k === "learner.round" || k.startsWith("reset.")) {
return heavier(num(local.v), num(remote.v));
}
if (k === "trainer.conjugation") {
return heavier(num(parseObject(local.v)?.n as Value), num(parseObject(remote.v)?.n as Value));
}
if (k === "grammar.learned" || k === "grammar.notes") {
const a = parseObject(local.v);
const b = parseObject(remote.v);
if (!a || !b) return ADOPT;
// Union. Where both have a key, the server's value for a flag; for a
// note, the longer one — a note is only ever grown by its owner.
const out: Record<string, unknown> = { ...a, ...b };
if (k === "grammar.notes") {
for (const key of Object.keys(a)) {
if (key in b && String(a[key] ?? "").length > String(b[key] ?? "").length) out[key] = a[key];
}
}
return settle(local, remote, { ...remote, v: JSON.stringify(out) });
}
// Preferences, recent task types: the server's copy.
return ADOPT;
}
export function resolve(tbl: string, local: Side, remote: Side, ctx: ResolveContext): Resolution {
if (local.deleted && remote.deleted) return ADOPT;
// A deliberate delete is obeyed, whichever side made it.
if (remote.deleted) return ADOPT;
if (local.deleted) return KEEP;
const a = local.data ?? {};
const b = remote.data ?? {};
switch (tbl) {
case "card":
return heavier(num(a.reps) + num(a.lapses), num(b.reps) + num(b.lapses));
case "evidence":
return heavier(
num(a.ok) + num(a.wrong) + num(a.lookups),
num(b.ok) + num(b.wrong) + num(b.lookups),
);
case "progress": {
const more = num(a.answers) > num(b.answers) ? a : b;
return settle(a, b, {
...b,
done: Math.max(num(a.done), num(b.done)),
answers: Math.max(num(a.answers), num(b.answers)),
confidence: more.confidence ?? 0,
note: more.note ?? "",
});
}
case "meta":
return resolveMeta(a, b, ctx);
// Keyed so that the same key means the same thing on every device:
// chat turns and per-device counters never genuinely conflict, and for
// a custom word's gloss, a confusion or a ledger tick the server's copy
// is as good as ours.
default:
return ADOPT;
}
}

66
app/src/ui/tutor/TutorTab.tsx
View File

@@ -15,14 +15,16 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { useStore } from "../../state/store.js";
import { editChatClear, editChatTrim, editChatTurn, seedChatTurn } from "../../db/writes.js";
import { editChatClear, editChatTurn, pruneChat, seedChatTurn } from "../../db/writes.js";
import { parseMessage } from "../../domain/gloss.js";
import {
assemblePrompt,
gateFor,
FOCUS_MODES,
UNITS,
VOCAB_CAP,
type BandQuery,
type FlatUnit,
} from "../../domain/gate.js";
import type { FocusMode } from "../../domain/gate.js";
import { applyProgressReport, currentUnit } from "../../domain/progress.js";
@@ -88,6 +90,24 @@ async function readBandWords(db: Db, band: number, ceiling: number): Promise<str
return rows.map((r) => r.headword);
}
/* ── the opening ─────────────────────────────────────────────────── */
/** The app's own first message — the artifact's wording. */
function openingTurn(unit: FlatUnit): string {
const first = unit.id === UNITS[0]!.id;
return [
"반가워요. 저는 선생님이에요 — your reading tutor.",
"",
first
? "We're starting at the beginning, and we'll go one small step at a time. Nothing will turn up in an exercise that I haven't taught you first — if it does, tell me and I'll drop it."
: "Picking up where you left off. Nothing will turn up in an exercise that I haven't taught you first — if it does, tell me and I'll drop it.",
"",
`The plan is six phases, ${UNITS.length} units, ending with you reading a manhwa page at speed. **Roadmap** above shows the whole thing; you can jump anywhere in it whenever you like.`,
"",
`Hit **시작 · Start unit** below, and we'll pick up ${unit.id} ${unit.ko} — ${unit.name}.`,
].join("\n");
}
/* ── the tab ─────────────────────────────────────────────────────── */
export function TutorTab() {
@@ -216,6 +236,11 @@ export function TutorTab() {
focus: prefs.focus,
});
const history = turns.map((t) => ({ role: t.role, content: t.body }));
// The transcript opens with the app's own assistant turn, and a
// conversation has to open with the learner — the artifact's fix.
if (history[0]?.role === "assistant") {
history.unshift({ role: "user", content: "Let's continue the lesson." });
}
try {
const result = await sample(
@@ -232,7 +257,7 @@ export function TutorTab() {
remounted it once the read returned — the reply visibly
disappeared and the log jumped by its height each time. */
await editChatTurn(db, "assistant", result.text);
await editChatTrim(db, KEEP_TURNS);
await pruneChat(db, KEEP_TURNS);
const committed = await readTurns();
/* All three in one batch. Leaving `busy` set until the finally
block meant one render where the reply was committed but the
@@ -288,7 +313,14 @@ export function TutorTab() {
/* Open the lesson if there is no transcript yet — EXACTLY ONCE.
The guard is claimed synchronously, before the first await: setting it
after one would let every concurrent run past it, which is precisely how
this managed to seed the opening turn three times over. */
this managed to seed the opening turn three times over.
The opening is written by the app, not asked of the model, and it is a
seed: unstamped, never dirty, never synced. Booting used to send "Start
unit" at once, so a fresh install wrote a stamped reply and a progress
edit before anyone could configure a server — boot-time writes that
looked like the newest work in the world. Now nothing is written until
the learner presses Start. */
const opened = useRef(false);
useEffect(() => {
if (opened.current) return;
@@ -296,12 +328,10 @@ export function TutorTab() {
void (async () => {
if ((await loadTurns()) > 0) return; // a transcript already exists
// The opening turn is app-supplied, not a user edit: unstamped.
await seedChatTurn(db, "user", `Start unit ${unit.id}.`, 0);
await seedChatTurn(db, "assistant", openingTurn(unit), 0);
await loadTurns();
await sendRef.current(`Start unit ${unit.id}.`, { record: false });
})();
// Mount only: `send` and `unit.id` are read through the ref / at mount.
// Mount only: `unit` is read at mount.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);
@@ -313,9 +343,12 @@ export function TutorTab() {
await editChatClear(db);
setRevealed(new Set());
setRecent([]);
await seedChatTurn(db, "assistant", openingTurn(unit), 0);
await loadTurns();
await sendRef.current(`Start unit ${unit.id}.`, { record: true });
}, [db, loadTurns, unit.id]);
}, [db, loadTurns, unit]);
/** Nothing asked yet: the opening is all there is, and Start begins it. */
const notStarted = !turns.some((t) => t.role === "user");
/* ── the rail ── */
@@ -536,6 +569,21 @@ export function TutorTab() {
</div>
{notStarted && !busy && (
<div className="chat-start">
<button
className="btn primary"
onClick={() =>
void send(
`Let's start unit ${unit.id} ${unit.ko} (${unit.name}). Give me the full introduction, then a first exercise.`,
)
}
>
시작 · Start unit
</button>
</div>
)}
{error && <div className="callout warn chat-error">{error}</div>}
<div className="chat-foot" ref={foot}>

7
app/src/ui/tutor/tutor.css
View File

@@ -189,3 +189,10 @@
border-top: 1px solid var(--line);
margin: 10px 0;
}
/* Start: the only action on a transcript that is just the app's opening. */
.chat-start {
display: flex;
justify-content: center;
padding: 10px 0 4px;
}