fabi/Hankan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(dict): stable lemma ids — a card names its word, not a build position

Browse Source 
Cards point at lemmas by id, and an id was the entry's position in the
sorted build. One word added near the top of the dictionary would have
moved every card below it onto a different word — silently, because loaded
bands were recorded by number and a rebuilt dictionary never reached an
existing install anyway. A custom word took max(id)+1 on whichever device
added it, so the same id meant different words on a phone and a laptop.

An id is now lemmaId(headword, pos), a 53-bit hash defined once in
shared/ and used by the build, the loader, custom words and the migration.
The build asserts all 30,520 entries are collision-free, and a test pins the
function itself, since changing it re-keys every card.

Band files are format 2: they carry no ids at all. The loader derives each
id from the word, and a surface names its lemma by row index in the same
file. Writing hashed ids out cost 0.5 MB of incompressible digits; leaving
them out makes the files smaller than before (1.1 MB -> 1.0 MB).

The loaded dictionary is now versioned by its band hashes, so a rebuild
reloads on the next boot — safe only now that a reload cannot move a card.

Migration 6 re-keys an existing install without stamping anything: cards
and their tombstones move through the lemma rows still loaded, custom words
become custom_word rows (the learner's data, which can travel) carrying the
time their card was made, and the dictionary is dropped to reload.

Sync is paused until the protocol that replaces it lands: the server still
holds rows under the old ids, and exchanging them would plant cards that
name no word.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-09-16 19:59:14 +02:00
parent 04a0900a2c
commit f8183d786f
24 changed files with 595 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
app/src/db/migrate.ts
View File

@@ -17,17 +17,26 @@ async function readVersion(db: Db): Promise<number> {
* Bring the database up to SCHEMA_VERSION. Idempotent: running it twice is a
* no-op, which is what makes it safe to call on every launch.
*
* Each migration's SQL and its data step run in ONE transaction with the
* version bump, so a migration either happened completely or not at all.
*
* The version row is app bookkeeping, not a user edit, so it is written with
* updated_at = 0 — see the timestamp rule in migrations.ts.
*
* `upTo` exists for tests that need a database as an older build left it.
*/
export async function migrate(db: Db): Promise<{ from: number; to: number }> {
export async function migrate(
db: Db,
upTo: number = SCHEMA_VERSION,
): Promise<{ from: number; to: number }> {
await db.exec(BOOTSTRAP_SQL);
const from = await readVersion(db);
for (const m of MIGRATIONS) {
if (m.id <= from) continue;
if (m.id <= from || m.id > upTo) continue;
await db.tx(async (tx) => {
await tx.exec(m.sql);
if (m.run) await m.run(tx);
await tx.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES (?, ?, 0)", [
VERSION_KEY,
String(m.id),
@@ -35,5 +44,5 @@ export async function migrate(db: Db): Promise<{ from: number; to: number }> {
});
}
return { from, to: SCHEMA_VERSION };
return { from, to: Math.min(upTo, SCHEMA_VERSION) };
}

107
app/src/db/migrations.ts
View File

@@ -15,10 +15,21 @@
Sync itself is out of scope for this pass; `change_seq` is deliberately
absent because it is server-assigned and arrives with the sync layer. */
import type { Db } from "./types.js";
import { lemmaId } from "@shared/lemma-id.mjs";
export interface Migration {
id: number;
name: string;
sql: string;
/**
* A data step, for what SQL cannot express. Runs after `sql`, inside the
* same transaction. It converts representation only: it must never stamp
* a row or invent user data — see "Migrations run after the first pull,
* never at boot" in PORT.md; a repair that empties something belongs in
* the sync layer's afterHydration hook, not here.
*/
run?: (db: Db) => Promise<void>;
}
export const MIGRATIONS: Migration[] = [
@@ -149,8 +160,104 @@ export const MIGRATIONS: Migration[] = [
DELETE FROM meta WHERE k = 'seed.known';
`,
},
{
id: 6,
name: "stable lemma ids — a card names its word, not a build position",
sql: /* sql */ `
-- The learner's own words, as data that can travel. A custom lemma row
-- is reference data derived from this; it used to BE the only record,
-- under an id that meant a different word on every device.
CREATE TABLE IF NOT EXISTS custom_word (
headword TEXT NOT NULL,
pos TEXT NOT NULL,
gloss TEXT NOT NULL DEFAULT '',
updated_at INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (headword, pos)
) WITHOUT ROWID;
`,
run: rekeyLemmas,
},
];
/**
* Migration 6's data step.
*
* Ids were positions in the sorted build, so a dictionary change moved cards
* onto other words. They are now lemmaId(headword, pos). Every reference to
* an old id is rewritten through the lemma rows still loaded — the only place
* the old id's word is recorded — and then the dictionary itself is dropped,
* so the bands reload under the new ids.
*
* Nothing is stamped: the rows are the same rows, renamed. A card whose lemma
* is not loaded cannot be named, so it is left exactly as it was.
*/
async function rekeyLemmas(db: Db): Promise<void> {
const named = await db.all<{ id: number; headword: string; pos: string }>(
"SELECT id, headword, pos FROM lemma",
);
const next = new Map(named.map((l) => [l.id, lemmaId(l.headword, l.pos)]));
/* Cards. Two passes through negative ids, so no new id can collide with
an old one that has not moved yet. */
const cards = await db.all<{ lemma_id: number }>("SELECT lemma_id FROM card");
for (const { lemma_id } of cards) {
const to = next.get(lemma_id);
if (to !== undefined && to !== lemma_id) {
await db.run("UPDATE card SET lemma_id = ? WHERE lemma_id = ?", [-to, lemma_id]);
}
}
await db.run("UPDATE card SET lemma_id = -lemma_id WHERE lemma_id < 0");
/* A card's tombstone names it by the same id, as text. */
const graves = await db.all<{ pk: string }>("SELECT pk FROM tombstone WHERE tbl = 'card'");
for (const { pk } of graves) {
const to = next.get(Number(pk));
if (to !== undefined) {
await db.run(
"INSERT OR REPLACE INTO tombstone (tbl, pk, updated_at) SELECT 'card', ?, updated_at FROM tombstone WHERE tbl = 'card' AND pk = ?",
[String(to), pk],
);
if (String(to) !== pk) await db.run("DELETE FROM tombstone WHERE tbl = 'card' AND pk = ?", [pk]);
}
}
/* The learner's own words become custom_word rows, carrying the time their
card was made — the moment he added them — and their lemma is re-made
under its stable id. */
const custom = await db.all<{ headword: string; pos: string; gloss_en: string }>(
"SELECT headword, pos, gloss_en FROM lemma WHERE source = 'custom'",
);
// The cards have already moved, so each is found under its new id.
for (const w of custom) {
const id = lemmaId(w.headword, w.pos);
const card = await db.get<{ updated_at: number }>("SELECT updated_at FROM card WHERE lemma_id = ?", [id]);
await db.run(
"INSERT OR IGNORE INTO custom_word (headword, pos, gloss, updated_at) VALUES (?, ?, ?, ?)",
[w.headword, w.pos, w.gloss_en, card?.updated_at ?? 0],
);
}
/* Drop the dictionary. The custom lemmas go too and are re-made from
custom_word; the shipped bands reload on the next boot, under new ids,
because nothing is recorded as loaded any more. */
await db.run("DELETE FROM surface");
await db.run("DELETE FROM lemma");
await db.run("DELETE FROM meta WHERE k IN ('dict.loadedBands', 'dict.version')");
for (const w of custom) {
const id = lemmaId(w.headword, w.pos);
await db.run(
`INSERT OR REPLACE INTO lemma (id, headword, pos, freq_rank, level, gloss_en, gloss_ko, unit_band, source)
VALUES (?, ?, ?, NULL, NULL, ?, '', 0, 'custom')`,
[id, w.headword, w.pos, w.gloss_en],
);
await db.run("INSERT OR REPLACE INTO surface (form, lemma_id, analysis) VALUES (?, ?, ?)", [
w.headword,
id,
"headword, custom",
]);
}
}
export const SCHEMA_VERSION = MIGRATIONS[MIGRATIONS.length - 1]!.id;
/** Bootstrap DDL: meta must exist before we can read the schema version. */

59
app/src/db/writes.ts
View File

@@ -26,6 +26,7 @@
import type { Db, Params } from "./types.js";
import type { Card, Grade } from "@lib/srs.js";
import { lemmaId } from "@shared/lemma-id.mjs";
/* ─────────────────────────────────────────────────────────────────────
The clock. This is the ONLY place in src/db/ that reads it.
@@ -215,17 +216,15 @@ export async function editPeek(db: Db, form: string): Promise<void> {
/* ═════════════════════════════════════════════════════════════════════
CUSTOM WORDS — the learner's own additions.
These live in `lemma` alongside the shipped dictionary, but their ids
come from a reserved range far above anything the build emits. Band ids
are assigned sequentially from 1, so a custom word placed in that range
would be silently overwritten the next time `npm run dict:build` runs and
the band files are reloaded. The reserved range is what keeps the
learner's own vocabulary from being collateral damage of a dictionary
rebuild.
═════════════════════════════════════════════════════════════════════ */
The record is a `custom_word` row: user data, stamped, and the thing that
travels between devices. The `lemma` and `surface` rows that let the rest
of the app treat it like any other word are derived from it.
/** First id available to custom words. The build never emits ids this high. */
export const CUSTOM_LEMMA_BASE = 10_000_000;
A custom word's lemma id is lemmaId(headword, pos) — the same function the
dictionary build uses — so it cannot collide with a shipped word, a later
rebuild cannot overwrite it with something else, and the same word added
on a phone and a laptop is the same card on both.
═════════════════════════════════════════════════════════════════════ */
export interface CustomWord {
headword: string;
@@ -271,15 +270,17 @@ export async function editAddCustomWord(db: Db, word: CustomWord): Promise<Added
return { lemmaId: existing.id, created: false };
}
const top = await tx.get<{ id: number | null }>(
"SELECT max(id) AS id FROM lemma WHERE id >= ?",
[CUSTOM_LEMMA_BASE],
);
const id = Math.max(CUSTOM_LEMMA_BASE, (top?.id ?? 0) + 1);
const id = lemmaId(headword, word.pos);
const t = now();
await tx.run(
`INSERT INTO lemma (id, headword, pos, freq_rank, level, gloss_en, gloss_ko,
unit_band, source)
`INSERT INTO custom_word (headword, pos, gloss, updated_at) VALUES (?, ?, ?, ?)
ON CONFLICT(headword, pos) DO UPDATE SET gloss = excluded.gloss, updated_at = excluded.updated_at`,
[headword, word.pos, gloss, t],
);
await tx.run(
`INSERT OR REPLACE INTO lemma (id, headword, pos, freq_rank, level, gloss_en, gloss_ko,
unit_band, source)
VALUES (?, ?, ?, NULL, NULL, ?, '', 0, 'custom')`,
[id, headword, word.pos, gloss],
);
@@ -290,20 +291,28 @@ export async function editAddCustomWord(db: Db, word: CustomWord): Promise<Added
]);
await tx.run(
`INSERT INTO card (lemma_id, state, ease, interval, due, reps, lapses, updated_at)
VALUES (?, 0, 2.5, 0, 0, 0, 0, ?)`,
[id, now()],
VALUES (?, 0, 2.5, 0, 0, 0, 0, ?)
ON CONFLICT(lemma_id) DO NOTHING`,
[id, t],
);
return { lemmaId: id, created: true };
});
}
export async function editRemoveCustomWord(db: Db, lemmaId: number): Promise<void> {
if (lemmaId < CUSTOM_LEMMA_BASE) return; // never touch shipped dictionary rows
await db.tx(async (tx) => {
const word = await tx.get<{ headword: string; pos: string }>(
"SELECT headword, pos FROM lemma WHERE id = ? AND source = 'custom'",
[lemmaId],
);
if (!word) return; // never touch shipped dictionary rows
await tx.run("DELETE FROM card WHERE lemma_id = ?", [lemmaId]);
await tx.run("DELETE FROM surface WHERE lemma_id = ?", [lemmaId]);
await tx.run("DELETE FROM lemma WHERE id = ?", [lemmaId]);
await tx.run("DELETE FROM custom_word WHERE headword = ? AND pos = ?", [word.headword, word.pos]);
await tombstone(tx, "card", lemmaId);
await tombstone(tx, "custom_word", JSON.stringify([word.headword, word.pos]));
});
}
@@ -338,11 +347,17 @@ export async function editReset(db: Db, scope: ResetScope): Promise<void> {
for (const r of await tx.all<{ form: string }>("SELECT form FROM peek"))
await tombstone(tx, "peek", r.form);
for (const r of await tx.all<{ headword: string; pos: string }>(
"SELECT headword, pos FROM custom_word",
))
await tombstone(tx, "custom_word", JSON.stringify([r.headword, r.pos]));
await tx.run("DELETE FROM card");
await tx.run("DELETE FROM study_log");
await tx.run("DELETE FROM peek");
await tx.run("DELETE FROM surface WHERE lemma_id >= ?", [CUSTOM_LEMMA_BASE]);
await tx.run("DELETE FROM lemma WHERE id >= ?", [CUSTOM_LEMMA_BASE]);
await tx.run("DELETE FROM custom_word");
await tx.run("DELETE FROM surface WHERE lemma_id IN (SELECT id FROM lemma WHERE source = 'custom')");
await tx.run("DELETE FROM lemma WHERE source = 'custom'");
// Preferences, grammar flags and notes, and the trainer score.
// Device bookkeeping (schema_version, dict.*) is left alone — it
// describes this install, not the learner.

76
app/src/domain/dictionary.ts
View File

@@ -18,6 +18,7 @@
import type { Db } from "../db/types.js";
import { insertBand, seedMeta, type LemmaRow, type SurfaceRow } from "../db/writes.js";
import { REFERENCE_BAND } from "@shared/bands.mjs";
import { lemmaId } from "@shared/lemma-id.mjs";
const BASE = `${import.meta.env.BASE_URL ?? "/"}dict/`;
@@ -32,6 +33,7 @@ export interface BandInfo {
}
export interface DictManifest {
format?: number;
builtWith: { dictionary: string; dictionaryEntries: number; frequencyForms: number };
totals: { lemmas: number; surfaces: number };
bands: BandInfo[];
@@ -41,11 +43,22 @@ export interface DictManifest {
interface BandPayload {
band: number;
format?: number;
columns: { lemma: string[]; surface: string[] };
lemmas: unknown[][];
surfaces: unknown[][];
}
/** A lemma as the file carries it: no id, which is derived from the word. */
type FileLemma = Omit<LemmaRow, "id">;
/** A surface as the file carries it: its lemma is a row index in the file. */
interface FileSurface {
form: string;
lemma: number;
analysis: string;
}
let manifestPromise: Promise<DictManifest> | null = null;
export function loadManifest(): Promise<DictManifest> {
@@ -95,7 +108,61 @@ function toRows<T>(columns: string[], rows: unknown[][]): T[] {
});
}
/**
* Band file rows, as database rows.
*
* The files carry no ids (format 2). A lemma's id is lemmaId(headword, pos) —
* the same function the build asserted collision-free, and the one the
* app uses for custom words — so the id means the same word on every device
* and across every rebuild. A surface points at its lemma by row index.
*/
function rowsOf(payload: BandPayload): { lemmas: LemmaRow[]; surfaces: SurfaceRow[] } {
if (payload.format !== 2) {
throw new Error(`band ${payload.band}: format ${payload.format ?? 1} is not supported — rebuild the dictionary`);
}
const lemmas = toRows<FileLemma>(payload.columns.lemma, payload.lemmas).map((l) => ({
...l,
id: lemmaId(l.headword, l.pos),
}));
const surfaces = toRows<FileSurface>(payload.columns.surface, payload.surfaces).map((s) => {
const lemma = lemmas[s.lemma];
if (!lemma) throw new Error(`band ${payload.band}: surface ${s.form} names a lemma row that does not exist`);
return { form: s.form, lemma_id: lemma.id, analysis: s.analysis };
});
return { lemmas, surfaces };
}
const LOADED_KEY = "dict.loadedBands";
const VERSION_KEY = "dict.version";
/** Which build of the dictionary is loaded: every band file's hash. */
const versionOf = (m: DictManifest): string =>
m.bands.map((b) => `${b.band}:${b.sha256.slice(0, 16)}`).join(",");
/**
* Drop the loaded dictionary when the shipped one has changed.
*
* Loaded bands used to be recorded by number alone, so a rebuilt dictionary
* never reached an existing install: band 0 was "already loaded". Reloading
* is safe now that an id names a word rather than a position — every card
* still points at the same word afterwards. The learner's own words are
* reference data too, but not the shipped kind, so they stay.
*/
async function ensureCurrent(db: Db): Promise<void> {
const want = versionOf(await loadManifest());
const have = await db.get<{ v: string }>("SELECT v FROM meta WHERE k = ?", [VERSION_KEY]);
if (have?.v === want) return;
await db.tx(async (tx) => {
await tx.run(
"DELETE FROM surface WHERE lemma_id IN (SELECT id FROM lemma WHERE source <> 'custom')",
);
await tx.run("DELETE FROM lemma WHERE source <> 'custom'");
await tx.run("DELETE FROM meta WHERE k = ?", [LOADED_KEY]);
// Bookkeeping, not a user edit — never stamped, never synced.
await tx.run("INSERT OR REPLACE INTO meta (k, v, updated_at) VALUES (?, ?, 0)", [VERSION_KEY, want]);
});
}
/**
* Band loading is a read-modify-write on one meta row, and it runs from two
@@ -138,18 +205,15 @@ async function rememberBand(db: Db, band: number): Promise<void> {
/** Load one band if it is not already in the database. */
export function loadBand(db: Db, band: number): Promise<boolean> {
return serialise(async () => {
await ensureCurrent(db);
if ((await loadedBands(db)).has(band)) return false;
const manifest = await loadManifest();
const info = manifest.bands.find((b) => b.band === band);
if (!info) return false;
const payload = await fetchBand(info.file);
await insertBand(
db,
toRows<LemmaRow>(payload.columns.lemma, payload.lemmas),
toRows<SurfaceRow>(payload.columns.surface, payload.surfaces),
);
const { lemmas, surfaces } = rowsOf(await fetchBand(info.file));
await insertBand(db, lemmas, surfaces);
await rememberBand(db, band);
return true;
});

6
app/src/state/store.tsx
View File

@@ -30,7 +30,7 @@ import {
writeServerConfig,
type ServerConfig,
} from "../domain/server-config.js";
import { trySync, type SyncResult } from "../sync/client.js";
import { SYNC_PAUSED, trySync, type SyncResult } from "../sync/client.js";
import type { ProgressState } from "@lib/gate.js";
import type { FocusMode } from "../domain/gate.js";
@@ -243,6 +243,10 @@ export function StoreProvider({
const running = useRef(false);
const syncNow = useCallback(async () => {
if (!store || !server || running.current) return;
if (SYNC_PAUSED) {
setSyncState({ at: Date.now(), result: null, error: SYNC_PAUSED, running: false });
return;
}
running.current = true;
setSyncState((s) => ({ ...s, running: true }));
const result = await trySync(store.db, server);

14
app/src/sync/client.ts
View File

@@ -46,6 +46,18 @@ export interface SyncResult {
cursor: number;
}
/**
* Why sync is not running, or null when it is.
*
* Cards are keyed by stable lemma ids now, while the server still holds rows
* written under the old positional ids. Exchanging them would plant cards
* under ids that name no word, on both sides. Everything stays local — which
* this app is built to do — until the sync protocol that replaces this one
* ships with its own server schema.
*/
export const SYNC_PAUSED: string | null =
"Sync is paused while the storage format changes. Everything is kept on this device";
const CURSOR_KEY = "sync.cursor";
const PUSHED_KEY = "sync.pushedAt";
@@ -248,7 +260,7 @@ export async function trySync(
cfg: SyncConfig | null,
signal?: AbortSignal,
): Promise<SyncResult | null> {
if (!cfg?.baseUrl || !cfg.token) return null;
if (!cfg?.baseUrl || !cfg.token || SYNC_PAUSED) return null;
try {
return await syncOnce(db, cfg, signal);
} catch (err) {

4
app/src/ui/tabs/VocabTab.tsx
View File

@@ -7,7 +7,7 @@ import { useEffect, useMemo, useState } from "react";
import { useStore } from "../../state/store.js";
import { useReview } from "../review/useReview.js";
import { deck, forget, markAsKnown, type CardStatus, type DeckEntry } from "../../domain/cards.js";
import { CUSTOM_LEMMA_BASE, editAddCustomWord, editRemoveCustomWord } from "../../db/writes.js";
import { editAddCustomWord, editRemoveCustomWord } from "../../db/writes.js";
import { search, type Entry } from "../../domain/lexicon.js";
import { ensureReferenceBand } from "../../domain/dictionary.js";
import { statusOf } from "@lib/srs.js";
@@ -238,7 +238,7 @@ export function VocabTab() {
Reset
</button>
)}
{e.lemmaId >= CUSTOM_LEMMA_BASE && (
{e.source === "custom" && (
<button
className="btn sm"
title="Remove this word entirely"