f8183d786f
Cards point at lemmas by id, and an id was the entry's position in the sorted build. One word added near the top of the dictionary would have moved every card below it onto a different word — silently, because loaded bands were recorded by number and a rebuilt dictionary never reached an existing install anyway. A custom word took max(id)+1 on whichever device added it, so the same id meant different words on a phone and a laptop. An id is now lemmaId(headword, pos), a 53-bit hash defined once in shared/ and used by the build, the loader, custom words and the migration. The build asserts all 30,520 entries are collision-free, and a test pins the function itself, since changing it re-keys every card. Band files are format 2: they carry no ids at all. The loader derives each id from the word, and a surface names its lemma by row index in the same file. Writing hashed ids out cost 0.5 MB of incompressible digits; leaving them out makes the files smaller than before (1.1 MB -> 1.0 MB). The loaded dictionary is now versioned by its band hashes, so a rebuild reloads on the next boot — safe only now that a reload cannot move a card. Migration 6 re-keys an existing install without stamping anything: cards and their tombstones move through the lemma rows still loaded, custom words become custom_word rows (the learner's data, which can travel) carrying the time their card was made, and the dictionary is dropped to reload. Sync is paused until the protocol that replaces it lands: the server still holds rows under the old ids, and exchanging them would plant cards that name no word. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
34 lines
1.5 KiB
JavaScript
34 lines
1.5 KiB
JavaScript
/* A lemma's id — derived from what the word IS, not where it sorts.
|
|
|
|
Ids used to be positions in the sorted build: the 12,000th entry got id
|
|
12000. Cards point at lemmas by id, so any change to the dictionary —
|
|
one entry added near the top — silently moved every card below it onto a
|
|
different word. And a custom word took max(id)+1 on the device that added
|
|
it, so the same id meant different words on a phone and a laptop.
|
|
|
|
Hashing (headword, pos) makes the id a property of the word: stable across
|
|
rebuilds, and identical on every device that adds the same word. The
|
|
build asserts the whole dictionary is collision-free.
|
|
|
|
cyrb53 (bryc, public domain): 53 bits, so the result is always a safe
|
|
JavaScript integer and fits SQLite's INTEGER PRIMARY KEY. It hashes
|
|
UTF-16 code units, which is deterministic across engines. The separator is
|
|
written as an escape, never a literal control byte. */
|
|
|
|
export function lemmaId(headword, pos) {
|
|
const str = `${headword}\u0001${pos}`;
|
|
let h1 = 0xdeadbeef;
|
|
let h2 = 0x41c6ce57;
|
|
for (let i = 0; i < str.length; i++) {
|
|
const ch = str.charCodeAt(i);
|
|
h1 = Math.imul(h1 ^ ch, 2654435761);
|
|
h2 = Math.imul(h2 ^ ch, 1597334677);
|
|
}
|
|
h1 = Math.imul(h1 ^ (h1 >>> 16), 2246822507);
|
|
h1 ^= Math.imul(h2 ^ (h2 >>> 13), 3266489909);
|
|
h2 = Math.imul(h2 ^ (h2 >>> 16), 2246822507);
|
|
h2 ^= Math.imul(h1 ^ (h1 >>> 13), 3266489909);
|
|
// 0 is never a valid rowid in practice; keep it out of the id space.
|
|
return 4294967296 * (2097151 & h2) + (h1 >>> 0) || 1;
|
|
}
|