strings extraction drops kanji-only Shift_JIS text, so name-based analyses miss real names #52

New Issue

fabi · 2026-09-17T19:02:11Z

fabi commented

2026-09-17 19:02:11 +00:00

crates/sylpheed-xexdb/src/strings.rs:237-241 requires at least one kana character (U+3040..U+30FF) before it accepts a decoded run:

// Require real kana somewhere. Arbitrary binary readily decodes to
// obscure kanji, but hiragana/katakana (U+3040..U+30FF) essentially never
let has_kana = t.chars().any(|c| ('\u{3040}'..='\u{30FF}').contains(&c));

The reason is sound and measured — without it the scan became a noise generator, 837 detections dominated by IEEE-754 arrays — but the consequence is that a string written only in kanji is never emitted. Japanese proper nouns routinely are.

Effect. Any analysis that starts from the strings table can miss real names, and cannot tell "no such name" from "not extractable". docs/re/data/name-block-bases.txt (182 KB) is one artifact built that way. Noted in PR #44 and in the desktop's CLAUDE.md; never filed here.

Directions (either needs the added noise measured before adoption):

accept a kanji-only run when it is NUL-terminated and sits in a region already known to hold text (a block whose neighbours decoded as text), or
emit kanji-only runs into a separate lower-confidence table, so a caller opts in and a negative result can say which table it searched.

`crates/sylpheed-xexdb/src/strings.rs:237-241` requires at least one kana character (U+3040..U+30FF) before it accepts a decoded run: ```rust // Require real kana somewhere. Arbitrary binary readily decodes to // obscure kanji, but hiragana/katakana (U+3040..U+30FF) essentially never let has_kana = t.chars().any(|c| ('\u{3040}'..='\u{30FF}').contains(&c)); ``` The reason is sound and measured — without it the scan became a noise generator, 837 detections dominated by IEEE-754 arrays — but the consequence is that **a string written only in kanji is never emitted**. Japanese proper nouns routinely are. **Effect.** Any analysis that starts from the strings table can miss real names, and cannot tell "no such name" from "not extractable". `docs/re/data/name-block-bases.txt` (182 KB) is one artifact built that way. Noted in PR #44 and in the desktop's `CLAUDE.md`; never filed here. **Directions** (either needs the added noise measured before adoption): * accept a kanji-only run when it is NUL-terminated *and* sits in a region already known to hold text (a block whose neighbours decoded as text), or * emit kanji-only runs into a separate lower-confidence table, so a caller opts in and a negative result can say which table it searched.

fabi added the state/proposed kind/defect labels 2026-09-17 19:02:11 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: fabi/Sylpheed#52

strings extraction drops kanji-only Shift_JIS text, so name-based analyses miss real names #52

`strings` extraction drops kanji-only Shift_JIS text, so name-based analyses miss real names #52