Add CJK coverage per writing system - #236
Conversation
Records how much of each East Asian writing system a font covers, as a count per tier in a new "cjk" object in fonts.json. The Han tiers are the character sets national standards define, enumerated by decoding their byte ranges with the matching stdlib codec, so this adds no dependency beyond the fonttools already in requirements.txt. Hangul and the kana are complete Unicode blocks, which for them is the practical set. cjk_verify.py re-derives the same sets from Unicode's own Unihan data (UAX braver#38 kIRG_*Source) and compares, so the hardcoded byte ranges can be checked rather than taken on trust. It is a one-off tool, not part of the build. Refs braver#231
|
Just so you know, I haven't abandoned this yet! Doing a lot of reading and thinking. 🧐 In the issue #231 we took Hanja, Hangul, Hiragana and Katakana separately. That no longer sounds super useful to me since Hyperglot already checks those. That data is there and I trust it for that. So that leaves this PR to do checks against the Chinese and Japanese standards. That's useful information, for sure... but it also kind of weirdly overlaps with the data we already have. I.e. it intersects with what Hyperglot already reports, especially around Kana. And all this code effectively adds is confirmation that Unifont can also do Japanese (Hyperglot already tells us it can do Chinese and Korean), that M PLUS 1 Code can do Japanese, and that Cilantro Code Mono can do Japanese and most of Chinese. We kind of already knew that 😅 Maybe Hyperglot changes their stance on what to require for Japanese (currently it's all of "Kanji" that's probably the culprit). If it's just those fonts, perhaps that data isn't even super useful (in this website, as a source for fonts-that-can-do-cjk), or it can be added manually. |
2 participants
Implements the CJK coverage data discussed in #231. Adds
cjk.py, which measures how much of each East Asian writing system a font covers, and wires it intoinfo.pyso the numbers land infonts.json.Every existing file is touched by pure addition — no line in
info.py,fonts-schema.jsonorREADME.mdis modified or removed.The front end is untouched. No changes to
filters.js,index.htmlor the styling — this only provides the data to filter on.Two places where I did something different from your comment
You proposed storing a percentage per facet, written only above a threshold of maybe 30%. I implemented counts, always written. Both are one-line changes if you prefer your version — I'd rather show you the reasoning than quietly do it my way:
Counts instead of percentages. It matches how
languagesalready works, so the front end reads both the same way, andfonts.jsonstays readable when someone reviews a PR by eye. It also survives a corrected denominator: if a tier total ever needs fixing, counts can be re-divided, percentages can't. To switch, divide bycjk.TOTALS[key]incjk.coverage().No write-time threshold. A threshold in
info.pybakes the decision into the data, so changing 30% to 20% means re-running the script over all 189 fonts. In the front end it's a comparison againstcjk_total[key], changeable any time — the same reasonlang_countlives in the front end today. To switch, drop entries below the cutoff incoverage().One thing worth knowing before you pick a threshold: a single cutoff means very different things per facet. 30% of Big5 level 2 is a reasonable "has some rarer hanzi". 30% of Hangul is useless — Korean composes syllables, so a font with 3,000 of the 11,172 fails on ordinary words. 30% of hiragana is 26 kana, which can't write anything. If you want one knob, the level-1 tiers and the syllabaries want something close to 95%, while the level-2 tiers are informative anywhere above ~20%.
Where the numbers come from
The four Han tiers are the character sets national standards define, enumerated by decoding that standard's byte ranges with the matching stdlib codec — no new dependency, no data file to maintain. Hangul and the kana are complete Unicode blocks, which for them genuinely is the practical set.
You asked to be able to trace the hardcoded ranges to an authoritative source. Rather than ask you to trust a comment,
cjk_verify.pyrebuilds the sets from Unicode's own data and compares: Unihan records which national standards each ideograph came from (UAX #38kIRG_*Source). Runpython3 cjk_verify.py— it downloads ~8MB from unicode.org, prints the comparison, and exits non-zero if anything differs unexpectedly. It is not part of the build; nothing on the site depends on it.Current result against UCD 17.0:
G0, 6,763J0, 6,356K0, 4,888T1+T2, 13,064cjk.pyalso checks each tier against the count its standard specifies when it builds the sets, so if a future Python changes its codec tables the script stops with an error instead of silently shifting every percentage on the site.Checked before opening this
cjk_verify.pyagainst UCD 17.0: all four comparisons as expected, exit 0cjk.pyover 29 fonts from this repo; 11 have CJK data, output parses as JSONinfo.py --nameend to end on three fonts;fonts.jsonvalidates against the updated schema with no errorscjkobjectcjkentry rather than leaving it behindNot included
fonts.jsondata. Runningpython3 info.pyregenerates it, and I'd rather not put a 189-font diff in the same PR as the code — also avoids colliding with #235.One caveat on reading the numbers
A count says whether a font can set text in a language, not whether it does so well. Han characters are shared between these standards but drawn to different regional conventions, and only the
cmapis inspected, so a font can cover Big5 with glyphs drawn to Japanese conventions. Worth keeping in mind if the filter ever grows a "best for" label.