diff options
| author | eunhhu <[email protected]> | 2026-09-28 14:49:05 +0100 |
|---|---|---|
| committer | eunhhu <[email protected]> | 2026-09-28 14:49:05 +0100 |
| commit | 60a3c0232bbca8d5d8c1107ab8197d272212ef42 (patch) | |
| tree | ba7b4bad98b79ad1abd40cdb038c493d50df0c19 | |
| parent | c23b79f1cb599ee7766002ad23756643e72113e5 (diff) | |
| download | vega-60a3c0232bbca8d5d8c1107ab8197d272212ef42.tar.gz vega-60a3c0232bbca8d5d8c1107ab8197d272212ef42.zip | |
Shape the nine Indic scripts as HarfBuzz does
The Indic shaper, from HarfBuzz 10.2's source: syllables found by the
Indic machine; each syllable's base consonant, its pre-base matras and
reph moved around it and masked for the basic features, applied one at
a time within the syllable; then matras, reph and pre-base-reordering
consonants placed again by what the features made. Vowel constraints,
Indic decompositions and compositions, dotted circles for broken
syllables, and per-syllable matching come with it.
Plans carry GSUB pauses (steps where the shaper does its part) and the
stages of the features the shaper asks whether they would substitute;
the runtime answers with each subtable's would_apply, as HarfBuzz does.
Masks are now the common ones and each shaper's own.
text/syllabic.py reads HarfBuzz's Ragel grammars (Indic, Khmer,
Myanmar) and compiles each to a minimal DFA with Ragel's scanner
semantics; beside the machines Ragel generated from the same grammars,
100,000 random runs each find no syllable that differs. It also reads
HarfBuzz's Indic table (Unicode 16) cell for cell.
Every installed font shapes as HarfBuzz does (213 fonts, 1,688 plans);
the Indic ones at 20,000 runs a plan (FreeSerif's eight scripts,
FreeSans, Unifont's nine without GSUB), and nine new-spec Noto Sans
fonts at 5,000.
| -rw-r--r-- | Makefile | 8 | ||||
| -rw-r--r-- | docs/design/004-text.md | 39 | ||||
| -rw-r--r-- | text/shape-build.c | 229 | ||||
| -rw-r--r-- | text/shape.c | 872 | ||||
| -rw-r--r-- | text/shape.h | 42 | ||||
| -rw-r--r-- | text/syllabic-data.c | 463 | ||||
| -rw-r--r-- | text/syllabic.h | 40 | ||||
| -rw-r--r-- | text/syllabic.py | 456 |
8 files changed, 2065 insertions, 84 deletions
@@ -48,9 +48,9 @@ HARFBUZZ_LIBS ?= $(shell pkg-config --libs harfbuzz) # Drawing them on the GPU takes OpenGL ES 3.1 through EGL. GL_LIBS ?= $(shell pkg-config --libs egl glesv2) SYMBOLS = symbols/symbols.c symbols/symbols-gl.c -SHAPE = text/shape.c text/layout.c text/bidi.c text/linebreak.c text/unicode-data.c +SHAPE = text/shape.c text/syllabic-data.c text/layout.c text/bidi.c text/linebreak.c text/unicode-data.c $(BUILD)/vega-symbols: symbols/symbols-tool.c $(SYMBOLS) symbols/symbols.h symbols/symbols-gl.h $(SHAPE) text/shape-build.c \ - text/shape.h text/shape-build.h text/layout.h text/text.h | $(BUILD) + text/shape.h text/shape-build.h text/layout.h text/text.h text/syllabic.h | $(BUILD) $(CC) $(CPPFLAGS) $(CFLAGS) $(FREETYPE_CFLAGS) $(HARFBUZZ_CFLAGS) -Itext symbols/symbols-tool.c $(SYMBOLS) $(SHAPE) \ text/shape-build.c $(LDFLAGS) $(FREETYPE_LIBS) $(HARFBUZZ_LIBS) $(GL_LIBS) -lm $(LDLIBS) -o $@ @@ -58,8 +58,8 @@ TEXT = text/bidi.c text/normalize.c text/linebreak.c text/grapheme.c text/word.c $(BUILD)/test-text: text/test-text.c $(TEXT) text/text.h | $(BUILD) $(CC) $(CPPFLAGS) $(CFLAGS) -Itext text/test-text.c $(TEXT) $(LDFLAGS) $(LDLIBS) -o $@ -LAYOUT = text/layout.c text/shape.c text/bidi.c text/linebreak.c text/unicode-data.c -$(BUILD)/test-layout: text/test-layout.c $(LAYOUT) text/layout.h text/shape.h text/text.h symbols/symbols.c \ +LAYOUT = text/layout.c text/shape.c text/syllabic-data.c text/bidi.c text/linebreak.c text/unicode-data.c +$(BUILD)/test-layout: text/test-layout.c $(LAYOUT) text/layout.h text/shape.h text/text.h text/syllabic.h symbols/symbols.c \ symbols/symbols.h | $(BUILD) $(CC) $(CPPFLAGS) $(CFLAGS) -Itext -Isymbols text/test-layout.c $(LAYOUT) symbols/symbols.c $(LDFLAGS) -lm $(LDLIBS) -o $@ diff --git a/docs/design/004-text.md b/docs/design/004-text.md index 6c8e09e..065c4db 100644 --- a/docs/design/004-text.md +++ b/docs/design/004-text.md @@ -129,11 +129,48 @@ HarfBuzz와 다른 파일은 만들어지지 않는다. `vega-symbols build --sh 않았다. Unifont의 히브리는 HarfBuzz보다 2.3배 빠르다(문자당 116 대 272 ns). HarfBuzz는 부호마다 CFF 외곽을 풀어 상자를 재고, Vega는 빌드할 때 잰 상자를 읽는다. +### 인도계 문자(데바나가리, 벵골, 구르무키, 구자라트, 오리야, 타밀, 텔루구, 칸나다, 말라얄람) + +HarfBuzz의 인도계 셰이퍼는 음절 단위로 일한다. 음절을 찾고(Ragel로 짠 문법), 음절마다 기저 자음을 찾아 그 앞뒤로 모음 부호와 +reph를 옮기고, 기본 기능(nukt, akhn, rphf, rkrf, pref, blwf, abvf, half, pstf, vatu, cjct)을 하나씩 음절 안에서만 적용한 뒤, +그 결과를 보고 모음 부호와 reph와 앞으로 가는 자음을 다시 옮긴다. 그 사이사이가 GSUB의 멈춤(pause)이다. + +- 계획: 멈춤은 조회 목록 안의 표지(`VEGA_STEP_PAUSE`와 멈춤 번호)로 싣는다. 셰이퍼가 "이 기능이 이 글리프들을 바꾸겠는가"를 + 묻는 기능(rphf, pref, blwf, pstf, vatu)은 그 단계의 조회 범위를 계획에 싣고, 실행 중에 조회의 하위 표를 문맥 없이 대어 본다 + (`would_apply`, 형식마다 HarfBuzz의 판단 그대로). 옛 규격(deva)과 새 규격(dev2)은 글꼴이 고른 문자 체계 태그로 가른다. +- 음절 기계: [text/syllabic.py](../../text/syllabic.py)가 HarfBuzz의 `.rl` 문법(인도계, 크메르, 미얀마)을 그대로 읽어 정규식에서 + NFA, DFA를 만들고 최소화한다. Ragel 스캐너의 뜻(각 위치에서 가장 긴 일치, 같으면 먼저 적은 패턴)을 따른다. 같은 문법으로 + Ragel이 만든 HarfBuzz의 기계(`hb-ot-shaper-*-machine.hh`)를 해석해 무작위 범주열 10만 개씩 대 보면 세 기계 모두 음절 경계와 + 종류가 하나도 다르지 않다(`--check`). 문자마다의 범주와 자리는 HarfBuzz 10.2의 `hb-ot-shaper-indic-table.cc`를 칸마다 읽는다 + (유니코드 16 기준이라 여기의 15.1 UCD에서 만들지 않는다). +- 실행: 모음 제약(다른 모음처럼 읽히는 모음 연쇄에 점선 원), 인도계 분해·합성(몇 글자는 분해하지 않고, 쪼갠 모음 부호는 다시 + 합치지 않는다), 음절 찾기, 초기 재배치(기저 찾기, 자리순 안정 정렬, 마스크), 깨진 음절의 점선 원, 음절 안에서만 맞추는 + 조회(`F_PER_SYLLABLE`), 최종 재배치(pref가 만든 기저, 말라얄람의 아래 형태, 모음 부호와 reph와 pref 자음의 자리, 단어 + 첫머리의 init). + +HarfBuzz 10.2의 원본을 기준으로 삼았다. 처음에는 rustybuzz 0.20(HarfBuzz 10.0.1을 옮긴 Rust)을 보고 옮겼는데, 설치된 +HarfBuzz 10.2와 대 보니 넷이 달랐다. rustybuzz가 틀린 것 둘: 새 규격에서 would-substitute를 문맥 없이 묻는 조건이 뒤집혀 +있고, 기저를 찾을 때 미얀마의 자음 집합을 쓴다. 10.0.1 뒤에 HarfBuzz가 바꾼 것 둘: 위첨자 2·3(U+00B2·00B3)이 SMPst라는 새 +범주가 됐고(문법에 `sm = SM | SMPst`), 텔루구 길이 부호의 수정 결합 등급이 0이 아니라 4·5다. 넷 모두 HarfBuzz 원본대로 고쳤다. +다섯째는 계획기에 있었다. HarfBuzz는 단계마다 조회를 번호로 정렬한 뒤 같은 조회를 합치는데, 합칠 때 음절 제한과 무작위 +깃발은 앞선 쪽의 것을 남긴다. 그 정렬이 자체 퀵정렬(hb_qsort)이라 안정적이지 않고, 앞뒤는 그 알고리즘이 정한다. Noto Sans +Malayalam은 psts와 rlig가 한 조회를 나눠 쓰는데, 안정 정렬로는 psts 쪽(음절 안에서만)이 남아 음절을 넘는 합자(ma와 결합 +점)가 생기지 않았다. 계획기가 같은 퀵정렬을 그대로 쓰게 했다. + +결과(Pi 5): + +- 설치된 글꼴 213벌, 계획 1,688개에 각 1천 실행: 다른 실행 0(글자 모양이 비트맵뿐인 Noto Color Emoji는 심볼 단계에서 + 빠진다). 인도계가 있는 글꼴은 따로 계획마다 2만 실행: FreeSerif(여덟 문자), FreeSans, 굵은 꼴들, FreeMono, Unifont(아홉 + 문자, GSUB 없음) 130개 계획 260만 실행, 다른 실행 0. +- 새 규격 글꼴(dev2 등): 인터넷에서 받은 Noto Sans 아홉 벌(데바나가리부터 말라얄람까지), 계획마다 5천 실행, 다른 실행 0. +- 속도: HarfBuzz의 1.25–1.3배 시간(Noto Sans Devanagari 문자당 1,614 대 1,235 ns, Tamil 674 대 526 ns). + 받지 않는 것(빌드 도구가 이유를 말하고 거부한다): - `kern` 표의 상태 기계(Apple 형식 1), 등급 형식(2, 3), 교차 방향 커닝. 설치된 글꼴의 `kern` 표는 모두 짝 형식(0)이다. - AAT 표(morx, kerx, trak), 늘이기(stch), 아랍 형태가 없어 HarfBuzz가 표시형으로 대체 셰이핑을 하는 글꼴(Unifont의 아랍). -- 기본·아랍·히브리·한글 셰이퍼가 아닌 문자 체계: 인도계, 태국·라오, 크메르, 미얀마, USE 문자 체계들, 시리아. +- 기본·아랍·히브리·한글·인도계 셰이퍼가 아닌 문자 체계: 태국·라오, 크메르, 미얀마, USE 문자 체계들(싱할라, 티베트, 자바 + 등), 시리아. - 이체 선택자(cmap 형식 14)는 싣지 않는다. 선택자는 숨기고 앞 글자는 기본 글리프로 둔다. ## 비례폭 글줄 diff --git a/text/shape-build.c b/text/shape-build.c index aa653d8..559517e 100644 --- a/text/shape-build.c +++ b/text/shape-build.c @@ -3,6 +3,7 @@ * masks and stages), what the section carries, and a check of it against * HarfBuzz on random runs. */ #include "shape-build.h" +#include "syllabic.h" #include "text.h" #include <hb-ot.h> #include <stdlib.h> @@ -25,7 +26,8 @@ typedef struct { uint32_t mask, one_mask; bool auto_zwnj, auto_zwj, random, per_syllable, needs_fallback; } FeatureMap; -typedef struct { unsigned index; uint32_t mask; bool auto_zwnj, auto_zwj, random, per_syllable; } LookupMap; +/* A lookup to run, or (index VEGA_STEP_PAUSE) the pause that ends a stage. */ +typedef struct { unsigned index; uint32_t mask; bool auto_zwnj, auto_zwj, random, per_syllable; unsigned pause; } LookupMap; typedef struct { hb_face_t *face; @@ -33,6 +35,7 @@ typedef struct { hb_tag_t chosen_script[2]; FeatureInfo infos[MAX_FEATURES]; unsigned info_count, current_stage[2]; + unsigned pause[2][MAX_STAGES]; /* what ends each stage: VEGA_PAUSE_..., or 0 */ bool is_simple; /* compiled */ FeatureMap features[MAX_FEATURES]; @@ -40,6 +43,7 @@ typedef struct { uint32_t global_mask; LookupMap *lookups[2]; unsigned lookup_count[2]; + unsigned stage_first[2][MAX_STAGES], stage_end[2][MAX_STAGES]; /* each stage's lookups */ } Map; static const hb_tag_t table_tags[2] = {HB_OT_TAG_GSUB, HB_OT_TAG_GPOS}; @@ -51,7 +55,10 @@ static void add_feature(Map *m, hb_tag_t tag, unsigned flags, unsigned value) { m->info_count++; } static void enable_feature(Map *m, hb_tag_t tag, unsigned flags, unsigned value) { add_feature(m, tag, flags | F_GLOBAL, value); } -static void add_gsub_pause(Map *m) { m->current_stage[0]++; } +static void add_gsub_pause(Map *m, unsigned pause) { + if (m->current_stage[0] < MAX_STAGES) m->pause[0][m->current_stage[0]] = pause; + m->current_stage[0]++; +} static bool has_feature(const Map *m, hb_tag_t tag) { for (int t = 0; t < 2; ++t) @@ -97,6 +104,17 @@ static uint32_t one_mask(const Map *m, const char *tag) { return f ? f->one_mask : 0; } +static bool push_lookup(Map *m, int table, LookupMap l, unsigned *allocated) { + if (m->lookup_count[table] == *allocated) { + *allocated = *allocated ? 2 * *allocated : 64; + LookupMap *grown = realloc(m->lookups[table], *allocated * sizeof(*grown)); + if (!grown) return false; + m->lookups[table] = grown; + } + m->lookups[table][m->lookup_count[table]++] = l; + return true; +} + static bool add_lookups(Map *m, int table, unsigned feature_index, uint32_t mask, bool auto_zwnj, bool auto_zwj, bool random, bool per_syllable, unsigned *allocated) { unsigned table_lookup_count = hb_ot_layout_table_get_lookup_count(m->face, table_tags[table]); @@ -104,17 +122,11 @@ static bool add_lookups(Map *m, int table, unsigned feature_index, uint32_t mask do { len = 32; hb_ot_layout_feature_get_lookups(m->face, table_tags[table], feature_index, offset, &len, indices); - for (unsigned i = 0; i < len; ++i) { - if (indices[i] >= table_lookup_count) continue; - if (m->lookup_count[table] == *allocated) { - *allocated = *allocated ? 2 * *allocated : 64; - LookupMap *grown = realloc(m->lookups[table], *allocated * sizeof(*grown)); - if (!grown) return false; - m->lookups[table] = grown; - } - m->lookups[table][m->lookup_count[table]++] = - (LookupMap){indices[i], mask, auto_zwnj, auto_zwj, random, per_syllable}; - } + for (unsigned i = 0; i < len; ++i) + if (indices[i] < table_lookup_count && + !push_lookup(m, table, (LookupMap){indices[i], mask, auto_zwnj, auto_zwj, random, per_syllable, 0}, + allocated)) + return false; offset += len; } while (len == 32); return true; @@ -254,9 +266,9 @@ static bool compile_map(Map *m) { map->one_mask = (1u << map->shift) & map->mask; } if (m->is_simple) qsort(m->features, m->feature_count, sizeof(*m->features), by_tag); - /* The final pause, then each stage's lookups: sorted, duplicates merged. */ - m->current_stage[0]++; - m->current_stage[1]++; + /* The final pause, then each stage's lookups: sorted, duplicates merged, + * the stage's pause after them. */ + if (++m->current_stage[0] > MAX_STAGES || ++m->current_stage[1] > MAX_STAGES) return false; for (int t = 0; t < 2; ++t) { unsigned allocated = 0, last = 0; for (unsigned stage = 0; stage < m->current_stage[t]; ++stage) { @@ -284,6 +296,12 @@ static bool compile_map(Map *m) { } m->lookup_count[t] = j + 1; } + m->stage_first[t][stage] = last; + m->stage_end[t][stage] = m->lookup_count[t]; + if (m->pause[t][stage] && + !push_lookup(m, t, (LookupMap){VEGA_STEP_PAUSE, 0, false, false, false, false, m->pause[t][stage]}, + &allocated)) + return false; last = m->lookup_count[t]; } } @@ -316,9 +334,9 @@ static int categorize(uint32_t script, hb_tag_t gsub_script) { if (script == TAG("Syrc") || script == TAG("Thai") || script == TAG("Laoo") || script == TAG("Khmr") || script == TAG("Qaag")) return -1; - if (listed(indic_scripts, sizeof(indic_scripts) / sizeof(*indic_scripts), script) || - listed(use_scripts, sizeof(use_scripts) / sizeof(*use_scripts), script)) - return dflt ? VEGA_SHAPER_DEFAULT : -1; + if (listed(indic_scripts, sizeof(indic_scripts) / sizeof(*indic_scripts), script)) + return dflt ? VEGA_SHAPER_DEFAULT : (gsub_script & 0xFF) == '3' ? -1 : VEGA_SHAPER_INDIC; + if (listed(use_scripts, sizeof(use_scripts) / sizeof(*use_scripts), script)) return dflt ? VEGA_SHAPER_DEFAULT : -1; if (script == TAG("Mymr")) return dflt || gsub_script == TAG("mymr") ? VEGA_SHAPER_DEFAULT : -1; return VEGA_SHAPER_DEFAULT; } @@ -330,14 +348,26 @@ static bool has_table(hb_face_t *face, hb_tag_t tag) { return has; } +typedef struct { unsigned first, count; } Stage; typedef struct { uint32_t script; int direction, native, shaper, flags; uint32_t global_mask, kern_mask, masks[VEGA_PLAN_MASKS]; LookupMap *lookups[2]; unsigned lookup_count[2]; + unsigned shaper_flags; + Stage stages[VEGA_PLAN_STAGES]; } Plan; +/* The GSUB stage a feature is in (hb_ot_map_t::get_stage_lookups), for a + * shaper's would-substitute questions. */ +static Stage stage_of(const Map *m, const char *tag) { + const FeatureMap *f = find_feature(m, TAG(tag)); + if (!f || f->stage[0] >= m->current_stage[0]) return (Stage){0, 0}; + unsigned first = m->stage_first[0][f->stage[0]]; + return (Stage){first, m->stage_end[0][f->stage[0]] - first}; +} + /* What HarfBuzz makes of the font's kern table (hb-ot-kern-table.hh): one * with subtables that pass its checks (data), Apple's state machines among * them (machine), cross-stream kerning (cross); and whether it has what @@ -415,7 +445,7 @@ static bool plan_for(hb_face_t *face, uint32_t script_tag, int direction, Plan * /* hb_ot_shape_collect_features */ m->is_simple = true; enable_feature(m, TAG("rvrn"), F_NONE, 1); - add_gsub_pause(m); + add_gsub_pause(m, 0); if (direction == VEGA_SHAPE_LTR) { enable_feature(m, TAG("ltra"), F_NONE, 1); enable_feature(m, TAG("ltrm"), F_NONE, 1); @@ -434,23 +464,57 @@ static bool plan_for(hb_face_t *face, uint32_t script_tag, int direction, Plan * if (shaper == VEGA_SHAPER_ARABIC) { m->is_simple = false; enable_feature(m, TAG("stch"), F_NONE, 1); - add_gsub_pause(m); /* record_stch */ + add_gsub_pause(m, 0); /* record_stch */ enable_feature(m, TAG("ccmp"), F_MANUAL_ZWJ, 1); enable_feature(m, TAG("locl"), F_MANUAL_ZWJ, 1); - add_gsub_pause(m); + add_gsub_pause(m, 0); for (int i = 0; i < 7; ++i) { bool syriac = arabic_features[i][3] == '2' || arabic_features[i][3] == '3'; add_feature(m, TAG(arabic_features[i]), F_MANUAL_ZWJ | (script == HB_SCRIPT_ARABIC && !syriac ? F_HAS_FALLBACK : 0), 1); - add_gsub_pause(m); + add_gsub_pause(m, 0); } enable_feature(m, TAG("rlig"), F_MANUAL_ZWJ | F_HAS_FALLBACK, 1); - if (script == HB_SCRIPT_ARABIC) add_gsub_pause(m); /* arabic_fallback_shape */ + if (script == HB_SCRIPT_ARABIC) add_gsub_pause(m, 0); /* arabic_fallback_shape */ enable_feature(m, TAG("calt"), F_MANUAL_ZWJ, 1); - if (!has_feature(m, TAG("rclt"))) add_gsub_pause(m); + if (!has_feature(m, TAG("rclt"))) add_gsub_pause(m, 0); enable_feature(m, TAG("liga"), F_MANUAL_ZWJ, 1); enable_feature(m, TAG("clig"), F_MANUAL_ZWJ, 1); enable_feature(m, TAG("mset"), F_MANUAL_ZWJ, 1); } + /* hb-ot-shaper-indic.cc: syllables found before any lookup; locl and + * ccmp; the initial reordering; the basic features one stage each; the + * final reordering; the rest. */ + static const struct { const char *tag; unsigned flags; } indic_features[17] = { + {"nukt", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"akhn", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"rphf", F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"rkrf", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"pref", F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"blwf", F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"abvf", F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"half", F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"pstf", F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"vatu", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"cjct", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"init", F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"pres", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"abvs", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"blws", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"psts", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}, + {"haln", F_GLOBAL | F_MANUAL_ZWNJ | F_MANUAL_ZWJ | F_PER_SYLLABLE}}; + if (shaper == VEGA_SHAPER_INDIC) { + m->is_simple = false; + add_gsub_pause(m, VEGA_PAUSE_SYLLABLES); + enable_feature(m, TAG("locl"), F_PER_SYLLABLE, 1); + enable_feature(m, TAG("ccmp"), F_PER_SYLLABLE, 1); + add_gsub_pause(m, VEGA_PAUSE_INITIAL_REORDERING); + for (int i = 0; i < 11; ++i) { + add_feature(m, TAG(indic_features[i].tag), indic_features[i].flags, 1); + add_gsub_pause(m, 0); + } + add_gsub_pause(m, VEGA_PAUSE_FINAL_REORDERING); + for (int i = 11; i < 17; ++i) add_feature(m, TAG(indic_features[i].tag), indic_features[i].flags, 1); + } if (shaper == VEGA_SHAPER_HANGUL) { m->is_simple = false; add_feature(m, TAG("ljmo"), F_NONE, 1); @@ -464,9 +528,14 @@ static bool plan_for(hb_face_t *face, uint32_t script_tag, int direction, Plan * add_feature(m, TAG(common[i]), F_GLOBAL | (i == 4 || i == 5 ? F_MANUAL_ZWNJ | F_MANUAL_ZWJ : 0), 1); static const char *const horizontal[] = {"calt", "clig", "curs", "dist", "kern", "liga", "rclt"}; for (int i = 0; i < 7; ++i) add_feature(m, TAG(horizontal[i]), F_GLOBAL | (i == 4 ? F_HAS_FALLBACK : 0), 1); - /* Hangul turns calt off: fonts put jamo lookups there. */ + /* Hangul turns calt off: fonts put jamo lookups there. Indic turns liga + * off, and is done with syllables after GSUB. */ if (shaper == VEGA_SHAPER_HANGUL) add_feature(m, TAG("calt"), F_GLOBAL, 0); - if (!compile_map(m)) FAIL("out of memory"); + if (shaper == VEGA_SHAPER_INDIC) { + add_feature(m, TAG("liga"), F_GLOBAL, 0); + add_gsub_pause(m, VEGA_PAUSE_CLEAR_SYLLABLES); + } + if (!compile_map(m)) FAIL("out of memory, or more than %d stages", MAX_STAGES); /* hb_ot_shape_planner_t::compile */ const FeatureMap *kern = find_feature(m, TAG("kern")); @@ -476,8 +545,9 @@ static bool plan_for(hb_face_t *face, uint32_t script_tag, int direction, Plan * bool apply_kern = (!has_gpos_kern || !apply_gpos) && kern_table.data; if (apply_kern && (kern_table.other || kern_table.cross)) FAIL("%s: the font's kern table has more than pairs (state machines, classes, cross-stream kerning)", name); - /* Hangul's shaper neither zeroes marks nor places them without GPOS. */ - bool script_marks = shaper != VEGA_SHAPER_HANGUL; + /* Hangul's and Indic's shapers neither zero marks nor place them + * without GPOS. */ + bool script_marks = shaper != VEGA_SHAPER_HANGUL && shaper != VEGA_SHAPER_INDIC; bool adjust = !apply_gpos && (!apply_kern || !kern_table.cross), fallback_marks = adjust && script_marks; if (shaper == VEGA_SHAPER_ARABIC) { if (one_mask(m, "stch")) FAIL("%s: the font stretches (stch)", name); @@ -499,15 +569,26 @@ static bool plan_for(hb_face_t *face, uint32_t script_tag, int direction, Plan * (apply_kern ? VEGA_PLAN_KERN : 0) | (fallback_marks ? VEGA_PLAN_FALLBACK_MARKS : 0) | (adjust ? VEGA_PLAN_ADJUST_ZEROING : 0), m->global_mask, kern ? kern->one_mask : 0, {0}, {m->lookups[0], m->lookups[1]}, - {m->lookup_count[0], m->lookup_count[1]}}; - for (int i = 0; i < 7; ++i) plan->masks[VEGA_MASK_ISOL + i] = one_mask(m, arabic_features[i]); + {m->lookup_count[0], m->lookup_count[1]}, 0, {{0, 0}}}; plan->masks[VEGA_MASK_RTLM] = one_mask(m, "rtlm"); plan->masks[VEGA_MASK_FRAC] = frac; plan->masks[VEGA_MASK_NUMR] = numr; plan->masks[VEGA_MASK_DNOM] = dnom; - plan->masks[VEGA_MASK_LJMO] = one_mask(m, "ljmo"); - plan->masks[VEGA_MASK_VJMO] = one_mask(m, "vjmo"); - plan->masks[VEGA_MASK_TJMO] = one_mask(m, "tjmo"); + if (shaper == VEGA_SHAPER_ARABIC) + for (int i = 0; i < 7; ++i) plan->masks[VEGA_MASK_ISOL + i] = one_mask(m, arabic_features[i]); + if (shaper == VEGA_SHAPER_HANGUL) { + plan->masks[VEGA_MASK_LJMO] = one_mask(m, "ljmo"); + plan->masks[VEGA_MASK_VJMO] = one_mask(m, "vjmo"); + plan->masks[VEGA_MASK_TJMO] = one_mask(m, "tjmo"); + } + if (shaper == VEGA_SHAPER_INDIC) { + /* the masks of the features set glyph by glyph (not global ones) */ + static const char *const masked[] = {"rphf", "pref", "blwf", "abvf", "half", "pstf", "init"}; + for (int i = 0; i < 7; ++i) plan->masks[VEGA_MASK_RPHF + i] = one_mask(m, masked[i]); + static const char *const staged[VEGA_PLAN_STAGES] = {"rphf", "pref", "blwf", "pstf", "vatu"}; + for (int i = 0; i < VEGA_PLAN_STAGES; ++i) plan->stages[i] = stage_of(m, staged[i]); + if ((m->chosen_script[0] & 0xFF) != '2') plan->shaper_flags |= VEGA_INDIC_OLD_SPEC; + } free(m); return true; fail: @@ -661,6 +742,13 @@ bool vega_shaping_compile(hb_face_t *face, const uint32_t *scripts, size_t scrip put32(&o, p->lookup_count[t]); step += p->lookup_count[t]; } + put32(&o, p->shaper_flags); + for (int k = 0; k < VEGA_PLAN_STAGES; ++k) { + put16(&o, p->stages[k].first); + put16(&o, p->stages[k].count); + put32(&o, 0); + } + put32(&o, 0); } set32(&o, 56, step); set32(&o, 60, (uint32_t)o.size); @@ -669,10 +757,11 @@ bool vega_shaping_compile(hb_face_t *face, const uint32_t *scripts, size_t scrip for (unsigned k = 0; k < plans[i].lookup_count[t]; ++k) { const LookupMap *l = &plans[i].lookups[t][k]; put16(&o, l->index); - unsigned char flags[2] = {(unsigned char)((l->auto_zwnj ? VEGA_STEP_AUTO_ZWNJ : 0) | - (l->auto_zwj ? VEGA_STEP_AUTO_ZWJ : 0) | - (l->random ? VEGA_STEP_RANDOM : 0) | - (l->per_syllable ? VEGA_STEP_PER_SYLLABLE : 0)), + unsigned char flags[2] = {(unsigned char)(l->index == VEGA_STEP_PAUSE ? l->pause + : (l->auto_zwnj ? VEGA_STEP_AUTO_ZWNJ : 0) | + (l->auto_zwj ? VEGA_STEP_AUTO_ZWJ : 0) | + (l->random ? VEGA_STEP_RANDOM : 0) | + (l->per_syllable ? VEGA_STEP_PER_SYLLABLE : 0)), 0}; put(&o, flags, 2); put32(&o, l->mask); @@ -708,7 +797,8 @@ bool vega_shaping_compile(hb_face_t *face, const uint32_t *scripts, size_t scrip if (hb_font_get_nominal_glyph(font, u, &g)) hb_set_add(glyphs, g); } for (size_t i = 0; i < plan_count; ++i) - for (unsigned k = 0; k < plans[i].lookup_count[0]; ++k) hb_set_add(lookups, plans[i].lookups[0][k].index); + for (unsigned k = 0; k < plans[i].lookup_count[0]; ++k) + if (plans[i].lookups[0][k].index != VEGA_STEP_PAUSE) hb_set_add(lookups, plans[i].lookups[0][k].index); hb_ot_layout_lookups_substitute_closure(face, lookups, glyphs); hb_set_destroy(wanted); hb_set_destroy(lookups); @@ -763,6 +853,46 @@ static uint32_t jamo(Random *r, const Pool *p, uint32_t first, uint32_t count) { return p->count && below(r, 2) ? pick(r, p) : first + below(r, count); } +/* An Indic syllable the ways text spells one (hb-ot-shaper-indic.cc): + * consonants joined by viramas (a joiner by some), a Ra and virama first + * for a reph, nuktas, matras, syllable modifiers, a virama at the end; a + * vowel with its modifiers. indic holds the script's consonants, Ra, + * viramas, nuktas, matras, modifiers and vowels. */ +enum { INDIC_CONSONANTS, INDIC_RA, INDIC_HALANTS, INDIC_NUKTAS, INDIC_MATRAS, INDIC_MODIFIERS, INDIC_VOWELS, INDIC_POOLS }; +static size_t indic_syllable(Random *r, const Pool *indic, uint32_t *text, size_t n) { + const Pool *c = &indic[INDIC_CONSONANTS], *ra = &indic[INDIC_RA], *h = &indic[INDIC_HALANTS], + *nukta = &indic[INDIC_NUKTAS], *m = &indic[INDIC_MATRAS], *sm = &indic[INDIC_MODIFIERS], + *v = &indic[INDIC_VOWELS]; + if (!c->count || !h->count) return n; + if (ra->count && !below(r, 4)) { + text[n++] = pick(r, ra); + text[n++] = pick(r, h); + if (!below(r, 8)) text[n++] = 0x200D; + } + if (v->count && !below(r, 6)) { + text[n++] = pick(r, v); + if (nukta->count && !below(r, 8)) text[n++] = pick(r, nukta); + } else + for (unsigned k = 0, count = 1 + below(r, 3); k < count; ++k) { + text[n++] = ra->count && !below(r, 5) ? pick(r, ra) : pick(r, c); + if (nukta->count && !below(r, 6)) text[n++] = pick(r, nukta); + if (k + 1 == count) break; + unsigned joiner = below(r, 8); + if (joiner == 0) text[n++] = 0x200D; + text[n++] = pick(r, h); + if (joiner == 1) text[n++] = 0x200D; + if (joiner == 2) text[n++] = 0x200C; + } + if (m->count && below(r, 3)) { + text[n++] = pick(r, m); + if (!below(r, 8)) text[n++] = pick(r, m); + } + if (sm->count && !below(r, 4)) text[n++] = pick(r, sm); + if (!below(r, 10)) text[n++] = pick(r, h); + if (!below(r, 12)) text[n++] = below(r, 2) ? 0x200C : 0x200D; + return n; +} + /* A Hangul syllable the ways text spells one (hb-ot-shaper-hangul.cc): jamo * in the orders that compose (L V, L V T, LV T), a syllable whole, jamo in * any order; now and then a tone mark after. jamo holds the font's L, V @@ -805,8 +935,9 @@ void vega_shaping_check(hb_font_t *font, const VegaShaping *s, uint32_t script, /* Letters of the script, marks it can take (its own and inherited * ones), the rest of its characters and Common's (spaces, digits, * punctuation), and those whose glyphs some substitution starts from. */ - Pool letters = {0}, marks = {0}, others = {0}, hot = {0}, jamo_pools[3] = {{0}}; + Pool letters = {0}, marks = {0}, others = {0}, hot = {0}, jamo_pools[3] = {{0}}, indic[INDIC_POOLS] = {{0}}; bool hangul = script == HB_TAG('H', 'a', 'n', 'g'); + bool syllabic = listed(indic_scripts, sizeof(indic_scripts) / sizeof(*indic_scripts), script); hb_set_t *inputs = hb_set_create(); uint32_t common = HB_TAG('Z', 'y', 'y', 'y'), inherited = HB_TAG('Z', 'i', 'n', 'h'); for (unsigned t = 0; t < s->lookup_count[0]; ++t) @@ -820,6 +951,19 @@ void vega_shaping_check(hb_font_t *font, const VegaShaping *s, uint32_t script, if (hangul && ((code >= 0x1100 && code <= 0x115F) || (code >= 0xA960 && code <= 0xA97C))) pool_add(&jamo_pools[0], code); if (hangul && ((code >= 0x1160 && code <= 0x11A7) || (code >= 0xD7B0 && code <= 0xD7C6))) pool_add(&jamo_pools[1], code); if (hangul && ((code >= 0x11A8 && code <= 0x11FF) || (code >= 0xD7CB && code <= 0xD7FB))) pool_add(&jamo_pools[2], code); + if (syllabic && (of == script || of == inherited)) { + int pool = -1; + switch (vega_indic_categories(code) & 0xFF) { + case VEGA_OT_C: case VEGA_OT_CS: pool = INDIC_CONSONANTS; break; + case VEGA_OT_RA: pool = INDIC_RA; break; + case VEGA_OT_H: pool = INDIC_HALANTS; break; + case VEGA_OT_N: pool = INDIC_NUKTAS; break; + case VEGA_OT_M: case VEGA_OT_MPST: pool = INDIC_MATRAS; break; + case VEGA_OT_SM: case VEGA_OT_A: pool = INDIC_MODIFIERS; break; + case VEGA_OT_V: pool = INDIC_VOWELS; break; + } + if (pool >= 0) pool_add(&indic[pool], code); + } if (mark && (of == script || of == inherited)) pool_add(&marks, code); else if (!mark && of == script && c >= VEGA_LU && c <= VEGA_LO) pool_add(&letters, code); else if (!mark && (of == script || of == common) && c != VEGA_ZL && c != VEGA_ZP) pool_add(&others, code); @@ -851,6 +995,10 @@ void vega_shaping_check(hb_font_t *font, const VegaShaping *s, uint32_t script, n = hangul_syllable(&random, jamo_pools, text, n); continue; } + if (syllabic && below(&random, 3)) { + n = indic_syllable(&random, indic, text, n); + continue; + } text[n++] = hot.count && below(&random, 3) == 0 ? pick(&random, &hot) : pick(&random, &letters); if (marks.count && below(&random, 100) < mark_rate) { text[n++] = pick(&random, &marks); @@ -916,4 +1064,5 @@ void vega_shaping_check(hb_font_t *font, const VegaShaping *s, uint32_t script, free(others.items); free(hot.items); for (int k = 0; k < 3; ++k) free(jamo_pools[k].items); + for (int k = 0; k < INDIC_POOLS; ++k) free(indic[k].items); } diff --git a/text/shape.c b/text/shape.c index 0813bae..3443b0b 100644 --- a/text/shape.c +++ b/text/shape.c @@ -1,15 +1,18 @@ -/* Shaping as HarfBuzz 10 does it (text/shape.h), with its default, Arabic - * and Hebrew shapers. The code follows HarfBuzz's own step for step - * (hb-ot-shape.cc, hb-ot-shape-normalize.cc, hb-buffer.cc, hb-ot-layout- - * gsubgpos.hh, the GSUB and GPOS lookups, hb-ot-shaper-arabic.cc and - * -hebrew.cc), under HarfBuzz's names, so that the two read side by side; - * the build tool compares the results with HarfBuzz's (symbols-tool.c). */ +/* Shaping as HarfBuzz 10 does it (text/shape.h), with its default, Arabic, + * Hebrew, Hangul and Indic shapers. The code follows HarfBuzz's own step + * for step (hb-ot-shape.cc, hb-ot-shape-normalize.cc, hb-buffer.cc, + * hb-ot-layout-gsubgpos.hh, the GSUB and GPOS lookups, hb-ot-shaper-*.cc), + * under HarfBuzz's names, so that the two read side by side; the build + * tool compares the results with HarfBuzz's (symbols-tool.c). */ #include "shape.h" +#include "syllabic.h" #include "text.h" #include <stdio.h> #include <stdlib.h> #include <string.h> +#define TAG4(a, b, c, d) ((uint32_t)(a) << 24 | (uint32_t)(b) << 16 | (uint32_t)(c) << 8 | (uint32_t)(d)) + /* ---- OpenType tables, read in place: a read past a table's end gives 0 */ static uint32_t get16(VegaTable t, uint32_t at) { @@ -223,17 +226,22 @@ bool vega_shaping_open(VegaShaping *s, const unsigned char *bytes, size_t size, REFUSE("out of memory"); for (uint32_t i = 0; i < s->plan_count; ++i) { const unsigned char *p = s->plans + (size_t)i * VEGA_PLAN_BYTES; - uint32_t ranges[2][2] = {{le32(p + 72), le32(p + 76)}, {le32(p + 80), le32(p + 84)}}; - if ((p[4] != VEGA_SHAPE_LTR && p[4] != VEGA_SHAPE_RTL) || p[5] > VEGA_SHAPE_RTL || p[6] > VEGA_SHAPER_HANGUL) + uint32_t ranges[2][2] = {{le32(p + 80), le32(p + 84)}, {le32(p + 88), le32(p + 92)}}; + if ((p[4] != VEGA_SHAPE_LTR && p[4] != VEGA_SHAPE_RTL) || p[5] > VEGA_SHAPE_RTL || p[6] > VEGA_SHAPER_INDIC) REFUSE("plan %u", i); if ((p[7] & VEGA_PLAN_FALLBACK_MARKS) && !s->extents) REFUSE("plan %u: marks placed without extents", i); for (int table = 0; table < 2; ++table) { if (ranges[table][0] > step_count || ranges[table][1] > step_count - ranges[table][0]) REFUSE("plan %u: steps", i); - for (uint32_t k = 0; k < ranges[table][1]; ++k) - if (le16(s->steps + (size_t)(ranges[table][0] + k) * VEGA_STEP_BYTES) >= s->lookup_count[table]) + for (uint32_t k = 0; k < ranges[table][1]; ++k) { + const unsigned char *step = s->steps + (size_t)(ranges[table][0] + k) * VEGA_STEP_BYTES; + bool pause = le16(step) == VEGA_STEP_PAUSE; + if (pause ? table || !step[2] || step[2] > VEGA_PAUSE_CLEAR_SYLLABLES : le16(step) >= s->lookup_count[table]) REFUSE("plan %u: a lookup the font lacks", i); + } } + for (int k = 0; k < VEGA_PLAN_STAGES; ++k) + if ((uint32_t)le16(p + 100 + 8 * k) + le16(p + 102 + 8 * k) > ranges[0][1]) REFUSE("plan %u: stages", i); } return true; } @@ -290,7 +298,13 @@ typedef struct { uint32_t codepoint; /* a character, then (map_glyphs) a glyph */ uint32_t mask, cluster, glyph_index; uint16_t uprops, gprops; - uint8_t lig_props, action; + uint8_t lig_props; + union { + uint8_t action; /* Arabic's joining action, Hangul's jamo feature */ + uint8_t position; /* Indic's */ + }; + uint8_t category; /* Indic's */ + uint8_t syllable; /* serial << 4 | type, once a shaper finds syllables */ } Info; typedef struct { int32_t x_advance, y_advance, x_offset, y_offset; @@ -311,7 +325,7 @@ struct VegaShaper { }; enum { HAS_NON_ASCII = 1, HAS_DEFAULT_IGNORABLES = 2, HAS_SPACE_FALLBACK = 4, HAS_GPOS_ATTACHMENT = 8, HAS_CGJ = 16, - HAS_VARIATION_SELECTOR_FALLBACK = 32 }; + HAS_VARIATION_SELECTOR_FALLBACK = 32, HAS_BROKEN_SYLLABLE = 64 }; VegaShaper *vega_shaper_new(void) { return calloc(1, sizeof(VegaShaper)); } @@ -390,6 +404,11 @@ static void output_glyph(VegaShaper *b, uint32_t glyph) { b->out[b->out_len++].codepoint = glyph; } +static void output_info(VegaShaper *b, Info info) { + if (!ensure(b, b->out_len + 1)) return; + b->out[b->out_len++] = info; +} + /* num_in glyphs become num_out, each a copy of the first with its own * codepoint; their clusters merge. */ static void merge_clusters(VegaShaper *b, size_t start, size_t end); @@ -674,12 +693,15 @@ static void form_clusters(VegaShaper *b) { /* ---- The plan */ +typedef struct { uint32_t first, count; } Stage; typedef struct { uint32_t script; int direction, native, shaper, flags; uint32_t global_mask, kern_mask, masks[VEGA_PLAN_MASKS]; const unsigned char *gsub, *gpos; uint32_t gsub_count, gpos_count; + unsigned shaper_flags; + Stage stages[VEGA_PLAN_STAGES]; } Plan; static bool find_plan(const VegaShaping *s, uint32_t script, int direction, Plan *plan) { @@ -691,9 +713,12 @@ static bool find_plan(const VegaShaping *s, uint32_t script, int direction, Plan } if (!found) return false; *plan = (Plan){le32(found), found[4], found[5], found[6], found[7], le32(found + 8), le32(found + 12), {0}, - s->steps + (size_t)le32(found + 72) * VEGA_STEP_BYTES, - s->steps + (size_t)le32(found + 80) * VEGA_STEP_BYTES, le32(found + 76), le32(found + 84)}; + s->steps + (size_t)le32(found + 80) * VEGA_STEP_BYTES, + s->steps + (size_t)le32(found + 88) * VEGA_STEP_BYTES, le32(found + 84), le32(found + 92), found[96], + {{0, 0}}}; for (int k = 0; k < VEGA_PLAN_MASKS; ++k) plan->masks[k] = le32(found + 16 + 4 * k); + for (int k = 0; k < VEGA_PLAN_STAGES; ++k) + plan->stages[k] = (Stage){le16(found + 100 + 8 * k), le16(found + 102 + 8 * k)}; return true; } @@ -783,8 +808,23 @@ static bool compose_hebrew(const Normalizer *c, uint32_t a, uint32_t b, uint32_t return *ab != 0; } +/* Indic's: no split matra composes again, and Bengali YA with nukta does, + * though Unicode excludes it. */ +static bool compose_indic(uint32_t a, uint32_t b, uint32_t *ab) { + if (is_mark_category(vega_category(a))) return false; + if (a == 0x09AF && b == 0x09BC) { + *ab = 0x09DF; + return true; + } + return compose_unicode(a, b, ab); +} + static bool compose(const Normalizer *c, uint32_t a, uint32_t b, uint32_t *ab) { - return c->plan->shaper == VEGA_SHAPER_HEBREW ? compose_hebrew(c, a, b, ab) : compose_unicode(a, b, ab); + switch (c->plan->shaper) { + case VEGA_SHAPER_HEBREW: return compose_hebrew(c, a, b, ab); + case VEGA_SHAPER_INDIC: return compose_indic(a, b, ab); + default: return compose_unicode(a, b, ab); + } } static bool nominal(const VegaShaping *s, uint32_t u, uint32_t *glyph) { return (*glyph = vega_shaping_glyph(s, u)) != 0; } @@ -801,10 +841,17 @@ static void next_char(VegaShaper *b, uint32_t glyph) { next_glyph(b); } +/* The shaper's decomposition: Indic keeps a few letters whole. */ +static bool decompose_char(const Normalizer *c, uint32_t ab, uint32_t *a, uint32_t *b) { + if (c->plan->shaper == VEGA_SHAPER_INDIC && (ab == 0x0931 || ab == 0x09DC || ab == 0x09DD || ab == 0x0B94)) + return false; + return decompose_unicode(ab, a, b); +} + /* Returns how many characters ab became, 0 if it did not decompose. */ static unsigned decompose(const Normalizer *c, bool shortest, uint32_t ab) { uint32_t a, bb, a_glyph = 0, b_glyph = 0; - if (!decompose_unicode(ab, &a, &bb) || (bb && !nominal(c->s, bb, &b_glyph))) return 0; + if (!decompose_char(c, ab, &a, &bb) || (bb && !nominal(c->s, bb, &b_glyph))) return 0; bool has_a = nominal(c->s, a, &a_glyph); if (shortest && has_a) { output_char(c->b, a, a_glyph); @@ -976,9 +1023,10 @@ static void normalize(const VegaShaping *s, const Plan *plan, VegaShaper *b) { if (!b->len) return; Normalizer c = {s, plan, b}; /* The shapers' preference is "auto" (composed diacritics); Hangul's is - * none: what the font has stays, and nothing is composed again. */ + * none: what the font has stays, and nothing is composed again; + * Indic's decomposes what decomposes even when the font has it whole. */ const bool none = plan->shaper == VEGA_SHAPER_HANGUL; - const bool might_short_circuit = true, always_short_circuit = none; + const bool always_short_circuit = none, might_short_circuit = none || plan->shaper != VEGA_SHAPER_INDIC; bool all_simple = true; clear_output(b); size_t count = b->len; @@ -1237,6 +1285,7 @@ typedef struct { uint32_t lookup_mask, lookup_props; bool auto_zwnj, auto_zwj, random, has_glyph_classes; int nesting_left; + bool per_syllable; /* GSUB matches within the syllable it starts in */ uint32_t random_state; long last_base; size_t last_base_until; @@ -1271,6 +1320,7 @@ typedef struct { Matcher matcher; uint32_t glyph_data; size_t idx, end; + uint8_t syllable; /* per syllable, from where the lookup is: only its glyphs match */ } Skippy; enum { SKIP_NO, SKIP_YES, SKIP_MAYBE }; @@ -1278,9 +1328,11 @@ enum { MATCH_NO, MATCH_YES, MATCH_MAYBE }; enum { MATCH, NOT_MATCH, SKIP }; static void skippy_init(Skippy *it, Apply *c, bool context_match, size_t start) { + VegaShaper *b = c->b; *it = (Skippy){c, c->lookup_props, context_match ? 0xFFFFFFFFu : c->lookup_mask, c->table == 1 || (context_match && c->auto_zwnj), context_match || c->auto_zwj, c->table == 1, - {MATCH_ANY, 0, 0}, 0, start, c->b->len}; + {MATCH_ANY, 0, 0}, 0, start, b->len, + !c->table && c->per_syllable && start == b->idx && b->idx < b->len ? b->info[b->idx].syllable : 0}; } static int may_skip(const Skippy *it, const Info *info) { @@ -1302,7 +1354,7 @@ static bool matches(const Apply *c, const Matcher *m, uint32_t index, uint32_t g } static int may_match(const Skippy *it, const Info *info) { - if (!(info->mask & it->mask)) return MATCH_NO; + if (!(info->mask & it->mask) || (it->syllable && it->syllable != info->syllable)) return MATCH_NO; if (it->matcher.kind != MATCH_ANY) return matches(it->c, &it->matcher, it->glyph_data, info->codepoint) ? MATCH_YES : MATCH_NO; return MATCH_MAYBE; } @@ -2105,13 +2157,23 @@ static VegaDigest buffer_digest(const VegaShaper *b) { return d; } -static void apply_table(const VegaShaping *s, VegaShaper *b, int table, const unsigned char *steps, uint32_t count) { +static bool shaper_pause(const VegaShaping *s, const Plan *plan, VegaShaper *b, unsigned pause); + +/* hb_ot_map_t::apply: the plan's lookups for a table in order; at a pause, + * the shaper's part (when it changed the buffer, the digest starts over). */ +static void apply_table(const VegaShaping *s, const Plan *plan, VegaShaper *b, int table) { + const unsigned char *steps = table ? plan->gpos : plan->gsub; + uint32_t count = table ? plan->gpos_count : plan->gsub_count; /* A lookup can apply only where the buffer has a glyph it starts from. */ - Apply c = {s, b, table, table ? s->gpos : s->gsub, 0, 0, true, true, false, s->glyph_classes != 0, - MAX_NESTING_LEVEL, 1, -1, 0, buffer_digest(b)}; + Apply c = {.s = s, .b = b, .table = table, .t = table ? s->gpos : s->gsub, .has_glyph_classes = s->glyph_classes != 0, + .nesting_left = MAX_NESTING_LEVEL, .random_state = 1, .last_base = -1, .digest = buffer_digest(b)}; for (uint32_t i = 0; i < count; ++i) { const unsigned char *step = steps + (size_t)i * VEGA_STEP_BYTES; uint32_t index = le16(step); + if (index == VEGA_STEP_PAUSE) { + if (shaper_pause(s, plan, b, step[2])) c.digest = buffer_digest(b); + continue; + } if (!digest_may_intersect(&s->lookups[table][index].digest, &c.digest)) continue; c.lookup_mask = le32(step + 4); c.last_base = -1; @@ -2119,6 +2181,7 @@ static void apply_table(const VegaShaping *s, VegaShaper *b, int table, const un c.auto_zwnj = step[2] & VEGA_STEP_AUTO_ZWNJ; c.auto_zwj = step[2] & VEGA_STEP_AUTO_ZWJ; c.random = step[2] & VEGA_STEP_RANDOM; + c.per_syllable = step[2] & VEGA_STEP_PER_SYLLABLE; apply_string(&c, &s->lookups[table][index]); } } @@ -2160,8 +2223,9 @@ static void kern_table(const VegaShaping *s, VegaShaper *b, const Plan *plan) { if ((apple && (coverage & 0x20)) || !horizontal || format != 0) continue; bool reverse = b->direction == VEGA_SHAPE_RTL; if (reverse) reverse_range(b, 0, b->len, true); - Apply c = {s, b, 1, s->gpos, plan->kern_mask, 0x08, true, true, false, s->glyph_classes != 0, MAX_NESTING_LEVEL, 1, - -1, 0, {{0, 0, 0}}}; + Apply c = {.s = s, .b = b, .table = 1, .t = s->gpos, .lookup_mask = plan->kern_mask, .lookup_props = 0x08, + .auto_zwnj = true, .auto_zwj = true, .has_glyph_classes = s->glyph_classes != 0, + .nesting_left = MAX_NESTING_LEVEL, .random_state = 1, .last_base = -1}; for (size_t idx = 0; idx < b->len;) { if (!(b->info[idx].mask & plan->kern_mask)) { idx++; @@ -2385,6 +2449,756 @@ static void fallback_marks(const VegaShaping *s, const Plan *plan, VegaShaper *b position_cluster(s, plan, b, start, b->len, adjust); } +/* ---- Would a lookup substitute glyphs as they are, alone + * (hb_ot_layout_lookup_would_substitute): the Indic shaper's question of + * its basic features. */ + +static bool would_match_input(const Apply *c, Matcher m, const uint32_t *glyphs, unsigned len, unsigned count) { + if (count != len) return false; + for (unsigned i = 1; i < count; ++i) + if (!matches(c, &m, i - 1, glyphs[i])) return false; + return true; +} + +static bool would_apply_rules(const Apply *c, int type, uint32_t set, Matcher input, const uint32_t *glyphs, + unsigned len, bool zero_context) { + VegaTable t = c->t; + for (uint32_t r = 0, n = set ? get16(t, set) : 0; r < n; ++r) { + uint32_t rule = offset16(t, set, set + 2 + 2 * r); + if (type == 5) { + input.values = rule + 4; + if (would_match_input(c, input, glyphs, len, get16(t, rule))) return true; + continue; + } + unsigned back_count = get16(t, rule); + uint32_t input_at = rule + 2 + 2 * back_count; + unsigned input_count = get16(t, input_at); + unsigned ahead_count = input_count ? get16(t, input_at + 2 + 2 * (input_count - 1)) : 0; + if (zero_context && (back_count || ahead_count)) continue; + input.values = input_at + 2; + if (would_match_input(c, input, glyphs, len, input_count)) return true; + } + return false; +} + +static bool would_apply_subtable(const Apply *c, const VegaSubtable *st, const uint32_t *glyphs, unsigned len, + bool zero_context) { + VegaTable t = c->t; + uint32_t at = st->at, format = get16(t, at); + switch (st->type) { + case 1: case 2: case 3: case 8: + if (format != 1 && !(st->type == 1 && format == 2)) return false; + return len == 1 && coverage(t, offset16(t, at, at + 2), glyphs[0]) != NOT_COVERED; + case 4: { + uint32_t index = coverage(t, offset16(t, at, at + 2), glyphs[0]); + if (format != 1 || index == NOT_COVERED || index >= get16(t, at + 4)) return false; + uint32_t set = offset16(t, at, at + 6 + 2 * index); + for (uint32_t k = 0, n = set ? get16(t, set) : 0; k < n; ++k) { + uint32_t lig = offset16(t, set, set + 2 + 2 * k); + if (would_match_input(c, (Matcher){MATCH_GLYPH, 0, lig + 4}, glyphs, len, get16(t, lig + 2))) return true; + } + return false; + } + case 5: case 6: { + bool chain = st->type == 6; + if (format == 1 || format == 2) { + /* format 2 takes the first glyph's class whatever the coverage */ + uint32_t classes = format == 2 ? offset16(t, at, at + (chain ? 6 : 4)) : 0; + uint32_t index = format == 1 ? coverage(t, offset16(t, at, at + 2), glyphs[0]) : class_of(t, classes, glyphs[0]); + uint32_t sets_at = format == 1 ? at + 4 : at + (chain ? 10 : 6); + if (index == NOT_COVERED || index >= get16(t, sets_at)) return false; + Matcher input = {format == 2 ? MATCH_CLASS : MATCH_GLYPH, classes, 0}; + return would_apply_rules(c, st->type, offset16(t, at, sets_at + 2 + 2 * index), input, glyphs, len, + zero_context); + } + if (format != 3) return false; + if (!chain) return would_match_input(c, (Matcher){MATCH_COVERAGE, at, at + 8}, glyphs, len, get16(t, at + 2)); + unsigned back_count = get16(t, at + 2); + uint32_t input_at = at + 4 + 2 * back_count; + unsigned input_count = get16(t, input_at); + unsigned ahead_count = get16(t, input_at + 2 + 2 * input_count); + if (zero_context && (back_count || ahead_count)) return false; + return would_match_input(c, (Matcher){MATCH_COVERAGE, at, input_at + 4}, glyphs, len, input_count); + } + default: return false; + } +} + +/* Whether the lookups of a feature's stage (all of them, whatever their + * masks) would substitute glyphs, len of them. */ +static bool would_substitute(const VegaShaping *s, const Plan *plan, int stage, const uint32_t *glyphs, unsigned len, + bool zero_context) { + const Stage *st = &plan->stages[stage]; + Apply c = {.s = s, .t = s->gsub}; + for (uint32_t k = 0; k < st->count; ++k) { + const unsigned char *step = plan->gsub + (size_t)(st->first + k) * VEGA_STEP_BYTES; + uint32_t index = le16(step); + if (index == VEGA_STEP_PAUSE) continue; + const VegaLookup *l = &s->lookups[0][index]; + if (!len || !digest_may_have(&l->digest, glyphs[0])) continue; + for (uint32_t i = 0; i < l->count; ++i) + if (would_apply_subtable(&c, &s->subtables[l->first + i], glyphs, len, zero_context)) return true; + } + return false; +} + +/* ---- Syllables (hb-ot-shaper-*-machine.rl, hb-ot-shaper-syllabic.cc) */ + +/* A shaper's machine over the buffer's categories: from each position the + * longest syllable it accepts; each numbered, 1 to 15 over again, beside + * its type. */ +static void find_syllables(VegaShaper *b, const VegaMachine *m) { + unsigned serial = 1; + for (size_t p = 0; p < b->len;) { + unsigned state = 0, type = 0; + size_t end = p + 1; + for (size_t q = p; q < b->len; ++q) { + size_t cell = (size_t)state * m->classes + m->class_of[b->info[q].category < 63 ? b->info[q].category : 63]; + state = m->width == 1 ? ((const uint8_t *)m->next)[cell] : ((const uint16_t *)m->next)[cell]; + if (state >= m->states) break; + if (m->accept[state]) type = m->accept[state], end = q + 1; + } + if (type == m->broken) b->scratch |= HAS_BROKEN_SYLLABLE; + for (size_t i = p; i < end; ++i) b->info[i].syllable = (uint8_t)(serial << 4 | (type ? type - 1 : 0)); + if (++serial == 16) serial = 1; + p = end; + } +} + +static size_t next_syllable(const VegaShaper *b, size_t start) { + size_t end = start + 1; + while (end < b->len && b->info[end].syllable == b->info[start].syllable) end++; + return end; +} + +/* A dotted circle at the start of each broken syllable (after a repha that + * starts it), for its marks to sit on; false when none went in. */ +static bool insert_dotted_circles(const VegaShaping *s, VegaShaper *b, unsigned broken, unsigned category, int repha, + unsigned position) { + uint32_t glyph; + if (!(b->scratch & HAS_BROKEN_SYLLABLE) || !nominal(s, 0x25CC, &glyph)) return false; + /* All of it zero but these: HarfBuzz's is a fresh glyph_info. */ + Info circle = {.codepoint = glyph, .uprops = VEGA_CC, .position = (uint8_t)position, .category = (uint8_t)category}; + clear_output(b); + unsigned last = 0; + while (b->idx < b->len && b->successful) { + unsigned syllable = b->info[b->idx].syllable; + if (last != syllable && (syllable & 0x0F) == broken) { + last = syllable; + Info c = circle; + c.cluster = b->info[b->idx].cluster; + c.mask = b->info[b->idx].mask; + c.syllable = (uint8_t)syllable; + if (repha >= 0) + while (b->idx < b->len && b->successful && b->info[b->idx].syllable == last && + b->info[b->idx].category == (unsigned)repha) + next_glyph(b); + output_info(b, c); + } else + next_glyph(b); + } + sync(b); + return true; +} + +/* Vowel sequences that read as another vowel get a dotted circle between + * (hb-ot-shaper-vowel-constraints.cc, from the Universal Shaping Engine's + * IndicShapingInvalidCluster.txt): each first letter with what may not + * follow it. */ +static const struct { uint32_t script, first, second[12]; } vowel_constraints[] = { + {TAG4('D', 'e', 'v', 'a'), 0x0905, {0x093A, 0x093B, 0x093E, 0x0945, 0x0946, 0x0949, 0x094A, 0x094B, 0x094C, 0x094F, 0x0956, 0x0957}}, + {TAG4('D', 'e', 'v', 'a'), 0x0906, {0x093A, 0x0945, 0x0946, 0x0947, 0x0948}}, + {TAG4('D', 'e', 'v', 'a'), 0x0909, {0x0941}}, + {TAG4('D', 'e', 'v', 'a'), 0x090F, {0x0945, 0x0946, 0x0947}}, + {TAG4('B', 'e', 'n', 'g'), 0x0985, {0x09BE}}, + {TAG4('B', 'e', 'n', 'g'), 0x098B, {0x09C3}}, + {TAG4('B', 'e', 'n', 'g'), 0x098C, {0x09E2}}, + {TAG4('G', 'u', 'r', 'u'), 0x0A05, {0x0A3E, 0x0A48, 0x0A4C}}, + {TAG4('G', 'u', 'r', 'u'), 0x0A72, {0x0A3F, 0x0A40, 0x0A47}}, + {TAG4('G', 'u', 'r', 'u'), 0x0A73, {0x0A41, 0x0A42, 0x0A4B}}, + {TAG4('G', 'u', 'j', 'r'), 0x0A85, {0x0ABE, 0x0AC5, 0x0AC7, 0x0AC8, 0x0AC9, 0x0ACB, 0x0ACC}}, + {TAG4('G', 'u', 'j', 'r'), 0x0AC5, {0x0ABE}}, + {TAG4('O', 'r', 'y', 'a'), 0x0B05, {0x0B3E}}, + {TAG4('O', 'r', 'y', 'a'), 0x0B0F, {0x0B57}}, + {TAG4('O', 'r', 'y', 'a'), 0x0B13, {0x0B57}}, + {TAG4('T', 'a', 'm', 'l'), 0x0B85, {0x0BC2}}, + {TAG4('T', 'e', 'l', 'u'), 0x0C12, {0x0C4C, 0x0C55}}, + {TAG4('T', 'e', 'l', 'u'), 0x0C3F, {0x0C55}}, + {TAG4('T', 'e', 'l', 'u'), 0x0C46, {0x0C55}}, + {TAG4('T', 'e', 'l', 'u'), 0x0C4A, {0x0C55}}, + {TAG4('K', 'n', 'd', 'a'), 0x0C89, {0x0CBE}}, + {TAG4('K', 'n', 'd', 'a'), 0x0C8B, {0x0CBE}}, + {TAG4('K', 'n', 'd', 'a'), 0x0C92, {0x0CCC}}, + {TAG4('M', 'l', 'y', 'm'), 0x0D07, {0x0D57}}, + {TAG4('M', 'l', 'y', 'm'), 0x0D09, {0x0D57}}, + {TAG4('M', 'l', 'y', 'm'), 0x0D0E, {0x0D46}}, + {TAG4('M', 'l', 'y', 'm'), 0x0D12, {0x0D3E, 0x0D57}}, + {TAG4('S', 'i', 'n', 'h'), 0x0D85, {0x0DCF, 0x0DD0, 0x0DD1}}, + {TAG4('S', 'i', 'n', 'h'), 0x0D8B, {0x0DDF}}, + {TAG4('S', 'i', 'n', 'h'), 0x0D8D, {0x0DD8}}, + {TAG4('S', 'i', 'n', 'h'), 0x0D8F, {0x0DDF}}, + {TAG4('S', 'i', 'n', 'h'), 0x0D91, {0x0DCA, 0x0DD9, 0x0DDA, 0x0DDC, 0x0DDD, 0x0DDE}}, + {TAG4('S', 'i', 'n', 'h'), 0x0D94, {0x0DDF}}, + {TAG4('B', 'r', 'a', 'h'), 0x11005, {0x11038}}, + {TAG4('B', 'r', 'a', 'h'), 0x1100B, {0x1103E}}, + {TAG4('B', 'r', 'a', 'h'), 0x1100F, {0x11042}}, + {TAG4('K', 'h', 'o', 'j'), 0x11200, {0x1122C, 0x11231, 0x11233}}, + {TAG4('K', 'h', 'o', 'j'), 0x11206, {0x1122C}}, + {TAG4('K', 'h', 'o', 'j'), 0x1122C, {0x11230, 0x11231}}, + {TAG4('K', 'h', 'o', 'j'), 0x11240, {0x1122E}}, + {TAG4('S', 'i', 'n', 'd'), 0x112B0, {0x112E0, 0x112E5, 0x112E6, 0x112E7, 0x112E8}}, + {TAG4('T', 'i', 'r', 'h'), 0x11481, {0x114B0}}, + {TAG4('T', 'i', 'r', 'h'), 0x1148B, {0x114BA}}, + {TAG4('T', 'i', 'r', 'h'), 0x1148D, {0x114BA}}, + {TAG4('T', 'i', 'r', 'h'), 0x114AA, {0x114B5, 0x114B6}}, + {TAG4('M', 'o', 'd', 'i'), 0x11600, {0x11639, 0x1163A}}, + {TAG4('M', 'o', 'd', 'i'), 0x11601, {0x11639, 0x1163A}}, + {TAG4('T', 'a', 'k', 'r'), 0x11680, {0x116AD, 0x116B4, 0x116B5}}, + {TAG4('T', 'a', 'k', 'r'), 0x11686, {0x116B2}}, +}; + +/* A dotted circle out before the current character, as a copy of it (its + * properties, less continuing the character before). */ +static void output_dotted_circle(VegaShaper *b) { + output_glyph(b, 0x25CC); + if (b->out_len) b->out[b->out_len - 1].uprops &= (uint16_t)~UPROPS_CONTINUATION; +} + +static void preprocess_vowel_constraints(const Plan *plan, VegaShaper *b) { + bool any = false; + for (size_t k = 0; k < sizeof(vowel_constraints) / sizeof(*vowel_constraints) && !any; ++k) + any = vowel_constraints[k].script == plan->script; + if (!any && plan->script != TAG4('D', 'e', 'v', 'a')) return; + clear_output(b); + while (b->idx + 1 < b->len && b->successful) { + uint32_t u = b->info[b->idx].codepoint, next = b->info[b->idx + 1].codepoint; + bool matched = false; + for (size_t k = 0; k < sizeof(vowel_constraints) / sizeof(*vowel_constraints) && !matched; ++k) + if (vowel_constraints[k].script == plan->script && vowel_constraints[k].first == u) + for (int j = 0; j < 12 && vowel_constraints[k].second[j] && !matched; ++j) + matched = vowel_constraints[k].second[j] == next; + /* Devanagari's RA, virama, I: the circle goes before the I */ + if (plan->script == TAG4('D', 'e', 'v', 'a') && u == 0x0930 && next == 0x094D && b->idx + 2 < b->len && + b->info[b->idx + 2].codepoint == 0x0907) { + next_glyph(b); + next_glyph(b); + output_dotted_circle(b); + } + next_glyph(b); + if (matched) { + output_dotted_circle(b); + next_glyph(b); + } + } + sync(b); +} + +/* ---- Indic (hb-ot-shaper-indic.cc): each consonant syllable's base found, + * its pre-base matras, reph and the like moved around the base and masked + * for the basic features; after those, the matras, reph and + * pre-base-reordering consonants placed again by what the features made. */ + +enum { REPH_IMPLICIT, REPH_EXPLICIT, REPH_LOG_REPHA }; +typedef struct { uint32_t script, virama; uint8_t reph_pos, reph_mode; bool blwf_post_only; } IndicConfig; +static const IndicConfig indic_configs[] = { + {0, 0, VEGA_POS_BEFORE_POST, REPH_IMPLICIT, false}, + {TAG4('D', 'e', 'v', 'a'), 0x094D, VEGA_POS_BEFORE_POST, REPH_IMPLICIT, false}, + {TAG4('B', 'e', 'n', 'g'), 0x09CD, VEGA_POS_AFTER_SUB, REPH_IMPLICIT, false}, + {TAG4('G', 'u', 'r', 'u'), 0x0A4D, VEGA_POS_BEFORE_SUB, REPH_IMPLICIT, false}, + {TAG4('G', 'u', 'j', 'r'), 0x0ACD, VEGA_POS_BEFORE_POST, REPH_IMPLICIT, false}, + {TAG4('O', 'r', 'y', 'a'), 0x0B4D, VEGA_POS_AFTER_MAIN, REPH_IMPLICIT, false}, + {TAG4('T', 'a', 'm', 'l'), 0x0BCD, VEGA_POS_AFTER_POST, REPH_IMPLICIT, false}, + {TAG4('T', 'e', 'l', 'u'), 0x0C4D, VEGA_POS_AFTER_POST, REPH_EXPLICIT, true}, + {TAG4('K', 'n', 'd', 'a'), 0x0CCD, VEGA_POS_AFTER_POST, REPH_IMPLICIT, true}, + {TAG4('M', 'l', 'y', 'm'), 0x0D4D, VEGA_POS_AFTER_MAIN, REPH_LOG_REPHA, false}, +}; + +typedef struct { + const VegaShaping *s; + const Plan *plan; + const IndicConfig *config; + bool old_spec, zero_context; + uint32_t virama; /* its glyph; 0 when the font has none */ +} Indic; + +static Indic indic_of(const VegaShaping *s, const Plan *plan) { + Indic d = {s, plan, &indic_configs[0], plan->shaper_flags & VEGA_INDIC_OLD_SPEC, false, 0}; + for (size_t i = 1; i < sizeof(indic_configs) / sizeof(*indic_configs); ++i) + if (indic_configs[i].script == plan->script) d.config = &indic_configs[i]; + /* New specifications match the basic features without context; so do + * scripts with one, but Malayalam's both take context. */ + d.zero_context = !d.old_spec && plan->script != TAG4('M', 'l', 'y', 'm'); + if (d.config->virama) nominal(s, d.config->virama, &d.virama); + return d; +} + +#define FLAG(x) (1u << (x)) +#define FLAG_UNSAFE(x) ((x) < 32 ? 1u << (x) : 0u) +static bool is_one_of(const Info *i, uint32_t flags) { + return !(i->gprops & LIGATED) && (FLAG_UNSAFE(i->category) & flags); /* a ligature is none of them */ +} +static bool is_consonant(const Info *i) { + return is_one_of(i, FLAG(VEGA_OT_C) | FLAG(VEGA_OT_CS) | FLAG(VEGA_OT_RA) | FLAG(VEGA_OT_CM) | FLAG(VEGA_OT_V) | + FLAG(VEGA_OT_PLACEHOLDER) | FLAG(VEGA_OT_DOTTEDCIRCLE)); +} +static bool is_joiner(const Info *i) { return is_one_of(i, FLAG(VEGA_OT_ZWJ) | FLAG(VEGA_OT_ZWNJ)); } +static bool is_halant(const Info *i) { return is_one_of(i, FLAG(VEGA_OT_H)); } +static bool ligated_and_didnt_multiply(const Info *i) { return (i->gprops & LIGATED) && !(i->gprops & MULTIPLIED); } + +static bool indic_would(const Indic *d, int stage, const uint32_t *glyphs, unsigned len) { + return would_substitute(d->s, d->plan, stage, glyphs, len, d->zero_context); +} + +/* Below-base, post-base or base: what the font's features would make of + * the consonant with a virama, either way round (fonts copied old lookups + * into new scripts). */ +static uint8_t consonant_position(const Indic *d, uint32_t consonant) { + uint32_t glyphs[3] = {d->virama, consonant, d->virama}; + if (indic_would(d, VEGA_STAGE_BLWF, glyphs, 2) || indic_would(d, VEGA_STAGE_BLWF, glyphs + 1, 2) || + indic_would(d, VEGA_STAGE_VATU, glyphs, 2) || indic_would(d, VEGA_STAGE_VATU, glyphs + 1, 2)) + return VEGA_POS_BELOW_C; + if (indic_would(d, VEGA_STAGE_PSTF, glyphs, 2) || indic_would(d, VEGA_STAGE_PSTF, glyphs + 1, 2) || + indic_would(d, VEGA_STAGE_PREF, glyphs, 2) || indic_would(d, VEGA_STAGE_PREF, glyphs + 1, 2)) + return VEGA_POS_POST_C; + return VEGA_POS_BASE_C; +} + +static void setup_masks_indic(VegaShaper *b) { + for (size_t i = 0; i < b->len; ++i) { + uint16_t categories = vega_indic_categories(b->info[i].codepoint); + b->info[i].category = categories & 0xFF; + b->info[i].position = categories >> 8; + } +} + +static void indic_reorder_consonant_syllable(const Indic *d, VegaShaper *b, size_t start, size_t end) { + Info *info = b->info; + const uint32_t *masks = d->plan->masks; + uint32_t script = d->plan->script; + /* Kannada's Ra, halant, ZWJ is Ra, ZWJ, halant. */ + if (script == TAG4('K', 'n', 'd', 'a') && start + 3 <= end && is_one_of(&info[start], FLAG(VEGA_OT_RA)) && + is_one_of(&info[start + 1], FLAG(VEGA_OT_H)) && is_one_of(&info[start + 2], FLAG(VEGA_OT_ZWJ))) { + merge_clusters(b, start + 1, start + 3); + Info t = info[start + 1]; + info[start + 1] = info[start + 2]; + info[start + 2] = t; + } + + /* 1. The base: from the end back, the first consonant without a + * below-base or post-base form (those follow below-base ones), or the + * first consonant. An initial Ra, halant that the font makes a reph + * is no candidate. */ + size_t base = end, limit = start; + bool has_reph = false; + if (masks[VEGA_MASK_RPHF] && start + 3 <= end && + ((d->config->reph_mode == REPH_IMPLICIT && !is_joiner(&info[start + 2])) || + (d->config->reph_mode == REPH_EXPLICIT && info[start + 2].category == VEGA_OT_ZWJ))) { + uint32_t glyphs[3] = {info[start].codepoint, info[start + 1].codepoint, + d->config->reph_mode == REPH_EXPLICIT ? info[start + 2].codepoint : 0}; + if (indic_would(d, VEGA_STAGE_RPHF, glyphs, 2) || + (d->config->reph_mode == REPH_EXPLICIT && indic_would(d, VEGA_STAGE_RPHF, glyphs, 3))) { + limit += 2; + while (limit < end && is_joiner(&info[limit])) limit++; + base = start; + has_reph = true; + } + } else if (d->config->reph_mode == REPH_LOG_REPHA && info[start].category == VEGA_OT_REPHA) { + limit += 1; + while (limit < end && is_joiner(&info[limit])) limit++; + base = start; + has_reph = true; + } + { + size_t i = end; + bool seen_below = false; + do { + i--; + if (is_consonant(&info[i])) { + if (info[i].position != VEGA_POS_BELOW_C && (info[i].position != VEGA_POS_POST_C || seen_below)) { + base = i; + break; + } + if (info[i].position == VEGA_POS_BELOW_C) seen_below = true; + base = i; + } else if (start < i && info[i].category == VEGA_OT_ZWJ && info[i - 1].category == VEGA_OT_H) + break; /* a ZWJ after a halant asks for a half form: the search stops */ + } while (i > limit); + } + if (has_reph && base == start && limit - base <= 2) has_reph = false; /* Ra alone is the base */ + + /* 2 and 3 (matras decomposed, marks in canonical order) normalization + * did. Reorder: */ + for (size_t i = start; i < base; ++i) + if (info[i].position > VEGA_POS_PRE_C) info[i].position = VEGA_POS_PRE_C; + if (base < end) info[base].position = VEGA_POS_BASE_C; + if (has_reph) info[start].position = VEGA_POS_RA_TO_BECOME_REPH; + + /* Old specifications: the first post-base halant after the last + * consonant (Kannada not past a halant already there). */ + if (d->old_spec) { + bool disallow_double_halants = script == TAG4('K', 'n', 'd', 'a'); + for (size_t i = base + 1; i < end; ++i) + if (info[i].category == VEGA_OT_H) { + size_t j; + for (j = end - 1; j > i; j--) + if (is_consonant(&info[j]) || (disallow_double_halants && info[j].category == VEGA_OT_H)) break; + if (info[j].category != VEGA_OT_H && j > i) { + Info t = info[i]; + memmove(&info[i], &info[i + 1], (j - i) * sizeof(Info)); + info[j] = t; + } + break; + } + } + + /* Joiners, nuktas, register shifters, medials and halants go with what + * is before them (a halant not with a left matra). */ + { + uint8_t last_pos = VEGA_POS_START; + for (size_t i = start; i < end; ++i) { + if (FLAG_UNSAFE(info[i].category) & (FLAG(VEGA_OT_ZWJ) | FLAG(VEGA_OT_ZWNJ) | FLAG(VEGA_OT_N) | + FLAG(VEGA_OT_RS) | FLAG(VEGA_OT_CM) | FLAG(VEGA_OT_H))) { + info[i].position = last_pos; + if (info[i].category == VEGA_OT_H && info[i].position == VEGA_POS_PRE_M) + for (size_t j = i; j > start; j--) + if (info[j - 1].position != VEGA_POS_PRE_M) { + info[i].position = info[j - 1].position; + break; + } + } else if (info[i].position != VEGA_POS_SMVD) { + if (info[i].category == VEGA_OT_MPST && i > start && info[i - 1].category == VEGA_OT_SM) + info[i - 1].position = info[i].position; + last_pos = info[i].position; + } + } + } + /* A post-base consonant owns what is before it since the last + * consonant or matra. */ + { + size_t last = base; + for (size_t i = base + 1; i < end; ++i) + if (is_consonant(&info[i])) { + for (size_t j = last + 1; j < i; ++j) + if (info[j].position < VEGA_POS_SMVD) info[j].position = info[i].position; + last = i; + } else if (FLAG_UNSAFE(info[i].category) & (FLAG(VEGA_OT_M) | FLAG(VEGA_OT_MPST))) + last = i; + } + + { + /* Sorted by position, stably; syllable holds each glyph's place + * before, to merge the clusters of what moved past base. */ + uint8_t syllable = info[start].syllable; + for (size_t i = start; i < end; ++i) info[i].syllable = (uint8_t)(i - start); + for (size_t i = start + 1; i < end; ++i) { + size_t j = i; + while (j > start && info[j - 1].position > info[i].position) j--; + if (j == i) continue; + Info t = info[i]; + memmove(&info[j + 1], &info[j], (i - j) * sizeof(Info)); + info[j] = t; + } + /* The base again; a run of left matras back in their order. */ + size_t first_left_matra = end, last_left_matra = end; + base = end; + for (size_t i = start; i < end; ++i) { + if (info[i].position == VEGA_POS_BASE_C) { + base = i; + break; + } + if (info[i].position == VEGA_POS_PRE_M) { + if (first_left_matra == end) first_left_matra = i; + last_left_matra = i; + } + } + if (first_left_matra < last_left_matra) { + reverse_range(b, first_left_matra, last_left_matra + 1, false); + size_t i = first_left_matra; + for (size_t j = i; j <= last_left_matra; ++j) + if (FLAG_UNSAFE(info[j].category) & (FLAG(VEGA_OT_M) | FLAG(VEGA_OT_MPST))) { + reverse_range(b, i, j + 1, false); + i = j + 1; + } + } + if (d->old_spec || end - start > 127) + merge_clusters(b, base, end); + else + for (size_t i = base; i < end; ++i) + if (info[i].syllable != 255) { + size_t min = i, max = i, j = start + info[i].syllable; + while (j != i) { + if (j < min) min = j; + if (j > max) max = j; + size_t next = start + info[j].syllable; + info[j].syllable = 255; + j = next; + } + merge_clusters(b, base > min ? base : min, max + 1); + } + for (size_t i = start; i < end; ++i) info[i].syllable = syllable; + } + + /* Masks: reph; pre-base (half, and blwf in new specifications that + * have it there); post-base. */ + for (size_t i = start; i < end && info[i].position == VEGA_POS_RA_TO_BECOME_REPH; ++i) + info[i].mask |= masks[VEGA_MASK_RPHF]; + uint32_t mask = masks[VEGA_MASK_HALF]; + if (!d->old_spec && !d->config->blwf_post_only) mask |= masks[VEGA_MASK_BLWF]; + for (size_t i = start; i < base; ++i) info[i].mask |= mask; + mask = masks[VEGA_MASK_BLWF] | masks[VEGA_MASK_ABVF] | masks[VEGA_MASK_PSTF]; + for (size_t i = base + 1; i < end; ++i) info[i].mask |= mask; + + /* Old-spec Devanagari: an eyelash Ra below a half form too, but not + * with a ZWJ. */ + if (d->old_spec && script == TAG4('D', 'e', 'v', 'a')) + for (size_t i = start; i + 1 < base; ++i) + if (info[i].category == VEGA_OT_RA && info[i + 1].category == VEGA_OT_H && + (i + 2 == base || info[i + 2].category != VEGA_OT_ZWJ)) { + info[i].mask |= masks[VEGA_MASK_BLWF]; + info[i + 1].mask |= masks[VEGA_MASK_BLWF]; + } + + /* A halant, Ra the font reorders before the base: marked for pref. */ + if (masks[VEGA_MASK_PREF] && base + 2 < end) + for (size_t i = base + 1; i + 1 < end; ++i) { + uint32_t glyphs[2] = {info[i].codepoint, info[i + 1].codepoint}; + if (indic_would(d, VEGA_STAGE_PREF, glyphs, 2)) { + info[i].mask |= masks[VEGA_MASK_PREF]; + info[i + 1].mask |= masks[VEGA_MASK_PREF]; + break; + } + } + + /* A ZWNJ keeps what is before it back to a consonant from half forms. */ + for (size_t i = start + 1; i < end; ++i) + if (is_joiner(&info[i])) { + bool non_joiner = info[i].category == VEGA_OT_ZWNJ; + size_t j = i; + do { + j--; + if (non_joiner) info[j].mask &= ~masks[VEGA_MASK_HALF]; + } while (j > start && !is_consonant(&info[j])); + } +} + +static bool indic_initial_reordering(const Indic *d, VegaShaper *b) { + if (d->virama) + for (size_t i = 0; i < b->len; ++i) + if (b->info[i].position == VEGA_POS_BASE_C) b->info[i].position = consonant_position(d, b->info[i].codepoint); + bool changed = insert_dotted_circles(d->s, b, 4, VEGA_OT_DOTTEDCIRCLE, VEGA_OT_REPHA, VEGA_POS_END); + for (size_t start = 0, end; start < b->len; start = end) { + end = next_syllable(b, start); + /* Vowel syllables look like consonant ones, and broken ones have + * their dotted circle, like a standalone cluster's. */ + unsigned type = b->info[start].syllable & 0x0F; + if (type <= 2 || type == 4) indic_reorder_consonant_syllable(d, b, start, end); + } + return changed; +} + +/* Categories of the preceding character that keep a left matra from + * 'init': HarfBuzz's Cf to Mn. */ +static bool word_character(unsigned c) { + switch (c) { + case VEGA_CF: case VEGA_CN: case VEGA_CO: case VEGA_CS: case VEGA_LL: case VEGA_LM: case VEGA_LO: case VEGA_LT: + case VEGA_LU: case VEGA_MC: case VEGA_ME: case VEGA_MN: + return true; + default: + return false; + } +} + +static void indic_final_syllable(const Indic *d, VegaShaper *b, size_t start, size_t end) { + Info *info = b->info; + const uint32_t *masks = d->plan->masks; + uint32_t script = d->plan->script; + bool malayalam_or_tamil = script == TAG4('M', 'l', 'y', 'm') || script == TAG4('T', 'a', 'm', 'l'); + /* A virama that ligated and multiplied back out is a halant again. */ + if (d->virama) + for (size_t i = start; i < end; ++i) + if (info[i].codepoint == d->virama && (info[i].gprops & LIGATED) && (info[i].gprops & MULTIPLIED)) { + info[i].category = VEGA_OT_H; + info[i].gprops &= (uint16_t)~(LIGATED | MULTIPLIED); + } + + /* 4. The base again: where a pref candidate formed nothing, or (in + * Malayalam) past unformed below-base forms. */ + bool try_pref = masks[VEGA_MASK_PREF] != 0; + size_t base; + for (base = start; base < end; base++) + if (info[base].position >= VEGA_POS_BASE_C) { + if (try_pref && base + 1 < end) + for (size_t i = base + 1; i < end; ++i) + if (info[i].mask & masks[VEGA_MASK_PREF]) { + if (!((info[i].gprops & SUBSTITUTED) && ligated_and_didnt_multiply(&info[i]))) { + base = i; + while (base < end && is_halant(&info[base])) base++; + if (base < end) info[base].position = VEGA_POS_BASE_C; + try_pref = false; + } + break; + } + if (script == TAG4('M', 'l', 'y', 'm')) + for (size_t i = base + 1; i < end; ++i) { + while (i < end && is_joiner(&info[i])) i++; + if (i == end || !is_halant(&info[i])) break; + i++; + while (i < end && is_joiner(&info[i])) i++; + if (i < end && is_consonant(&info[i]) && info[i].position == VEGA_POS_BELOW_C) { + base = i; + info[base].position = VEGA_POS_BASE_C; + } + } + if (start < base && info[base].position > VEGA_POS_BASE_C) base--; + break; + } + if (base == end && start < base && is_one_of(&info[base - 1], FLAG(VEGA_OT_ZWJ))) base--; + if (base < end) + while (start < base && is_one_of(&info[base], FLAG(VEGA_OT_N) | FLAG(VEGA_OT_H))) base--; + + /* Pre-base matras: after the last halant before the base (not one a + * ZWJ follows, nor the matra's own), else where they are. */ + if (start + 1 < end && start < base) { + size_t new_pos = base == end ? base - 2 : base - 1; + if (!malayalam_or_tamil) { + for (;;) { + while (new_pos > start && + !is_one_of(&info[new_pos], FLAG(VEGA_OT_M) | FLAG(VEGA_OT_MPST) | FLAG(VEGA_OT_H))) + new_pos--; + if (is_halant(&info[new_pos]) && info[new_pos].position != VEGA_POS_PRE_M) { + if (new_pos + 1 < end && info[new_pos + 1].category == VEGA_OT_ZWJ && new_pos > start) { + new_pos--; + continue; /* keep searching */ + } + } else + new_pos = start; + break; + } + } + if (start < new_pos && info[new_pos].position != VEGA_POS_PRE_M) { + for (size_t i = new_pos; i > start; i--) + if (info[i - 1].position == VEGA_POS_PRE_M) { + size_t old_pos = i - 1; + if (old_pos < base && base <= new_pos) base--; + Info t = info[old_pos]; + memmove(&info[old_pos], &info[old_pos + 1], (new_pos - old_pos) * sizeof(Info)); + info[new_pos] = t; + merge_clusters(b, new_pos, end < base + 1 ? end : base + 1); + new_pos--; + } + } else + for (size_t i = start; i < base; ++i) + if (info[i].position == VEGA_POS_PRE_M) { + merge_clusters(b, i, end < base + 1 ? end : base + 1); + break; + } + } + + /* Reph: moved only if it formed (or, encoded as a repha, if it did + * not ligate), to where the script puts it. */ + if (start + 1 < end && info[start].position == VEGA_POS_RA_TO_BECOME_REPH && + ((info[start].category == VEGA_OT_REPHA) ^ ligated_and_didnt_multiply(&info[start]))) { + size_t new_reph_pos = start; + unsigned reph_pos = d->config->reph_pos; + bool found = false; + if (reph_pos != VEGA_POS_AFTER_POST) { + /* 2. after the first explicit halant before the base (and a + * joiner after it) */ + new_reph_pos = start + 1; + while (new_reph_pos < base && !is_halant(&info[new_reph_pos])) new_reph_pos++; + if (new_reph_pos < base && is_halant(&info[new_reph_pos])) { + if (new_reph_pos + 1 < base && is_joiner(&info[new_reph_pos + 1])) new_reph_pos++; + found = true; + } + /* 3. after the main consonant */ + if (!found && reph_pos == VEGA_POS_AFTER_MAIN) { + new_reph_pos = base; + while (new_reph_pos + 1 < end && info[new_reph_pos + 1].position <= VEGA_POS_AFTER_MAIN) new_reph_pos++; + found = new_reph_pos < end; + } + /* 4. before the first post-base consonant or matra */ + if (!found && reph_pos == VEGA_POS_AFTER_SUB) { + new_reph_pos = base; + while (new_reph_pos + 1 < end && + !(FLAG_UNSAFE(info[new_reph_pos + 1].position) & + (FLAG(VEGA_POS_POST_C) | FLAG(VEGA_POS_AFTER_POST) | FLAG(VEGA_POS_SMVD)))) + new_reph_pos++; + found = new_reph_pos < end; + } + } + if (!found) { + /* 5. as step 2 */ + new_reph_pos = start + 1; + while (new_reph_pos < base && !is_halant(&info[new_reph_pos])) new_reph_pos++; + if (new_reph_pos < base && is_halant(&info[new_reph_pos])) { + if (new_reph_pos + 1 < base && is_joiner(&info[new_reph_pos + 1])) new_reph_pos++; + found = true; + } + } + if (!found) { + /* 6. the end, before any syllable modifiers; before a halant + * that follows a matra */ + new_reph_pos = end - 1; + while (new_reph_pos > start && info[new_reph_pos].position == VEGA_POS_SMVD) new_reph_pos--; + if (is_halant(&info[new_reph_pos])) + for (size_t i = base + 1; i < new_reph_pos; ++i) + if (FLAG_UNSAFE(info[i].category) & (FLAG(VEGA_OT_M) | FLAG(VEGA_OT_MPST))) new_reph_pos--; + } + merge_clusters(b, start, new_reph_pos + 1); + Info reph = info[start]; + memmove(&info[start], &info[start + 1], (new_reph_pos - start) * sizeof(Info)); + info[new_reph_pos] = reph; + if (start < base && base <= new_reph_pos) base--; + } + + /* A pre-base-reordering consonant that pref formed: before the base, + * where a pre-base matra would go. */ + if (try_pref && base + 1 < end) + for (size_t i = base + 1; i < end; ++i) + if (info[i].mask & masks[VEGA_MASK_PREF]) { + if (ligated_and_didnt_multiply(&info[i])) { + size_t new_pos = base; + if (!malayalam_or_tamil) + while (new_pos > start && + !is_one_of(&info[new_pos - 1], FLAG(VEGA_OT_M) | FLAG(VEGA_OT_MPST) | FLAG(VEGA_OT_H))) + new_pos--; + if (new_pos > start && is_halant(&info[new_pos - 1]) && new_pos < end && is_joiner(&info[new_pos])) + new_pos++; + size_t old_pos = i; + merge_clusters(b, new_pos, old_pos + 1); + Info t = info[old_pos]; + memmove(&info[new_pos + 1], &info[new_pos], (old_pos - new_pos) * sizeof(Info)); + info[new_pos] = t; + if (new_pos <= base && base < old_pos) base++; + } + break; + } + + /* 'init' on a left matra that starts a word. */ + if (info[start].position == VEGA_POS_PRE_M && (!start || !word_character(gen_cat(&info[start - 1])))) + info[start].mask |= masks[VEGA_MASK_INDIC_INIT]; +} + +static void indic_final_reordering(const Indic *d, VegaShaper *b) { + for (size_t start = 0, end; start < b->len; start = end) { + end = next_syllable(b, start); + indic_final_syllable(d, b, start, end); + } +} + +/* The shaper's part of the work at a pause in GSUB; whether it changed the + * buffer. */ +static bool shaper_pause(const VegaShaping *s, const Plan *plan, VegaShaper *b, unsigned pause) { + if (plan->shaper != VEGA_SHAPER_INDIC) return false; + Indic d = indic_of(s, plan); + switch (pause) { + case VEGA_PAUSE_SYLLABLES: find_syllables(b, &vega_indic_machine); return false; + case VEGA_PAUSE_INITIAL_REORDERING: return indic_initial_reordering(&d, b); + case VEGA_PAUSE_FINAL_REORDERING: indic_final_reordering(&d, b); return false; + default: return false; + } +} + /* ---- Positions */ static void fallback_spaces(const VegaShaping *s, VegaShaper *b) { @@ -2449,7 +3263,7 @@ static void position(const VegaShaping *s, const Plan *plan, VegaShaper *b) { for (size_t i = 0; i < b->len; ++i) b->pos[i].x_advance = advance_of(s, b->info[i].codepoint); if (b->scratch & HAS_SPACE_FALLBACK) fallback_spaces(s, b); bool adjust = (plan->flags & VEGA_PLAN_ADJUST_ZEROING) && b->direction == VEGA_SHAPE_LTR; - if (plan->flags & VEGA_PLAN_GPOS) apply_table(s, b, 1, plan->gpos, plan->gpos_count); + if (plan->flags & VEGA_PLAN_GPOS) apply_table(s, plan, b, 1); if (plan->flags & VEGA_PLAN_KERN) kern_table(s, b, plan); if (plan->flags & VEGA_PLAN_ZERO_MARKS) for (size_t i = 0; i < b->len; ++i) @@ -2484,13 +3298,14 @@ long vega_shape(VegaShaper *b, const VegaShaping *s, const uint32_t *codes, size b->max_len = count * 64 > 16384 ? count * 64 : 16384; b->max_ops = (long)(count * 1024 > 16384 ? count * 1024 : 16384); if (!ensure(b, count + 1)) return -1; - for (size_t i = 0; i < count; ++i) b->info[i] = (Info){codes[i], plan.global_mask, (uint32_t)i, 0, 0, 0, 0, 0}; + for (size_t i = 0; i < count; ++i) b->info[i] = (Info){.codepoint = codes[i], .mask = plan.global_mask, .cluster = (uint32_t)i}; b->len = count; set_unicode_props(b); form_clusters(b); ensure_native_direction(b, &plan); if (plan.shaper == VEGA_SHAPER_HANGUL) preprocess_hangul(s, b); + if (plan.shaper == VEGA_SHAPER_INDIC) preprocess_vowel_constraints(&plan, b); /* Mirror what the font mirrors; mark the rest for rtlm. */ if (direction == VEGA_SHAPE_RTL) @@ -2502,6 +3317,7 @@ long vega_shape(VegaShaper *b, const VegaShaping *s, const uint32_t *codes, size normalize(s, &plan, b); setup_masks_fraction(b, &plan); if (plan.shaper == VEGA_SHAPER_ARABIC) setup_masks_arabic(b, &plan); + if (plan.shaper == VEGA_SHAPER_INDIC) setup_masks_indic(b); if (plan.shaper == VEGA_SHAPER_HANGUL) for (size_t i = 0; i < b->len; ++i) if (b->info[i].action >= LJMO && b->info[i].action <= TJMO) @@ -2517,7 +3333,7 @@ long vega_shape(VegaShaper *b, const VegaShaping *s, const uint32_t *codes, size if (plan.flags & VEGA_PLAN_GUESS_CLASSES) info->gprops = gen_cat(info) != VEGA_MN || is_default_ignorable(info) ? BASE_GLYPH : MARK; } - apply_table(s, b, 0, plan.gsub, plan.gsub_count); + apply_table(s, &plan, b, 0); if (!b->successful || !ensure(b, b->len)) return -1; position(s, &plan, b); diff --git a/text/shape.h b/text/shape.h index fdec7ee..7b12009 100644 --- a/text/shape.h +++ b/text/shape.h @@ -6,7 +6,8 @@ /* Shaping (docs/design/004-text.md): a run of characters (one script, one * direction) to glyphs and their positions, as HarfBuzz 10 shapes it with - * its default, Arabic and Hebrew shapers, glyph for glyph and unit for unit. + * its default, Arabic, Hebrew, Hangul and Indic shapers, glyph for glyph and + * unit for unit. * * What a font's shaping needs travels in a symbol file (its shaping * section), written by the build tool: @@ -31,15 +32,22 @@ * no plan positions marks without * GPOS; x_bearing INT16_MIN: none), * u32 0 - * Offsets are from the section's start. A plan: + * Offsets are from the section's start. A plan (VEGA_PLAN_BYTES): * u32 script (ISO 15924 as a tag: 'Latn'), u8 direction, u8 native * direction (of its script; 0 when it has none), u8 shaper, u8 flags, * u32 global mask, u32 kern mask, u32 masks[VEGA_PLAN_MASKS], u32 first - * GSUB step, u32 GSUB steps, u32 first GPOS step, u32 GPOS steps. - * A step, a lookup to run: u16 lookup index, u8 flags, u8 0, u32 mask. */ -enum { VEGA_SHAPING_HEADER = 80, VEGA_PLAN_BYTES = 88, VEGA_STEP_BYTES = 8, VEGA_PLAN_MASKS = 14 }; + * GSUB step, u32 GSUB steps, u32 first GPOS step, u32 GPOS steps, + * u8 shaper flags, u8 0[3], + * the stages the shaper asks whether a feature would substitute in + * (VEGA_PLAN_STAGES of: u16 first step, counted from the plan's first + * GSUB step, u16 steps, u32 0), u32 0. + * A step, a lookup to run: u16 lookup index, u8 flags, u8 0, u32 mask; or + * a pause, where the shaper does its part: u16 VEGA_STEP_PAUSE, u8 which + * (VEGA_PAUSE_...), u8 0, u32 0. */ +enum { VEGA_SHAPING_HEADER = 80, VEGA_PLAN_BYTES = 144, VEGA_STEP_BYTES = 8, VEGA_PLAN_MASKS = 16, + VEGA_PLAN_STAGES = 5 }; enum { VEGA_SHAPE_LTR = 1, VEGA_SHAPE_RTL = 2 }; -enum { VEGA_SHAPER_DEFAULT, VEGA_SHAPER_ARABIC, VEGA_SHAPER_HEBREW, VEGA_SHAPER_HANGUL }; +enum { VEGA_SHAPER_DEFAULT, VEGA_SHAPER_ARABIC, VEGA_SHAPER_HEBREW, VEGA_SHAPER_HANGUL, VEGA_SHAPER_INDIC }; /* Plan flags: GPOS runs; mark advances are zeroed after it (by GDEF); glyph * classes come from Unicode (the font has none); the font positions marks * (a 'mark' feature); automatic fractions; the kern table kerns; marks are @@ -47,12 +55,24 @@ enum { VEGA_SHAPER_DEFAULT, VEGA_SHAPER_ARABIC, VEGA_SHAPER_HEBREW, VEGA_SHAPER_ * it where it was. */ enum { VEGA_PLAN_GPOS = 1, VEGA_PLAN_ZERO_MARKS = 2, VEGA_PLAN_GUESS_CLASSES = 4, VEGA_PLAN_GPOS_MARK = 8, VEGA_PLAN_FRACTIONS = 16, VEGA_PLAN_KERN = 32, VEGA_PLAN_FALLBACK_MARKS = 64, VEGA_PLAN_ADJUST_ZEROING = 128 }; -/* The masks a plan names, for the features set glyph by glyph. */ -enum { VEGA_MASK_ISOL, VEGA_MASK_FINA, VEGA_MASK_FIN2, VEGA_MASK_FIN3, VEGA_MASK_MEDI, VEGA_MASK_MED2, - VEGA_MASK_INIT, VEGA_MASK_RTLM, VEGA_MASK_FRAC, VEGA_MASK_NUMR, VEGA_MASK_DNOM, VEGA_MASK_LJMO, VEGA_MASK_VJMO, - VEGA_MASK_TJMO }; -/* Step flags. */ +/* The masks a plan names, for the features set glyph by glyph: those of + * every shaper, then each shaper's own. */ +enum { VEGA_MASK_RTLM, VEGA_MASK_FRAC, VEGA_MASK_NUMR, VEGA_MASK_DNOM, VEGA_MASK_SHAPER }; +enum { VEGA_MASK_ISOL = VEGA_MASK_SHAPER, VEGA_MASK_FINA, VEGA_MASK_FIN2, VEGA_MASK_FIN3, VEGA_MASK_MEDI, + VEGA_MASK_MED2, VEGA_MASK_INIT }; /* Arabic */ +enum { VEGA_MASK_LJMO = VEGA_MASK_SHAPER, VEGA_MASK_VJMO, VEGA_MASK_TJMO }; /* Hangul */ +enum { VEGA_MASK_RPHF = VEGA_MASK_SHAPER, VEGA_MASK_PREF, VEGA_MASK_BLWF, VEGA_MASK_ABVF, VEGA_MASK_HALF, + VEGA_MASK_PSTF, VEGA_MASK_INDIC_INIT }; /* Indic */ +/* The stages a plan names: Indic's, whose features decide a consonant's + * position by whether they would substitute it. */ +enum { VEGA_STAGE_RPHF, VEGA_STAGE_PREF, VEGA_STAGE_BLWF, VEGA_STAGE_PSTF, VEGA_STAGE_VATU }; +/* Shaper flags: Indic's font follows the old specification (deva, not + * dev2). */ +enum { VEGA_INDIC_OLD_SPEC = 1 }; +/* Step flags; pauses, which the shaper tells apart. */ enum { VEGA_STEP_AUTO_ZWNJ = 1, VEGA_STEP_AUTO_ZWJ = 2, VEGA_STEP_RANDOM = 4, VEGA_STEP_PER_SYLLABLE = 8 }; +enum { VEGA_STEP_PAUSE = 0xFFFF }; +enum { VEGA_PAUSE_SYLLABLES = 1, VEGA_PAUSE_INITIAL_REORDERING, VEGA_PAUSE_FINAL_REORDERING, VEGA_PAUSE_CLEAR_SYLLABLES }; typedef struct { const unsigned char *p; uint32_t size; } VegaTable; /* Which glyphs may be in a set, as three 64-bit patterns of glyph bits diff --git a/text/syllabic-data.c b/text/syllabic-data.c new file mode 100644 index 0000000..3b3e6ff --- /dev/null +++ b/text/syllabic-data.c @@ -0,0 +1,463 @@ +/* Generated by text/syllabic.py from HarfBuzz 10.2.0's sources; do not edit. */ +#include "syllabic.h" + +/* Each character's Indic category and position (hb-ot-shaper-indic-table.cc, Unicode 16): + * category | position << 8. */ +static const uint16_t indic_cells[1769] = { + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x0E00, 0x0E00, 0x040A, 0x040A, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x040A, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0D39, 0x0D39, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x0D08, 0x0D08, 0x0D08, 0x0D08, + 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, + 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x040F, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0907, 0x0907, 0x0E03, 0x0D11, 0x0907, 0x0207, + 0x0907, 0x0907, 0x0907, 0x0907, 0x0907, 0x0907, 0x0907, 0x0907, 0x0907, 0x0907, 0x0907, 0x0907, + 0x0907, 0x0804, 0x0207, 0x0907, 0x0E00, 0x0D09, 0x0D09, 0x0D08, 0x0D08, 0x0907, 0x0907, 0x0907, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0402, 0x0402, 0x0907, 0x0907, + 0x0E00, 0x0E00, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, + 0x0E00, 0x0E00, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x040A, 0x0D08, 0x0D08, 0x0D08, 0x0E00, 0x0402, 0x0402, 0x0402, + 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0E00, 0x0E00, 0x0402, 0x0402, 0x0E00, 0x0E00, 0x0402, + 0x0402, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x040F, 0x0E00, 0x0401, 0x0E00, 0x0E00, 0x0E00, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0E00, 0x0E00, 0x0E03, 0x0D11, 0x0C07, 0x0207, 0x0C07, 0x0907, 0x0907, 0x0907, + 0x0907, 0x0E00, 0x0E00, 0x0207, 0x0207, 0x0E00, 0x0E00, 0x0C07, 0x0C07, 0x0804, 0x0401, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0C07, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0401, 0x0401, 0x0E00, 0x0401, 0x0402, 0x0402, 0x0907, 0x0907, 0x0E00, 0x0E00, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040F, 0x0401, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x0E00, 0x0D08, 0x0E00, + 0x0E00, 0x0D08, 0x0D08, 0x0D08, 0x0E00, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0402, 0x0402, 0x0E00, 0x0E00, 0x0402, 0x0402, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x040F, 0x0E00, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0401, 0x0E00, 0x0E00, + 0x0E03, 0x0E00, 0x0C07, 0x0207, 0x0C0D, 0x0C07, 0x0C07, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0C07, + 0x0C07, 0x0E00, 0x0E00, 0x0C07, 0x0C07, 0x0804, 0x0E00, 0x0E00, 0x0E00, 0x0807, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x0D08, 0x0D08, 0x0401, 0x0401, 0x0E00, 0x0410, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0D08, 0x0D08, 0x0D08, + 0x0E00, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0E00, 0x0402, + 0x0402, 0x0402, 0x0E00, 0x0402, 0x0402, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0E00, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x040F, 0x0E00, 0x0401, 0x0401, + 0x0E00, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0E00, 0x0E03, 0x0D11, 0x0C07, 0x0207, + 0x0C07, 0x0C07, 0x0C07, 0x0C07, 0x0C07, 0x0907, 0x0E00, 0x0907, 0x0907, 0x0C07, 0x0E00, 0x0C07, + 0x0C07, 0x0804, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0402, 0x0402, 0x0C07, 0x0C07, + 0x0E00, 0x0E00, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0401, 0x0D09, 0x0E03, + 0x0D09, 0x0E03, 0x0E03, 0x0E03, 0x0E00, 0x0708, 0x0D08, 0x0D08, 0x0E00, 0x0402, 0x0402, 0x0402, + 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0E00, 0x0E00, 0x0402, 0x0402, 0x0E00, 0x0E00, 0x0402, + 0x0402, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x040F, 0x0E00, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0E00, 0x0E00, 0x0E03, 0x0D11, 0x0C07, 0x0507, 0x0C07, 0x0907, 0x0907, 0x0907, + 0x0907, 0x0E00, 0x0E00, 0x0207, 0x0507, 0x0E00, 0x0E00, 0x0C07, 0x0C07, 0x0804, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E03, 0x0507, 0x0C07, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0401, 0x0401, 0x0E00, 0x0401, 0x0402, 0x0402, 0x0907, 0x0907, 0x0E00, 0x0E00, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x0E00, 0x0401, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0D08, 0x0E00, 0x0E00, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0E00, + 0x0E00, 0x0E00, 0x0402, 0x0402, 0x0402, 0x0E00, 0x0402, 0x0402, 0x0402, 0x0401, 0x0E00, 0x0E00, + 0x0E00, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0E00, 0x0401, 0x0401, 0x0E00, 0x0E00, 0x0E00, 0x0401, + 0x0401, 0x0E00, 0x0E00, 0x0E00, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0E00, 0x0E00, 0x0401, 0x0401, + 0x040F, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0C07, 0x0C07, 0x0907, 0x0C07, 0x0C07, 0x0E00, 0x0E00, 0x0E00, 0x0207, 0x0207, + 0x0207, 0x0E00, 0x0C07, 0x0C07, 0x0C07, 0x0604, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0C07, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0D08, 0x0D08, 0x0D08, 0x0D08, + 0x0D08, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0E00, 0x0402, 0x0402, + 0x0402, 0x0E00, 0x0402, 0x0402, 0x0402, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0E00, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x040F, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0E00, 0x0E03, 0x0D11, 0x0707, 0x0707, + 0x0707, 0x0707, 0x0707, 0x0907, 0x0907, 0x0E00, 0x0707, 0x0707, 0x0707, 0x0E00, 0x0707, 0x0707, + 0x0707, 0x0604, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0707, 0x0707, 0x0E00, + 0x0401, 0x0401, 0x0401, 0x0E00, 0x0E00, 0x0401, 0x0E00, 0x0E00, 0x0402, 0x0402, 0x0707, 0x0707, + 0x0E00, 0x0E00, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x0D08, 0x0D08, 0x0D08, 0x0E00, 0x0402, 0x0402, 0x0402, + 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0E00, 0x0402, 0x0402, 0x0402, 0x0E00, 0x0402, 0x0402, + 0x0402, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x040F, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0E00, 0x0E00, 0x0E03, 0x0D11, 0x0707, 0x0707, 0x0707, 0x0707, 0x0707, 0x0907, + 0x0907, 0x0E00, 0x0707, 0x0907, 0x0907, 0x0E00, 0x0907, 0x0907, 0x0707, 0x0604, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0907, 0x0907, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0401, 0x0401, 0x0E00, 0x0402, 0x0402, 0x0707, 0x0707, 0x0E00, 0x0E00, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x0E00, 0x0412, 0x0412, 0x0D08, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0D08, 0x0D08, 0x0D08, 0x0D08, 0x040A, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, + 0x0402, 0x0E00, 0x0402, 0x0402, 0x0402, 0x0E00, 0x0402, 0x0402, 0x0402, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x040F, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0907, + 0x0907, 0x0D11, 0x0C07, 0x0C07, 0x0C07, 0x0C07, 0x0C07, 0x0C07, 0x0C07, 0x0E00, 0x0207, 0x0207, + 0x0207, 0x0E00, 0x0C07, 0x0C07, 0x0C07, 0x0604, 0x0E0E, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0401, 0x0401, 0x0401, 0x0C07, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0402, + 0x0402, 0x0402, 0x0C07, 0x0C07, 0x0E00, 0x0E00, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x040F, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x040F, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, + 0x0402, 0x0402, 0x0402, 0x0B17, 0x0B17, 0x0614, 0x0614, 0x0815, 0x0815, 0x0316, 0x0D09, 0x0614, + 0x0614, 0x0614, 0x0D09, 0x0E03, 0x0D08, 0x0E04, 0x0E20, 0x0E26, 0x0E24, 0x0E25, 0x0E23, 0x0401, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, + 0x0E00, 0x0E00, 0x0401, 0x0E00, 0x0401, 0x0401, 0x0402, 0x0402, 0x0402, 0x0402, 0x0B17, 0x0B17, + 0x0815, 0x0815, 0x040F, 0x0401, 0x0401, 0x0401, 0x0E26, 0x0E26, 0x0E29, 0x0401, 0x0B17, 0x0E27, + 0x0E27, 0x0401, 0x0401, 0x0B17, 0x0B17, 0x0E27, 0x0E27, 0x0E27, 0x0E27, 0x0E27, 0x0401, 0x0401, + 0x0401, 0x0614, 0x0614, 0x0614, 0x0614, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E25, 0x0B17, 0x0316, 0x0614, 0x0614, 0x0D08, + 0x0D08, 0x0D08, 0x0D08, 0x0D08, 0x0D08, 0x0D08, 0x0401, 0x0D08, 0x040A, 0x040A, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x0D08, 0x0D08, 0x0D08, 0x0614, 0x0E00, 0x0E00, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x040F, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0402, + 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, 0x0402, + 0x0402, 0x0402, 0x0402, 0x0402, 0x0E00, 0x0E00, 0x0B17, 0x0614, 0x0614, 0x0614, 0x0614, 0x0815, + 0x0815, 0x0815, 0x0614, 0x0B17, 0x0B17, 0x0316, 0x0316, 0x0316, 0x0B17, 0x0B17, 0x0E1A, 0x0E1B, + 0x0E1B, 0x0E19, 0x0E19, 0x0E1A, 0x0E19, 0x0E1A, 0x0E1A, 0x0E1A, 0x0E1A, 0x0E1A, 0x0E04, 0x0E1B, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x0E00, 0x0E00, 0x0D11, 0x0E1B, 0x0E00, 0x0E00, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0D09, 0x0D09, 0x0D09, 0x0E00, 0x0D09, 0x0D09, 0x0D09, 0x0D09, + 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, + 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D11, 0x0D11, 0x0D11, 0x0D11, 0x0D09, 0x0D11, 0x0D11, + 0x0D11, 0x0D11, 0x0401, 0x0401, 0x0D09, 0x0401, 0x0401, 0x0D09, 0x0D09, 0x0D09, 0x040A, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E05, 0x0E06, 0x0E00, 0x0E00, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0D39, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0D39, 0x0D39, 0x0D39, 0x0E00, 0x0E00, 0x0E00, + 0x040B, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x040A, 0x040A, 0x040A, 0x0E00, 0x0D09, 0x0D09, 0x0D09, + 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, 0x0D09, + 0x0D09, 0x0D09, 0x0D09, 0x0D11, 0x0D11, 0x0D11, 0x0D11, 0x0D11, 0x0D11, 0x0E00, 0x0E00, 0x0E00, + 0x0E00, 0x0E00, 0x0E00, 0x0402, 0x0907, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0614, 0x0E00, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x040A, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0E00, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, + 0x0401, 0x0401, 0x0401, 0x0401, 0x0401, 0x0E00, 0x0401, 0x0401, 0x0401, 0x040A, 0x040A, 0x040A, + 0x0E00, 0x0E00, 0x0E00, 0x0401, 0x0E27, 0x0E03, 0x0E03, 0x0401, 0x0401, 0x0E28, 0x0E28, 0x0E28, + 0x0E28, 0x0E28, 0x0E28, 0x0E28, 0x0E28, 0x0E28, 0x0E28, 0x0E28, 0x0E28, 0x0E28, 0x0E28, 0x0E28, + 0x0E28, 0x0E00, 0x0D08, 0x0D08, 0x0D08, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, 0x0E00, + 0x0E03, 0x0E03, 0x0E00, 0x0E00, 0x0E00, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, + 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, 0x040A, + 0x040A, 0x0E00, 0x0E00, 0x0E00, 0x0E00, +}; +static const struct { uint32_t first, last, at; } indic_runs[17] = { + {0x0028, 0x003F, 0}, + {0x00A0, 0x00D7, 24}, + {0x0900, 0x0D7F, 80}, + {0x1000, 0x109F, 1232}, + {0x1780, 0x17EF, 1392}, + {0x1CD0, 0x1CFF, 1504}, + {0x2008, 0x2027, 1552}, + {0x2070, 0x2087, 1584}, + {0x25CC, 0x25CC, 1608}, + {0x25F8, 0x25FF, 1609}, + {0xA8E0, 0xA8FF, 1617}, + {0xA9E0, 0xA9FF, 1649}, + {0xAA60, 0xAA7F, 1681}, + {0xFE00, 0xFE0F, 1713}, + {0x11300, 0x11307, 1729}, + {0x11338, 0x1133F, 1737}, + {0x116D0, 0x116E7, 1745}, +}; + +uint16_t vega_indic_categories(uint32_t u) { + for (unsigned i = 0; i < 17 && u >= indic_runs[i].first; ++i) + if (u <= indic_runs[i].last) return indic_cells[indic_runs[i].at + u - indic_runs[i].first]; + return 0x0E00; +} + +/* The syllable machines: a DFA over classes of symbol; a state's accept is + * the syllable type it ends, plus one. */ +/* indic (hb-ot-shaper-indic-machine.rl): 137 states, 20 classes of symbol; types indic_consonant_syllable, indic_vowel_syllable, indic_standalone_cluster, indic_symbol_cluster, indic_broken_cluster, indic_non_indic_cluster. */ +static const uint8_t indic_class[64] = { + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0, 0, 0, +}; +static const uint8_t indic_next[137][20] = { + {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 11, 12, 8, 13, 14, 15, 16, 17, 18}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 19, 20, 21, 22, 23, 24, 25, 137, 137, 26, 23, 137, 137, 27, 137, 137, 24}, + {137, 137, 137, 28, 29, 30, 31, 32, 33, 34, 137, 137, 35, 32, 137, 137, 36, 137, 137, 33}, + {137, 137, 137, 37, 5, 38, 38, 8, 9, 10, 137, 137, 137, 8, 137, 137, 15, 137, 137, 9}, + {137, 39, 137, 137, 137, 40, 41, 137, 42, 10, 137, 137, 137, 137, 137, 39, 137, 137, 137, 42}, + {137, 137, 137, 137, 43, 44, 44, 8, 9, 137, 137, 137, 12, 8, 137, 137, 137, 137, 137, 9}, + {137, 137, 137, 137, 43, 44, 44, 8, 9, 137, 137, 137, 137, 8, 137, 137, 137, 137, 137, 9}, + {137, 137, 137, 45, 46, 47, 47, 8, 9, 10, 137, 137, 137, 8, 137, 137, 137, 137, 137, 9}, + {137, 137, 137, 137, 137, 10, 137, 137, 48, 10, 137, 137, 137, 8, 137, 137, 137, 137, 137, 48}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 10, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 49, 50, 51, 52, 53, 54, 55, 137, 137, 56, 53, 137, 137, 57, 137, 137, 54}, + {137, 137, 137, 4, 5, 38, 38, 8, 9, 10, 137, 137, 137, 8, 137, 137, 15, 137, 137, 9}, + {137, 2, 3, 4, 5, 58, 38, 8, 9, 10, 11, 11, 12, 8, 137, 2, 15, 137, 137, 9}, + {137, 137, 137, 19, 59, 21, 22, 23, 24, 25, 137, 137, 26, 23, 137, 137, 27, 137, 137, 24}, + {137, 137, 137, 137, 60, 61, 61, 8, 9, 10, 137, 137, 137, 8, 137, 137, 137, 137, 137, 9}, + {137, 137, 137, 62, 137, 63, 63, 137, 64, 65, 137, 137, 137, 137, 137, 137, 137, 137, 137, 64}, + {137, 2, 137, 137, 137, 137, 137, 137, 137, 137, 11, 137, 137, 137, 137, 2, 137, 137, 137, 137}, + {137, 137, 137, 137, 137, 10, 137, 137, 48, 10, 137, 137, 137, 8, 137, 137, 137, 137, 137, 48}, + {137, 137, 137, 66, 20, 67, 67, 23, 24, 25, 137, 137, 137, 23, 137, 137, 27, 137, 137, 24}, + {137, 2, 137, 137, 137, 68, 69, 137, 70, 25, 137, 137, 137, 137, 137, 2, 137, 137, 137, 70}, + {137, 137, 137, 137, 71, 72, 72, 23, 24, 137, 137, 137, 26, 23, 137, 137, 137, 137, 137, 24}, + {137, 137, 137, 19, 20, 21, 67, 23, 24, 25, 137, 137, 26, 23, 137, 137, 27, 137, 137, 24}, + {137, 137, 137, 73, 74, 75, 75, 23, 24, 25, 137, 137, 137, 23, 137, 137, 137, 137, 137, 24}, + {137, 137, 137, 137, 137, 25, 137, 137, 76, 25, 137, 137, 137, 23, 137, 137, 137, 137, 137, 76}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 25, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 19, 20, 67, 67, 23, 24, 25, 137, 137, 137, 23, 137, 137, 27, 137, 137, 24}, + {137, 137, 137, 137, 77, 78, 78, 23, 24, 25, 137, 137, 137, 23, 137, 137, 137, 137, 137, 24}, + {137, 137, 137, 79, 29, 80, 31, 32, 33, 34, 137, 137, 137, 32, 137, 137, 36, 137, 137, 33}, + {137, 81, 137, 137, 137, 82, 83, 137, 84, 34, 137, 137, 137, 137, 137, 81, 137, 137, 137, 84}, + {137, 137, 137, 137, 85, 86, 86, 32, 33, 137, 137, 137, 35, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 137, 85, 86, 86, 32, 33, 137, 137, 137, 137, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 87, 88, 89, 89, 32, 33, 34, 137, 137, 137, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 137, 137, 34, 137, 137, 90, 34, 137, 137, 137, 32, 137, 137, 137, 137, 137, 90}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 34, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 28, 29, 80, 31, 32, 33, 34, 137, 137, 137, 32, 137, 137, 36, 137, 137, 33}, + {137, 137, 137, 137, 91, 92, 92, 32, 33, 34, 137, 137, 137, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 137, 5, 38, 38, 8, 9, 10, 137, 137, 137, 8, 137, 137, 15, 137, 137, 9}, + {137, 137, 137, 137, 43, 44, 44, 8, 9, 137, 137, 137, 137, 8, 137, 137, 137, 137, 137, 9}, + {137, 137, 137, 4, 5, 58, 93, 8, 9, 10, 137, 137, 12, 8, 137, 137, 15, 137, 137, 9}, + {137, 137, 137, 137, 137, 94, 94, 137, 42, 10, 137, 137, 137, 137, 137, 137, 137, 137, 137, 42}, + {137, 39, 137, 95, 137, 94, 94, 137, 42, 10, 137, 137, 137, 137, 137, 39, 137, 137, 137, 42}, + {137, 137, 137, 137, 137, 10, 137, 137, 48, 10, 137, 137, 137, 137, 137, 137, 137, 137, 137, 48}, + {137, 39, 137, 137, 137, 94, 41, 137, 42, 10, 137, 137, 137, 137, 137, 39, 137, 137, 137, 42}, + {137, 137, 137, 137, 137, 44, 44, 8, 96, 137, 137, 137, 137, 8, 137, 137, 137, 137, 137, 96}, + {137, 137, 137, 137, 46, 47, 47, 8, 9, 10, 137, 137, 137, 8, 137, 137, 137, 137, 137, 9}, + {137, 137, 137, 137, 137, 47, 47, 8, 9, 10, 137, 137, 137, 8, 137, 137, 137, 137, 137, 9}, + {137, 137, 137, 137, 137, 44, 44, 8, 9, 137, 137, 137, 137, 8, 137, 137, 137, 137, 137, 9}, + {137, 137, 137, 137, 137, 10, 137, 137, 137, 10, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 97, 50, 52, 52, 53, 54, 55, 137, 137, 137, 53, 137, 137, 57, 137, 137, 54}, + {137, 98, 137, 137, 137, 99, 100, 137, 101, 55, 137, 137, 137, 137, 137, 98, 137, 137, 137, 101}, + {137, 137, 137, 137, 102, 103, 103, 53, 54, 137, 137, 137, 56, 53, 137, 137, 137, 137, 137, 54}, + {137, 137, 137, 137, 102, 103, 103, 53, 54, 137, 137, 137, 137, 53, 137, 137, 137, 137, 137, 54}, + {137, 137, 137, 104, 105, 106, 106, 53, 54, 55, 137, 137, 137, 53, 137, 137, 137, 137, 137, 54}, + {137, 137, 137, 137, 137, 55, 137, 137, 107, 55, 137, 137, 137, 53, 137, 137, 137, 137, 137, 107}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 55, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 49, 50, 52, 52, 53, 54, 55, 137, 137, 137, 53, 137, 137, 57, 137, 137, 54}, + {137, 137, 137, 137, 108, 109, 109, 53, 54, 55, 137, 137, 137, 53, 137, 137, 137, 137, 137, 54}, + {137, 137, 137, 137, 43, 44, 44, 8, 9, 137, 137, 137, 12, 8, 137, 137, 137, 137, 137, 9}, + {137, 2, 3, 4, 5, 110, 111, 8, 112, 25, 137, 11, 12, 8, 137, 2, 15, 137, 137, 112}, + {137, 137, 137, 137, 137, 40, 113, 137, 42, 10, 137, 137, 137, 137, 137, 137, 137, 137, 137, 42}, + {137, 137, 137, 137, 114, 44, 44, 8, 9, 137, 137, 137, 137, 8, 137, 137, 137, 137, 137, 9}, + {137, 137, 137, 137, 137, 63, 63, 137, 64, 65, 137, 137, 137, 137, 137, 137, 137, 137, 137, 64}, + {137, 137, 137, 137, 137, 137, 137, 137, 64, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 64}, + {137, 137, 137, 137, 137, 65, 137, 137, 115, 65, 137, 137, 137, 137, 137, 137, 137, 137, 137, 115}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 65, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 137, 20, 67, 67, 23, 24, 25, 137, 137, 137, 23, 137, 137, 27, 137, 137, 24}, + {137, 137, 137, 137, 71, 72, 72, 23, 24, 137, 137, 137, 137, 23, 137, 137, 137, 137, 137, 24}, + {137, 137, 137, 137, 137, 116, 116, 137, 70, 25, 137, 137, 137, 137, 137, 137, 137, 137, 137, 70}, + {137, 2, 137, 117, 137, 116, 116, 137, 70, 25, 137, 137, 137, 137, 137, 2, 137, 137, 137, 70}, + {137, 137, 137, 137, 137, 25, 137, 137, 76, 25, 137, 137, 137, 137, 137, 137, 137, 137, 137, 76}, + {137, 2, 137, 137, 137, 116, 69, 137, 70, 25, 137, 137, 137, 137, 137, 2, 137, 137, 137, 70}, + {137, 137, 137, 137, 137, 72, 72, 23, 118, 137, 137, 137, 137, 23, 137, 137, 137, 137, 137, 118}, + {137, 137, 137, 137, 74, 75, 75, 23, 24, 25, 137, 137, 137, 23, 137, 137, 137, 137, 137, 24}, + {137, 137, 137, 137, 137, 75, 75, 23, 24, 25, 137, 137, 137, 23, 137, 137, 137, 137, 137, 24}, + {137, 137, 137, 137, 137, 72, 72, 23, 24, 137, 137, 137, 137, 23, 137, 137, 137, 137, 137, 24}, + {137, 137, 137, 137, 137, 25, 137, 137, 137, 25, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 137, 137, 68, 119, 137, 70, 25, 137, 137, 137, 137, 137, 137, 137, 137, 137, 70}, + {137, 137, 137, 137, 120, 72, 72, 23, 24, 137, 137, 137, 137, 23, 137, 137, 137, 137, 137, 24}, + {137, 137, 137, 137, 29, 80, 31, 32, 33, 34, 137, 137, 137, 32, 137, 137, 36, 137, 137, 33}, + {137, 137, 137, 137, 85, 86, 86, 32, 33, 137, 137, 137, 137, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 121, 29, 122, 123, 32, 33, 34, 137, 137, 124, 32, 137, 137, 36, 137, 137, 33}, + {137, 137, 137, 137, 137, 125, 125, 137, 84, 34, 137, 137, 137, 137, 137, 137, 137, 137, 137, 84}, + {137, 81, 137, 126, 137, 125, 125, 137, 84, 34, 137, 137, 137, 137, 137, 81, 137, 137, 137, 84}, + {137, 137, 137, 137, 137, 34, 137, 137, 90, 34, 137, 137, 137, 137, 137, 137, 137, 137, 137, 90}, + {137, 81, 137, 137, 137, 125, 83, 137, 84, 34, 137, 137, 137, 137, 137, 81, 137, 137, 137, 84}, + {137, 137, 137, 137, 137, 86, 86, 32, 127, 137, 137, 137, 137, 32, 137, 137, 137, 137, 137, 127}, + {137, 137, 137, 137, 88, 89, 89, 32, 33, 34, 137, 137, 137, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 137, 137, 89, 89, 32, 33, 34, 137, 137, 137, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 137, 137, 86, 86, 32, 33, 137, 137, 137, 137, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 137, 137, 34, 137, 137, 137, 34, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 137, 137, 82, 128, 137, 84, 34, 137, 137, 137, 137, 137, 137, 137, 137, 137, 84}, + {137, 137, 137, 137, 129, 86, 86, 32, 33, 137, 137, 137, 137, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 4, 5, 58, 38, 8, 9, 10, 137, 137, 12, 8, 137, 137, 15, 137, 137, 9}, + {137, 137, 137, 137, 137, 137, 137, 137, 42, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 42}, + {137, 39, 137, 137, 137, 94, 94, 137, 42, 10, 137, 137, 137, 137, 137, 39, 137, 137, 137, 42}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 8, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 137, 50, 52, 52, 53, 54, 55, 137, 137, 137, 53, 137, 137, 57, 137, 137, 54}, + {137, 137, 137, 49, 50, 51, 11, 53, 54, 55, 137, 137, 56, 53, 137, 137, 57, 137, 137, 54}, + {137, 137, 137, 137, 137, 130, 130, 137, 101, 55, 137, 137, 137, 137, 137, 137, 137, 137, 137, 101}, + {137, 98, 137, 131, 137, 130, 130, 137, 101, 55, 137, 137, 137, 137, 137, 98, 137, 137, 137, 101}, + {137, 137, 137, 137, 137, 55, 137, 137, 107, 55, 137, 137, 137, 137, 137, 137, 137, 137, 137, 107}, + {137, 98, 137, 137, 137, 130, 100, 137, 101, 55, 137, 137, 137, 137, 137, 98, 137, 137, 137, 101}, + {137, 137, 137, 137, 137, 103, 103, 53, 132, 137, 137, 137, 137, 53, 137, 137, 137, 137, 137, 132}, + {137, 137, 137, 137, 105, 106, 106, 53, 54, 55, 137, 137, 137, 53, 137, 137, 137, 137, 137, 54}, + {137, 137, 137, 137, 137, 106, 106, 53, 54, 55, 137, 137, 137, 53, 137, 137, 137, 137, 137, 54}, + {137, 137, 137, 137, 137, 103, 103, 53, 54, 137, 137, 137, 137, 53, 137, 137, 137, 137, 137, 54}, + {137, 137, 137, 137, 137, 55, 137, 137, 137, 55, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 137, 137, 99, 133, 137, 101, 55, 137, 137, 137, 137, 137, 137, 137, 137, 137, 101}, + {137, 137, 137, 137, 134, 103, 103, 53, 54, 137, 137, 137, 137, 53, 137, 137, 137, 137, 137, 54}, + {137, 137, 137, 137, 43, 135, 135, 8, 112, 25, 137, 137, 12, 8, 137, 137, 137, 137, 137, 112}, + {137, 2, 137, 117, 43, 135, 135, 8, 112, 25, 137, 137, 137, 8, 137, 2, 137, 137, 137, 112}, + {137, 137, 137, 137, 137, 25, 137, 137, 76, 25, 137, 137, 137, 8, 137, 137, 137, 137, 137, 76}, + {137, 137, 137, 40, 137, 94, 94, 137, 42, 10, 137, 137, 137, 137, 137, 137, 137, 137, 137, 42}, + {137, 137, 137, 137, 137, 94, 113, 137, 42, 10, 137, 137, 137, 137, 137, 137, 137, 137, 137, 42}, + {137, 137, 137, 137, 137, 65, 137, 137, 137, 65, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 137, 137, 137, 137, 137, 70, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 70}, + {137, 2, 137, 137, 137, 116, 116, 137, 70, 25, 137, 137, 137, 137, 137, 2, 137, 137, 137, 70}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 23, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 68, 137, 116, 116, 137, 70, 25, 137, 137, 137, 137, 137, 137, 137, 137, 137, 70}, + {137, 137, 137, 137, 137, 116, 119, 137, 70, 25, 137, 137, 137, 137, 137, 137, 137, 137, 137, 70}, + {137, 137, 137, 136, 29, 80, 80, 32, 33, 34, 137, 137, 137, 32, 137, 137, 36, 137, 137, 33}, + {137, 137, 137, 137, 85, 86, 86, 32, 33, 137, 137, 137, 124, 32, 137, 137, 137, 137, 137, 33}, + {137, 137, 137, 121, 29, 122, 80, 32, 33, 34, 137, 137, 124, 32, 137, 137, 36, 137, 137, 33}, + {137, 137, 137, 121, 29, 80, 80, 32, 33, 34, 137, 137, 137, 32, 137, 137, 36, 137, 137, 33}, + {137, 137, 137, 137, 137, 137, 137, 137, 84, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 84}, + {137, 81, 137, 137, 137, 125, 125, 137, 84, 34, 137, 137, 137, 137, 137, 81, 137, 137, 137, 84}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 32, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 82, 137, 125, 125, 137, 84, 34, 137, 137, 137, 137, 137, 137, 137, 137, 137, 84}, + {137, 137, 137, 137, 137, 125, 128, 137, 84, 34, 137, 137, 137, 137, 137, 137, 137, 137, 137, 84}, + {137, 137, 137, 137, 137, 137, 137, 137, 101, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 101}, + {137, 98, 137, 137, 137, 130, 130, 137, 101, 55, 137, 137, 137, 137, 137, 98, 137, 137, 137, 101}, + {137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 137, 53, 137, 137, 137, 137, 137, 137}, + {137, 137, 137, 99, 137, 130, 130, 137, 101, 55, 137, 137, 137, 137, 137, 137, 137, 137, 137, 101}, + {137, 137, 137, 137, 137, 130, 133, 137, 101, 55, 137, 137, 137, 137, 137, 137, 137, 137, 137, 101}, + {137, 137, 137, 137, 137, 44, 44, 8, 112, 137, 137, 137, 137, 8, 137, 137, 137, 137, 137, 112}, + {137, 137, 137, 137, 29, 80, 80, 32, 33, 34, 137, 137, 137, 32, 137, 137, 36, 137, 137, 33}, +}; +static const uint8_t indic_accept[137] = { + 5, 6, 1, 2, 5, 5, 6, 6, 5, 5, 5, 3, 5, 5, 1, 5, 4, 6, 6, 1, 1, 0, 1, 1, 1, 1, 1, 1, 2, 2, 0, 2, + 2, 2, 2, 2, 2, 5, 0, 5, 5, 5, 5, 5, 0, 5, 5, 0, 5, 3, 3, 0, 0, 3, 3, 3, 3, 3, 0, 1, 5, 0, 4, 0, + 4, 4, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 2, 0, 2, 2, 2, 2, 2, 0, 2, 2, 0, 2, 2, 0, 5, 0, 5, + 0, 3, 3, 3, 3, 3, 3, 0, 3, 3, 0, 3, 3, 0, 1, 1, 1, 5, 5, 4, 0, 1, 0, 1, 1, 2, 0, 2, 2, 0, 2, 0, + 2, 2, 0, 3, 0, 3, 3, 0, 2, +}; +const VegaMachine vega_indic_machine = {indic_class, 20, 137, indic_next[0], sizeof(indic_next[0][0]), indic_accept, 5}; + +/* khmer (hb-ot-shaper-khmer-machine.rl): 44 states, 12 classes of symbol; types khmer_consonant_syllable, khmer_broken_cluster, khmer_non_khmer_cluster. */ +static const uint8_t khmer_class[64] = { + 0, 1, 1, 0, 2, 3, 3, 0, 0, 0, 4, 4, 0, 0, 0, 1, 0, 0, 0, 0, 5, 6, 7, 8, 0, 9, 10, 11, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +}; +static const uint8_t khmer_next[44][12] = { + {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}, + {44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44}, + {44, 44, 13, 14, 44, 15, 16, 17, 18, 5, 19, 20}, + {44, 21, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44}, + {44, 44, 44, 22, 44, 6, 44, 44, 44, 44, 11, 44}, + {44, 44, 13, 23, 44, 15, 16, 17, 18, 24, 19, 20}, + {44, 44, 25, 26, 44, 44, 44, 44, 9, 44, 6, 12}, + {44, 44, 25, 27, 44, 6, 44, 44, 9, 44, 7, 12}, + {44, 44, 25, 28, 44, 6, 7, 44, 9, 44, 8, 12}, + {44, 44, 25, 29, 44, 44, 44, 44, 44, 44, 9, 12}, + {44, 44, 3, 30, 44, 6, 7, 8, 9, 44, 11, 12}, + {44, 44, 25, 30, 44, 6, 7, 8, 9, 44, 11, 12}, + {44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 12}, + {44, 31, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44}, + {44, 44, 44, 32, 44, 15, 44, 44, 44, 5, 19, 44}, + {44, 44, 33, 34, 44, 44, 44, 44, 18, 44, 15, 20}, + {44, 44, 33, 35, 44, 15, 44, 44, 18, 44, 16, 20}, + {44, 44, 33, 36, 44, 15, 16, 44, 18, 44, 17, 20}, + {44, 44, 33, 37, 44, 44, 44, 44, 44, 44, 18, 20}, + {44, 44, 33, 23, 44, 15, 16, 17, 18, 44, 19, 20}, + {44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 20}, + {44, 44, 3, 38, 44, 6, 7, 8, 9, 10, 11, 12}, + {44, 44, 44, 22, 44, 44, 44, 44, 44, 44, 11, 44}, + {44, 44, 44, 32, 44, 15, 44, 44, 44, 44, 19, 44}, + {44, 44, 13, 23, 44, 15, 16, 17, 18, 44, 19, 20}, + {44, 12, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44}, + {44, 44, 44, 26, 44, 44, 44, 44, 44, 44, 6, 44}, + {44, 44, 44, 39, 44, 6, 44, 44, 44, 44, 7, 44}, + {44, 44, 44, 40, 44, 6, 44, 44, 44, 44, 8, 44}, + {44, 44, 44, 29, 44, 44, 44, 44, 44, 44, 9, 44}, + {44, 44, 44, 22, 44, 6, 44, 44, 44, 44, 11, 44}, + {44, 44, 13, 41, 44, 15, 16, 17, 18, 24, 19, 20}, + {44, 44, 44, 32, 44, 44, 44, 44, 44, 44, 19, 44}, + {44, 20, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44}, + {44, 44, 44, 34, 44, 44, 44, 44, 44, 44, 15, 44}, + {44, 44, 44, 42, 44, 15, 44, 44, 44, 44, 16, 44}, + {44, 44, 44, 43, 44, 15, 44, 44, 44, 44, 17, 44}, + {44, 44, 44, 37, 44, 44, 44, 44, 44, 44, 18, 44}, + {44, 44, 44, 22, 44, 6, 44, 44, 44, 10, 11, 44}, + {44, 44, 44, 39, 44, 44, 44, 44, 44, 44, 7, 44}, + {44, 44, 44, 40, 44, 44, 44, 44, 44, 44, 8, 44}, + {44, 44, 44, 32, 44, 15, 44, 44, 44, 24, 19, 44}, + {44, 44, 44, 42, 44, 44, 44, 44, 44, 44, 16, 44}, + {44, 44, 44, 43, 44, 44, 44, 44, 44, 44, 17, 44}, +}; +static const uint8_t khmer_accept[44] = { + 2, 3, 1, 2, 3, 1, 2, 2, 2, 2, 2, 2, 2, 1, 0, 1, 1, 1, 1, 1, 1, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +}; +const VegaMachine vega_khmer_machine = {khmer_class, 12, 44, khmer_next[0], sizeof(khmer_next[0][0]), khmer_accept, 2}; + +/* myanmar (hb-ot-shaper-myanmar-machine.rl): 57 states, 23 classes of symbol; types myanmar_consonant_syllable, myanmar_broken_cluster, myanmar_non_myanmar_cluster. */ +static const uint8_t myanmar_class[64] = { + 0, 1, 1, 2, 3, 4, 4, 0, 5, 6, 7, 7, 0, 0, 0, 8, 0, 0, 9, 0, 10, 11, 12, 13, 0, 0, 0, 0, 0, 0, 0, 0, + 14, 0, 0, 15, 16, 17, 18, 19, 20, 21, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0, 0, 0, +}; +static const uint8_t myanmar_next[57][23] = { + {1, 2, 3, 4, 1, 5, 6, 2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21}, + {57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57}, + {57, 57, 22, 23, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 25}, + {57, 57, 57, 57, 39, 5, 57, 57, 57, 57, 57, 57, 57, 12, 40, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 41, 57, 57, 57, 57, 57, 57, 41, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57}, + {57, 57, 57, 57, 39, 5, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 57, 57, 57, 12, 57, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 22, 23, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 42, 32, 33, 34, 35, 36, 37, 38, 25}, + {57, 2, 57, 57, 57, 57, 57, 2, 2, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 57, 12, 57, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 57, 10, 57, 12, 57, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 57, 57, 57, 57, 57, 18, 43, 57, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 44, 57, 57, 12, 45, 46, 57, 57, 57, 18, 57, 45, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 57, 20, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 43, 57, 57, 57, 57, 18, 57, 20, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 57, 14, 57, 16, 57, 18, 57, 20, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 43, 14, 57, 57, 57, 18, 57, 20, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 47, 14, 15, 16, 57, 18, 57, 20, 5}, + {57, 57, 48, 57, 39, 5, 18, 57, 57, 57, 57, 57, 57, 57, 5, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 3, 4, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 57, 20, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 43, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 57, 57, 39, 5, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 57, 57, 24, 25, 57, 57, 57, 57, 57, 57, 57, 30, 49, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 2, 57, 57, 57, 57, 57, 57, 2, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57}, + {57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57}, + {57, 57, 57, 57, 24, 25, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 57, 57, 57, 30, 57, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 57, 30, 57, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 57, 28, 57, 30, 57, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 57, 57, 57, 57, 57, 36, 50, 57, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 51, 57, 57, 30, 52, 53, 57, 57, 57, 36, 57, 52, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 57, 38, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 50, 57, 57, 57, 57, 36, 57, 38, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 57, 32, 57, 34, 57, 36, 57, 38, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 50, 32, 57, 57, 57, 36, 57, 38, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 54, 32, 33, 34, 57, 36, 57, 38, 25}, + {57, 57, 55, 57, 24, 25, 36, 57, 57, 57, 57, 57, 57, 57, 25, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 22, 23, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 57, 38, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 50, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57}, + {57, 57, 57, 57, 39, 5, 57, 57, 57, 57, 57, 57, 57, 12, 57, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 3, 4, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 5}, + {57, 57, 22, 56, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 57, 38, 25}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 57, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 44, 57, 57, 12, 57, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 44, 57, 57, 12, 45, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 44, 57, 57, 12, 45, 57, 57, 57, 57, 18, 57, 45, 5}, + {57, 57, 3, 57, 39, 5, 6, 57, 57, 57, 9, 10, 11, 12, 57, 14, 15, 16, 57, 18, 57, 20, 5}, + {57, 57, 57, 57, 39, 5, 57, 57, 57, 57, 57, 57, 57, 57, 5, 57, 57, 57, 57, 18, 57, 57, 5}, + {57, 57, 57, 57, 24, 25, 57, 57, 57, 57, 57, 57, 57, 30, 57, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 57, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 51, 57, 57, 30, 57, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 51, 57, 57, 30, 52, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 51, 57, 57, 30, 52, 57, 57, 57, 57, 36, 57, 52, 25}, + {57, 57, 22, 57, 24, 25, 26, 57, 57, 57, 27, 28, 29, 30, 57, 32, 33, 34, 57, 36, 57, 38, 25}, + {57, 57, 57, 57, 24, 25, 57, 57, 57, 57, 57, 57, 57, 57, 25, 57, 57, 57, 57, 36, 57, 57, 25}, + {57, 2, 3, 4, 39, 5, 6, 2, 2, 57, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 5}, +}; +static const uint8_t myanmar_accept[57] = { + 2, 3, 1, 2, 2, 2, 2, 1, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 2, +}; +const VegaMachine vega_myanmar_machine = {myanmar_class, 23, 57, myanmar_next[0], sizeof(myanmar_next[0][0]), myanmar_accept, 2}; + diff --git a/text/syllabic.h b/text/syllabic.h new file mode 100644 index 0000000..8ab8699 --- /dev/null +++ b/text/syllabic.h @@ -0,0 +1,40 @@ +#ifndef VEGA_SYLLABIC_H +#define VEGA_SYLLABIC_H +#include <stdint.h> + +/* What HarfBuzz's syllabic shapers read, as text/syllabic.py makes it + * (text/syllabic-data.c): characters' categories, and the machines that + * find syllables in runs of them. */ + +/* Categories (hb-ot-shaper-indic.hh's ot_category_t: Indic's, and Khmer's + * and Myanmar's after them) and positions (ot_position_t). */ +enum { + VEGA_OT_X = 0, VEGA_OT_C = 1, VEGA_OT_V = 2, VEGA_OT_N = 3, VEGA_OT_H = 4, VEGA_OT_ZWNJ = 5, VEGA_OT_ZWJ = 6, + VEGA_OT_M = 7, VEGA_OT_SM = 8, VEGA_OT_A = 9, VEGA_OT_VD = 9, VEGA_OT_PLACEHOLDER = 10, VEGA_OT_DOTTEDCIRCLE = 11, + VEGA_OT_RS = 12, VEGA_OT_MPST = 13, VEGA_OT_REPHA = 14, VEGA_OT_RA = 15, VEGA_OT_CM = 16, VEGA_OT_SYMBOL = 17, + VEGA_OT_CS = 18, VEGA_OT_SMPST = 57, VEGA_OT_VABV = 20, VEGA_OT_VBLW = 21, VEGA_OT_VPRE = 22, VEGA_OT_VPST = 23, VEGA_OT_ROBATIC = 25, + VEGA_OT_XGROUP = 26, VEGA_OT_YGROUP = 27, VEGA_OT_AS = 32, VEGA_OT_MH = 35, VEGA_OT_MR = 36, VEGA_OT_MW = 37, + VEGA_OT_MY = 38, VEGA_OT_PT = 39, VEGA_OT_VS = 40, VEGA_OT_ML = 41 +}; +enum { + VEGA_POS_START, VEGA_POS_RA_TO_BECOME_REPH, VEGA_POS_PRE_M, VEGA_POS_PRE_C, VEGA_POS_BASE_C, VEGA_POS_AFTER_MAIN, + VEGA_POS_ABOVE_C, VEGA_POS_BEFORE_SUB, VEGA_POS_BELOW_C, VEGA_POS_AFTER_SUB, VEGA_POS_BEFORE_POST, VEGA_POS_POST_C, + VEGA_POS_AFTER_POST, VEGA_POS_SMVD, VEGA_POS_END +}; +/* A character's category | position << 8 (hb_indic_get_categories). */ +uint16_t vega_indic_categories(uint32_t u); + +/* A syllable machine: a DFA over classes of symbols (categories, those + * past 63 taken as 63). From each position the longest run it accepts is + * a syllable, ties to the pattern its grammar lists first (Ragel's + * scanner), which the accepting state already says. */ +typedef struct { + const uint8_t *class_of; /* [64] */ + unsigned classes, states; + const void *next; /* [states][classes]: the next state, states when none */ + unsigned width; /* bytes of a next state: 1 or 2 */ + const uint8_t *accept; /* [states]: the syllable type a state ends, plus one; 0 for none */ + unsigned broken; /* the broken cluster's type, plus one */ +} VegaMachine; +extern const VegaMachine vega_indic_machine, vega_khmer_machine, vega_myanmar_machine; +#endif diff --git a/text/syllabic.py b/text/syllabic.py new file mode 100644 index 0000000..6eb7c93 --- /dev/null +++ b/text/syllabic.py @@ -0,0 +1,456 @@ +#!/usr/bin/env python3 +"""The tables HarfBuzz's syllabic shapers read, as C (text/syllabic-data.c). + + python3 text/syllabic.py HARFBUZZ_SRC > text/syllabic-data.c + python3 text/syllabic.py HARFBUZZ_SRC --check + +HARFBUZZ_SRC is HarfBuzz 10.2.0's src/ directory; what is read from it: + +- Each character's Indic category and position: hb-ot-shaper-indic- + table.cc, the table HarfBuzz's gen-indic-table.py made from Unicode 16's + IndicSyllabicCategory and IndicPositionalCategory with its own overrides, + read cell for cell (Unicode here is 15.1: the table is taken as HarfBuzz + has it). Khmer and Myanmar read the same table. + +- The syllable machines: hb-ot-shaper-*-machine.rl, the Ragel grammars. + This compiles each to a DFA with Ragel's scanner semantics (the longest + match from each position, ties to the pattern listed first), its symbols + folded into classes that behave alike. --check runs the DFAs beside the + machines Ragel generated from the same grammars (hb-ot-shaper-*- + machine.hh) on random runs of symbols, and counts where they differ. + +The output is deterministic: the same inputs give the same file. +""" +import random +import re +import sys +from pathlib import Path + +HB = Path(sys.argv[1] if len(sys.argv) > 1 else '.') +MACHINES = ['indic', 'khmer', 'myanmar'] +ALPHABET = 64 # categories past 63 are folded into 63, which only `any` takes + + +# ---- Grammars: Ragel's regular expressions over symbols + +class Parser: + """Ragel's expressions as the machines use them: names, `a | b`, + `a . b` or `a b`, postfix `? * +`, parentheses.""" + + def __init__(self, text, names): + self.tokens = re.findall(r'[A-Za-z_][A-Za-z_0-9]*|[()|.?*+]', text) + self.at = 0 + self.names = names + + def peek(self): + return self.tokens[self.at] if self.at < len(self.tokens) else None + + def take(self): + self.at += 1 + return self.tokens[self.at - 1] + + def alternation(self): + node = self.concatenation() + while self.peek() == '|': + self.take() + node = ('alt', node, self.concatenation()) + return node + + def concatenation(self): + node = self.postfix() + while self.peek() not in (None, '|', ')'): + if self.peek() == '.': + self.take() + node = ('cat', node, self.postfix()) + return node + + def postfix(self): + node = self.atom() + while self.peek() in ('?', '*', '+'): + node = ({'?': 'opt', '*': 'star', '+': 'plus'}[self.take()], node) + return node + + def atom(self): + token = self.take() + if token == '(': + node = self.alternation() + assert self.take() == ')' + return node + return self.names[token] + + +def parse(text, names): + parser = Parser(text, names) + node = parser.alternation() + assert parser.peek() is None, (text, parser.peek()) + return node + + +class Grammar: + """A machine's .rl: its symbols (exports), syllable types (the enum), and + the scanner's patterns in order, each with the type it finds.""" + + def __init__(self, name): + text = (HB / f'hb-ot-shaper-{name}-machine.rl').read_text() + enum = re.search(r'enum \w+_syllable_type_t \{(.*?)\};', text, re.S).group(1) + self.types = [t.strip() for t in enum.split(',') if t.strip()] + body = '\n'.join(block for block in re.findall(r'%%\{(.*?)\}%%', text, re.S) if 'main :=' in block) + body = re.sub(r'#[^\n]*', '', body) + main = re.search(r'main := \|\*(.*?)\*\|;', body, re.S).group(1) + rest = body.replace(re.search(r'main := \|\*.*?\*\|;', body, re.S).group(0), '') + self.symbols = {name: int(value) for name, value in re.findall(r'export (\w+)\s*=\s*(\d+);', rest)} + names = {name: ('sym', frozenset([value])) for name, value in self.symbols.items()} + names['any'] = ('any',) + for statement in rest.split(';'): + statement = statement.strip() + if not statement or statement.startswith('export'): + continue + name, expression = statement.split('=', 1) + names[name.strip()] = parse(expression, names) + self.patterns = [] + for pattern, action in re.findall(r'([^;{}]+?)=>\s*\{(.*?)\};', main, re.S): + kind = re.search(r'found_syllable \((\w+)\)', action).group(1) + self.patterns.append((parse(pattern, names), self.types.index(kind))) + self.broken = next(i for i, t in enumerate(self.types) if t.endswith('broken_cluster')) + + +class NFA: + def __init__(self): + self.edges = [] # state: [(symbols or None for epsilon, target)] + self.accept = {} # state: pattern index + + def state(self): + self.edges.append([]) + return len(self.edges) - 1 + + def build(self, node): + """Thompson's construction: (start, end) of node.""" + kind = node[0] + if kind in ('sym', 'any'): + a, b = self.state(), self.state() + symbols = frozenset(range(ALPHABET)) if kind == 'any' else frozenset(min(s, ALPHABET - 1) for s in node[1]) + self.edges[a].append((symbols, b)) + return a, b + if kind == 'alt': + a, b = self.state(), self.state() + for part in node[1:]: + s, e = self.build(part) + self.edges[a].append((None, s)) + self.edges[e].append((None, b)) + return a, b + if kind == 'cat': + s1, e1 = self.build(node[1]) + s2, e2 = self.build(node[2]) + self.edges[e1].append((None, s2)) + return s1, e2 + s, e = self.build(node[1]) + a, b = self.state(), self.state() + self.edges[a].append((None, s)) + self.edges[e].append((None, b)) + if kind in ('opt', 'star'): + self.edges[a].append((None, b)) + if kind in ('star', 'plus'): + self.edges[e].append((None, s)) + return a, b + + def closure(self, states): + seen, stack = set(states), list(states) + while stack: + for symbols, target in self.edges[stack.pop()]: + if symbols is None and target not in seen: + seen.add(target) + stack.append(target) + return frozenset(seen) + + +def machine(grammar): + """The scanner's DFA, minimal: each state's transitions by symbol (None + for none) and the syllable type it accepts (the first pattern's), or + None; the start is state 0.""" + nfa = NFA() + start = nfa.state() + for index, (tree, kind) in enumerate(grammar.patterns): + s, e = nfa.build(tree) + nfa.edges[start].append((None, s)) + nfa.accept[e] = index + first = nfa.closure([start]) + states, order, transitions = {first: 0}, [first], [] + for current in order: + row = [] + for symbol in range(ALPHABET): + targets = [t for s in current for symbols, t in nfa.edges[s] if symbols is not None and symbol in symbols] + if not targets: + row.append(None) + continue + target = nfa.closure(targets) + if target not in states: + states[target] = len(order) + order.append(target) + row.append(states[target]) + transitions.append(row) + accepts = [] + for current in order: + matched = [nfa.accept[s] for s in current if s in nfa.accept] + accepts.append(grammar.patterns[min(matched)][1] if matched else None) + # Moore's partition refinement + n = len(transitions) + block = list(accepts) + while True: + signature = [(block[s],) + tuple(None if t is None else block[t] for t in transitions[s]) for s in range(n)] + numbering = {} + refined = [numbering.setdefault(signature[s], len(numbering)) for s in range(n)] + if len(numbering) == len(set(block)): + break + block = refined + new = {} + for s in range(n): + new.setdefault(block[s], len(new)) + table, accept = [None] * len(new), [None] * len(new) + for s in range(n): + table[new[block[s]]] = [None if t is None else new[block[t]] for t in transitions[s]] + accept[new[block[s]]] = accepts[s] + # every symbol starts a syllable: the scanner never stalls + assert all(table[0][symbol] is not None and accept[table[0][symbol]] is not None for symbol in range(ALPHABET)) + return table, accept + + +def classes(table): + """Symbols whose columns are the same, folded.""" + columns, of = {}, [] + for symbol in range(ALPHABET): + of.append(columns.setdefault(tuple(row[symbol] for row in table), len(columns))) + return of, len(columns) + + +def scan(table, accept, symbols): + """Syllables of a run of symbols by a DFA: (start, end, type).""" + found, p = [], 0 + while p < len(symbols): + state, end, kind = 0, None, None + for q in range(p, len(symbols)): + state = table[state][min(symbols[q], ALPHABET - 1)] + if state is None: + break + if accept[state] is not None: + end, kind = q + 1, accept[state] + found.append((p, end, kind)) + p = end + return found + + +# ---- The Indic table + +def indic_table(grammars): + """Code point to (category, position) for the characters HarfBuzz's + table has, and the default for the others.""" + text = (HB / 'hb-ot-shaper-indic-table.cc').read_text() + positions = {name: int(value) for name, value in + re.findall(r'(POS_\w+) = (\d+)', (HB / 'hb-ot-shaper-indic.hh').read_text())} + machines = {'I': grammars['indic'].symbols, 'K': grammars['khmer'].symbols, 'M': grammars['myanmar'].symbols} + long_names = {name: machines[m][cat] for name, m, cat in re.findall(r'#define (OT_\w+) ([IKM])_Cat\((\w+)\)', text)} + short = {name: long_names[target] for name, target in re.findall(r'#define (_OT_\w+)\s+(OT_\w+)', text)} + short.update({name: positions[target] for name, target in re.findall(r'#define (_POS_\w+)\s+(POS_\w+)', text)}) + + def cell(s, m): + return short['_OT_' + s], short['_POS_' + m] + + cells = {} + table = text[text.index('static const uint16_t indic_table[]'):text.index('}; /* Table items:')] + for start, row in re.findall(r'/\* ([0-9A-F]{4,5}) \*/(.*)', table): + for k, (s, m) in enumerate(re.findall(r'_\((\w+),(\w+)\)', row)): + cells[int(start, 16) + k] = cell(s, m) + lookup = text[text.index('hb_indic_get_categories (hb_codepoint_t u)'):] + for code, s, m in re.findall(r'u == 0x([0-9A-F]+)u\)\) return _\((\w+),(\w+)\);', lookup): + cells[int(code, 16)] = cell(s, m) + default = cell(*re.search(r'return _\((\w+),(\w+)\);\s*\}', lookup).groups()) + return cells, default + + +def runs_of(cells): + """Runs of covered code points, gaps under 16 kept inside.""" + runs = [] + for code in sorted(cells): + if runs and code - runs[-1][1] <= 16: + runs[-1][1] = code + else: + runs.append([code, code]) + return runs + + +# ---- Checking against Ragel's machines (hb-ot-shaper-*-machine.hh) + +def ragel(name): + """A scanner from the flat tables and actions Ragel generated: symbols + to (start, end, type).""" + text = (HB / f'hb-ot-shaper-{name}-machine.hh').read_text() + grammar = Grammar(name) + + def array(field): + body = re.search(rf'_{name}_syllable_machine_{field}\[\] = \{{(.*?)\}};', text, re.S).group(1) + return [int(v) for v in re.findall(r'-?\d+', body)] + + keys, spans, offsets, indicies = array('trans_keys'), array('key_spans'), array('index_offsets'), array('indicies') + targets, actions = array('trans_targs'), array('trans_actions') + to_state, from_state, eof_trans = array('to_state_actions'), array('from_state_actions'), array('eof_trans') + start = int(re.search(rf'{name}_syllable_machine_start = (\d+);', text).group(1)) + + def cases(switch): + """The action cases of the switch on an array: number to code.""" + at = text.index(f'switch ( _{name}_syllable_machine_{switch}[') + depth, begin = 0, text.index('{', at) + for k in range(begin, len(text)): + depth += {'{': 1, '}': -1}.get(text[k], 0) + if depth == 0: + body = text[begin + 1:k] + break + out, depth, current, k = {}, 0, None, 0 + while k < len(body): + m = re.compile(r'case (\d+):').match(body, k) if depth == 0 else None + if m: + current = int(m.group(1)) + out[current] = '' + k = m.end() + continue + depth += {'{': 1, '}': -1}.get(body[k], 0) + if current is not None: + out[current] += body[k] + k += 1 + return out + + trans_cases, to_cases, from_cases = cases('trans_actions'), cases('to_state_actions'), cases('from_state_actions') + + def run(code, st): + # the `switch( act )` of a backtracking action: only the chosen case + m = re.search(r'switch\( act \) \{(.*)\}', code, re.S) + if m: + chosen = re.search(rf'case {st["act"]}:(.*?)break;', m.group(1), re.S) + code = code[:m.start()] + (chosen.group(1) if chosen else '') + for statement in re.findall(r'ts = p;|ts = 0;|te = p\+1;|te = p;|p--;|p = \(\(te\)\)-1;|act = \d+;|' + r'found_syllable \(\w+\)', code): + if statement == 'ts = p;': st['ts'] = st['p'] + elif statement == 'ts = 0;': st['ts'] = 0 + elif statement == 'te = p+1;': st['te'] = st['p'] + 1 + elif statement == 'te = p;': st['te'] = st['p'] + elif statement == 'p--;': st['p'] -= 1 + elif statement == 'p = ((te))-1;': st['p'] = st['te'] - 1 + elif statement.startswith('act'): st['act'] = int(statement[6:-1]) + else: + kind = statement[len('found_syllable ('):-1] + st['found'].append((st['ts'], st['te'], grammar.types.index(kind))) + + def scanner(symbols): + st = {'ts': 0, 'te': 0, 'act': 0, 'p': 0, 'found': []} + cs, pe = start, len(symbols) + state = 'resume' if pe else 'test_eof' + while True: + if state == 'resume': + if from_state[cs]: + run(from_cases[from_state[cs]], st) + c, low, high, span = symbols[st['p']], keys[2 * cs], keys[2 * cs + 1], spans[cs] + trans = indicies[offsets[cs] + (c - low if span > 0 and low <= c <= high else span)] + state = 'eof_trans' + if state == 'eof_trans': + cs = targets[trans] + if actions[trans]: + run(trans_cases[actions[trans]], st) + if to_state[cs]: + run(to_cases[to_state[cs]], st) + st['p'] += 1 + state = 'resume' if st['p'] != pe else 'test_eof' + continue + if state == 'test_eof': + if st['p'] == pe and eof_trans[cs] > 0: + trans = eof_trans[cs] - 1 + state = 'eof_trans' + continue + return st['found'] + return scanner + + +def check(runs=100000): + rng = random.Random(1) + for name in MACHINES: + grammar = Grammar(name) + table, accept = machine(grammar) + theirs = ragel(name) + symbols = sorted(set(grammar.symbols.values())) + [0, 19, 63] + differ = 0 + for _ in range(runs): + run = [rng.choice(symbols) for _ in range(rng.randint(1, 14))] + a, b = scan(table, accept, run), theirs(run) + if a != b: + differ += 1 + if differ <= 3: + print(name, 'differ:', run, a, b, file=sys.stderr) + print(f'{name}: {len(table)} states, {runs} runs, {differ} differ') + + +# ---- The C file + +def emit_machine(out, name, grammar): + table, accept = machine(grammar) + of, count = classes(table) + dead = len(table) + column_of = {} + for symbol in range(ALPHABET): + column_of.setdefault(of[symbol], symbol) + width = 'uint8_t' if dead < 256 else 'uint16_t' + out.append(f'/* {name} (hb-ot-shaper-{name}-machine.rl): {len(table)} states, {count} classes of symbol;' + f' types {", ".join(grammar.types)}. */') + out.append(f'static const uint8_t {name}_class[{ALPHABET}] = {{') + for k in range(0, ALPHABET, 32): + out.append(' ' + ', '.join(str(c) for c in of[k:k + 32]) + ',') + out.append('};') + out.append(f'static const {width} {name}_next[{len(table)}][{count}] = {{') + for row in table: + out.append(' {' + ', '.join(str(dead if row[column_of[k]] is None else row[column_of[k]]) + for k in range(count)) + '},') + out.append('};') + out.append(f'static const uint8_t {name}_accept[{len(table)}] = {{') + cells = [str(0 if a is None else a + 1) for a in accept] + for k in range(0, len(cells), 32): + out.append(' ' + ', '.join(cells[k:k + 32]) + ',') + out.append('};') + out.append(f'const VegaMachine vega_{name}_machine = {{{name}_class, {count}, {len(table)}, {name}_next[0], ' + f'sizeof({name}_next[0][0]), {name}_accept, {grammar.broken + 1}}};') + out.append('') + + +def main(): + if len(sys.argv) > 2 and sys.argv[2] == '--check': + check() + return + grammars = {name: Grammar(name) for name in MACHINES} + cells, default = indic_table(grammars) + flat, starts = [], [] + for a, b in runs_of(cells): + starts.append((a, b, len(flat))) + flat.extend(cells.get(code, default) for code in range(a, b + 1)) + out = ['/* Generated by text/syllabic.py from HarfBuzz 10.2.0\'s sources; do not edit. */', + '#include "syllabic.h"', '', + '/* Each character\'s Indic category and position (hb-ot-shaper-indic-table.cc, Unicode 16):', + ' * category | position << 8. */', + f'static const uint16_t indic_cells[{len(flat)}] = {{'] + for k in range(0, len(flat), 12): + out.append(' ' + ', '.join(f'0x{c | p << 8:04X}' for c, p in flat[k:k + 12]) + ',') + out.append('};') + out.append(f'static const struct {{ uint32_t first, last, at; }} indic_runs[{len(starts)}] = {{') + for a, b, at in starts: + out.append(f' {{0x{a:04X}, 0x{b:04X}, {at}}},') + out.append('};') + out.append('') + out.append('uint16_t vega_indic_categories(uint32_t u) {') + out.append(f' for (unsigned i = 0; i < {len(starts)} && u >= indic_runs[i].first; ++i)') + out.append(' if (u <= indic_runs[i].last) return indic_cells[indic_runs[i].at + u - indic_runs[i].first];') + out.append(f' return 0x{default[0] | default[1] << 8:04X};') + out.append('}') + out.append('') + out.append('/* The syllable machines: a DFA over classes of symbol; a state\'s accept is') + out.append(' * the syllable type it ends, plus one. */') + for name in MACHINES: + emit_machine(out, name, grammars[name]) + sys.stdout.write('\n'.join(out) + '\n') + + +if __name__ == '__main__': + main() |
