| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
| |
Run against HarfBuzz 14.5.0 and Unicode 18.0.0, text/unicode.py met a
line breaking class it does not know (HH) and text/syllabic.py a table
laid out otherwise, and the tool had already emptied both files. A
generator's output now replaces its file only when it succeeds, and the
tool stops before building when one fails: those are what to port first.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
text/upstream.py brings in a HarfBuzz release (default: the one the
build links) and the UCD of the Unicode version its tables came from,
builds that HarfBuzz as one static library from its amalgamated source
(no system package, any version), regenerates the Unicode tables and the
syllabic ones, holds the syllable machines to HarfBuzz's compiled ones,
and runs the text checks built against it: Unicode's conformance tests on
that UCD, random shaping against that HarfBuzz, the layout's promises,
the kern class tests, HarfBuzz's own suite. --corpus adds every installed
font (text/fonts.py, the corpus runner, now in the tree). What a new
version breaks shows as failing checks with the font and the input to
port a fix from; the generated files are left changed for review.
make check-text is those checks alone; UCD and HARFBUZZ_SRC default to
build/upstream's when the tool fetched them. Its first run found that
text/shape-build.c used clock_gettime without asking for POSIX, building
only because the system HarfBuzz's headers did. Against HarfBuzz 10.2.0
built from source and the 16.0.0 UCD, every check passes in about a
minute.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
HarfBuzz 10.2's tables are Unicode 16.0, Vega's were the installed UCD's
15.1: characters new in 16.0 could be read two ways. The tables now come
from the 16.0 UCD, and the line breaking rules are revision 53's: LB13's
'x IS' split into LB15c (SP / IS NU, as in 'subtract .5') and LB15d; the
numbers of section 8.2's Example 7 made LB25 (new: IS x NU, PR x OP IS
NU); LB19 split, with LB19a letting quotation marks between East Asian
characters break; LB20a keeping a word-initial hyphen to its word; LB21a
narrowed to Hebrew, hyphen, then not Hebrew. vega_east_asian now covers
all of East_Asian_Width F, W and H.
Unicode 16.0's conformance tests all pass (bidi 770241 and 91707, line
breaks 16672, graphemes 1093, words 1826, sentences 512, normalization
19965). test-text reads NormalizationTest.txt plain or compressed, and
passes over a UCD of another version than the tables' (the installed
one here is 15.1). HarfBuzz's suite (5018 the same), the installed fonts'
and Noto's random runs, and the surfaces' checks agree as before.
|
| |
|
|
|
|
|
|
|
|
| |
A font whose GSUB grows the buffer past its bounds (text-rendering-tests'
GSUB-3) made vega_shape fail. HarfBuzz goes on with the glyphs of the last
pass that finished: every later pass does nothing, positions are the
default advances. Vega's buffer operations already stand still once the
buffer is unsuccessful, so it now goes on too, and a broken font shows
what HarfBuzz shows rather than nothing. HarfBuzz's suite: 5018 of 5199
the same, none failing.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The kern table's class subtables, OpenType's and Apple's, as HarfBuzz
reads them (hb-aat-layout-kerx-table.hh, hb-ot-kern-table.hh): format 2's
class tables give byte offsets into a kerning array, format 3's byte
classes index a value list. Both go through the same pass as pairs
(marks skipped, the kerning split between the pair). Format 2 filters the
glyphs by the class tables' digests, as HarfBuzz's accelerator does, so a
digest's false positive reads class 0 there too; what a subtable reads
must lie within it, the last one's within the table. The build checks the
subtables as HarfBuzz's sanitizer does, and still refuses state machines
(format 1) and cross-stream kerning.
text/test-kern.py makes fonts with format 2 and 3 subtables in every
order, with the edge cases that reading has (value 1, glyphs outside a
table, offsets below the array or odd, values past a subtable's or the
table's end, classes and indexes past their counts): 24 fonts, 96 plans,
the same as HarfBuzz. make check runs it where fontTools is. HarfBuzz's
suite: 5017 of 5199 the same.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The section now carries each sequence the font maps, a base and a
selector, with the glyph HarfBuzz gives the pair (a default one's the
base's own). As HarfBuzz's normalizer does, a sequence the font has
becomes that one glyph, the two characters' clusters merged; one it lacks
still passes on as two, the selector hidden. The header grows to 88 bytes
for the table.
The random check now mixes in the font's own sequences and stray
selectors: 30000 runs over Noto Sans CJK's 14787 ideographic variation
sequences are HarfBuzz's, and so are the installed fonts' and Noto's.
HarfBuzz's suite: 5014 of 5199 the same, none different; what is left is
what the build refuses and one run without a plan.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The shaping section's character map came from hb_face_collect_unicodes.
HarfBuzz's own suite showed where that list and its mapping part: it cuts
a format 13 map short (AdobeBlank2's whole Unicode to one glyph), and a
symbol map (3,0) it searches through the private use area, Latin and, in an
old Arabic font, Arabic too. Now every code point is asked with
hb_font_get_nominal_glyph (about 30 ms a font) and the map carried as
groups: codes going to consecutive glyphs, or all to one (12 bytes each,
so a format 13 map stays small).
HarfBuzz's suite: 5011 of 5199 cases the same (1355 before). The installed
fonts (213) and the Noto ones (21) still shape as HarfBuzz does on random
runs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
text/harfbuzz-suite.py reads a HarfBuzz release's test/shape/data
(in-house, text-rendering-tests, aots: fonts and inputs, many added when a
font or a shaper once went wrong) and shapes every case both ways with the
new vega-symbols shape-cases: it compiles a font's section for the scripts
its cases need (guessed as HarfBuzz guesses them), then compares each run
with HarfBuzz's, glyph for glyph and unit for unit. The oracle is HarfBuzz
run live, not the recorded outputs, which depend on options Vega has no
counterpart for (font functions, sizes, output formats). Cases asking for
what Vega does not do (user features, variations, languages, context,
cluster levels, buffer flags, vertical text) and fonts of other systems are
left out and counted by why.
text/harfbuzz-suite.known lists what differs today, and why: AAT, the
character map built from what HarfBuzz lists rather than how it maps
(format 13, symbol maps into the private use area), variation selectors,
Arabic fallback with shared glyphs, kern format 2, Zawgyi, and one run
without a plan. Any other difference fails the run; a listed case that
agrees is named. make check-harfbuzz HARFBUZZ_SRC=... runs it (make check
too, when given): 5929 cases, 5199 compared, 1355 the same.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
vega_layout takes fonts in order of preference (VegaFonts). A grapheme
cluster goes to the first font that has all its characters (default
ignorables aside: shaping hides them); a cluster of Common and Inherited
characters stays in the font before it when that one has it, so the spaces
between Arabic words stay in the Arabic font; failing that, the first font
with its first character, else the first font. Runs split where the font
changes too. Positions are in the first font's units, a run's pen scaled
as a whole so rounding does not add up. Callers can narrow what a font
has (a symbol file holds only the glyphs its codes reach).
vega_default_ignorable is shape.c's HarfBuzz 10.2 set, now public.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The rest of HarfBuzz 10.2's Arabic shaper. Joining types now come from
HarfBuzz's own table (Unicode 16, with Syriac's Alaph and Dalath-Rish
groups the state machine's last two columns use), the general category
deciding where it has none, so Syriac shapes with fin2, fin3 and med2.
What stch multiplies is recorded and, after positioning, stretched under
the word before it, repeating pieces copied as often as fit. A font with
presentation forms but no Arabic features gets HarfBuzz's fallback
lookups, which the build tool writes as OpenType and the runtime applies
at their pause like any GSUB lookup.
The generator now reads HarfBuzz's generated tables through their lookup
functions' ranges and offsets: rows whose range starts mid-row have
blank cells in front, which the row comments alone miscounted (Arabic
Supplement); the Indic table, read again this way, did not change.
Noto Sans Syriac, FreeSans's Syriac, Unifont's Arabic and Noto Sans
Arabic shape as HarfBuzz does at 3,000 runs a plan; no installed font
is refused any more (213 fonts, 1,822 plans).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The Thai shaper, from HarfBuzz 10.2's source: SARA AM comes apart into
NIKHAHIT and SARA AA, the NIKHAHIT moving back over the above-base marks
(and counting as a mark when advances are zeroed). A font with no Thai
script of its own (only DFLT or latn in GSUB) gets its marks shifted to
its private-use glyphs, Windows' before the Mac's, as the stacking
around the base asks; the planner records, as HarfBuzz's does, whether
the font had the script under its own tag.
Noto Sans Thai and Lao, TLWG Loma, FreeSerif's Thai, DejaVu Sans's Lao
and Unifont shape as HarfBuzz does at 3,000 runs a plan, and Loma with
its thai script taken out (so the private-use path runs) at 5,000.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The USE shaper, from HarfBuzz 10.2's source, for Sinhala, Tibetan,
Javanese, Balinese, Mongolian and the rest of its eighty-odd scripts
(and Indic fonts with the new dev3-style tags). The syllable machine
reads past CGJs and ZWNJs that marks follow; rphf goes on each
syllable's first glyphs and records the repha it made, pref records its
pre-base glyph, with substitution marks cleared between; a repha moves
before the first post-base glyph and pre-base vowels move back. Scripts
that join like Arabic take Arabic's forms (Mongolian's free variation
selectors their base's), the others join syllable to syllable. Marks
lose their advances before GPOS.
text/syllabic.py carries HarfBuzz's compressed USE table over as it is,
compiles the USE grammar with the others (100,000 random runs against
Ragel's machine: no syllable differs), and checks at compile time that
the categories syllabic.h names are the machines' own.
Noto Sans Sinhala, Javanese, Balinese, Mongolian, Noto Serif Tibetan,
FreeSerif's Sinhala and Unifont's five shape as HarfBuzz does at 5,000
runs a plan; all installed fonts (1,748 plans) and 21 Noto fonts too.
|
| |
|
|
|
|
|
| |
HarfBuzz's is_default_ignorable takes U+180B..180E from the Mongolian
block, not the fourth free variation selector (U+180F, default
ignorable since Unicode 14), so it keeps its own glyph rather than the
invisible one. Unifont's Mongolian showed it.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Two more shapers on the syllabic groundwork, from HarfBuzz 10.2's
source. Khmer reorders before any lookup: a coeng and Ro go before the
base, masked for pref, what follows them for cfar, a left matra goes
first; split matras decompose with U+17C1. Myanmar finds the base after
a kinzi, gives each glyph its place around it and sorts by place, and
zeroes mark advances before GPOS rather than after.
Noto Sans Khmer and Noto Sans Myanmar shape as HarfBuzz does at 20,000
runs a plan, and so do Unifont's Khmer and Myanmar (no GSUB).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The Indic shaper, from HarfBuzz 10.2's source: syllables found by the
Indic machine; each syllable's base consonant, its pre-base matras and
reph moved around it and masked for the basic features, applied one at
a time within the syllable; then matras, reph and pre-base-reordering
consonants placed again by what the features made. Vowel constraints,
Indic decompositions and compositions, dotted circles for broken
syllables, and per-syllable matching come with it.
Plans carry GSUB pauses (steps where the shaper does its part) and the
stages of the features the shaper asks whether they would substitute;
the runtime answers with each subtable's would_apply, as HarfBuzz does.
Masks are now the common ones and each shaper's own.
text/syllabic.py reads HarfBuzz's Ragel grammars (Indic, Khmer,
Myanmar) and compiles each to a minimal DFA with Ragel's scanner
semantics; beside the machines Ragel generated from the same grammars,
100,000 random runs each find no syllable that differs. It also reads
HarfBuzz's Indic table (Unicode 16) cell for cell.
Every installed font shapes as HarfBuzz does (213 fonts, 1,688 plans);
the Indic ones at 20,000 runs a plan (FreeSerif's eight scripts,
FreeSans, Unifont's nine without GSUB), and nine new-spec Noto Sans
fonts at 5,000.
|
| |
|
|
|
|
|
| |
HarfBuzz reorders marks by a modified combining class; for U+0C55 and
U+0C56 (canonical classes 84 and 91) it is 4 and 5, so they sort before
the virama. Vega had 0, which rustybuzz 0.20 uses; HarfBuzz 10.2's
hb-unicode.hh says otherwise, and Unifont's Telugu showed it.
|
| |
|
|
|
|
|
|
|
|
| |
HarfBuzz sorts each stage's lookups by index with hb_qsort, its copy of
Isaac Turner's quicksort, and then merges entries of the same lookup:
masks and joiner flags combine, but the per-syllable and random flags are
the first entry's. Which entry comes first is the quicksort's doing, so
the planner now sorts the same way instead of stably. Noto Sans
Malayalam, whose psts and rlig share a lookup, needed it: ma with a
combining dot joined across syllables in HarfBuzz and not here.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
HarfBuzz does
The planner now picks who positions as HarfBuzz 10.2 does: GPOS, the old
kern table when GPOS has no kern feature, and marks placed by combining
class and ink boxes when there is no GPOS; the Hangul shaper composes
syllables the font has whole, takes apart those it lacks into jamo for
ljmo, vjmo and tjmo, and moves tone marks before their syllable.
The section grows to an 80-byte header (the kern table, and each glyph's
ink box when a plan places marks without GPOS) and 88-byte plans (the
kern mask, the jamo masks). Kern tables are applied pair subtable by
pair subtable with HarfBuzz's own binary search; state machines, class
subtables and cross-stream kerning, which no installed font has, are
refused. The check spells Hangul syllables every way text does.
Every installed font Vega accepts shapes as HarfBuzz does (209 fonts,
1,520 plans, a thousand runs each), and the fonts refused before at
20,000 runs a plan: Noto Sans CJK KR and Baekmuk for Hangul, IPA Gothic
and Noto Mono for marks without GPOS, FreeSans, FreeSerif, Liberation
and DejaVu Sans ExtraLight for kern tables, Unifont's Latin, Greek,
Cyrillic, Hebrew and Hangul.
|
| |
|
|
|
|
| |
HarfBuzz's Hebrew presentation forms join U+05F2 and patah into U+FB1F;
Vega had U+05D9 there. Only fonts without a GPOS mark feature take the
rule, so no font Vega accepted showed it; Unifont's Hebrew did.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
order
text/layout.c puts the checked parts together: a paragraph's levels,
scripts resolved for Common and Inherited characters, break
opportunities, one shaping to measure, lines broken at the last
opportunity that fits (trailing spaces hang, and a line that shapes
wider than measured backs off), then per line L1, runs of one level and
one script shaped again, the runs in L2's order and their glyphs placed.
test-layout checks 30,000 random mixed-script paragraphs: lines cover
the text, mandatory breaks end lines, lines fit when they can, and
glyphs come in the order UAX #9 gives the characters. vega-symbols
paragraph draws a layout through the symbol file's glyphs.
|
| |
|
|
|
|
|
|
|
| |
UAX #14 line breaking (LB1-LB31, numbers tailored as Unicode's tests
are) and UAX #29's grapheme cluster, word and sentence boundaries, each
from generated property tables, the rules taken in order, in linear time.
They pass all of Unicode 15.1's LineBreakTest (10,274 cases),
GraphemeBreakTest (1,187), WordBreakTest (1,826) and SentenceBreakTest
(512).
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The build tool compiles a font's shaping plans the way HarfBuzz's planner
does (features, stages, masks, lookups per script and direction) and
carries them with the font's character map, advances and GDEF, GSUB and
GPOS tables in a section of the symbol file (VEGASYM4); glyphs shaping
reaches become symbols of their own. A small engine ports HarfBuzz 10's
default, Arabic and Hebrew shaping step for step and reads the tables in
place, every read bounds-checked. The build refuses a file whose shaping
differs from HarfBuzz's on random runs: DejaVu Sans agrees on 600,000
runs in six scripts both ways, and every installed font the tool accepts
agrees too, at 1.05-1.18 times HarfBuzz's time.
|
| |
|
|
|
|
| |
Joining types, general categories and scripts (with their ISO 15924
codes) as two-stage tables, canonical decompositions one step at a time,
and Extended_Pictographic, from the UCD by text/unicode.py.
|
| | |
|
| |
|
