Build the generated Unicode tables from Unicode 18.0.0

[jakejackson1]

The follow-up to #251, which pinned its generator to Unicode 17.0.0 and said which Unicode mPDF claims
was a maintainer decision. It is 18.0.0 now. One edit — UcdnTables::DEFAULT_VERSION — then
composer ucdn:update and composer arabicjoining:update.

Stacked on #251 (PR #261), so this branch contains its commits too. #261 should merge first.

composer ucdn:update
  Unicode 18.0.0: 1017 records, 178 scripts (3 new), 438 mirror pairs, 219791 bytes written
  New scripts: Jurchen, Proto_Cuneiform, Seal
composer arabicjoining:update
  Unicode 18.0.0: 284 left-joining, 375 right-joining characters, 23538 bytes written

composer otlanguages:update is not part of this and was checked rather than assumed:
OtLanguageTags::DEFAULT_VERSION is a HarfBuzz release, not a Unicode one, and running it over the
rebuilt src/Ucdn.php produces no diff.

The cost, stated up front: the independent oracle goes dark

UcdnAgainstIcuTest compares every one of the 1,114,112 codepoints against ICU — a wholly separate
implementation of the same data — and skips itself when ICU's Unicode release does not match the repo's.
Local ICU 78.2 carries Unicode 17.0, so this change switches that test off until ICU ships Unicode 18.

That matters more than a skip count: it is the only check on src/Ucdn.php that is not the generator
grading its own homework. After this, the regenerated tables are pinned by the round-trip test (which
proves the generator agrees with itself), the golden masters and the shaping suite — and nothing
independent.

Its last word on the tables is on record. At #251's tip with DEFAULT_VERSION still 17.0.0:

$ vendor/bin/phpunit tests/Mpdf/UcdnAgainstIcuTest.php
OK (1 test, 2 assertions)

Every codepoint agreed with ICU 78.2 on category, combining class, bidi class, mirrored, East Asian
width and script. The comparison itself is untouched; the skip is the test protecting itself from
grading Unicode 18 tables against Unicode 17 data, and it is correct to do so.

What moved in the joining tables: exactly 22 characters

$leftJoining 262 → 284, $rightJoining 375 → unchanged. The 22 are contiguous,
U+10ED9..U+10EEE, and all Joining_Type=L — which is why only the left table moves.

This is bounded at the source rather than inferred from the output: the entire data delta between
ArabicShaping-17.0.0.txt and -18.0.0.txt is those 22 added lines, 834 → 856, with nothing removed
and nothing changed. So no character of any other script could have moved.

10ED9; CROWN BEH; L; CROWN BEH
… through …
10EEE; DOTLESS CROWN BEH WITH 2 DOTS BELOW; L; CROWN BEH

At Unicode 17 no in-scope character had Joining_Type=L at all, so L was a column the tables could
not reach.

Why the new test's oracle is not a plain hb-shape run

Released HarfBuzz does not carry Unicode 18's joining data yet. Both 14.3.1 and 14.4.0 (the current
release) generate src/hb-ot-shaper-arabic-table.hh from ArabicShaping-17.0.0.txt — the header says so
— and only main has been regenerated. So hb-shape gives U+10ED9 no joining behaviour, and that is a
gap in the oracle, not a disagreement. For these 22 characters mPDF is ahead of released HarfBuzz.

The oracle is therefore in two parts. The joining type is normative UCD data, the same
ArabicShaping-18.0.0.txt line that HarfBuzz main reads. The semantics of L come from hb-shape
14.3.1 read on a character Unicode 17 already typed L — and HarfBuzz's arabic_state_table, the code
that turns a type into a form, is byte-identical between 14.3.1 and main, so only the data differs:

$ hb-shape --font-file=packages/Middle-East-Scripts-Bundle/fonts/LateefRegOT.ttf \
    --script=arab --direction=rtl --no-clusters --no-positions --unicodes=10ACD,0628
[uni0628.fina|.notdef]        U+10ACD is L at Unicode 17: the Beh after it takes fina
$ ... --unicodes=0628,10ACD
[.notdef|uni0628]             and before it, nominal
$ ... --unicodes=0628,10EC2
[.notdef|uni0628.init]        R, the mirror case, on a character released HarfBuzz does know
$ ... --unicodes=10ED9,0628
[uni0628|.notdef]             the crown letter: released HarfBuzz has no joining for it

Both new tests fail at #251's tip:

1) …testALetterAfterALeftJoiningCrownLetterTakesItsFinalForm
-    0 => 'B_FINA'   -    1 => 1
+    0 => 'B_ISOL'   +    1 => 0
2) …testABehAfterALeftJoiningCrownLetterDrawsTheSameFormAsAfterAMeem
-'U+E064'            (Lateef's uni0628.fina)
+'U+0628'            (the nominal Beh)
Tests: 2, Assertions: 2, Failures: 2.

The shape of the src/Ucdn.php diff

2,145 changed lines, and it decomposes completely:

• 13,010 codepoints changed record, of which 13,007 were unassigned at 17 and now carry a script.
  By script: Seal 11,328, Jurchen 965, Proto-Cuneiform 164, Latin 161, Cuneiform 159, Common 153,
  Arabic 39, Inherited 11, then 16 scripts with six or fewer. No codepoint moved from one named script
  to another.
• The 39 new Arabic codepoints are U+10EC9..10ECF, U+10ED9..10EEE and U+10EF0..10EF9 — the 22 crown
  letters plus 17 marks and digits.
• 3 new scripts, appended at 175/176/177 with the ISO 15924 codes jurc, pcun, seal.
• 10 new mirror pairs, matching BidiMirroring.txt's ten new lines exactly; none removed or changed.
• 3 already-assigned characters reclassified: U+1B3A, U+1B3C and U+1B42 (Balinese vowel signs) move
  Bidi_Class NSM → L. That is the whole of the non-assignment delta.
• The remaining bulk is the index tables re-packing. $ucd_records 1013 → 1017, the first new record
  inserted at slot 383, so record numbers from 383 up shift.

No existing script's number moved. The only change to the SCRIPT_* constants is three appended
lines; SCRIPT_INHERITED is still 40 and SCRIPT_UNKNOWN still 102, which three call sites depend on.

Three literals, because there were three copies

src/Otl.php:359 read if ($sbl && $sbl != 40 && $sbl != 102) — SCRIPT_INHERITED and
SCRIPT_UNKNOWN as bare numbers, on the line that decides which script starts a run and therefore
which shaper a character reaches. Otl::analyseCharacters()'s loop is copy-pasted verbatim into
src/Mpdf.php:26932 and src/Image/Svg.php:3343, and both carried the same two literals. Fixing one of
three would be worse than fixing none, so all three now name the constants. Behaviour-identical — the
numbers held — and one grep now finds every site.

The rule is still stated in four places and the three src/ sites are the same ~20-line loop. That is a
refactor of its own: #266.

Two tests this invalidated

• UcdnTablesTest::testANewScriptIsAppendedAndTheRestKeepTheirNumbers hardcoded Adlam 175 and Toto 176,
  which is one past the last number in use and therefore moves at every Unicode release. It now derives
  the next number. Every failure the literals caught is still caught — appending into the holes Adlam
  (103) and Toto (168) leave, appending in read order, appending with a gap. SCRIPT_UNKNOWN === 102
  stays hardcoded as the stability anchor. Verified passing at both 17 and 18.
• ArabicJoiningTest's Unknown-script case used U+10EE8 to stand for "a joining type Ucdn's script table
  has not caught up with". At 18 that character has a script, so the case went dead while staying green.
  It is U+FDD0 now, a noncharacter Unicode's stability policy will never assign, so it holds at every
  release. Mutation-checked: removing the SCRIPT_UNKNOWN clause from inScope() fails it.

Verified

composer test                                        2534 tests, 6712 assertions, 1 skipped, OK
composer test -- --group=snapshot                    79 tests, 109 assertions, OK
composer cs                                          clean
phpstan --no-progress --memory-limit=2G              33 errors, all pre-existing

Base at #251's tip: 2532 / 6702 / 3 skipped. The one remaining skip is UcdnAgainstIcuTest, for the
reason above.

No golden master and no snapshot moved. All four generators were re-run over the whole corpus after
deleting tests/Mpdf/tmp/mpdf; every fixture was rewritten and every one came back byte-identical.

utils/data/ucd/18.0.0/ is gitignored and not committed, as the other generators' sources are.

🤖 Generated with Claude Code