Skip to content

Fix \X after regional indicators and prepend characters - #1056

Open
wleeaf wants to merge 1 commit into
PCRE2Project:mainfrom
wleeaf:fix-extuni-gbtable
Open

wleeaf wants to merge 1 commit into
PCRE2Project:mainfrom
wleeaf:fix-extuni-gbtable

Conversation

@wleeaf

@wleeaf wleeaf commented Oct 26, 2026

Copy link
Copy Markdown

I checked \X against GraphemeBreakTest.txt from Unicode 17.0.0, which matches PCRE2's unicode_version. I used the 32-bit library and matched an anchored \X from each boundary in turn, with pcre2_match (interpreter and JIT) and with pcre2_dfa_match. 43 of the 766 lines failed, the same ones in all three. Two rows of PRIV(ucp_gbtable) in pcre2_tables.c cause 27 of them:

  • The Regional Indicator row has only the RI bit. \X therefore ends the cluster before an Extend, SpacingMark or ZWJ character that follows a regional indicator. UAX Update to Unicode 14.0.0 #29 rules GB9 and GB9a say not to break there. So does pcre2pattern ("Do not end before extending characters or spacing marks or the zero-width joiner (ZWJ) character"). Examples:
    • \x{1F1E6}\x{300} gives two clusters instead of one.
    • In \x{1F1E6}\x{1F1E7}\x{200d}\x{1F1E8}, the ZWJ ends up on its own instead of joining the flag.
  • The Prepend row has every bit except Extended_Pictographic. GB9b says not to break after a prepend character, and pcre2pattern says "Do not end after prepend characters". So \x{6dd}\x{a9} (ARABIC END OF AYAH, COPYRIGHT SIGN) gives two clusters instead of one.

The change adds ESZ to the Regional Indicator row and the Extended_Pictographic bit to the Prepend row. PRIV(extuni), the matcher's own \X loop and the JIT read the same table, so all engines change together. The RI pairing code is unchanged: a following Extend becomes the new left property, so a regional indicator after it still breaks.

The other 16 failing lines all involve GB9c, the Indic conjunct rule from Unicode 15.1, which PCRE2 doesn't implement (pcre2pattern doesn't list it). Supporting it would need the Indic_Conjunct_Break property in the UCD tables, so I left it out.

Tests:

  • testinput4 and testinput7 (DFA) get the same new subject lines in the ^\X block:
    • a regional indicator followed by extend, spacing mark and ZWJ;
    • prepend followed by Extended_Pictographic, other and CR.
  • The new lines replace the subject line "There are no Prepend characters, so we can't test Prepend, CR". Prepend characters such as U+0600 and U+06DD exist now.
  • With the old table, the RI and Prepend + Extended_Pictographic lines produce different output. The Prepend + other and Prepend + CR lines are there for coverage and pass either way.
  • Perl 5.38.2's \X gives the same clusters for all six new subjects, so test 4 stays Perl-compatible.

Configurations run:

  • gcc 13 Release, 8/16/32-bit with JIT: ctest passes all 4 tests, and RunTest 4 7 passes with and without JIT.
  • gcc 13 Debug with -fsanitize=address,undefined: RunTest 4 7 passes for all three widths.
  • I did not run the clang devcontainer presets or Valgrind.

ChangeLog: entry 4 under 10.50.

Two rows of the grapheme break table didn't follow UAX PCRE2Project#29:

- The Regional Indicator row lacked Extend, SpacingMark and ZWJ, so \X
  ended the cluster before them (GB9, GB9a): U+1F1E6 U+0300 gave two
  clusters.
- The Prepend row lacked Extended_Pictographic (GB9b): U+06DD U+00A9 gave
  two clusters.

The interpreter, DFA and JIT all read this table. GraphemeBreakTest.txt
17.0.0 had 43 failing lines; 16 remain, all for the Indic conjunct rule
GB9c, which PCRE2 doesn't implement.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant