From 843c859242a25770cddccce12dd6f248c7add8d5 Mon Sep 17 00:00:00 2001 From: Wleeaf <179125857+wleeaf@users.noreply.github.com> Date: Mon, 26 Oct 2026 16:11:09 +0000 Subject: [PATCH] Fix \X after regional indicators and prepend characters Two rows of the grapheme break table didn't follow UAX #29: - The Regional Indicator row lacked Extend, SpacingMark and ZWJ, so \X ended the cluster before them (GB9, GB9a): U+1F1E6 U+0300 gave two clusters. - The Prepend row lacked Extended_Pictographic (GB9b): U+06DD U+00A9 gave two clusters. The interpreter, DFA and JIT all read this table. GraphemeBreakTest.txt 17.0.0 had 43 failing lines; 16 remain, all for the Indic conjunct rule GB9c, which PCRE2 doesn't implement. --- ChangeLog | 5 +++++ src/pcre2_tables.c | 4 ++-- testdata/testinput4 | 9 ++++++++- testdata/testinput7 | 9 ++++++++- testdata/testoutput4 | 25 +++++++++++++++++++++++-- testdata/testoutput7 | 21 ++++++++++++++++++++- 6 files changed, 66 insertions(+), 7 deletions(-) diff --git a/ChangeLog b/ChangeLog index bdc6c5e93..3e3254370 100644 --- a/ChangeLog +++ b/ChangeLog @@ -22,6 +22,11 @@ backtracking after the recursive call. 3. (#1040) Fixed the start_match field passed to interpreter callouts after \K resets the start of a match, including restoration on backtracking. +4. Fixed \X so that an extended grapheme cluster continues after a regional +indicator when extending characters, spacing marks or ZWJ follow (GB9 and GB9a +in UAX #29), and after a prepend character when an Extended_Pictographic +character follows (GB9b). Previously \X ended the cluster in both places. + Version 10.49 28-September-2026 ------------------------------- diff --git a/src/pcre2_tables.c b/src/pcre2_tables.c index dedf0d3d5..f8c4cedf6 100644 --- a/src/pcre2_tables.c +++ b/src/pcre2_tables.c @@ -207,7 +207,7 @@ const uint32_t PRIV(ucp_gbtable)[] = { ESZ|(1u<\X{2})X/utf,aftertext \x{1111}\x{ae4c}\x{1111}\x{ae4c}X diff --git a/testdata/testoutput4 b/testdata/testoutput4 index bdb19fd7d..b5a989b15 100644 --- a/testdata/testoutput4 +++ b/testdata/testoutput4 @@ -2733,9 +2733,30 @@ No match \x{261d}\x{200d}\x{1f308}B Extended_Pictographic ZWJ Extended_Pictographic 0: \x{261d}\x{200d}\x{1f308} 0+ B Extended_Pictographic ZWJ Extended_Pictographic - *There are no Prepend characters, so we can't test Prepend, CR + *Test regional indicators followed by extend, spacing mark and ZWJ 0: * - 0+ There are no Prepend characters, so we can't test Prepend, CR + 0+ Test regional indicators followed by extend, spacing mark and ZWJ + \x{1F1E6}\x{300}B RegionalIndicator Extend + 0: \x{1f1e6}\x{300} + 0+ B RegionalIndicator Extend + \x{1F1E6}\x{903}B RegionalIndicator SpacingMark + 0: \x{1f1e6}\x{903} + 0+ B RegionalIndicator SpacingMark + \x{1F1E6}\x{1F1E7}\x{200d}\x{1F1E8}B RegionalIndicator RegionalIndicator ZWJ RegionalIndicator + 0: \x{1f1e6}\x{1f1e7}\x{200d} + 0+ \x{1f1e8}B RegionalIndicator RegionalIndicator ZWJ RegionalIndicator + *Test Prepend + 0: * + 0+ Test Prepend + \x{6dd}\x{a9}B Prepend Extended_Pictographic + 0: \x{6dd}\x{a9} + 0+ B Prepend Extended_Pictographic + \x{600}AB Prepend Other + 0: \x{600}A + 0+ B Prepend Other + \x{600}\x{0d}B Prepend CR + 0: \x{600} + 0+ \x{0d}B Prepend CR /^.\X/utf,aftertext A\x{1F1E6}\x{1F1E7}\x{1F1E8}B diff --git a/testdata/testoutput7 b/testdata/testoutput7 index aa36e9c67..21fb78cc1 100644 --- a/testdata/testoutput7 +++ b/testdata/testoutput7 @@ -3036,7 +3036,26 @@ Partial match: AA \x09\x{1b04}Control, spacingmark 0: \x{09} 0+ \x{1b04}Control, spacingmark -\= There are no Prepend characters, so we can't test Prepend, CR +\= Test regional indicators followed by extend, spacing mark and ZWJ + \x{1F1E6}\x{300}B RegionalIndicator Extend + 0: \x{1f1e6}\x{300} + 0+ B RegionalIndicator Extend + \x{1F1E6}\x{903}B RegionalIndicator SpacingMark + 0: \x{1f1e6}\x{903} + 0+ B RegionalIndicator SpacingMark + \x{1F1E6}\x{1F1E7}\x{200d}\x{1F1E8}B RegionalIndicator RegionalIndicator ZWJ RegionalIndicator + 0: \x{1f1e6}\x{1f1e7}\x{200d} + 0+ \x{1f1e8}B RegionalIndicator RegionalIndicator ZWJ RegionalIndicator +\= Test Prepend + \x{6dd}\x{a9}B Prepend Extended_Pictographic + 0: \x{6dd}\x{a9} + 0+ B Prepend Extended_Pictographic + \x{600}AB Prepend Other + 0: \x{600}A + 0+ B Prepend Other + \x{600}\x{0d}B Prepend CR + 0: \x{600} + 0+ \x{0d}B Prepend CR /^(?>\X{2})X/utf,aftertext \x{1111}\x{ae4c}\x{1111}\x{ae4c}X