Repository navigation
Conversation
Two rows of the grapheme break table didn't follow UAX PCRE2Project#29: - The Regional Indicator row lacked Extend, SpacingMark and ZWJ, so \X ended the cluster before them (GB9, GB9a): U+1F1E6 U+0300 gave two clusters. - The Prepend row lacked Extended_Pictographic (GB9b): U+06DD U+00A9 gave two clusters. The interpreter, DFA and JIT all read this table. GraphemeBreakTest.txt 17.0.0 had 43 failing lines; 16 remain, all for the Indic conjunct rule GB9c, which PCRE2 doesn't implement.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I checked
\Xagainst GraphemeBreakTest.txt from Unicode 17.0.0, which matches PCRE2'sunicode_version. I used the 32-bit library and matched an anchored\Xfrom each boundary in turn, withpcre2_match(interpreter and JIT) and withpcre2_dfa_match. 43 of the 766 lines failed, the same ones in all three. Two rows ofPRIV(ucp_gbtable)inpcre2_tables.ccause 27 of them:\Xtherefore ends the cluster before an Extend, SpacingMark or ZWJ character that follows a regional indicator. UAX Update to Unicode 14.0.0 #29 rules GB9 and GB9a say not to break there. So does pcre2pattern ("Do not end before extending characters or spacing marks or the zero-width joiner (ZWJ) character"). Examples:\x{1F1E6}\x{300}gives two clusters instead of one.\x{1F1E6}\x{1F1E7}\x{200d}\x{1F1E8}, the ZWJ ends up on its own instead of joining the flag.\x{6dd}\x{a9}(ARABIC END OF AYAH, COPYRIGHT SIGN) gives two clusters instead of one.The change adds
ESZto the Regional Indicator row and the Extended_Pictographic bit to the Prepend row.PRIV(extuni), the matcher's own\Xloop and the JIT read the same table, so all engines change together. The RI pairing code is unchanged: a following Extend becomes the new left property, so a regional indicator after it still breaks.The other 16 failing lines all involve GB9c, the Indic conjunct rule from Unicode 15.1, which PCRE2 doesn't implement (pcre2pattern doesn't list it). Supporting it would need the Indic_Conjunct_Break property in the UCD tables, so I left it out.
Tests:
testinput4andtestinput7(DFA) get the same new subject lines in the^\Xblock:\Xgives the same clusters for all six new subjects, so test 4 stays Perl-compatible.Configurations run:
ctestpasses all 4 tests, andRunTest 4 7passes with and without JIT.-fsanitize=address,undefined:RunTest 4 7passes for all three widths.ChangeLog: entry 4 under 10.50.