Repository navigation
Conversation
load_shex_file() trusted chardet whenever it was > 0.9 confident, but
chardet samples only the start of the data: for FHIR R4's fhir.shex
(690 KB, first non-ASCII byte ~250 KB in) it reports ascii with
confidence 1.0, and decode('ascii') raises UnicodeDecodeError. ShExC is
UTF-8 by definition, so try that first (utf-8-sig, which also strips a
BOM) and only guess for files that aren't valid UTF-8.
The old code also crashed on short non-UTF-8 files: chardet's low
confidence sent them to the UTF-8 default. Now chardet's guess is used
whatever its confidence, with latin-1 as a last resort that can't fail.
Same fix as linkml/PyShEx#117.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
load_shex_file()(inpyshexc/parser_impl/generate_shexj.py) decoded with whatever chardet guessed whenever its confidence was above 0.9. chardet only samples the start of the data, so a large file that is ASCII for a long stretch before its first non-ASCII character gets reported asasciiwith confidence 1.0, and loading crashes. For example, chardet 7.4.3 on FHIR R4's 690 KBfhir.shex:Fix: ShExC is UTF-8 by definition, so decode as UTF-8 first (
utf-8-sig, which also strips a BOM). Only if that fails, use chardet's guess, withlatin-1as a last resort that can't fail. This also fixes a second crash: a short non-UTF-8 file got low confidence from chardet, so the old code sent it to the UTF-8 default and failed. This is the same fix as linkml/PyShEx#117.Tests: new
tests/test_encoding.py, usingtmp_pathso nothing tracked is written:asciierror)utf-8error)Known unrelated failure:
tests/test_shexc.py::TestShExCValidation::test_shex_schemafails onmaintoo (CI has been red since #4). One shexTest schema containing<http://a.example/dir1/dir2/s/p1?q=v>gets a lexertoken recognition error, soparse()returnsNone. All other tests pass: 17 passed, 2 skipped.🤖 Generated with Claude Code