Repository navigation
decode: do not split charset decoding across the first read - #535
Merged
imroc merged 1 commit intoSep 6, 2026
Merged
Conversation
autoDecodeReadCloser peeked the first chunk with Decoder.Bytes and decoded the remainder with Decoder.Reader. A multi-byte character that straddled that boundary was treated as a complete input, so HTML meta charset sniffing could corrupt the body. Replay the first chunk through the same decoder stream. Fixes imroc#525
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #525.
When the charset is only declared in an HTML
<meta>tag,autoDecodeReadCloser.peekReaddecoded the first body chunk withDecoder.Bytesand the remainder withDecoder.Reader.Decoder.Bytestreats its input as a complete sequence, so a multi-byte character that straddles that first read (512 bytes forio.ReadAll) is replaced and the leftover trailing byte then starts the streaming decoder mid-character.The sniffing read is now replayed through the same decoder as the rest of the body via
io.MultiReader, which is the approach suggested on the issue. A regression test places GBK中国across the 512-byte boundary and checks that the decoded body still contains中国.