Skip to content

fix: decode extractor responses with declared charset - #87

Merged
Priveetee merged 1 commit into
TypeType-Video:devfrom
tam1m:fix/response-charset
Sep 26, 2026
Merged

Priveetee merged 1 commit into
TypeType-Video:devfrom
tam1m:fix/response-charset

Conversation

@tam1m

@tam1m tam1m commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

Searching for anything non-ASCII turned the suggestions into � (U+FFFD) chars. Googles suggestions endpoint answers in a locale dependent charset (ISO-8859-1, Shift_JIS or windows-1251), but OkHttpExtractorResponseMapper decoded every body as UTF-8.

The mapper now decodes with the charset from the response Content-Type, falling back to UTF-8 when nothing is declared, so all other traffic is unchanged.

The mapper sits in front of all extractor traffic, but all the other endpoints already answers in UTF-8 afaik, so really only the suggestions endpoint is effected here.

Added regression tests in OkHttpExtractorResponseMapperTest.

To test the upstream endpoint:

curl -sD /tmp/headers 'https://suggestqueries.google.com/complete/search?client=youtube&jsonp=JP&ds=yt&q=%C3%8B' -o /tmp/body
grep -i content-type /tmp/headers
xxd /tmp/body | head -2
iconv -f iso-8859-1 -t utf-8 /tmp/body | head -c 80
  1. query is Ë (%C3%8B)
  2. Headers say ISO-8859-1
  3. hexdump shows 0xCB byte
  4. iconv decodes it the same way the mapper does now

The charset also seems to follow the requesters locale:

curl -sD - -o /dev/null -H 'Accept-Language: ja' 'https://suggestqueries.google.com/complete/search?client=youtube&ds=yt&q=test' | grep -i content-type
curl -sD - -o /dev/null -H 'Accept-Language: ru' 'https://suggestqueries.google.com/complete/search?client=youtube&ds=yt&q=test' | grep -i content-type

First returns Shift_JIS and second returns windows-1251. Just here to demonstrate locale dependency. We never send one though, so ISO-8859-1 is all we ever see.

@Priveetee

Copy link
Copy Markdown
Member

Thx a lot for ur contribution, merging to dev :) !

@Priveetee
Priveetee merged commit da5ac6f into TypeType-Video:dev Sep 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants