fix(compress): read zlib wrapped deflate, and close the body

[Mootjelh]

Two defects in internal/compress/deflate_reader.go, one commit each. No issue open for either, so the measurements are here.

1. A spec conformant deflate body fails the request

RFC 9110 defines Content-Encoding: deflate as the zlib format of RFC 1950. DeflateReader calls flate.NewReader unconditionally, which reads raw RFC 1951, so a zlib body is rejected outright.

Measured through the public API, a httptest server answering Content-Encoding: deflate and a client with EnableAutoDecompress():

                                          master          with this change
zlib  (RFC 1950, what the spec says)   flate: corrupt input        OK
                                       before offset 5
raw   (RFC 1951, common variant)       OK                          OK

So today the conformant encoding is the one that does not work, and the request fails rather than degrades. This is on all three transports, since h1, h2 and h3 all go through compress.NewCompressReader.

The fix sniffs the two header bytes on the first Read, puts them back with io.MultiReader, and picks zlib.NewReader or flate.NewReader. The reader stays lazy, so nothing is buffered and nothing is read at construction time.

On the heuristic. A raw deflate stream can begin with two bytes that satisfy the zlib rule, so this cannot be exact. It is what browsers do, and looksLikeZlib is checked against compress/zlib itself over all 65536 possible two byte headers: both accept exactly the same 66, with no disagreement in either direction. That keeps it from drifting and means no header had to be worked out by hand.

34 of those 66 name a preset dictionary. Those are still zlib headers, so the reader surfaces zlib's own error for them instead of reading the body as raw deflate and returning garbage.

2. Close leaks the response body

func (df *DeflateReader) Close() error {
	if df.dr != nil {
		return df.dr.Close()   // flate's Close does not close df.Body
	}
	return df.Body.Close()
}

flate.NewReader's Close does not close the reader underneath it, so once the body has been read df.Body is never closed and the connection is not released. GzipReader, BrotliReader and ZstdReader in the same package all close theirs, and ZstdReader already does exactly what this changes DeflateReader to do.

TestDeflateReaderCloseClosesTheBody wraps the body in a closer that records the call.

Tests

Three added, each checked against a broken build rather than assumed to work:

mutation	caught by
Read never uses zlib, master's behaviour	TestDeflateReaderReadsBothVariants/zlib
drop the CM check	TestLooksLikeZlibAgreesWithCompressZlib
drop the CINFO check	same
drop the FCHECK check	same
Close as it is on master	TestDeflateReaderCloseClosesTheBody

The first version of the header test did not catch the CM mutation, because its only negative case was one raw stream that failed the CINFO check anyway. Comparing against compress/zlib exhaustively is what fixed that.

go test ./... gives the same two failures with and without the change, TestTraceInfo and TestSetFile. Both are on master here as well and look environment specific to Windows.

If you would rather not

If supporting only raw deflate is deliberate, say so and I will close this. The second commit stands on its own either way.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[imroc]

Verified locally (Go 1.27.0, Linux — the fork CI hasn't run and needs maintainer approval). Both defects are real, and the branch is green: go build ./... && go vet ./... && go test ./... all pass. The TestTraceInfo / TestSetFile failures you mention don't reproduce on Linux, consistent with what we saw on #538 — they look Windows-specific and unrelated to this change.

Bug 1 — zlib framing (295b0a2). Confirmed independently: on master, a Content-Encoding: deflate body in the RFC 1950 zlib form fails (flate: corrupt input), so the spec-conformant variant is the broken one. internal/compress/reader.go:15 wires deflate to NewDeflateReader, so this hits the auto-decompress path on all three transports.

The implementation reads well:

• looksLikeZlib (CM=8, CINFO≤7, FCHECK) is exactly the RFC 1950 header check, and TestLooksLikeZlibAgreesWithCompressZlib pins it against compress/zlib over all 65536 two-byte headers — I re-ran it and it passes, which rules out drift in the heuristic itself.
• Sniffing stays lazy: two bytes via io.ReadFull are pushed back with io.MultiReader before the reader is chosen, nothing is buffered up front.
• Short-body edge cases are handled: a real read error becomes the sticky derr; io.EOF/io.ErrUnexpectedEOF with 0–1 bytes falls through to the raw path, so an empty body still yields a clean EOF.
• The misdetection surface is the same one browsers accept (a raw stream whose first two bytes satisfy the zlib check); zlib's own error surfaces for the FDICT subset instead of silently decoding garbage.

Bug 2 — Close leak (438c877). Confirmed independently: on master, after reading the body, DeflateReader.Close only calls flate's Close, which does not close the underlying reader, so Body is never closed and the connection is never released. The fix matches ZstdReader.Close exactly (dr.Close() then Body.Close()), and GzipReader/BrotliReader in the same package close theirs too, so this brings DeflateReader in line with its siblings. Unambiguous.

I also re-verified both regressions black-box on master code (restored deflate_reader.go, kept equivalent tests): the zlib read fails and the close-tracking test fails, so neither test is vacuous.

On merging. The second commit stands on its own and is safe to take as-is. The first is a framing-policy decision rather than a straight bug fix: RFC 9110 defines deflate as zlib, but raw deflate is what many servers actually send, and sniffing (rather than erroring on one variant) is the browser behavior — as you note, this can't be exact. Since there's no open issue where the desired behavior was settled, I'm leaving the merge decision to the maintainer (@imroc): both fixes are verified, the approach is sound, but whether req should auto-decode both variants with a 2-byte heuristic is a product call. If the answer is yes, this LGTM as-is; if deflate support is meant to stay raw-only, the second commit is still worth taking separately.