Skip to content

zlib: finish async convenience calls in one write - #66854

Open
anonrig wants to merge 1 commit into
nodejs:mainfrom
anonrig:zlib-perf
Open

anonrig wants to merge 1 commit into
nodejs:mainfrom
anonrig:zlib-perf

Conversation

@anonrig

@anonrig anonrig commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

The async convenience methods (zlib.deflate(), zlib.inflate(), zlib.gunzip(), ...) write the input to a Transform and then end it. That's two threadpool round trips per call, one for the data and one for the final flush, plus the stream machinery on top. In a CPU profile of benchmark/zlib/inflate.js method=inflate, the main thread sits idle about 71% of the time waiting on the threadpool.

We already have the whole input up front, so this hands it to the handle in a single write with the finish flush, the same way the *Sync methods do, and collects the output in the write callback. A call whose output fits in one buffer now takes one round trip. The work still runs on the threadpool.

These stay on the stream path:

  • info: true, since the engine is handed back to the caller.
  • A non-default flush, which can change the output bytes.
  • zstdCompress(). A single ZSTD_e_end write makes frames 3–18 bytes larger than ZSTD_e_continue followed by ZSTD_e_end, so I left it alone.

The output doesn't change. For deflate, gzip, raw deflate and brotli, the old two-write sequence already produced the same bytes as the sync methods. I compared all the async methods plus gzipSync()/deflateSync() on four inputs (source text, text plus random bytes, a short string, empty) against a build of the base commit: 28/28 outputs are byte-identical, and each build decompresses the other's output.

There are two observable differences. The success callback now runs straight from the write completion, so async_hooks sees it in the ZLIB resource instead of a TickObject, which is how fs callbacks behave too. AsyncLocalStorage context is unchanged, and the new test checks it. Errors still arrive through the 'error' event like before. Also, this inside a non-arrow callback is still the internal engine, but it's never ended now, so writableEnded, writableFinished and readableEnded are false (it's still destroyed, with bytesWritten set), and info: true, the documented way to get the engine, keeps the old stream path.

Two new tests:

  • test-zlib-convenience-async.js covers the contract: every input type, interop with the sync methods, invalid and truncated input, maxOutputLength, info, rejectGarbageAfterEnd, dictionaries, finishFlush, and that the callback never runs synchronously. It passes on the base commit too.
  • test-zlib-convenience-async-round-trip.js uses async_hooks to check that a small call takes exactly one round trip. It fails on the base commit, which takes two.

Benchmarks

benchmark/compare.js, 40 runs, n=100000, against a build of the base commit with the same configure flags. The default method/inputLen are kept and only n is scaled down.

benchmark run 1 (loaded machine) run 2 (quiet machine) A/A, quiet
inflate.js method='inflate' +98.11% *** +94.91% *** ±0.59% +0.10%
deflate.js method='deflate' +62.96% *** +66.27% *** ±2.34% +0.34%

--max-regression 5 over deflate.js, inflate.js, pipe.js and creation.js (20 runs) passes. The sync methods, createDeflate writes, pipes and stream creation all stay within about 2%.

compare.js, run 2 (40 runs)
                                                         confidence   improvement   accuracy (*)    (**)   (***)
zlib/inflate.js n=100000 inputLen=1024 method='inflate'  ***            +94.91 %   ±0.59%  ±0.78%  ±1.02%

                                                         confidence   improvement   accuracy (*)    (**)   (***)
zlib/deflate.js n=100000 inputLen=1024 method='deflate'  ***            +66.27 %   ±2.34%  ±3.11%  ±4.06%
compare.js --max-regression 5 (20 runs)
                                                               confidence   improvement   accuracy (*)    (**)   (***)
zlib/deflate.js n=100000 inputLen=1024 method='createDeflate'                  +0.84 %   ±2.03%  ±2.73%  ±3.58%
zlib/deflate.js n=100000 inputLen=1024 method='deflate'        ***            +75.51 %   ±1.87%  ±2.51%  ±3.31%
zlib/deflate.js n=100000 inputLen=1024 method='deflateSync'                    +0.12 %   ±1.06%  ±1.42%  ±1.86%

                                                             confidence   improvement   accuracy (*)    (**)   (***)
zlib/inflate.js n=100000 inputLen=1024 method='inflate'      ***            +95.85 %   ±3.53%  ±4.79%  ±6.43%
zlib/inflate.js n=100000 inputLen=1024 method='inflateSync'                  -1.56 %   ±3.28%  ±4.41%  ±5.84%

                                                                        confidence   improvement   accuracy (*)    (**)   (***)
zlib/pipe.js algorithm='gzip' type='string' duration=1 inputLen=1024                    +1.74 %   ±2.32%  ±3.13%  ±4.18%
zlib/pipe.js algorithm='brotli' type='string' duration=1 inputLen=1024                  +0.12 %   ±3.43%  ±4.59%  ±6.04%
zlib/pipe.js algorithm='zstd' type='string' duration=1 inputLen=1024                    -0.15 %   ±1.40%  ±1.88%  ±2.47%
zlib/pipe.js algorithm='gzip' type='buffer' duration=1 inputLen=1024                    -0.90 %   ±1.75%  ±2.35%  ±3.10%
zlib/pipe.js algorithm='brotli' type='buffer' duration=1 inputLen=1024                  +0.74 %   ±1.50%  ±2.06%  ±2.79%
zlib/pipe.js algorithm='zstd' type='buffer' duration=1 inputLen=1024      *             -1.54 %   ±1.16%  ±1.55%  ±2.05%

                                                                   confidence   improvement   accuracy (*)    (**)   (***)
zlib/creation.js n=500000 options='true' type='Deflate'                            +0.49 %   ±1.14%  ±1.53%  ±2.01%
zlib/creation.js n=500000 options='false' type='Deflate'                           -0.43 %   ±0.72%  ±0.96%  ±1.26%
zlib/creation.js n=500000 options='true' type='DeflateRaw'           *             -2.01 %   ±1.55%  ±2.10%  ±2.81%
zlib/creation.js n=500000 options='false' type='DeflateRaw'                        +0.81 %   ±1.64%  ±2.23%  ±2.98%
zlib/creation.js n=500000 options='true' type='Inflate'                            -0.19 %   ±1.01%  ±1.36%  ±1.80%
zlib/creation.js n=500000 options='false' type='Inflate'                           +1.17 %   ±1.68%  ±2.27%  ±3.02%
zlib/creation.js n=500000 options='true' type='InflateRaw'                         -1.01 %   ±2.40%  ±3.23%  ±4.25%
zlib/creation.js n=500000 options='false' type='InflateRaw'                        +0.72 %   ±1.35%  ±1.82%  ±2.41%
zlib/creation.js n=500000 options='true' type='Gzip'                               +0.91 %   ±1.44%  ±1.93%  ±2.56%
zlib/creation.js n=500000 options='false' type='Gzip'                              +0.34 %   ±1.60%  ±2.16%  ±2.87%
zlib/creation.js n=500000 options='true' type='Gunzip'                             -0.17 %   ±1.15%  ±1.55%  ±2.04%
zlib/creation.js n=500000 options='false' type='Gunzip'                            +0.61 %   ±1.74%  ±2.33%  ±3.06%
zlib/creation.js n=500000 options='true' type='Unzip'                              -1.05 %   ±1.72%  ±2.31%  ±3.06%
zlib/creation.js n=500000 options='false' type='Unzip'                             +0.09 %   ±0.97%  ±1.31%  ±1.74%
zlib/creation.js n=500000 options='true' type='BrotliCompress'                     -2.17 %   ±3.48%  ±4.74%  ±6.39%
zlib/creation.js n=500000 options='false' type='BrotliCompress'                    -1.24 %   ±2.90%  ±3.92%  ±5.25%
zlib/creation.js n=500000 options='true' type='BrotliDecompress'                   -0.96 %   ±2.39%  ±3.21%  ±4.23%
zlib/creation.js n=500000 options='false' type='BrotliDecompress'                  -1.13 %   ±2.70%  ±3.62%  ±4.76%
zlib/creation.js n=500000 options='true' type='ZstdCompress'                       +0.13 %   ±1.42%  ±1.91%  ±2.52%
zlib/creation.js n=500000 options='false' type='ZstdCompress'                      -0.24 %   ±1.38%  ±1.85%  ±2.44%
zlib/creation.js n=500000 options='true' type='ZstdDecompress'                     -0.28 %   ±2.36%  ±3.17%  ±4.21%
zlib/creation.js n=500000 options='false' type='ZstdDecompress'                    +0.13 %   ±1.49%  ±2.00%  ±2.63%

Below, Chrome talks to two local servers, one on the base build and one on this PR. It decodes zlib.gzip()/zlib.deflate()/zlib.brotliCompress() responses, and sends CompressionStream uploads that the servers decode with zlib.gunzip()/zlib.inflate(). The payloads match on both, the round trip count drops from 2 to 1, and sequential zlib.inflate() calls get faster.

Chrome checking both builds: gzip, deflate and brotli responses decode to the same bytes, CompressionStream uploads decode with zlib.gunzip()/zlib.inflate(), one zlib.inflate() takes 2 threadpool round trips on the base build and 1 on this PR, and 20,000 sequential 1 KiB zlib.inflate() calls run at 59,335 vs 111,603 ops/s

The asynchronous convenience methods (`zlib.deflate()`,
`zlib.inflate()`, `zlib.gunzip()`, ...) wrote the input to a
Transform stream and then ended it. That took two threadpool round
trips, one for the data and one for the final flush, and ran the
result through the stream machinery.

The whole input is known up front, so unless `info` or a custom
`flush` is requested, process it with the finish flush directly, the
way the synchronous methods already do, and collect the output in the
write callback. A call whose output fits in one buffer now takes a
single threadpool round trip. The output is unchanged: the previous
two-write sequence already produced the same bytes as the synchronous
methods. `zstdCompress()` keeps the separate end-of-frame write,
because a single `ZSTD_e_end` call can produce slightly larger frames.

benchmark/compare.js, 40 runs, n=100000:

                                          confidence  improvement
    zlib/inflate.js method='inflate'      ***          +94.91 %
    zlib/deflate.js method='deflate'      ***          +66.27 %

Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>
Assisted-by: Grok
@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. zlib Issues and PRs related to the zlib module and its compression dependencies. labels Oct 19, 2026
@anonrig
anonrig requested a review from mcollina October 19, 2026 16:26
@anonrig
anonrig requested a review from ronag October 19, 2026 16:28
@anonrig anonrig added the semver-minor PRs that contain new features and should be released in the next minor version. label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ci PRs that need a full CI run. semver-minor PRs that contain new features and should be released in the next minor version. zlib Issues and PRs related to the zlib module and its compression dependencies.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants