Repository navigation
zlib: add crc32c - #66828
Open
pipobscure wants to merge 1 commit into
Open
zlib: add crc32c#66828pipobscure wants to merge 1 commit into
pipobscure wants to merge 1 commit into
Conversation
Add zlib.crc32c(data[, value]), which computes the CRC-32C checksum (Castagnoli polynomial, RFC 3720 section 12.1). CRC-32C has better error detection properties than the CRC-32 of zlib.crc32() and is the checksum required by many current protocols, formats and services, among them: - iSCSI header and data digests (RFC 7143) - SCTP packet checksums (RFC 9260) - iWARP MPA framing (RFC 5044) - NVMe/TCP header and data digests - ext4, Btrfs and XFS metadata checksums - the Snappy framing format - Apache Kafka record batches - LevelDB and RocksDB block checksums - TensorFlow TFRecord files - Google Cloud Storage and Amazon S3 object checksums Implementing any of these in JavaScript currently needs a userland CRC-32C, which is far slower than native code. The function mirrors zlib.crc32(): it accepts a string (encoded as UTF-8) or an ArrayBufferView, takes an optional 32-bit unsigned starting value so a checksum can be continued across chunks, and has a V8 fast API path. The checksum is computed by absl::ExtendCrc32c() from the bundled Abseil, which is already built and linked into node, so this adds no new dependency and no new checksum implementation. absl_crc32c is added to the libraries linked with --shared-abseil. Abseil selects its hardware-accelerated CRC-32C path (SSE4.2 and PCLMULQDQ on x64, the CRC32 and PMULL extensions on arm64) at compile time, so with node's default compiler flags the portable table-driven implementation is used. It runs at about the same speed as zlib.crc32() (about 6.5 GB/s on a recent x64 machine), while the hardware path reaches about 38 GB/s. Builds whose target CPU baseline already includes those instructions get the hardware path without changes. Enabling it for generic builds is left out on purpose: on x64, Abseil uses the accelerated code for unrecognized CPUs without a runtime feature check, so compiling it in unconditionally would fault on CPUs without SSE4.2. Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
Collaborator
|
Review requested:
|
nodejs-github-bot
added
c++
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #66828 +/- ##
==========================================
- Coverage 90.49% 90.49% -0.01%
==========================================
Files 797 797
Lines 277686 277824 +138
Branches 53362 53376 +14
==========================================
+ Hits 251292 251413 +121
+ Misses 16796 16795 -1
- Partials 9598 9616 +18
🚀 New features to boost your workflow:
|
Contributor
Author
richardlau
added
semver-minor
github-actions
Bot
removed
the
request-ci
Collaborator
ShogunPanda
approved these changes
Oct 19, 2026
trivikr
approved these changes
Oct 19, 2026
trivikr
added
the
resume-ci
github-actions
Bot
removed
the
resume-ci
Collaborator
Contributor
Author
|
Awesome. CI passes. This is as ready as I can get it. |
trivikr
added
commit-queue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add zlib.crc32c(data[, value]), which computes the CRC-32C checksum (Castagnoli polynomial, RFC 3720 section 12.1). CRC-32C has better error detection properties than the CRC-32 of zlib.crc32() and is the checksum required by many current protocols, formats and services, among them:
Implementing any of these in JavaScript currently needs a userland CRC-32C, which is far slower than native code.
The function mirrors zlib.crc32(): it accepts a string (encoded as UTF-8) or an ArrayBufferView, takes an optional 32-bit unsigned starting value so a checksum can be continued across chunks, and has a V8 fast API path.
The checksum is computed by absl::ExtendCrc32c() from the bundled Abseil, which is already built and linked into node, so this adds no new dependency and no new checksum implementation. absl_crc32c is added to the libraries linked with --shared-abseil.
Abseil selects its hardware-accelerated CRC-32C path (SSE4.2 and PCLMULQDQ on x64, the CRC32 and PMULL extensions on arm64) at compile time, so with node's default compiler flags the portable table-driven implementation is used. It runs at about the same speed as zlib.crc32() (about 6.5 GB/s on a recent x64 machine), while the hardware path reaches about 38 GB/s. Builds whose target CPU baseline already includes those instructions get the hardware path without changes. Enabling it for generic builds is left out on purpose: on x64, Abseil uses the accelerated code for unrecognized CPUs without a runtime feature check, so compiling it in unconditionally would fault on CPUs without SSE4.2.
I had agentic help here, but I did review and check every single line.