GH-50944: [C++] Replace RapidJSON with simdjson in JSON chunker by Reranko05 · Pull Request #50945 · apache/arrow

Reranko05 · 2026-08-21T13:39:30Z

Rationale for this change

This PR continues the simdjson migration by replacing the RapidJSON-based JSON boundary detection used by the JSON chunker.

The existing implementation uses RapidJSON's streaming parser to identify complete JSON values. This change replaces that logic with structural boundary detection and simdjson validation.

Changes

Replace the RapidJSON custom stream and boundary detection.
Detect complete JSON values with structural scanning.
Validate complete candidates with simdjson::dom::parser.
Preserve incomplete-value, block-boundary, and error handling behavior.
Update the affected error expectation.

Fixes: #50944

GitHub Issue: [C++] Replace RapidJSON with simdjson in JSON chunker #50944

Reranko05 · 2026-08-21T14:05:22Z

While working on this migration, @rok's earlier implementation rok#47 of the simdjson-based JSON chunker. It was very helpful. Thanks :)

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

pitrou

I don't understand why this is parsing JSON by hand?

It seems that we might be able to use simdjson::ondemand::parser::iterate_many.

pitrou · 2026-08-24T09:29:09Z


 namespace arrow {

 using std::string_view;


Can we remove this if it's not useful anymore?

pitrou · 2026-08-24T09:31:16Z

+    if (escape_next) {
+      escape_next = false;
+      continue;
+    }
+
+    if (c == '\\' && in_string) {
+      escape_next = true;
+      continue;
+    }


This does not account for unicode escapes. Do we have to parse JSON by hand like this?

pitrou · 2026-08-24T09:34:22Z

+    std::string combined;
+    combined.reserve(partial.size() + block.size());
+    combined.append(partial);
+    combined.append(block);


We should only concatenate if both substrings are non-empty.

pitrou · 2026-08-24T09:35:07Z

-      return Status::Invalid("JSON chunk error: invalid data at end of document");
+      return Status::Invalid("JSON parse error: Invalid value");


Why not keep the error message?

pitrou · 2026-08-24T09:36:43Z

+        const size_t start = ConsumeWhitespace(block);
+
+        if (start < block.size()) {
+          const char first_char = block[start];
+
+          if (first_char != '{' && first_char != '[') {
+            const size_t remaining_len = block.size() - start;
+
+            if (remaining_len > 1 || (first_char != '}' && first_char != ']')) {
+              return Status::Invalid("JSON parse error: Invalid value");
+            }
+          }
+        }


Please add comments explaining what this does and why it is necessary.

Reranko05 · 2026-08-24T10:05:59Z

I don't understand why this is parsing JSON by hand?
It seems that we might be able to use simdjson::ondemand::parser::iterate_many.

I initially tried using simdjson::ondemand::parser::iterate_many, but after a few attempts I ran into boundary-handling issues with the chunker semantics. I then referred to an earlier implementation by @rok as a reference and followed that approach.

That said, I agree that parsing JSON manually here is not ideal. I'll revisit this using iterate_many and address the other review comments as well.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-08-25T12:46:28Z

@pitrou I tried using simdjson::ondemand::parser::iterate_many(). Most tests pass, but PropagateErrorsNonLinewiseChunker behaves differently: malformed input is split into a separate document and the error is then reported by the JSON parser, whereas the current implementation reports a chunker error.

Manual structural parsing approach preserved the existing test behavior. @rok, since your earlier implementation was helpful here, do you have any suggestions on how to preserve the current error behavior with iterate_many()?

pitrou · 2026-08-25T12:51:17Z

Most tests pass, but PropagateErrorsNonLinewiseChunker behaves differently: malformed input is split into a separate document and the error is then reported by the JSON parser, whereas the current implementation reports a chunker error.

As long as an error is reported while reading the JSON stream, I don't think we care if it's reported by the chunker or the parser.

Reranko05 · 2026-08-26T11:33:30Z

@pitrou I gave iterate_many() more that a few attempts, but I am running into semantic differences with the previous RapidJSON implementation, especially around stopping after a complete value when it is followed by a partial or malformed value. At this point, I think manually finding the boundary of the first complete object/array and then validating that slice with simdjson may be simpler and closer to the existing behavior. Do you think that would be a reasonable approach?

pitrou · 2026-08-26T11:43:13Z

@pitrou I gave iterate_many() more that a few attempts, but I am running into semantic differences with the previous RapidJSON implementation, especially around stopping after a complete value when it is followed by a partial or malformed value.

Hmm, I see. Thanks for trying anyway :-)

At this point, I think manually finding the boundary of the first complete object/array and then validating that slice with simdjson may be simpler and closer to the existing behavior. Do you think that would be a reasonable approach?

Well, as a last resort, yes. The problem:

We're writing our own JSON parser, which means we must careful test it.
We're losing performance unless we implement our own SIMD optimizations.

Thoughts @HuaHuaY @cyb70289 ?

cyb70289 · 2026-08-27T02:47:23Z

Try to understand the issue. Is it that json strings legal for rapidjson may fail on simdjson, makes future Arrow release potentially incompatible to old version?

Writing our own optimized version looks not ideal. Can we just use simdjson? It's state-of-the-art, and even with self written object delimiter, there's still incompatibility risk I'm afraid.

Reranko05 · 2026-08-27T08:18:07Z

@cyb70289 I don't think the issue is JSON compatibility between RapidJSON and simdjson. The main issue I ran into is the boundary/streaming semantics.

The previous RapidJSON implementation uses kParseStopWhenDoneFlag, so it stops as soon as the first complete top-level value is parsed, even if the remaining input contains a partial or malformed value.

With iterate_many(), I wasn't able to reproduce that behavior: errors in subsequent/incomplete data can affect whether we successfully identify the preceding complete value.

I agree that writing our own optimized JSON parser would not be ideal. The manual approach I used, and which @rok also implemented in rok#47, only scans for the boundary of the first complete object or array while respecting strings and escapes, and then lets simdjson perform the actual JSON validation. But I agree this still introduces complexity and needs careful testing.

If there is a way to use simdjson directly while preserving the old stop-after-one-value semantics, that would definitely be preferable.

pitrou · 2026-08-27T09:06:21Z

With iterate_many(), I wasn't able to reproduce that behavior: errors in subsequent/incomplete data can affect whether we successfully identify the preceding complete value.

Is that a problem? We want to keep compatibility when parsing valid JSON streams. The failure mode for an invalid JSON stream can change.

cyb70289 · 2026-08-27T09:27:33Z

A discussion about ignoring trailing garbage in simdjson. Looks there're real use cases lenient parsing can be useful.
simdjson/simdjson#2502

pitrou · 2026-08-27T09:48:52Z

Trailing garbage is not the problem here. We are parsing a stream of valid JSON documents. We are happy to error out on trailing garbage.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions Bot added Component: C++ awaiting review Awaiting review labels Aug 21, 2026

Reranko05 added CI: Extra: C++ Run extra C++ CI and removed awaiting review Awaiting review labels Aug 21, 2026

Reranko05 marked this pull request as ready for review August 21, 2026 16:33

Reranko05 requested a review from pitrou as a code owner August 21, 2026 16:33

Copilot AI lite review requested due to automatic review settings August 21, 2026 16:33

Copilot AI reviewed Aug 21, 2026

Reranko05 requested review from kou and rok August 21, 2026 16:33

github-actions Bot added the awaiting review Awaiting review label Aug 21, 2026

Replace RapidJSON with simdjson in JSON chunker

be4c2a1

Reranko05 force-pushed the gh-35460-chunker branch from e9fc9fe to be4c2a1 Compare August 24, 2026 07:52

pitrou requested changes Aug 24, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 24, 2026

WIP: preserve chunker behavior before iterate_many

aa6e20d

Copilot AI review requested due to automatic review settings August 24, 2026 20:06

Copilot AI reviewed Aug 24, 2026

WIP: Iterate Many

a5b89a7

Copilot AI review requested due to automatic review settings August 27, 2026 11:31

Copilot AI reviewed Aug 27, 2026

use iterate_many approach

3f15314

Copilot AI review requested due to automatic review settings August 27, 2026 11:36

Reranko05 force-pushed the gh-35460-chunker branch from 227961f to 3f15314 Compare August 27, 2026 11:36

Copilot AI reviewed Aug 27, 2026

Copilot AI review requested due to automatic review settings August 27, 2026 12:33

Reranko05 requested review from AlenkaF and raulcd as code owners August 27, 2026 12:33

Copilot AI reviewed Aug 27, 2026

github-actions Bot added the Component: Python label Aug 27, 2026

Change expectation of python test

b9af8ce

Copilot AI review requested due to automatic review settings August 27, 2026 13:27

Reranko05 force-pushed the gh-35460-chunker branch from 626bb0d to b9af8ce Compare August 27, 2026 13:27

Copilot AI reviewed Aug 27, 2026

		return Status::Invalid("JSON chunk error: invalid data at end of document");
		return Status::Invalid("JSON parse error: Invalid value");

Uh oh!

Conversation

Reranko05 commented Aug 21, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Changes

Uh oh!

Reranko05 commented Aug 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Aug 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

Reranko05 commented Aug 26, 2026

Uh oh!

pitrou commented Aug 26, 2026

Uh oh!

cyb70289 commented Aug 27, 2026

Uh oh!

Reranko05 commented Aug 27, 2026

Uh oh!

pitrou commented Aug 27, 2026

Uh oh!

cyb70289 commented Aug 27, 2026

Uh oh!

pitrou commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Reranko05 commented Aug 21, 2026 •

edited by github-actions Bot

Loading