Skip to content

processor: fuse compatible body content modifier chains - #12505

Merged
edsiper merged 13 commits into
masterfrom
perf/5.2-candidates
Oct 26, 2026
Merged

edsiper merged 13 commits into
masterfrom
perf/5.2-candidates

Conversation

@edsiper

@edsiper edsiper commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Compatible native body-edit chains still spend CPU materializing CFL objects after the FTS merge. Fuse a complete compatible segment into one MessagePack decode/edit/encode pass for ordered body insert, upsert, delete and rename actions.

The planner respects starting stages and filter boundaries, checks every unit, acquires stage locks in configuration order, and updates each stage's metrics. Conditions, mixed plugins, group markers, legacy records and unsupported/deep payloads retain the current FTS windowed/CFL path. Partial private output is discarded before fallback.

The rebase preserves FTS's per-invocation codecs, bounded windows, window flags and flb_processor_run_with_count() interface. Known logical counts propagate through raw segments. Fully decoded CFL chains count live records for following stages, fixing stale per-stage input/drop accounting after native drops; that bug reproduces on unmodified post-FTS master.

The SQL envelope fix remains independent in #12506. Worker codec isolation is already in master; #12511 now supplies additional regression tests rather than a production prerequisite.

Validation

  • 58 content-modifier integration cases pass on a build containing this PR's production changes alone, normally and under strict Valgrind.
  • Coverage includes input/output contexts, two-route fan-out, filesystem storage, concurrent workers, ordered dependent edits, conditions, filter boundaries, no-op ownership, native drops, nested/boundary values, fallback after a rewritten prefix, growing maps and grouped envelopes.
  • Memcheck processes report zero errors and zero definite/indirect/possible leaked bytes.
  • Four focused CTest targets pass: processor, processor_conditional, mp_chunk_cobj and processor_content_modifier. Raw/CFL differential tests also verify returned logical counts.
  • The earlier combined review candidate passed 98 integration cases normally and under strict Valgrind, plus 13 focused/broader CTest targets covering FTS composition, routing/chunks and SQL.
  • Full rebased PR range passes the repository prefix linter. Commits remain separated by core, plugin, tests and benchmarks and retain DCO sign-offs.

From the repository root, with FLUENT_BIT_BINARY pointing at this branch's build:

python -m pytest tests/integration/scenarios/processor_content_modifier -q
VALGRIND=1 VALGRIND_STRICT=1 python -m pytest tests/integration/scenarios/processor_content_modifier -q
ctest --test-dir build -R '^flb-(it-(processor|processor_conditional|mp_chunk_cobj)|rt-processor_content_modifier)$' --output-on-failure

Fresh post-FTS measurements

Post-FTS master 05048e35d versus the reviewed candidate, verified -O3 -DNDEBUG, Linux/Ryzen 9 7940HS, CPU 7 affinity, alternating fresh processes, four clients, 1,000 records per batch and 10,000 warmup records. Confirmation uses five pairs per case, 800,000 measured records/run for 12 fields and 200,000 for 128 fields.

Fields Upserts CPU s/M master → candidate CPU reduction Throughput change
12 0 0.8375 → 0.8250 1.5% +0.1%
12 6 3.7375 → 3.1500 15.7% approximately 0%
128 0 7.0000 → 6.8000 2.9% -0.1%
128 6 26.3000 → 21.2000 19.4% +20.0%

These replace the pre-FTS performance claims. Compatible plain-log fusion remains useful; CPU savings do not always increase throughput. Mixed Forward chains with a CFL-only hash tail changed CPU by only 1.2–1.8%, so no meaningful mixed-chain speedup is claimed.

Grouped OTLP declines raw fusion and exercises live counting instead. Initial ingestion controls varied by about ±6%, so not every observed grouped-fallback change is attributed to this PR. No general fallback speedup or memory-saving claim is made. The host was shared; confirmation ran after the competing benchmark and integration tests finished, without reserving a CPU.

Across the initial and confirmation campaigns, all 160 runs delivered exactly 44 million measured records, plus 1.6 million warmup records. The measured review candidate also contained #12506's SQL fix; none of these benchmark pipelines invokes SQL. Binary/payload hashes, per-run CSVs, invocation metadata and the full report are committed under benchmarks/results/post-fts-12505.

python benchmarks/pipeline_compare.py --baseline /path/to/master/fluent-bit \
  --candidate build/bin/fluent-bit --input forward --fields 128 --modifiers 6 \
  --records 200000 --runs 5 --cpu 7 --output wide.csv

Repeat with --modifiers 0 for a control. Set FLB_DEBUG=Off and FLB_RELEASE=Off with CMAKE_BUILD_TYPE=Release, and inspect the actual compiler flags.

No public configuration or bundled-library changes. Stress tests and Memcheck are not a race-detector proof. Target: master.

@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@edsiper edsiper added this to the Fluent Bit v5.2.0 milestone Oct 2, 2026
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
@edsiper
edsiper force-pushed the perf/5.2-candidates branch from 4c27c12 to d9662be Compare October 26, 2026 19:30
@edsiper edsiper changed the title processor: fuse compatible content modifier chains processor: fuse compatible body content modifier chains Oct 26, 2026
@edsiper
edsiper marked this pull request as ready for review October 26, 2026 21:38
@edsiper
edsiper requested a review from cosmo0920 as a code owner October 26, 2026 21:38
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T21:39:58.880576Z d9662be Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@edsiper
edsiper merged commit 42e2226 into master Oct 26, 2026
62 of 65 checks passed
@edsiper
edsiper deleted the perf/5.2-candidates branch October 26, 2026 21:38

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant