[mabrarov]

Implementation of Upstream feature for the Elasticsearch output plugin.

Refer to fluent/fluent-bit#7608 (replaced with mabrarov/fluent-bit#1) for the changes in Fluent Bit code.

Summary by CodeRabbit

• Documentation
  • Added Elasticsearch to the list of plugins that support Upstream.
  • Expanded Elasticsearch docs with full Upstream configuration support and examples for Classic mode and YAML, including sample upstream server layouts.
  • Added per-key override guidance and a dedicated overrides table for Upstream NODE usage.
  • Added AWS SigV4 authentication guidance and notes on how AWS/TLS settings interact with Upstream overrides.
  • Clarified write-operation behavior and field-name handling in Upstream contexts.
• Style
  • Added "AWS SigV4" to heading-capitalization exceptions.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[mabrarov]

This pull request is waiting for review of fluent/fluent-bit#7608

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[cnorris-cs]

I left suggestions, but I've added my technical writing-based approval based on that to keep this moving.

Primarily, I was looking to help the table be more consistent and readable, while implementing some style-based suggestions for consistency.

[mabrarov]

Hi @cnorris-cs,

I applied some (!) parts of your suggestions as well as introduced additional changes for consistency with your suggestions. Please take a look at this pull request one more time when you have time. Some of the changes are implemented in dedicated commits (for ease of revert in case of concerns) - please let me know if I need to squash all commits into a single one.

Thank you for your review.

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@pipeline/outputs/elasticsearch.md`:
- Around line 207-218: The YAML example uses an unquoted asterisk for the match
value ("match: *"), which YAML treats as an anchor reference; update the example
so the match key uses a quoted literal asterisk (e.g., change match: * to match:
"*" ) to ensure the YAML parses as intended and preserves the wildcard
semantics.
- Around line 226-249: The YAML has an invalid unquoted wildcard in the pipeline
output config: change the match field from an unquoted asterisk to a quoted
string (e.g., change match: * to match: "*" or match: "'*'") in the pipeline
outputs block so the YAML parser accepts the wildcard value; update the pipeline
outputs entry with the quoted wildcard for the match key.

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d8aa628 and 9f9a5ee.

📒 Files selected for processing (2)

• administration/configuring-fluent-bit/classic-mode/upstream-servers.md
• pipeline/outputs/elasticsearch.md

🧰 Additional context used

🧠 Learnings (1)

📚 Learning: 2025-12-12T14:30:18.461Z

Learnt from: kalavt
Repo: fluent/fluent-bit-docs PR: 2294
File: pipeline/inputs/kafka.md:147-168
Timestamp: 2025-12-12T14:30:18.461Z
Learning: In Fluent Bit v4.0.4+, when using AWS MSK IAM authentication (rdkafka.sasl.mechanism: aws_msk_iam), the rdkafka.security.protocol is automatically set to SASL_SSL and the AWS region is auto-detected from the broker hostname for standard MSK endpoints. The aws_msk_iam_cluster_arn parameter was removed - users only need to set rdkafka.sasl.mechanism: aws_msk_iam (and optionally aws_region for custom DNS/PrivateLink).

Applied to files:

• pipeline/outputs/elasticsearch.md

🔇 Additional comments (7)

administration/configuring-fluent-bit/classic-mode/upstream-servers.md (1)

8-8: LGTM! Elasticsearch correctly added to Upstream-supported plugins.

The addition properly documents Elasticsearch as an Upstream-capable plugin, consistent with the extensive Upstream configuration documentation added in the elasticsearch.md file.

pipeline/outputs/elasticsearch.md (6)

13-20: LGTM! Clear documentation of Upstream override capability.

The introduction and table header effectively explain which configuration parameters can be overridden in Upstream NODE sections.

---

21-58: LGTM! Comprehensive override documentation for Elasticsearch parameters.

The "Allows overrides" column correctly identifies which parameters can be overridden in Upstream NODE sections, with helpful explanations for Host and Port requirements.

---

66-68: LGTM! Important clarification about AWS Sigv4 and Upstream interaction.

This section correctly distinguishes between TLS parameters for AWS authentication (not overridable in NODE) and TLS parameters for Elasticsearch connections (overridable in NODE), preventing potential configuration confusion.

---

163-163: Note: Vale style check flagged this heading.

Per the PR objectives, Vale reported that this heading should use sentence-style capitalization. However, "Upstream" appears to be used as a proper noun (feature name) throughout the documentation, so the current capitalization may be intentional. Please verify whether this should be "Classic mode configuration file with upstream" or if "Upstream" should remain capitalized as a feature name.

---

165-201: LGTM! Well-documented Classic mode Upstream configuration.

The Classic mode configuration examples and Upstream server definitions are syntactically correct and clearly demonstrate how to configure Elasticsearch with Upstream load balancing.

---

220-224: LGTM! Helpful explanation of YAML Upstream configuration options.

The explanation clearly documents that Upstream configuration can use either Classic mode format or YAML format, and can optionally be embedded in the main configuration file when using YAML.

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

[eschabell]

@mabrarov I ran the two waiting checks and you've got Lint errors to fix before I can review.

[mabrarov]

Hi @eschabell,

@mabrarov I ran the two waiting checks and you've got Lint errors to fix before I can review.

Just curious is it a new linter? The only changes I did today is rebase on master branch.

[mabrarov]

Hi @eschabell,

I ran the two waiting checks and you've got Lint errors to fix before I can review.

I believe I fixed these errors. Could you please run same checks again?

Thank you.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pipeline/outputs/elasticsearch.md`:
- Line 40: In the table entries for the Logstash-related settings, fix two
grammar issues: in the description for `Logstash_Prefix` where the sentence
reads ", For example" change the "For" to lowercase ("for example"), and in the
other Logstash description that currently says "it append" change the verb to
third-person singular "it appends"; search for the strings `Logstash_Prefix` and
`Logstash_Format` to locate and update the two descriptions accordingly.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a09c65d7-1a25-46ef-a453-4c2039fc8dd2

📥 Commits

Reviewing files that changed from the base of the PR and between 2c92d49 and 96f70d0.

📒 Files selected for processing (2)

• pipeline/outputs/elasticsearch.md
• vale-styles/FluentBit/Headings.yml

[eschabell]

@mabrarov now just waiting on code PR merging.

[eschabell]

@mabrarov,

Status update: the Elasticsearch upstream feature has merged, but not through the PR this one references, so the waiting-on-code-merge label is stale and the content needs revalidation.

Three code PRs exist for this feature:

┌───────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────┐
│                        PR                         │                                   State                                   │
├───────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ fluent-bit#1560                                   │ Open since 2019, superseded                                               │
├───────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ fluent-bit#7608 (linked in this PR's description) │ Closed 2026-07-29, never merged                                           │
├───────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ fluent-bit#11647                                  │ Merged 2026-08-03 as fc41bcc69, by @cosmo0920 with @mabrarov as co-author │
└───────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────┘

Because the merged implementation is a reimplementation rather than #7608, the Allows overrides column needs checking against fc41bcc69. In the merged code, a key is overridable only if it is resolved through es_get_property*() in plugins/out_es/es.c, or read via flb_upstream_node_get_property() in plugins/out_es/es_conf.c. Eight keys currently marked Yes aren't:

┌─────────────────┬────────────┬───────────────────────────────────────────────────────────────────┐
│       Key       │ Table says │                            Merged code                            │
├─────────────────┼────────────┼───────────────────────────────────────────────────────────────────┤
│ cloud_auth      │ Yes        │ No — parsed into ctx->cloud_user/cloud_passwd at config time only │
├─────────────────┼────────────┼───────────────────────────────────────────────────────────────────┤
│ include_tag_key │ Yes        │ No — ctx-> only                                                   │
├─────────────────┼────────────┼───────────────────────────────────────────────────────────────────┤
│ tag_key         │ Yes        │ No — ctx-> only                                                   │
├─────────────────┼────────────┼───────────────────────────────────────────────────────────────────┤
│ time_key        │ Yes        │ No — ctx-> only                                                   │
├─────────────────┼────────────┼───────────────────────────────────────────────────────────────────┤
│ time_key_format │ Yes        │ No — ctx-> only                                                   │
├─────────────────┼────────────┼───────────────────────────────────────────────────────────────────┤
│ time_key_nanos  │ Yes        │ No — ctx-> only                                                   │
├─────────────────┼────────────┼───────────────────────────────────────────────────────────────────┤
│ trace_error     │ Yes        │ No — ctx-> only                                                   │
├─────────────────┼────────────┼───────────────────────────────────────────────────────────────────┤
│ trace_output    │ Yes        │ No — ctx-> only                                                   │
└─────────────────┴────────────┴───────────────────────────────────────────────────────────────────┘

The complete set of overridable keys in the merged implementation is:

• Strings (es_get_property, es.c:216-217, 602-665, 1320-1322): path, pipeline, write_operation, id_key, logstash_prefix, logstash_prefix_separator, logstash_dateformat, index, type, http_user, http_passwd, http_api_key
• Booleans (es_get_property_bool, es.c:592-600, 1370): logstash_format, suppress_type_name, replace_dots, current_time_index, generate_id, aws_auth
• Other scalars (es.c:1247-1248): compress, buffer_size
• AWS (es_conf.c:122-151): aws_region, aws_service_name, aws_sts_endpoint, aws_profile, aws_role_arn, aws_external_id
• Record accessor (es_conf.c:323): logstash_prefix_key
• Plus host and port, which are NODE fields

The existing No entries (cloud_id, upstream, workers) are correct, and the AWS SigV4 note still holds — those http_proxy/no_proxy/TLS parameters aren't node-resolved.

Also worth a rebase: pipeline/outputs/elasticsearch.md has drifted since this branch was written, and #2648 touches the same table.

[mabrarov]

I rebased source branch of this PR onto master to match out_es_upstream_support_extended branch (which I also rebased on master branch, including removal of changes made in fluent/fluent-bit#11647) of fluent-bit repository fork.

Making this PR matching master branch of fluent-bit repository requires investigation of fluent/fluent-bit#11647, which I am not an author of and which approach I find debatable due to it seems to contradict with Fluent Bit key feature (lightweight replacement for Fluentd / Filebeat), so I converted this PR into draft.

I'll try to to synchronize this PR with master branch of fluent-bit repository later autumn this year.