diff --git a/ARCHIVED_WORKSHOPS.md b/ARCHIVED_WORKSHOPS.md index ef22b99934..e7cb95aa68 100644 --- a/ARCHIVED_WORKSHOPS.md +++ b/ARCHIVED_WORKSHOPS.md @@ -9,3 +9,6 @@ must not be deployed without first reviewing and updating their dependencies. | AWS | [Browse archived files](https://github.com/splunk/observability-workshop/tree/cd5bfe5e9d625d6c4e2b842258f3aa559b9aa07d/workshop/aws) | | GCP | [Browse archived files](https://github.com/splunk/observability-workshop/tree/1e1d80ac4240df97e31543d49868af47715277bb/workshop/gcp) | | Legacy content | [Browse archived files](https://github.com/splunk/observability-workshop/tree/b29d80b77a1ccec7d13ec085ceba3283b037459e/legacy-content) | +| Advanced OpenTelemetry Collector (agent + gateway) | [Browse archived files](https://github.com/splunk/observability-workshop/tree/12db13707828d859f4d201cde2036261ca82fae8/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector) | +| Advanced Collector Configuration (earlier in-tree copy) | [Browse archived files](https://github.com/splunk/observability-workshop/tree/12db13707828d859f4d201cde2036261ca82fae8/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old) | +| Advanced OpenTelemetry Collector lab assets (agent + gateway) | [Browse archived files](https://github.com/splunk/observability-workshop/tree/8332606fb8cf275dcbd563f5523158ed2fda399e/workshop/ninja/advanced-otel) | diff --git a/content/en/conf/3-obs1184/_index.md b/content/en/conf/3-obs1184/_index.md deleted file mode 100644 index e9ad2a0784..0000000000 --- a/content/en/conf/3-obs1184/_index.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Advanced OpenTelemetry Collector - .conf26 -description: Practice reviewing and modifying a host-installed OpenTelemetry Collector configuration. -weight: 3 -type: chapter -authors: ["Kyle Wang", "Antoine Toulme"] -original_authors: ["Robert Castley", "Charity Anderson", "Pieter Hagen", "Geoff Higginbottom"] -ai_assistance: "Codex" -time: 55 minutes ---- - -{{% notice title="Workshop credits" style="info" %}} -**.conf26 edition:** Kyle Wang and Antoine Toulme. - -**Original workshop:** Robert Castley, Charity Anderson, Pieter Hagen, and -Geoff Higginbottom. -{{% /notice %}} - -In this workshop, you run one Splunk Distribution of the OpenTelemetry -Collector in **agent mode** on a Linux host or Apple silicon Mac. The Collector -receives sample traces, reads sample logs, and collects host metrics. You then -use OTel Collector Config Builder to improve the data before it leaves the -Collector. - -## Workshop overview - -During this workshop, you will: - -- Run version `0.157.0` of the Collector as one Collector process. -- Generate traces and logs, and collect host metrics. -- Check all three signals locally and, when available, in Splunk Observability - Cloud. -- Upload `agent_config.yaml` and inspect it in OTel Collector Config Builder. -- Filter noisy spans, protect sensitive attributes, and transform logs. -- Download the completed configuration, apply it to the Collector, and verify the - results. - -Chapter 6 includes more ways to continue learning after the workshop. diff --git a/content/en/conf/3-obs1184/prerequisites.md b/content/en/conf/3-obs1184/prerequisites.md deleted file mode 100644 index 7e678a9e92..0000000000 --- a/content/en/conf/3-obs1184/prerequisites.md +++ /dev/null @@ -1,170 +0,0 @@ ---- -title: Prerequisites -weight: 2.1 -archetype: chapter -time: 5 minutes ---- - -## Before you begin - -1. Pair up with other attendees (optional). - -2. Open Splunk Observability Cloud. Choose one of these options: - - - Sign in to the Observability Workshop organization provided with your - Splunk Show instance. - - Register for a free Splunk Observability Cloud organization and sign in. - - **Registration:** [Register for Splunk Observability Cloud Free](https://www.splunk.com/en_us/download/observability-cloud-free-edition.html) - -3. Choose one supported execution path. - - The workshop requires `bash`, `curl`, `jq`, a text editor, outbound HTTPS, - and free local ports `2222`, `4318`, and `13133`. The Splunk Show path also - requires `ssh` on your local computer. The `scp` command is optional and is - needed only if you choose to transfer files manually. - - - **Splunk Show instance:** Open a terminal on your computer and connect to - the Splunk Show instance with the supplied SSH command. Keep the supplied - SSH command and password handy because you use them in each new terminal. - For example, enter: - - ```bash - ssh -p 2222 splunk@127.0.0.1 - ``` - - At the prompt, enter the provided password. - - **Linux laptop or Apple silicon Mac:** Use Terminal locally. Linux systems - can use an `x86_64`/`amd64` or `arm64`/`aarch64` processor. - -{{% notice title="Windows and Intel-based Mac computers" style="warning" %}} -Connect to a Splunk Show instance to participate in this workshop remotely. -Contact a facilitator if you need help. -{{% /notice %}} - -{{% exercise title="Set up the workshop" %}} - -{{< step "Create a folder" "1" >}} - -```bash -mkdir -p ~/advanced-otel-workshop -cd ~/advanced-otel-workshop -``` - -The remaining pages refer to this folder as `[WORKSHOP]`. - -{{< /step >}} - -{{< step "Get the three workshop files" "2" >}} - -Select the tab for the computer that runs the Collector. - -{{% tabs %}} -{{% tab title="Splunk Show instance" %}} - -{{% notice title="Keep your SSH details handy" style="info" %}} -The SSH command and password for your Splunk Show instance are provided by -email or by the workshop facilitator. Keep them in a convenient, secure place. -You must use the SSH command and password each time you open a new terminal and -connect to the instance. -{{% /notice %}} - -Connect to the Splunk Show instance with the supplied SSH command, then run: - -```bash -cd ~/advanced-otel-workshop -curl -fL https://github.com/signalfx/splunk-otel-collector/releases/download/v0.157.0/otelcol_linux_amd64 -o otelcol -curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/obs1184/loadgen/build/loadgen-linux-amd64 -o loadgen -curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/obs1184/setup-workshop-conf2026.sh -o setup-workshop.sh -chmod +x setup-workshop.sh -``` - -Keep the supplied SSH details available for each new terminal. The guided -workshop does not require `scp`. - -{{% /tab %}} -{{% tab title="Linux x86_64" %}} - -Use this tab for an `x86_64` or `amd64` Linux laptop. - -```bash -curl -fL https://github.com/signalfx/splunk-otel-collector/releases/download/v0.157.0/otelcol_linux_amd64 -o otelcol -curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/obs1184/loadgen/build/loadgen-linux-amd64 -o loadgen -curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/obs1184/setup-workshop-conf2026.sh -o setup-workshop.sh -chmod +x setup-workshop.sh -``` - -{{% /tab %}} -{{% tab title="Linux ARM64" %}} - -Use this tab when `uname -m` reports `arm64` or `aarch64`. - -```bash -curl -fL https://github.com/signalfx/splunk-otel-collector/releases/download/v0.157.0/otelcol_linux_arm64 -o otelcol -curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/obs1184/loadgen/build/loadgen-linux-arm64 -o loadgen -curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/obs1184/setup-workshop-conf2026.sh -o setup-workshop.sh -chmod +x setup-workshop.sh -``` - -{{% /tab %}} -{{% tab title="Apple silicon" %}} - -```bash -curl -fL https://github.com/signalfx/splunk-otel-collector/releases/download/v0.157.0/otelcol_darwin_arm64 -o otelcol -curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/obs1184/loadgen/build/loadgen-darwin-arm64 -o loadgen -curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/obs1184/setup-workshop-conf2026.sh -o setup-workshop.sh -chmod +x setup-workshop.sh -``` - -{{% /tab %}} -{{% /tabs %}} - -{{< /step >}} - -{{< step "Run setup" "3" >}} - -```bash -./setup-workshop.sh -``` - -At the cloud-export prompt, press **Enter** to send metrics and traces to -Splunk Observability Cloud. To keep all workshop data local, enter `n`. -The script then asks for your realm and access token. On a Splunk Show -instance, the supplied realm appears as the default, and an access token is -already available. Press **Enter** to use each supplied value, or enter a -replacement, such as the realm and access token for your Splunk Observability -Cloud Free organization. Token characters are hidden while you type. - -Setup creates one Collector configuration: - -```text -[WORKSHOP] -├── 1-agent -│ └── agent_config.yaml -├── loadgen -├── otelcol -├── setup-workshop.sh -└── workshop-env.sh -``` - -You learn about the components and pipelines in `agent_config.yaml` in Step -1.6. - -{{% expand title="Optional: find your realm and access token when using your own organization" %}} - -Skip this section when you are using a Splunk Show instance. - -If you use your own Splunk Observability Cloud organization: - -- Find the realm in the organization URL. For example, a URL containing `us1` - uses the `us1` realm. -- Go to **Settings > Access Tokens**. Create a token or use an existing token - that has ingest authorization. - -{{% /expand %}} - -{{< /step >}} - -{{% /exercise %}} - -{{< checkpoint "One Collector configuration is ready." >}} diff --git a/content/en/conf/_index.md b/content/en/conf/_index.md index d702f60fa3..0d605d757f 100644 --- a/content/en/conf/_index.md +++ b/content/en/conf/_index.md @@ -7,3 +7,11 @@ hidden: true layout: hero --- +{{< cards >}} +{{< card title="Advanced OpenTelemetry Collector" href="/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/" >}} +Practice reviewing and modifying a host-installed OpenTelemetry Collector configuration. +{{< /card >}} +{{< /cards >}} + +{{% children type="card" description="true" %}} + diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-1-validation.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-1-validation.md deleted file mode 100644 index 4235a86c38..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-1-validation.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: 1.1 Validation & Load Generation -linkTitle: 1.1 Validation & Load Generation -weight: 1 ---- - -In this workshop, we’ll use [**https://otelbin.io**](https://otelbin.io/) to quickly validate YAML syntax and ensure your OpenTelemetry configurations are accurate. This step helps avoid errors before running tests during the session. - -{{% notice title="Exercise" style="green" icon="running" %}} -Here’s how to validate your configuration: - -1. Open [**https://otelbin.io**](https://otelbin.io/) and replace the existing configuration by pasting your YAML into the left pane. - > [!INFO] - > If are on a Mac and **not** using a Splunk Workshop instance, you can quickly copy the contents of the `agent.yaml` file to your clipboard by running the following command: - > - > ```bash - > cat agent.yaml | pbcopy - > ``` - -2. At the top of the page, make sure **Splunk OpenTelemetry Collector** is selected as the validation target. If you **don't** select this option, then you will see warnings in the UI stating `Receiver "hostmetrics" is unused. (Line 8)`. - -3. Once validated, refer to the image representation below to confirm your pipelines are set up correctly. - -In most cases, we’ll display only the **key pipeline**. However, if all three pipelines (Traces, Metrics, and Logs) share the same structure, we’ll note this instead of showing each one individually. - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(resourcedetection
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - EXP1( debug 
fa:fa-upload):::exporter - %% Links - subID1:::sub-traces - subgraph " " - subgraph subID1[**Traces/Metrics/Logs**] - direction LR - REC1 --> PRO1 - PRO1 --> PRO2 - PRO2 --> PRO3 - PRO3 --> EXP1 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-traces stroke:#fff,stroke-width:1px, color:#fff,stroke-dasharray: 3 3; -``` - -{{% /notice %}} - ---- - -## Load Generation Tool - -For this workshop we have specifically developed a `loadgen` tool. `loadgen` is a flexible load generator for simulating traces and logging activities. It supports base, health, and security traces by default, along with optional logging of random quotes to a file, either in plain text or JSON format. - -The output generated by `loadgen` mimic those produced by an OpenTelemetry instrumentation library, allowing us to test the Collector’s processing logic and offers a simple yet powerful way to mimic real-world scenarios. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-2-test-agent.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-2-test-agent.md deleted file mode 100644 index 1f327d36dc..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-2-test-agent.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: 1.2 Test Agent Configuration -linkTitle: 1.2 Test Agent Configuration -weight: 2 ---- - -You’re ready to start the OpenTelemetry Collector with the newly created `agent.yaml`. This exercise sets the foundation for understanding how data flows through the OpenTelemetry Collector. - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Agent**: In the **Agent terminal** window run the following command: - -```bash { title="Start Collector" } -../otelcol --config=agent.yaml -``` - -**Verify debug output**: If everything is configured correctly, the first and last lines of the output will look like: - -```text -2025/01/13T12:43:51 settings.go:478: Set config to [agent.yaml] - -2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data. -``` - -**Send Test Span**: Instead of instrumenting an application, we’ll simulate sending trace data to the OpenTelemetry Collector using the `loadgen` tool. - -In the **Spans terminal** window, change into the `1-agent` directory and run the following command to send a single span: - -{{% tabs %}} -{{% tab title="Start Load Generator" %}} - -```bash -../loadgen -count 1 -``` - -{{% /tab %}} -{{% tab title="Load Generator Output" %}} - -```text -Sending traces. Use Ctrl-C to stop. -Response: {"partialSuccess":{}} - -Base trace sent with traceId: 1aacb1db8a6d510f10e52f154a7fdb90 and spanId: 7837a3a2d3635d9f - ``` - -`{"partialSuccess":{}}`: Indicates 100% success, as the `partialSuccess` field is empty. In case of a partial failure, this field will include details about any failed parts. - -{{% /tab %}} -{{% /tabs %}} - -**Verify Debug Output**: - -In the **Agent terminal** window check the collector's debug output: - -```text -2025-03-06T10:11:35.174Z info Traces {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1} -2025-03-06T10:11:35.174Z info ResourceSpans #0 -Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1 -Resource attributes: - -> service.name: Str(cinema-service) - -> deployment.environment: Str(production) - -> host.name: Str(workshop-instance) - -> os.type: Str(linux) - -> otelcol.service.mode: Str(agent) -ScopeSpans #0 -ScopeSpans SchemaURL: -InstrumentationScope cinema.library 1.0.0 -InstrumentationScope attributes: - -> fintest.scope.attribute: Str(Starwars, LOTR) -Span #0 - Trace ID : 0ef4daa44a259a7199a948231bc383c0 - Parent ID : - ID : e8fdd442c36cbfb1 - Name : /movie-validator - Kind : Server - Start time : 2025-03-06 10:11:35.163557 +0000 UTC - End time : 2025-03-06 10:11:36.163557 +0000 UTC - Status code : Ok - Status message : Success -Attributes: - -> user.name: Str(George Lucas) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - -> payment.amount: Double(86.48) - {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"} -``` - -> [!IMPORTANT] -> Stop the `agent` in the **Agent terminal** window using `Ctrl-C`. - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-3-fileexporter/1-test-fileexporter.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-3-fileexporter/1-test-fileexporter.md deleted file mode 100644 index 60526f0842..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-3-fileexporter/1-test-fileexporter.md +++ /dev/null @@ -1,188 +0,0 @@ ---- -title: 1.3.1 Test File Exporter -linkTitle: 1.3.1 Test File Exporter -weight: 1 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Restart your agent**: Find your **Agent terminal** window, and (re)start the `agent` using the modified configuration: - -{{% tabs %}} -{{% tab title="Start the Agent" %}} - -```bash -../otelcol --config=agent.yaml -``` - -{{% /tab %}} -{{% tab title="Agent Output" %}} - -```text -2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data. -``` - -{{% /tab %}} -{{% /tabs %}} - -**Send a Trace**: From the **Spans terminal** window, send another span and verify you get the same output on the console as we saw previously: - -```bash { title="Start Load Generator" } -../loadgen -count 1 -``` - -We can now stop the `agent` in the **Agent terminal** window using `Ctrl-C` so that we can verify the `agent.out` file was written. - -**Verify that the `agent.out` file is written**: Check that a file named `agent.out` is written in the current directory. - -```text { title="Updated Directory Structure" } -. -├── agent.out # OTLP/Json output created by the File Exporter -└── agent.yaml -``` - -**Verify the span format**: - -1. Verify the format used by the File Exporter to write the span to `agent.out`. -2. The output will be a single line in **OTLP/JSON** format. -3. To view the contents of `agent.out`, you can use the `cat ./agent.out` command. For a more readable formatted view, pipe the output to `jq` like this: `cat ./agent.out | jq`: - -{{% tabs %}} -{{% tab title="cat ./agent.out" %}} - -```json -{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"cinema-service"}},{"key":"deployment.environment","value":{"stringValue":"production"}},{"key":"host.name","value":{"stringValue":"workshop-instance"}},{"key":"os.type","value":{"stringValue":"linux"}},{"key":"otelcol.service.mode","value":{"stringValue":"agent"}}]},"scopeSpans":[{"scope":{"name":"cinema.library","version":"1.0.0","attributes":[{"key":"fintest.scope.attribute","value":{"stringValue":"Starwars, LOTR"}}]},"spans":[{"traceId":"d824a28db5aa5f5a3011f19c452e5af0","spanId":"ab4cde146f77eacf","parentSpanId":"","name":"/movie-validator","kind":2,"startTimeUnixNano":"1741256991405300000","endTimeUnixNano":"1741256992405300000","attributes":[{"key":"user.name","value":{"stringValue":"George Lucas"}},{"key":"user.phone_number","value":{"stringValue":"+1555-867-5309"}},{"key":"user.email","value":{"stringValue":"george@deathstar.email"}},{"key":"user.password","value":{"stringValue":"LOTR\u003eStarWars1-2-3"}},{"key":"user.visa","value":{"stringValue":"4111 1111 1111 1111"}},{"key":"user.amex","value":{"stringValue":"3782 822463 10005"}},{"key":"user.mastercard","value":{"stringValue":"5555 5555 5555 4444"}},{"key":"payment.amount","value":{"doubleValue":56.24}}],"status":{"message":"Success","code":1}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.6.1"}]} -``` - -{{% /tab %}} -{{% tab title="cat ./agent.out | jq" %}} - -```json -{ - "resourceSpans": [ - { - "resource": { - "attributes": [ - { - "key": "service.name", - "value": { - "stringValue": "cinema-service" - } - }, - { - "key": "deployment.environment", - "value": { - "stringValue": "production" - } - }, - { - "key": "host.name", - "value": { - "stringValue": "RCASTLEY-M-YQRY.local" - } - }, - { - "key": "os.type", - "value": { - "stringValue": "darwin" - } - }, - { - "key": "otelcol.service.mode", - "value": { - "stringValue": "agent" - } - } - ] - }, - "scopeSpans": [ - { - "scope": { - "name": "cinema.library", - "version": "1.0.0", - "attributes": [ - { - "key": "fintest.scope.attribute", - "value": { - "stringValue": "Starwars, LOTR" - } - } - ] - }, - "spans": [ - { - "traceId": "d824a28db5aa5f5a3011f19c452e5af0", - "spanId": "ab4cde146f77eacf", - "parentSpanId": "", - "name": "/movie-validator", - "kind": 2, - "startTimeUnixNano": "1741256991405300000", - "endTimeUnixNano": "1741256992405300000", - "attributes": [ - { - "key": "user.name", - "value": { - "stringValue": "George Lucas" - } - }, - { - "key": "user.phone_number", - "value": { - "stringValue": "+1555-867-5309" - } - }, - { - "key": "user.email", - "value": { - "stringValue": "george@deathstar.email" - } - }, - { - "key": "user.password", - "value": { - "stringValue": "LOTR>StarWars1-2-3" - } - }, - { - "key": "user.visa", - "value": { - "stringValue": "4111 1111 1111 1111" - } - }, - { - "key": "user.amex", - "value": { - "stringValue": "3782 822463 10005" - } - }, - { - "key": "user.mastercard", - "value": { - "stringValue": "5555 5555 5555 4444" - } - }, - { - "key": "payment.amount", - "value": { - "doubleValue": 56.24 - } - } - ], - "status": { - "message": "Success", - "code": 1 - } - } - ] - } - ], - "schemaUrl": "https://opentelemetry.io/schemas/1.6.1" - } - ] -} -``` - -{{% /tab %}} -{{% /tabs %}} - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-3-fileexporter/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-3-fileexporter/_index.md deleted file mode 100644 index 57695c0f4c..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-3-fileexporter/_index.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: 1.3 File Exporter -linkTitle: 1.3 File Exporter -weight: 2 ---- - -To capture more than just debug output on the screen, we also want to generate output during the export phase of the pipeline. For this, we'll add a **File Exporter** to write OTLP data to files for comparison. - -The difference between the OpenTelemetry **debug exporter** and the **file exporter** lies in their purpose and output destination: - -| Feature | Debug Exporter | File Exporter | -|---------------------|---------------------------------|-------------------------------| -| **Output Location** | Console/Log | File on disk | -| **Purpose** | Real-time debugging | Persistent offline analysis | -| **Best for** | Quick inspection during testing | Temporary storage and sharing | -| **Production Use** | No | Rare, but possible | -| **Persistence** | No | Yes | - -In summary, the **Debug Exporter** is great for real-time, in-development troubleshooting, while the **File Exporter** is better suited for storing telemetry data locally for later use. - -{{% notice title="Exercise" style="green" icon="running" %}} - -In the **Agent terminal** window ensure the collector is not running then edit the `agent.yaml` and configure the **File Exporter**: - -1. **Configuring a `file` exporter**: The [**File Exporter**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/fileexporter/README.md) writes telemetry data to files on disk. - - ```yaml - file: # File Exporter - path: "./agent.out" # Save path (OTLP/JSON) - append: false # Overwrite the file each time - ``` - -1. **Update the Pipelines Section**: Add the `file` exporter to the `traces` pipeline only: - - ```yaml - pipelines: - traces: - receivers: - - otlp # OTLP Receiver - processors: - - memory_limiter # Memory Limiter processor - - resourcedetection # Add system attributes to the data - - resource/add_mode # Add collector mode metadata - exporters: - - debug # Debug Exporter - - file # File Exporter - metrics: - receivers: - - otlp - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - exporters: - - debug - logs: - receivers: - - otlp - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - exporters: - - debug - ``` - -{{% /notice %}} - -Validate the agent configuration using [**https://otelbin.io**](https://otelbin.io/): - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(resourcedetection
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - EXP1( debug 
fa:fa-upload):::exporter - EXP2( file 
fa:fa-upload):::exporter - %% Links - subID1:::sub-traces - subgraph " " - subgraph subID1["`**Traces**`"] - direction LR - REC1 --> PRO1 - PRO1 --> PRO2 - PRO2 --> PRO3 - PRO3 --> EXP1 - PRO3 --> EXP2 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-4-metadata/1-4-1-test-metadata.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-4-metadata/1-4-1-test-metadata.md deleted file mode 100644 index 1a0cbb7224..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-4-metadata/1-4-1-test-metadata.md +++ /dev/null @@ -1,182 +0,0 @@ ---- -title: 1.4.1 Test Resource Metadata -linkTitle: 1.4.1 Test Resource Metadata -weight: 1 ---- -{{% notice title="Exercise" style="green" icon="running" %}} - -**Restart your Agent**: In your **Agent terminal** window, and restart the `agent` using the updated configuration to test the changes: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -If everything is set up correctly, the last line of the output should confirm the collector is running: - -```text - 2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data. -``` - -**Send a Trace**: From the **Spans terminal** window (making sure you are in the `1-agent` directory), send spans again with the `loadgen` binary to create a new `agent.out`: - -```bash { title="Start Load Generator" } -../loadgen -``` - -**Check the Agent’s debug output**: You should see three new lines in the `resource attributes` section: (`host.name`, `os.type` & `otelcol.service.mode`): - -```text - -Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1 -Resource attributes: - -> service.name: Str(cinema.service) - -> deployment.environment: Str(production) - -> host.name: Str([MY_HOST_NAME]) - -> os.type: Str([MY_OS]) - -> otelcol.service.mode: Str(agent) - -``` - -**Verify that metadata is added to spans**: Stop `loadgen` using `Ctrl-C`. In the new `agent.out` file: - -1. Check for the existence of the`otelcol.service.mode` attribute in the `resourceSpans` section and that it has a value of `agent`. -2. Verify that the `resourcedetection` attributes (`host.name` and `os.type`) exist too. - -These values are automatically added based on your device by the processors configured in the pipeline. - -{{% tabs %}} -{{% tab title="cat ./agent.out" %}} - -```json -{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"cinema-service"}},{"key":"deployment.environment","value":{"stringValue":"production"}},{"key":"host.name","value":{"stringValue":"RCASTLEY-M-YQRY.local"}},{"key":"os.type","value":{"stringValue":"darwin"}},{"key":"otelcol.service.mode","value":{"stringValue":"agent"}}]},"scopeSpans":[{"scope":{"name":"cinema.library","version":"1.0.0","attributes":[{"key":"fintest.scope.attribute","value":{"stringValue":"Starwars, LOTR"}}]},"spans":[{"traceId":"ae921957a4d93fa11cee640cd7908eb8","spanId":"f6b0f29825efe585","parentSpanId":"","name":"/movie-validator","kind":2,"startTimeUnixNano":"1740994347431796000","endTimeUnixNano":"1740994348431796000","attributes":[{"key":"user.name","value":{"stringValue":"George Lucas"}},{"key":"user.phone_number","value":{"stringValue":"+1555-867-5309"}},{"key":"user.email","value":{"stringValue":"george@deathstar.email"}},{"key":"user.account_password","value":{"stringValue":"LOTR\u003eStarWars1-2-3"}},{"key":"user.visa","value":{"stringValue":"4111 1111 1111 1111"}},{"key":"user.amex","value":{"stringValue":"3782 822463 10005"}},{"key":"user.mastercard","value":{"stringValue":"5555 5555 5555 4444"}}],"status":{"message":"Success","code":1}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.6.1"}]} -``` - -{{% /tab %}} -{{% tab title="cat ./agent.out | jq" %}} - -```json -{ - "resourceSpans": [ - { - "resource": { - "attributes": [ - { - "key": "service.name", - "value": { - "stringValue": "cinema-service" - } - }, - { - "key": "deployment.environment", - "value": { - "stringValue": "production" - } - }, - { - "key": "host.name", - "value": { - "stringValue": "RCASTLEY-M-YQRY.local" - } - }, - { - "key": "os.type", - "value": { - "stringValue": "darwin" - } - }, - { - "key": "otelcol.service.mode", - "value": { - "stringValue": "agent" - } - } - ] - }, - "scopeSpans": [ - { - "scope": { - "name": "cinema.library", - "version": "1.0.0", - "attributes": [ - { - "key": "fintest.scope.attribute", - "value": { - "stringValue": "Starwars, LOTR" - } - } - ] - }, - "spans": [ - { - "traceId": "ab984cd113463aa919ac200751fcfc1d", - "spanId": "db651e116290a8f2", - "parentSpanId": "", - "name": "/movie-validator", - "kind": 2, - "startTimeUnixNano": "1740994462515044000", - "endTimeUnixNano": "1740994463515044000", - "attributes": [ - { - "key": "user.name", - "value": { - "stringValue": "George Lucas" - } - }, - { - "key": "user.phone_number", - "value": { - "stringValue": "+1555-867-5309" - } - }, - { - "key": "user.email", - "value": { - "stringValue": "george@deathstar.email" - } - }, - { - "key": "user.account_password", - "value": { - "stringValue": "LOTR>StarWars1-2-3" - } - }, - { - "key": "user.visa", - "value": { - "stringValue": "4111 1111 1111 1111" - } - }, - { - "key": "user.amex", - "value": { - "stringValue": "3782 822463 10005" - } - }, - { - "key": "user.mastercard", - "value": { - "stringValue": "5555 5555 5555 4444" - } - } - ], - "status": { - "message": "Success", - "code": 1 - } - } - ] - } - ], - "schemaUrl": "https://opentelemetry.io/schemas/1.6.1" - } - ] -} -``` - -{{% /tab %}} -{{% /tabs %}} - -> [!IMPORTANT] -> Stop the `agent` and `loadgen` processes by using `Ctrl-C` in the respective terminal windows. - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-4-metadata/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-4-metadata/_index.md deleted file mode 100644 index 37aacbfe59..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/1-4-metadata/_index.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: 1.4 Resource Metadata -linkTitle: 1.4 Resource Metadata -weight: 3 -hidden: true ---- - -So far, we've simply exported an exact copy of the span sent through the OpenTelemetry Collector. - -Now, let's improve the base span by adding metadata with processors. This extra information can be helpful for troubleshooting and correlation. - -{{% notice title="Exercise" style="green" icon="running" %}} -**Stop the collector**: In your **Agent terminal** window, and stop the running collector by pressing `Ctrl-C`. Once the `agent` has stopped, open the `agent.yaml`. - -**Update All Pipelines**: Add both processors (`resourcedetection` and `resource/add_mode`) to the `processors` array in **all pipelines**. Ensure `memory_limiter` remains the first processor. - -- The [**Resource Detection Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/README.md) is used to detect resource information from the host and append or override the resource value in telemetry data with this information. -- The [**Resource Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourceprocessor/README.md) is used to apply changes on resource attributes. In this case, the default configuration adds a new attribute `otelcol.service.mode` with the value `agent`. - -```yaml -service: # Services configured for this Collector - extensions: # Enabled extensions - - health_check - pipelines: # Array of configured pipelines - traces: - receivers: - - otlp # OTLP Receiver - processors: - - memory_limiter # Memory Limiter Processor - - resourcedetection # Adds system attributes to the data - - resource/add_mode # Adds collector mode metadata - exporters: - - debug # Debug Exporter - - file # File Exporter - metrics: - receivers: - - otlp # OTLP Receiver - processors: - - memory_limiter # Memory Limiter Processor - - resourcedetection # Adds system attributes to the data - - resource/add_mode # Adds collector mode metadata - exporters: - - debug # Debug Exporter - - file # File Exporter - logs: - receivers: - - otlp # OTLP Receiver - processors: - - memory_limiter # Memory Limiter Processor - - resourcedetection # Adds system attributes to the data - - resource/add_mode # Adds collector mode metadata - exporters: - - debug # Debug Exporter - - file # File Exporter - -``` - -{{% /notice %}} - -By adding these processors, we enrich the data with system metadata and the agent’s operational mode, which aids in troubleshooting and provides useful context for related content. - -Validate the agent configuration using **[otelbin.io](https://www.otelbin.io/)**: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(resourcedetection
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - EXP1( debug 
fa:fa-upload):::exporter - EXP2( file 
fa:fa-upload):::exporter - %% Links - subID1:::sub-traces - subgraph " " - subgraph subID1[**Traces/Metrics/Logs**] - direction LR - REC1 --> PRO1 - PRO1 --> PRO2 - PRO2 --> PRO3 - PRO3 --> EXP1 - PRO3 --> EXP2 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-traces stroke:#fff,stroke-width:1px, color:#fff,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/_index.md deleted file mode 100644 index 28fcdd8936..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/1-agent/_index.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: 1. Agent Configuration -linkTitle: 1. Agent Setup -time: 10 minutes -weight: 3 ---- - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -During this workshop, you will be using up to five terminal windows simultaneously. To stay organized, consider customizing each terminal or shell with unique names and colors. This will help you quickly identify and switch between them as needed. - -We will refer to these terminals as: **Agent**, **Gateway**, **Spans**, **Logs** and **Tests**. -{{% /notice %}} - -{{% notice title="Exercise" style="green" icon="running" %}} - -1. In the **Agent terminal** window, change into the `[WORKSHOP]` directory and create a new subdirectory named `1-agent`. - - ```bash - mkdir 1-agent && \ - cd 1-agent - ``` - -2. Create a file named `agent.yaml`. This file will define the basic structure of an OpenTelemetry Collector configuration. - -3. Copy and paste the following initial configuration into `agent.yaml`: - - ```yaml - # Extensions - extensions: - health_check: # Health Check Extension - endpoint: 0.0.0.0:13133 # Health Check Endpoint - - # Receivers - receivers: - hostmetrics: # Host Metrics Receiver - collection_interval: 3600s # Collection Interval (1hr) - scrapers: - cpu: # CPU Scraper - otlp: # OTLP Receiver - protocols: - http: # Configure HTTP protocol - endpoint: "0.0.0.0:4318" # Endpoint to bind to - - # Exporters - exporters: - debug: # Debug Exporter - verbosity: detailed # Detailed verbosity level - - # Processors - processors: - memory_limiter: # Limits memory usage - check_interval: 2s # Check interval - limit_mib: 512 # Memory limit in MiB - resourcedetection: # Resource Detection Processor - detectors: [system] # Detect system resources - override: true # Overwrites existing attributes - resource/add_mode: # Resource Processor - attributes: - - action: insert # Action to perform - key: otelcol.service.mode # Key name - value: "agent" # Key value - - # Connectors - #connectors: # leave this commented out; we will uncomment in an upcoming exercise - - # Service Section - Enabled Pipelines - service: - extensions: - - health_check # Health Check Extension - pipelines: - traces: - receivers: - - otlp # OTLP Receiver - processors: - - memory_limiter # Memory Limiter processor - - resourcedetection # Add system attributes to the data - - resource/add_mode # Add collector mode metadata - exporters: - - debug # Debug Exporter - metrics: - receivers: - - otlp - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - exporters: - - debug - logs: - receivers: - - otlp - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - exporters: - - debug - ``` - -4. Your directory structure should now look like this: - - ```text - . - └── agent.yaml # OpenTelemetry Collector configuration file - ``` - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-1-start-gateway.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-1-start-gateway.md deleted file mode 100644 index 31f7c9f25b..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-1-start-gateway.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: 2.1 Start Gateway -linkTitle: 2.1 Start Gateway -weight: 1 ---- - -The configuration for the `gateway` does not need any additional configuration changes to function. This has been done to save time and focus on the core concepts of the **Gateway**. - -Validate the `gateway` configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `logs:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(resource
fa:fa-microchip
add_mode):::processor - PRO3(batch
fa:fa-microchip):::processor - EXP1( file 
fa:fa-upload
logs):::exporter - EXP2( debug 
fa:fa-upload):::exporter - %% Links - subID1:::sub-logs - subgraph " " - subgraph subID1[**Logs**] - direction LR - REC1 --> PRO1 - PRO1 --> PRO2 - PRO2 --> PRO3 - PRO3 --> EXP2 - PRO3 --> EXP1 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3; -``` - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Gateway**: In the **Gateway terminal** window, run the following command to start the `gateway`: - -```bash {title="Start the Gateway"} -../otelcol --config=gateway.yaml -``` - -If everything is configured correctly, the first and last lines of the output should look like: - -```text -2025/01/15 15:33:53 settings.go:478: Set config to [gateway.yaml] - -2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data. -``` - -{{% /notice %}} - -Next, we will configure the `agent` to send data to the newly created `gateway`. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-2-configure-agent.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-2-configure-agent.md deleted file mode 100644 index 5c5c44e041..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-2-configure-agent.md +++ /dev/null @@ -1,111 +0,0 @@ ---- -title: 2.2 Configure Agent -linkTitle: 2.2 Configure Agent -weight: 2 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Add the `otlphttp` exporter**: The [**OTLP/HTTP Exporter**](https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/collector-components/exporters/otlphttp-exporter) is used to send data from the agent to the gateway using the **OTLP/HTTP** protocol. - -1. Switch to your **Agent terminal** window. -2. Validate that the newly generated `gateway-logs.out`, `gateway-metrics.out`, and `gateway-traces.out` are present in the directory. -3. Open the `agent.yaml` file in your editor. -4. Add the `otlphttp` exporter configuration to the `exporters:` section: - -```yaml - otlphttp: # Exporter Type - endpoint: "http://localhost:5318" # Gateway OTLP endpoint -``` - -**Add a Batch Processor configuration**: The [**Batch Processor**](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md) will accept spans, metrics, or logs and place them into batches. Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data. It is highly recommended configuring the batch processor on every collector. - -1. Add the `batch` processor configuration to the `processors:` section: - -```yaml - batch: # Processor Type -``` - -**Update the pipelines**: - -1. **Enable Hostmetrics Receiver**: - - Add `hostmetrics` to the `metrics` pipeline. The [**HostMetrics Receiver**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver#readme) will generate host CPU metrics once per hour with the current configuration. -2. **Enable Batch Processor**: - - Add the `batch` processor (after the `resource/add_mode` processor) to the `traces`, `metrics`, and `logs` pipelines. -3. **Enable OTLPHTTP Exporter**: - - Add the `otlphttp` exporter to the `traces`, `metrics`, and `logs` pipelines. - -```yaml - pipelines: - traces: - receivers: - - otlp # OTLP Receiver - processors: - - memory_limiter # Memory Limiter processor - - resourcedetection # Add system attributes to the data - - resource/add_mode # Add collector mode metadata - - batch # Batch processor - exporters: - - debug # Debug Exporter - - file # File Exporter - - otlphttp # OTLP/HTTP Exporter - metrics: - receivers: - - otlp - - hostmetrics # Host Metrics Receiver - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - otlphttp - logs: - receivers: - - otlp - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - otlphttp -``` - -{{% /notice %}} - -Validate the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `metrics:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(    otlp    
fa:fa-download):::receiver - REC2(hostmetrics
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(resourcedetection
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(batch
fa:fa-microchip):::processor - EXP1(otlphttp
fa:fa-upload):::exporter - EXP2( debug 
fa:fa-upload):::exporter - %% Links - subID1:::sub-metrics - subgraph " " - subgraph subID1["`**Metrics**`"] - direction LR - REC1 --> PRO1 - REC2 --> PRO1 - PRO1 --> PRO2 - PRO2 --> PRO3 - PRO3 --> PRO4 - PRO4 --> EXP2 - PRO4 --> EXP1 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-3-send-metrics.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-3-send-metrics.md deleted file mode 100644 index c7cf1a5762..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-3-send-metrics.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -title: 2.3 Send metrics from the Agent to the Gateway -linkTitle: 2.3 Send Metrics -weight: 3 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Agent**: In the **Agent terminal** window start the agent with the updated configuration: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Verify CPU Metrics**: - -1. Check that when the `agent` starts, it immediately starts sending **CPU** metrics. -2. Both the `agent` and the `gateway` will display this activity in their debug output. The output should resemble the following snippet: - -```text - -NumberDataPoints #37 -Data point attributes: - -> cpu: Str(cpu0) - -> state: Str(system) -StartTimestamp: 2024-12-09 14:18:28 +0000 UTC -Timestamp: 2025-01-15 15:27:51.319526 +0000 UTC -Value: 9637.660000 -``` - -At this stage, the `agent` continues to collect **CPU** metrics once per hour or upon each restart and sends them to the gateway. - -The `gateway` processes these metrics and exports them to a file named `./gateway-metrics.out`. This file stores the exported metrics as part of the pipeline service. - -**Verify Data arrived at Gateway**: To confirm that CPU metrics, specifically for `cpu0`, have successfully reached the gateway, we’ll inspect the `gateway-metrics.out` file using the `jq` command. - -The following command filters and extracts the `system.cpu.time` metric, focusing on `cpu0`. It displays the metric’s state (e.g., `user`, `system`, `idle`, `interrupt`) along with the corresponding values. - -Run the command below in the **Tests terminal** to check the `system.cpu.time` metric: - -{{% tabs %}} -{{% tab title="Check CPU Metrics" %}} - -```bash -jq '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "system.cpu.time") | .sum.dataPoints[] | select(.attributes[0].value.stringValue == "cpu0") | {cpu: .attributes[0].value.stringValue, state: .attributes[1].value.stringValue, value: .asDouble}' gateway-metrics.out -``` - -{{% /tab %}} -{{% tab title="Example Output" %}} - -```json -{ - "cpu": "cpu0", - "state": "user", - "value": 123407.02 -} -{ - "cpu": "cpu0", - "state": "system", - "value": 64866.6 -} -{ - "cpu": "cpu0", - "state": "idle", - "value": 216427.87 -} -{ - "cpu": "cpu0", - "state": "interrupt", - "value": 0 -} -``` - -{{% /tab %}} -{{% /tabs %}} - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-4-send-traces.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-4-send-traces.md deleted file mode 100644 index 15ec3dc9e6..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-4-send-traces.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -title: 2.4 Send traces from the Agent to the Gateway -linkTitle: 2.4 Send Traces -weight: 4 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Send a Test Trace**: - -1. Validate `agent` and `gateway` are still running. -2. In the **Spans terminal** window, run the following command to send 5 spans and validate the output of the `agent` and `gateway` debug logs: - -{{% tabs %}} -{{% tab title="Start the Load Generator" %}} - -```bash -../loadgen -count 5 -``` - -{{% /tab %}} -{{% tab title="Agent/Gateway Debug Output" %}} - -```text -2025-03-06T11:49:00.456Z info Traces {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1} -2025-03-06T11:49:00.456Z info ResourceSpans #0 -Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1 -Resource attributes: - -> service.name: Str(cinema-service) - -> deployment.environment: Str(production) - -> host.name: Str(workshop-instance) - -> os.type: Str(linux) - -> otelcol.service.mode: Str(agent) -ScopeSpans #0 -ScopeSpans SchemaURL: -InstrumentationScope cinema.library 1.0.0 -InstrumentationScope attributes: - -> fintest.scope.attribute: Str(Starwars, LOTR) -Span #0 - Trace ID : 97fb4e5b13400b5689e3306da7cff077 - Parent ID : - ID : 413358465e5b4f15 - Name : /movie-validator - Kind : Server - Start time : 2025-03-06 11:49:00.431915 +0000 UTC - End time : 2025-03-06 11:49:01.431915 +0000 UTC - Status code : Ok - Status message : Success -Attributes: - -> user.name: Str(George Lucas) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - -> payment.amount: Double(87.01) - {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"} -``` - -{{% /tab %}} -{{% /tabs %}} - -**Verify the Gateway has handled the spans**: Once the gateway processes incoming spans, it writes the trace data to a file named `gateway-traces.out`. To confirm that the spans have been successfully handled, you can inspect this file. - -In your **Tests terminal**, use the `jq` command to extract and display key details about each span, such as its `spanId` and its position in the file. Also, we can extract the attributes that the **Hostmetrics Receiver** added to the spans. - -{{% tabs %}} -{{% tab title="Inspect the Gateway Trace File" %}} - -```bash -jq -c '.resourceSpans[] as $resource | $resource.scopeSpans[].spans[] | "Span \(input_line_number) found with spanId \(.spanId), hostname \($resource.resource.attributes[] | select(.key == "host.name") | .value.stringValue), os \($resource.resource.attributes[] | select(.key == "os.type") | .value.stringValue)"' ./gateway-traces.out -``` - -{{% /tab %}} -{{% tab title="Example Output" %}} - -```text -"Span 1 found with spanId d71fe6316276f97d, hostname workshop-instance, os linux" -"Span 2 found with spanId e8d19489232f8c2a, hostname workshop-instance, os linux" -"Span 3 found with spanId 9dfaf22857a6bd05, hostname workshop-instance, os linux" -"Span 4 found with spanId c7f544a4b5fef5fc, hostname workshop-instance, os linux" -"Span 5 found with spanId 30bb49261315969d, hostname workshop-instance, os linux" -``` - -{{% /tab %}} -{{% /tabs %}} - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-5-addendum.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-5-addendum.md deleted file mode 100644 index 41c5baef77..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/2-5-addendum.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -title: 2.5 Addendum - Info on Access Tokens and Batch Processing -linkTitle: 2.5 Addendum -weight: 5 -hidden: true ---- - - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} - -## Introduction to the otlphttp Exporter - -The `otlphttp` exporter is now the default method for sending metrics and traces to Splunk Observability Cloud. This exporter provides a standardized and efficient way to transmit telemetry data using the OpenTelemetry Protocol (OTLP) over HTTP. - -When deploying the Splunk Distribution of the OpenTelemetry Collector in host monitoring (agent) mode, the `otlphttp` exporter is included by default. This replaces older exporters such as `sapm` and `signalfx`, which are gradually being phased out. - -{{% /notice %}} - -## Configuring Splunk Access Tokens - -To authenticate and send data to Splunk Observability Cloud, you need to configure access tokens properly. -In OpenTelemetry, authentication is handled via HTTP headers. To pass an access token, use the `headers:` key with the sub-key `X-SF-Token:`. This configuration works in both agent and gateway mode. - -Example: - -```yaml -exporters: - otlphttp: - endpoint: "https://ingest..signalfx.com" - headers: - X-SF-Token: "your-access-token" -``` - -### Pass-Through Mode - -If you need to forward headers through the pipeline, enable pass-through mode by setting `include_metadata:` to `true` in the OTLP receiver configuration. This ensures that any authentication headers received by the collector are retained and forwarded along with the data. - -Example: - -```yaml -receivers: - otlp: - protocols: - http: - include_metadata: true -``` - -This is particularly useful in gateway mode, where data from multiple agents may pass through a centralized gateway before being sent to Splunk. - -## Understanding Batch Processing - -The Batch Processor is a key component in optimizing data transmission efficiency. It groups traces, metrics, and logs into batches before sending them to the backend. Batching improves performance by: - -- Reducing the number of outgoing requests. -- Improving compression efficiency. -- Lowering network overhead. - -### Configuring the Batch Processor - -To enable batching, configure the `batch:` section and include the `X-SF-Token:` key. This ensures that data is grouped correctly before being sent to Splunk Observability Cloud. - -Example: - -```yaml -processors: - batch: - metadata_keys: [X-SF-Token] # Array of metadata keys to batch - send_batch_size: 100 - timeout: 5s -``` - -### Best Practices for Batch Processing - -For optimal performance, it is recommended to use the Batch Processor in every collector deployment. The best placement for the Batch Processor is **after the memory limiter and sampling processors**. This ensures that only necessary data is batched, avoiding unnecessary processing of dropped data. - -### Gateway Configuration with Batch Processor - -When deploying a gateway, ensure that the Batch Processor is included in the pipeline: - -```yaml -service: - pipelines: - traces: - processors: [memory_limiter, tail_sampling, batch] -``` - -## Conclusion - -The `otlphttp` exporter is now the preferred method for sending telemetry data to Splunk Observability Cloud. Properly configuring Splunk Access Tokens ensures secure data transmission, while the Batch Processor helps optimize performance by reducing network overhead. By implementing these best practices, you can efficiently collect and transmit observability data at scale. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/_index.md deleted file mode 100644 index 87ccfb0fcd..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/2-gateway/_index.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -title: 2. Gateway Configuration -linkTitle: 2. Gateway Setup -time: 10 minutes -weight: 4 ---- - -The OpenTelemetry Gateway is designed to receive, process, and export telemetry data. It acts as an intermediary between telemetry sources (e.g. applications, services) and backends (e.g., observability platforms like Prometheus, Jaeger, or Splunk Observability Cloud). - -The gateway is useful because it centralizes telemetry data collection, enabling features like data filtering, transformation, and routing to multiple destinations. It also reduces the load on individual services by offloading telemetry processing and ensures consistent data formats across distributed systems. This makes it easier to manage, scale, and analyze telemetry data in complex environments. - -{{% notice title="Exercise" style="green" icon="running" %}} - -- In the **Gateway terminal** window, change into the `[WORKSHOP]` directory and create a new subdirectory named `2-gateway`. - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `[WORKSHOP]/2-gateway` directory.** - -- Back in the **Gateway terminal** window, copy `agent.yaml` from the `1-agent` directory into `2-gateway`. -- Create a file called `gateway.yaml` and add the following initial configuration: - -```yaml { title="gateway.yaml" } -########################### This section holds all the -## Configuration section ## configurations that can be -########################### used in this OpenTelemetry Collector -extensions: # List of extensions - health_check: # Health check extension - endpoint: 0.0.0.0:14133 # Custom port to avoid conflicts - -receivers: - otlp: # OTLP receiver - protocols: - http: # HTTP protocol - endpoint: "0.0.0.0:5318" # Custom port to avoid conflicts - include_metadata: true # Required for token pass-through - -exporters: # List of exporters - debug: # Debug exporter - verbosity: detailed # Enable detailed debug output - file/traces: # Exporter Type/Name - path: "./gateway-traces.out" # Path for OTLP JSON output - append: false # Overwrite the file each time - file/metrics: # Exporter Type/Name - path: "./gateway-metrics.out" # Path for OTLP JSON output - append: false # Overwrite the file each time - file/logs: # Exporter Type/Name - path: "./gateway-logs.out" # Path for OTLP JSON output - append: false # Overwrite the file each time - -processors: # List of processors - memory_limiter: # Limits memory usage - check_interval: 2s # Memory check interval - limit_mib: 512 # Memory limit in MiB - batch: # Batches data before exporting - metadata_keys: # Groups data by token - - X-SF-Token - resource/add_mode: # Adds metadata - attributes: - - action: upsert # Inserts or updates a key - key: otelcol.service.mode # Key name - value: "gateway" # Key value - -# Connectors -#connectors: # leave this commented out; we will uncomment in an upcoming exercise - -########################### -### Activation Section ### -########################### -service: # Service configuration - telemetry: - metrics: - level: none # Disable metrics - extensions: [health_check] # Enabled extensions - pipelines: # Configured pipelines - traces: # Traces pipeline - receivers: - - otlp # OTLP receiver - processors: # Processors for traces - - memory_limiter - - resource/add_mode - - batch - exporters: - - debug # Debug exporter - - file/traces - metrics: # Metrics pipeline - receivers: - - otlp # OTLP receiver - processors: # Processors for metrics - - memory_limiter - - resource/add_mode - - batch - exporters: - - debug # Debug exporter - - file/metrics - logs: # Logs pipeline - receivers: - - otlp # OTLP receiver - processors: # Processors for logs - - memory_limiter - - resource/add_mode - - batch - exporters: - - debug # Debug exporter - - file/logs -``` - -> [!NOTE] -> When the `gateway` is started it will generate three files: `gateway-traces.out`, `gateway-metrics.out`, and `gateway-logs.out`. These files will eventually contain the telemetry data received by the gateway. - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/3-filelog/3-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/3-filelog/3-1-configuration.md deleted file mode 100644 index bf6c78f8c0..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/3-filelog/3-1-configuration.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -title: 3.1 Configuration -linkTitle: 3.1 Configuration -weight: 1 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} -In the **Agent terminal** window edit the `agent.yaml` and configure the FileLog receiver. - -1. **Configure FileLog Receiver**: The `filelog` receiver reads log data from a file and includes custom resource attributes in the log data: - - ```yaml - filelog/quotes: # Receiver Type/Name - include: ./quotes.log # The file to read log data from - include_file_path: true # Include file path in the log data - include_file_name: false # Exclude file name from the log data - resource: # Add custom resource attributes - com.splunk.source: ./quotes.log # Source of the log data - com.splunk.sourcetype: quotes # Source type of the log data - ``` - -2. **Update `logs` pipeline**: Add the`filelog/quotes` receiver to the `logs` pipeline only: - - ```yaml - logs: - receivers: - - otlp - - filelog/quotes # Filelog Receiver - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - otlphttp - ``` - -3. **Validate Configuration**: Paste the updated `agent.yaml` into **[otelbin.io](https://www.otelbin.io/)**. For reference, the `logs:` section of your pipelines will look similar to this: - - ```mermaid - %%{init:{"fontFamily":"monospace"}}%% - graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - REC2(filelog
fa:fa-download
quotes):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(resourcedetection
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(batch
fa:fa-microchip):::processor - EXP1( debug 
fa:fa-upload):::exporter - EXP2(otlphttp
fa:fa-upload):::exporter - %% Links - subID1:::sub-logs - subgraph " " - subgraph subID1[**Logs**] - direction LR - REC1 --> PRO1 - REC2 --> PRO1 - PRO1 --> PRO2 - PRO2 --> PRO3 - PRO3 --> PRO4 - PRO4 --> EXP1 - PRO4 --> EXP2 - end - end - classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; - classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; - classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; - classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3; - ``` - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/3-filelog/3-2-test-filelog.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/3-filelog/3-2-test-filelog.md deleted file mode 100644 index a45347603b..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/3-filelog/3-2-test-filelog.md +++ /dev/null @@ -1,150 +0,0 @@ ---- -title: 3.2 Test FileLog Receiver -linkTitle: 3.2 Test FileLog Receiver -weight: 4 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Gateway**: In your **Gateway terminal** window start the `gateway`. - -**Start the Agent**: In your **Agent terminal** window start the `agent`. - -A continuous stream of log data from the `quotes.log` will be in the `agent` and `gateway` debug logs: - -```text { title="Agent/Gateway Debug Output" } -Timestamp: 1970-01-01 00:00:00 +0000 UTC -SeverityText: -SeverityNumber: Unspecified(0) -Body: Str(2025-03-06 15:18:32 [ERROR] - There is some good in this world, and it's worth fighting for. LOTR) -Attributes: - -> log.file.path: Str(quotes.log) -Trace ID: -Span ID: -Flags: 0 -LogRecord #1 -``` - -**Stop the `loadgen`**: In the **Logs terminal** window, stop the `loadgen` using `Ctrl-C`. - -**Verify the gateway**: Check if the `gateway` has written a `./gateway-logs.out` file. - -At this point, your directory structure will appear as follows: - -```text { title="Updated Directory Structure" } -. -├── agent.out -├── agent.yaml -├── gateway-logs.out # Output from the logs pipeline -├── gateway-metrics.out # Output from the metrics pipeline -├── gateway-traces.out # Output from the traces pipeline -├── gateway.yaml -└── quotes.log # File containing Random log lines -``` - -**Examine a log line**: In `gateway-logs.out` compare a log line with the snippet below. Verify that the log entry includes the same attributes as we have seen in metrics and traces data previously: - -{{% tabs %}} -{{% tab title="cat /gateway-logs.out" %}} - -```json -{"resourceLogs":[{"resource":{"attributes":[{"key":"com.splunk.source","value":{"stringValue":"./quotes.log"}},{"key":"com.splunk.sourcetype","value":{"stringValue":"quotes"}},{"key":"host.name","value":{"stringValue":"workshop-instance"}},{"key":"os.type","value":{"stringValue":"linux"}},{"key":"otelcol.service.mode","value":{"stringValue":"gateway"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1741274312475540000","body":{"stringValue":"2025-03-06 15:18:32 [DEBUG] - All we have to decide is what to do with the time that is given us. LOTR"},"attributes":[{"key":"log.file.path","value":{"stringValue":"quotes.log"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1741274312475560000","body":{"stringValue":"2025-03-06 15:18:32 [DEBUG] - Your focus determines your reality. SW"},"attributes":[{"key":"log.file.path","value":{"stringValue":"quotes.log"}}],"traceId":"","spanId":""}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.6.1"}]} -``` - -{{% /tab %}} -{{% tab title="cat ./gateway-logs.out | jq" %}} - -```json -{ - "resourceLogs": [ - { - "resource": { - "attributes": [ - { - "key": "com.splunk.source", - "value": { - "stringValue": "./quotes.log" - } - }, - { - "key": "com.splunk.sourcetype", - "value": { - "stringValue": "quotes" - } - }, - { - "key": "host.name", - "value": { - "stringValue": "workshop-instance" - } - }, - { - "key": "os.type", - "value": { - "stringValue": "linux" - } - }, - { - "key": "otelcol.service.mode", - "value": { - "stringValue": "gateway" - } - } - ] - }, - "scopeLogs": [ - { - "scope": {}, - "logRecords": [ - { - "observedTimeUnixNano": "1741274312475540000", - "body": { - "stringValue": "2025-03-06 15:18:32 [DEBUG] - All we have to decide is what to do with the time that is given us. LOTR" - }, - "attributes": [ - { - "key": "log.file.path", - "value": { - "stringValue": "quotes.log" - } - } - ], - "traceId": "", - "spanId": "" - }, - { - "observedTimeUnixNano": "1741274312475560000", - "body": { - "stringValue": "2025-03-06 15:18:32 [DEBUG] - Your focus determines your reality. SW" - }, - "attributes": [ - { - "key": "log.file.path", - "value": { - "stringValue": "quotes.log" - } - } - ], - "traceId": "", - "spanId": "" - } - ] - } - ], - "schemaUrl": "https://opentelemetry.io/schemas/1.6.1" - } - ] -} -``` - -{{% /tab %}} -{{% /tabs %}} - -You may also have noticed that every log line contains empty placeholders for `"traceId":""` and `"spanId":""`. -The FileLog receiver will populate these fields only if they are not already present in the log line. -For example, if the log line is generated by an application instrumented with an OpenTelemetry instrumentation library, these fields will already be included and will not be overwritten. - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/3-filelog/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/3-filelog/_index.md deleted file mode 100644 index 50cf08dcac..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/3-filelog/_index.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -title: 3. FileLog Setup -linkTitle: 3. FileLog Setup -time: 10 minutes -weight: 5 ---- - -The [**FileLog Receiver**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md) in the OpenTelemetry Collector is used to ingest logs from files. - -It monitors specified files for new log entries and streams those logs into the Collector for further processing or exporting. It is also useful for testing and development purposes. - -For this part of the workshop, the `loadgen` will generate logs using random quotes: - -```golang -lotrQuotes := []string{ - "One does not simply walk into Mordor.", - "Even the smallest person can change the course of the future.", - "All we have to decide is what to do with the time that is given us.", - "There is some good in this world, and it's worth fighting for.", -} - -starWarsQuotes := []string{ - "Do or do not, there is no try.", - "The Force will be with you. Always.", - "I find your lack of faith disturbing.", - "In my experience, there is no such thing as luck.", -} -``` - -The **FileLog receiver** in the `agent` collector will read these log lines and send them to the `gateway`. - -{{% notice title="Exercise" style="green" icon="running" %}} - -- In the **Logs terminal** window, change into the `[WORKSHOP]` directory and create a new subdirectory named `3-filelog`. -- Next, copy `*.yaml` from `2-gateway` into `3-filelog`. - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `[WORKSHOP]/3-filelog` directory.** - -Start the `loadgen` and this will begin writing lines to a file named `quotes.log`: - -{{% tabs %}} -{{% tab title="Log Load Generator" %}} - -```bash -../loadgen -logs -``` - -{{% /tab %}} -{{% tab title="Log Load Generator Output" %}} - -```text -Writing logs to quotes.log. Press Ctrl+C to stop. -``` - -{{% /tab %}} -{{% /tabs %}} - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -├── gateway.yaml -└── quotes.yaml -``` - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-1-configuration.md deleted file mode 100644 index 2d388d685b..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-1-configuration.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -title: 4.1 File Storage Configuration -linkTitle: 4.1 Configuration -weight: 1 ---- - -In this exercise, we will update the `extensions:` section of the `agent.yaml` file. This section is part of the OpenTelemetry configuration YAML and defines optional components that enhance or modify the OpenTelemetry Collector’s behavior. - -While these components do not process telemetry data directly, they provide valuable capabilities and services to improve the Collector’s functionality. - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Update the `agent.yaml`**: In the **Agent terminal** window, add the `file_storage` extension and name it `checkpoint`: - -```yaml - file_storage/checkpoint: # Extension Type/Name - directory: "./checkpoint-dir" # Define directory - create_directory: true # Create directory - timeout: 1s # Timeout for file operations - compaction: # Compaction settings - on_start: true # Start compaction at Collector startup - # Define compaction directory - directory: "./checkpoint-dir/tmp" - max_transaction_size: 65536 # Max. size limit before compaction occurs -``` - -**Add `file_storage` to existing `otlphttp` exporter**: Modify the `otlphttp:` exporter to configure retry and queuing mechanisms, ensuring data is retained and resent if failures occur: - -```yaml - otlphttp: - endpoint: "http://localhost:5318" - retry_on_failure: - enabled: true # Enable retry on failure - sending_queue: # - enabled: true # Enable sending queue - num_consumers: 10 # No. of consumers - queue_size: 10000 # Max. queue size - storage: file_storage/checkpoint # File storage extension -``` - -**Update the `services` section**: Add the `file_storage/checkpoint` extension to the existing `extensions:` section. This will cause the extension to be enabled: - -```yaml -service: - extensions: - - health_check - - file_storage/checkpoint # Enabled extensions for this collector -``` - -**Update the `metrics` pipeline**: For this exercise we are going to remove the `hostmetrics` receiver from the Metric pipeline to reduce debug and log noise: - -```yaml - metrics: - receivers: - - otlp - # - hostmetrics # Hostmetrics Receiver -``` - -{{% /notice %}} - -Validate the `agent` configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `metrics:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(resourcedetection
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(batch
fa:fa-microchip):::processor - EXP1( debug 
fa:fa-upload):::exporter - EXP2(otlphttp
fa:fa-upload):::exporter - %% Links - subID1:::sub-metrics - subgraph " " - subgraph subID1["`**Metrics**`"] - direction LR - REC1 --> PRO1 - PRO1 --> PRO2 - PRO2 --> PRO3 - PRO3 --> PRO4 - PRO4 --> EXP1 - PRO4 --> EXP2 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-2-test-environment.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-2-test-environment.md deleted file mode 100644 index 9639afb169..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-2-test-environment.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: 4.2 Setup environment for Resilience Testing -linkTitle: 4.2 Setup environment -weight: 2 ---- - -Next, we will configure our environment to be ready for testing the **File Storage** configuration. - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Gateway**: In the **Gateway terminal** window navigate to the `[WORKSHOP]/4-resilience` directory and run: - -```bash { title="Start the Gateway" } -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: In the **Agent terminal** window navigate to the `[WORKSHOP]/4-resilience` directory and run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Send five test spans**: In the **Spans terminal** window navigate to the `[WORKSHOP]/4-resilience` directory and run: - -```bash { title="Start Load Generator" } -../loadgen -count 5 -``` - -Both the `agent` and `gateway` should display debug logs, and the `gateway` should create a `./gateway-traces.out` file. - -{{% /notice %}} - -If everything functions correctly, we can proceed with testing system resilience. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-3-failure.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-3-failure.md deleted file mode 100644 index 3db252e6aa..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-3-failure.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: 4.3 Simulate Failure -linkTitle: 4.3 Simulate Failure -weight: 3 ---- - -To assess the **Agent's** resilience, we'll simulate a temporary `gateway` outage and observe how the `agent` handles it: - -**Summary**: - -1. **Send Traces to the Agent** – Generate traffic by sending traces to the `agent`. -2. **Stop the Gateway** – This will trigger the `agent` to enter retry mode. -3. **Restart the Gateway** – The `agent` will recover traces from its persistent queue and forward them successfully. Without the persistent queue, these traces would have been lost permanently. - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Simulate a network failure**: In the **Gateway terminal** stop the `gateway` with `Ctrl-C` and wait until the gateway console shows that it has stopped: - -```text -2025-01-28T13:24:32.785+0100 info service@v0.120.0/service.go:309 Shutdown complete. -``` - -**Send traces**: In the **Spans terminal** window send five more traces using the `loadgen`. - -Notice that the agent’s retry mechanism is activated as it continuously attempts to resend the data. In the agent’s console output, you will see repeated messages similar to the following: - -```text -2025-01-28T14:22:47.020+0100 info internal/retry_sender.go:126 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:5318/v1/traces\": dial tcp 127.0.0.1:5318: connect: connection refused", "interval": "9.471474933s"} -``` - -**Stop the Agent**: In the **Agent terminal** window, use `Ctrl-C` to stop the agent. Wait until the agent’s console confirms it has stopped: - -```text -2025-01-28T14:40:28.702+0100 info extensions/extensions.go:66 Stopping extensions... -2025-01-28T14:40:28.702+0100 info service@v0.120.0/service.go:309 Shutdown complete. -``` - -{{% /notice %}} - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -Stopping the agent will halt its retry attempts and prevent any future retry activity. - -If the agent runs for too long without successfully delivering data, it may begin dropping traces, depending on the retry configuration, to conserve memory. By stopping the agent, any metrics, traces, or logs currently stored in memory are lost before being dropped, ensuring they remain available for recovery. - -This step is essential for clearly observing the recovery process when the agent is restarted. -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-4-recovery.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-4-recovery.md deleted file mode 100644 index 96ac29a2c7..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/4-4-recovery.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: 4.4 Recovery -linkTitle: 4.4 Recovery -weight: 4 ---- - -In this exercise, we’ll test how the **OpenTelemetry Collector** recovers from a network outage by restarting the **Gateway** collector. When the `gateway` becomes available again, the `agent` will resume sending data from its last checkpointed state, ensuring no data loss. - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Restart the Gateway**: In the **Gateway terminal** window run: - -```bash {title="Gateway"} -../otelcol --config=gateway.yaml -``` - -**Restart the Agent**: In the **Agent terminal** window run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -{{% /notice %}} - -After the `agent` is up and running, the **File_Storage** extension will detect buffered data in the checkpoint folder. -It will start to dequeue the stored spans from the last checkpoint folder, ensuring no data is lost. - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Verify the Agent Debug output** -Note that the Agent Debug Screen does **NOT** change and still shows the following line indicating no new data is being exported. - - ```text - 2025-02-07T13:40:12.195+0100 info service@v0.120.0/service.go:253 Everything is ready. Begin running and processing data. - ``` - -**Watch the Gateway Debug output** -You should see from the `gateway` debug screen, it has started receiving the previously missed traces without requiring any additional action on your part. - - ```txt - 2025-02-07T12:44:32.651+0100 info service@v0.120.0/service.go:253 Everything is ready. Begin running and processing data. - 2025-02-07T12:47:46.721+0100 info Traces {"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 4, "spans": 4} - 2025-02-07T12:47:46.721+0100 info ResourceSpans #0 - Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1 - Resource attributes: - ``` - -**Check the `gateway-traces.out` file** -Using `jq`, count the number of traces in the recreated `gateway-traces.out`. It should match the number you send when the `gateway` was down. - -{{% tabs %}} -{{% tab title="Check Gateway Traces Out File" %}} - -```bash -jq '.resourceSpans | length | "\(.) resourceSpans found"' gateway-traces.out -``` - -{{% /tab %}} - -{{% tab title="Example output" %}} - -```text -"5 resourceSpans found" -``` - -{{% /tab %}} -{{% /tabs %}} - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} - -### Conclusion - -This exercise demonstrated how to enhance the resilience of the OpenTelemetry Collector by configuring the `file_storage` extension, enabling retry mechanisms for the `otlp` exporter, and using a file-backed queue for temporary data storage. - -By implementing file-based checkpointing and queue persistence, you ensure the telemetry pipeline can gracefully recover from temporary interruptions, making it a more robust and reliable for production environments. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/_index.md deleted file mode 100644 index cc27da85c8..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/4-building-resilience/_index.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -title: 4. Building In Resilience -linkTitle: 4. Building Resilience -time: 10 minutes -weight: 6 ---- - -The OpenTelemetry Collector’s [**FileStorage Extension**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/19bc7d6ee854c0c1b5c97d8d348e5b9d1199e8aa/extension/storage/filestorage/README.md) enhances the resilience of your telemetry pipeline by providing reliable checkpointing, managing retries, and handling temporary failures effectively. - -With this extension enabled, the OpenTelemetry Collector can store intermediate states on disk, preventing data loss during network disruptions and allowing it to resume operations seamlessly. - -{{% notice note %}} - -This solution will work for metrics as long as the connection downtime is brief—up to 15 minutes. If the downtime exceeds this, Splunk Observability Cloud will drop data due to datapoints being out of order. - -For logs, there are plans to implement a more enterprise-ready solution in one of the upcoming Splunk OpenTelemetry Collector releases. - -{{% /notice %}} - -{{% notice title="Exercise" style="green" icon="running" %}} - -- Inside the `[WORKSHOP]` directory, create a new subdirectory named `4-resilience`. -- Next, copy `*.yaml` from the `3-filelog` directory into `4-resilience`. - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `[WORKSHOP]/4-resilience` directory.** - -Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/5-dropping-spans/5-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/5-dropping-spans/5-1-configuration.md deleted file mode 100644 index 0c6dfd6529..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/5-dropping-spans/5-1-configuration.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -title: 5.1 Configuration -linkTitle: 5.1 Configuration -weight: 1 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -Switch to your **Gateway terminal** window and open the `gateway.yaml` file. Update the `processors` section with the following configuration: - -1. **Add a `filter` processor**: - Configure the gateway to exclude spans with the name `/_healthz`. The `error_mode: ignore` directive ensures that any errors encountered during filtering are ignored, allowing the pipeline to continue running smoothly. The `traces` section defines the filtering rules, specifically targeting spans named `/_healthz` for exclusion. - - ```yaml - filter/health: # Defines a filter processor - error_mode: ignore # Ignore errors - traces: # Filtering rules for traces - span: # Exclude spans named "/_healthz" - - 'name == "/_healthz"' - ``` - -2. **Add the `filter` processor to the `traces` pipeline**: - Include the `filter/health` processor in the `traces` pipeline. For optimal performance, place the filter as early as possible—right after the `memory_limiter` and before the `batch` processor. Here’s how the configuration should look: - - ```yaml - traces: - receivers: - - otlp - processors: - - memory_limiter - - filter/health # Filters data based on rules - - resource/add_mode - - batch - exporters: - - debug - - file/traces - ``` - -This setup ensures that health check-related spans (`/_healthz`) are filtered out early in the pipeline, reducing unnecessary noise in your telemetry data. - -{{% /notice %}} - -Validate the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `traces:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(filter
fa:fa-microchip
health):::processor - PRO5(batch
fa:fa-microchip):::processor - EXP1( debug 
fa:fa-upload):::exporter - EXP2(  file  
fa:fa-upload
traces):::exporter - %% Links - subID1:::sub-traces - subgraph " " - subgraph subID1["`**Traces**`"] - direction LR - REC1 --> PRO1 - PRO1 --> PRO4 - PRO4 --> PRO3 - PRO3 --> PRO5 - PRO5 --> EXP1 - PRO5 --> EXP2 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/5-dropping-spans/5-2-test-filter.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/5-dropping-spans/5-2-test-filter.md deleted file mode 100644 index 770573c1b0..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/5-dropping-spans/5-2-test-filter.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: 5.2 Test Filter Processor -linkTitle: 5.2 Test Filter Processor -weight: 2 ---- - -To test your configuration, you'll need to generate some trace data that includes a span named `"/_healthz"`. - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Gateway**: In your **Gateway terminal** window start the `gateway`. - -```bash -../otelcol --config ./gateway.yaml -``` - -**Start the Agent**: In your **Agent terminal** window start the `agent`. - -```bash -../otelcol --config ./agent.yaml -``` - -**Start the Loadgen**: In the **Spans terminal** window run the `loadgen` with the flag to also send `healthz` spans along with base spans: - -```bash -../loadgen -health -count 5 -``` - -**Verify `agent.out`**: Using `jq` confirm the name of the spans received by the `agent`: - -{{% tabs %}} -{{% tab title="Check spans in agent.out" %}} - -```bash -jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./agent.out -``` - -{{% /tab %}} -{{% tab title="Example output" %}} - -```text -"Span 1 found with name /movie-validator" -"Span 2 found with name /_healthz" -"Span 3 found with name /movie-validator" -"Span 4 found with name /_healthz" -"Span 5 found with name /movie-validator" -"Span 6 found with name /_healthz" -"Span 7 found with name /movie-validator" -"Span 8 found with name /_healthz" -"Span 9 found with name /movie-validator" -"Span 10 found with name /_healthz" -``` - -{{% /tab %}} -{{% /tabs %}} - -**Check the Gateway Debug output**: Using `jq` confirm the name of the spans received by the `gateway`: - -{{% tabs %}} -{{% tab title="Check spans in gateway-traces.out" %}} - -```bash { title="Check spans in gateway-traces.out" } -jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./gateway-traces.out -``` - -{{% /tab %}} -{{% tab title="Example output" %}} - -The `gateway-metrics.out` file will not contain any spans named `/_healthz`. - -```text -"Span 1 found with name /movie-validator" -"Span 2 found with name /movie-validator" -"Span 3 found with name /movie-validator" -"Span 4 found with name /movie-validator" -"Span 5 found with name /movie-validator" -``` - -{{% /tab %}} -{{% /tabs %}} - -{{% /notice %}} - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} - -When using the `Filter` processor, make sure you understand the look of your incoming data and test the configuration thoroughly. In general, use **as specific a configuration as possible** to lower the risk of the wrong data being dropped. - -You can further extend this configuration to filter out spans based on different attributes, tags, or other criteria, making the OpenTelemetry Collector more customizable and efficient for your observability needs. - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} - diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/5-dropping-spans/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/5-dropping-spans/_index.md deleted file mode 100644 index 0accea59f5..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/5-dropping-spans/_index.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -title: 5. Dropping Spans -linkTitle: 5. Dropping Spans -time: 10 minutes -weight: 7 ---- - -In this section, we will explore how to use the [**Filter Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md) to selectively drop spans based on certain conditions. - -Specifically, we will drop traces based on the span name, which is commonly used to filter out unwanted spans such as health checks or internal communication traces. In this case, we will be filtering out spans whose name is `"/_healthz"`, typically associated with health check requests and usually are quite "**noisy**". - -{{% notice title="Exercise" style="green" icon="running" %}} - -- Inside the `[WORKSHOP]` directory, create a new subdirectory named `5-dropping-spans`. -- Next, copy `*.yaml` from the `4-resilience` directory into `5-dropping-spans`. - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `[WORKSHOP]/5-dropping-spans` directory.** - -Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /notice %}} - -Next, we will configure the **filter processor** and the respective pipelines. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/6-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/6-1-configuration.md deleted file mode 100644 index cc7e5b77a0..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/6-1-configuration.md +++ /dev/null @@ -1,126 +0,0 @@ ---- -title: 6.1 Configuration -linkTitle: 6.1 Configuration -weight: 1 ---- - -In this step, we'll modify `agent.yaml` to include the `attributes` and `redaction` processors. These processors will help ensure that sensitive data within span attributes is properly handled before being logged or exported. - -Previously, you may have noticed that some span attributes displayed in the console contained personal and sensitive data. We'll now configure the necessary processors to filter out and redact this information effectively. - -```text - -Attributes: - -> user.name: Str(George Lucas) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.account_password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - {"kind": "exporter", "data_type": "traces", "name": "debug"} -``` - -{{% notice title="Exercise" style="green" icon="running" %}} - -Switch to your **Agent terminal** window and open the `agent.yaml` file in your editor. We’ll add two processors to enhance the security and privacy of your telemetry data: the Attributes Processor and the Redaction Processor. - -**Add an `attributes` Processor**: The [**Attributes Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor) allows you to modify span attributes (tags) by updating, deleting, or hashing their values. This is particularly useful for obfuscating sensitive information before it is exported. - -In this step, we’ll: - -1. **Update** the `user.phone_number` attribute to a static value `("UNKNOWN NUMBER")`. -2. **Hash** the `user.email` attribute to ensure the original email is not exposed. -3. **Delete** the `user.password` attribute to remove it entirely from the span. - -```yaml - attributes/update: - actions: # Actions - - key: user.phone_number # Target key - action: update # Update action - value: "UNKNOWN NUMBER" # New value - - key: user.email # Target key - action: hash # Hash the email value - - key: user.password # Target key - action: delete # Delete the password - ``` - -**Add a `redaction` Processor**: The [**The Redaction Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/redactionprocessor) detects and redacts sensitive data in span attributes based on predefined patterns, such as credit card numbers or other personally identifiable information (PII). - -In this step: - -- We set `allow_all_keys: true` to ensure all attributes are processed (if set to `false`, only explicitly allowed keys are retained). - -- We define `blocked_values` with regular expressions to detect and redact **Visa** and **MasterCard** credit card numbers. - -- The `summary: debug` option logs detailed information about the redaction process for debugging purposes. - -```yaml - redaction/redact: - allow_all_keys: true # If false, only allowed keys will be retained - blocked_values: # List of regex patterns to block - - '\b4[0-9]{3}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b' # Visa - - '\b5[1-5][0-9]{2}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b' # MasterCard - summary: debug # Show debug details about redaction -``` - -**Update the `traces` Pipeline**: Integrate both processors into the `traces` pipeline. Make sure that you comment out the redaction processor at first (we will enable it later in a separate exercise): - -> [!NOTE] -> Leave the `redaction/redact` processor commented out in this exercise. We will enable it in an upcoming exercise. - -```yaml - traces: - receivers: - - otlp - processors: - - memory_limiter - - attributes/update # Update, hash, and remove attributes - #- redaction/redact # Redact sensitive fields using regex - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - file - - otlphttp -``` - -{{% /notice %}} - -Validate the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `traces:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRML(memory_limiter
fa:fa-microchip):::processor - PRRD(resourcedetection
fa:fa-microchip):::processor - PRRS(resource
fa:fa-microchip
add_mode):::processor - PRBA(batch
fa:fa-microchip):::processor - PRUP(attributes
fa:fa-microchip
update):::processor - EXP1(otlphttp
fa:fa-upload):::exporter - EXP2(  debug  
fa:fa-upload):::exporter - EXP3(file
fa:fa-upload):::exporter - - %% Links - subID1:::sub-traces - subgraph " " - subgraph subID1["`**Traces**`"] - direction LR - REC1 --> PRML - PRML --> PRUP - PRUP --> PRRD - PRRD --> PRRS - PRRS --> PRBA - PRBA --> EXP2 - PRBA --> EXP3 - PRBA --> EXP1 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/6-2-test-delete-tag.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/6-2-test-delete-tag.md deleted file mode 100644 index 90b6eeb449..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/6-2-test-delete-tag.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -title: 6.2 Test Attribute Processor -linkTitle: 6.2 Test Attribute Processor -weight: 2 ---- - -In this exercise, we will **delete** the `user.account_password`, **update** the `user.phone_number` **attribute** and **hash** the `user.email` in the span data before it is exported by the `agent`. - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Gateway**: In your **Gateway terminal** window start the `gateway`. - -```bash -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: In your **Agent terminal** window start the `agent`. - -```bash -../otelcol --config=agent.yaml -``` - -**Start the Load Generator**: In the **Spans terminal** window start the `loadgen`: - -```bash -../loadgen -count 1 -``` - -**Check the debug output**: For both the `agent` and `gateway` debug output, confirm that `user.account_password` has been removed, and both `user.phone_number` & `user.email` have been updated. - -{{% tabs %}} -{{% tab title="New Debug Output" %}} - - ```text - -> user.name: Str(George Lucas) - -> user.phone_number: Str(UNKNOWN NUMBER) - -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287) - -> payment.amount: Double(51.71) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - ``` - -{{% /tab %}} -{{% tab title="Original Debug Output" %}} - - ```text - -> user.name: Str(George Lucas) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - -> payment.amount: Double(95.22) - ``` - -{{% /tab %}} -{{% /tabs %}} - -**Check file output**: Using `jq` validate that `user.account_password` has been removed, and `user.phone_number` & `user.email` have been updated in `gateway-taces.out`: - -{{% tabs %}} -{{% tab title="Validate attribute changes" %}} - -```bash -jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.password" or .key == "user.phone_number" or .key == "user.email") | {key: .key, value: .value.stringValue}' ./gateway-traces.out -``` - -{{% /tabs %}} -{{% tab title="Output" %}} - -Notice that the `user.account_password` has been removed, and the `user.phone_number` & `user.email` have been updated: - -```json -{ - "key": "user.phone_number", - "value": "UNKNOWN NUMBER" -} -{ - "key": "user.email", - "value": "62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287" -} -``` - -{{% /tab %}} -{{% /tabs %}} - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/6-3-test-redaction.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/6-3-test-redaction.md deleted file mode 100644 index 929837a64e..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/6-3-test-redaction.md +++ /dev/null @@ -1,141 +0,0 @@ ---- -title: 6.3 Test Redaction Processor -linkTitle: 6.3 Test Redaction Processor -weight: 3 ---- - -The `redaction` processor gives precise control over which attributes and values are **permitted** or **removed** from telemetry data. - -In this exercise, we will **redact** the `user.visa` & `user.mastercard` **values** in the span data before it is exported by the `agent`. -{{% notice title="Exercise" style="green" icon="running" %}} - -**Prepare the terminals**: Delete the `*.out` files and clear the screen. - -**Start the Gateway**: In your **Gateway terminal** window start the `gateway`. - -```bash -../otelcol --config=gateway.yaml -``` - -**Enable the `redaction/redact` processor**: In the **Agent terminal** window, edit `agent.yaml` and remove the `#` we inserted in the previous exercise. - -```yaml - traces: - receivers: - - otlp - processors: - - memory_limiter - - attributes/update # Update, hash, and remove attributes - - redaction/redact # Redact sensitive fields using regex - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - file - - otlphttp -``` - -**Start the Agent**: In your **Agent terminal** window start the `agent`. - -```bash -../otelcol --config=agent.yaml -``` - -**Start the Load Generator**: In the **Spans terminal** window start the `loadgen`: - -```bash -../loadgen -count 1 -``` - -**Check the debug output**: For both the `agent` and `gateway` confirm the values for `user.visa` & `user.mastercard` have been updated. Notice `user.amex` attribute value was NOT redacted because a matching regex pattern was not added to `blocked_values` - -{{% tabs %}} -{{% tab title="New Debug Output" %}} - - ```text - -> user.name: Str(George Lucas) - -> user.phone_number: Str(UNKNOWN NUMBER) - -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287) - -> payment.amount: Double(69.71) - -> user.visa: Str(****) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(****) - -> redaction.masked.keys: Str(user.mastercard,user.visa) - -> redaction.masked.count: Int(2) - ``` - -{{% /tab %}} -{{% tab title="Original Debug Output" %}} - - ```text - -> user.name: Str(George Lucas) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - -> payment.amount: Double(65.54) - ``` - -{{% /tab %}} -{{% /tabs %}} - -{{% notice note %}} -By including `summary:debug` in the redaction processor, the debug output will include summary information about which matching key values were redacted, along with the count of values that were masked. - -```text - -> redaction.masked.keys: Str(user.mastercard,user.visa) - -> redaction.masked.count: Int(2) - ``` - -{{% /notice %}} - -**Check file output**: Using `jq` verify that `user.visa` & `user.mastercard` have been updated in the `gateway-traces.out`. - -{{% tabs %}} -{{% tab title="Validate attribute changes" %}} - -```bash -jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.visa" or .key == "user.mastercard" or .key == "user.amex") | {key: .key, value: .value.stringValue}' ./gateway-traces.out -``` - -{{% /tabs %}} -{{% tab title="Output" %}} - -Notice that `user.amex` has not been redacted because a matching regex pattern was not added to `blocked_values`: - -```json -{ - "key": "user.visa", - "value": "****" -} -{ - "key": "user.amex", - "value": "3782 822463 10005" -} -{ - "key": "user.mastercard", - "value": "****" -} -``` - -{{% /tab %}} -{{% /tabs %}} - -These are just a few examples of how `attributes` and `redaction` processors can be configured to protect sensitive data. - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} - diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/_index.md deleted file mode 100644 index f3268e0b59..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/6-sensitive-data/_index.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: 6. Redacting Sensitive Data -linkTitle: 6. Sensitive Data -time: 10 minutes -weight: 8 ---- - -In this section, you'll learn how to configure the OpenTelemetry Collector to remove specific tags and redact sensitive data from telemetry spans. This is crucial for protecting sensitive information such as credit card numbers, personal data, or other security-related details that must be anonymized before being processed or exported. - -We'll walk through configuring key processors in the OpenTelemetry Collector, including: - -- **[Attributes Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/README.md)**: Modifies or removes specific span attributes. -- [**Redaction Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/redactionprocessor/README.md): Ensures sensitive data is sanitized before being stored or transmitted. - -{{% notice title="Exercise" style="green" icon="running" %}} - -- Inside the `[WORKSHOP]` directory, create a new subdirectory named `6-sensitive-data`. -- Next, copy `*.yaml` from the `5-dropping-spans` directory into `6-sensitive-data`. - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `[WORKSHOP]/6-sensitive-data` directory.** - -Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/7-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/7-1-configuration.md deleted file mode 100644 index ec65eb6fd8..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/7-1-configuration.md +++ /dev/null @@ -1,118 +0,0 @@ ---- -title: 7.1 Configuration -linkTitle: 7.1 Configuration -weight: 1 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} -**Add a `transform` processor**: Switch to your **Agent terminal** window and edit the `agent.yaml` and add the following `transform` processor: - -```yaml - transform/logs: # Processor Type/Name - log_statements: # Log Processing Statements - - context: resource # Log Context - statements: # List of attribute keys to keep - - keep_keys(attributes, ["com.splunk.sourcetype", "host.name", "otelcol.service.mode"]) -``` - -By using the `-context: resource` key we are targeting the `resourceLog` attributes of logs. - -This configuration ensures that only the relevant resource attributes (`com.splunk.sourcetype`, `host.name`, `otelcol.service.mode`) are retained, improving log efficiency and reducing unnecessary metadata. - -**Adding a Context Block for Log Severity Mapping**: To properly set the `severity_text` and `severity_number` fields of a log record, we add a `log` context block within `log_statements`. This configuration extracts the `level` value from the log body, maps it to `severity_text`, and assigns the corresponding `severity_number` based on the log level: - -```yaml - - context: log # Log Context - statements: # Transform Statements Array - - set(cache, ParseJSON(body)) where IsMatch(body, "^\\{") # Parse JSON log body into a cache object - - flatten(cache, "") # Flatten nested JSON structure - - merge_maps(attributes, cache, "upsert") # Merge cache into attributes, updating existing keys - - set(severity_text, attributes["level"]) # Set severity_text from the "level" attribute - - set(severity_number, 1) where severity_text == "TRACE" # Map severity_text to severity_number - - set(severity_number, 5) where severity_text == "DEBUG" - - set(severity_number, 9) where severity_text == "INFO" - - set(severity_number, 13) where severity_text == "WARN" - - set(severity_number, 17) where severity_text == "ERROR" - - set(severity_number, 21) where severity_text == "FATAL" -``` - -The `merge_maps` function is used to combine two maps (dictionaries) into one. In this case, it merges the `cache` object (containing parsed JSON data from the log body) into the `attributes` map. - -- **Parameters**: - - `attributes`: The target map where the data will be merged. - - `cache`: The source map containing the parsed JSON data. - - `"upsert"`: This mode ensures that if a key already exists in the `attributes` map, its value will be updated with the value from `cache`. If the key does not exist, it will be inserted. - -This step is crucial because it ensures that all relevant fields from the log body (e.g., `level`, `message`, etc.) are added to the `attributes` map, making them available for further processing or exporting. - -**Summary of Key Transformations**: - -- **Parse JSON**: Extracts structured data from the log body. -- **Flatten JSON**: Converts nested JSON objects into a flat structure. -- **Merge Attributes**: Integrates extracted data into log attributes. -- **Map Severity Text**: Assigns severity_text from the log’s level attribute. -- **Assign Severity Numbers**: Converts severity levels into standardized numerical values. - -You should have a **single** `transform` processor containing two context blocks: one whose context is for `resource` and one whose context is for `log`. - -This configuration ensures that log severity is correctly extracted, standardized, and structured for efficient processing. - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -This method of mapping all JSON fields to top-level attributes should only be used for **testing and debugging OTTL**. It will result in high cardinality in a production scenario. -{{% /notice %}} - -**Update the `logs` pipeline**: Add the `transform/logs:` processor into the `logs:` pipeline: - -```yaml - logs: - receivers: - - otlp - - filelog/quotes - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - transform/logs # Transform logs processor - - batch - exporters: - - debug - - otlphttp -``` - -{{% /notice %}} - -Validate the agent configuration using [**https://otelbin.io**](https://otelbin.io/). For reference, the `logs:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - REC2(filelog
fa:fa-download
quotes):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(resourcedetection
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(transform
fa:fa-microchip
logs):::processor - PRO5(batch
fa:fa-microchip):::processor - EXP1(otlphttp
fa:fa-upload):::exporter - EXP2(  debug  
fa:fa-upload):::exporter - %% Links - subID1:::sub-logs - subgraph " " - subgraph subID1[**Logs**] - direction LR - REC1 --> PRO1 - REC2 --> PRO1 - PRO1 --> PRO2 - PRO2 --> PRO3 - PRO3 --> PRO4 - PRO4 --> PRO5 - PRO5 --> EXP2 - PRO5 --> EXP1 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/7-2-setup.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/7-2-setup.md deleted file mode 100644 index f62e4ebc26..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/7-2-setup.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: 7.2 Setup Environment -linkTitle: 7.2 Setup Environment -weight: 2 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Gateway**: In the **Gateway terminal** run: - -```bash { title="Start the Gateway" } -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: In the **Agent terminal** run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Start the Load Generator**: Open the **Logs terminal** window and run the `loadgen`. - -> [!IMPORTANT] -> To ensure the logs are structured in JSON format, include the `-json` flag when starting the script. - -```bash { title="Log Generator" } -../loadgen -logs -json -count 5 -``` - -The `loadgen` will write 5 log lines to `./quotes.log` in JSON format. - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/7-3-test-transform.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/7-3-test-transform.md deleted file mode 100644 index 766ba6cd02..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/7-3-test-transform.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: 7.3 Test Transform Processor -linkTitle: 7.3 Test Transform Processor -weight: 3 ---- - -This test verifies that the `com.splunk/source` and `os.type` metadata have been **removed** from the log resource attributes before being exported by the `agent`. Additionally, the test ensures that: - -1. The log body is parsed to extract severity information. - - `SeverityText` and `SeverityNumber` are set on the `LogRecord`. -2. JSON fields from the log body are promoted to log `attributes`. - -This ensures proper metadata filtering, severity mapping, and structured log enrichment before export. - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Check the debug output**: For both the `agent` and `gateway` confirm that `com.splunk/source` and `os.type` have been removed: - -{{% tabs %}} -{{% tab title="New Debug Output" %}} - - ```text -Resource attributes: - -> com.splunk.sourcetype: Str(quotes) - -> host.name: Str(workshop-instance) - -> otelcol.service.mode: Str(agent) - ``` - -{{% /tab %}} -{{% tab title="Original Debug Output" %}} - - ```text -Resource attributes: - -> com.splunk.source: Str(./quotes.log) - -> com.splunk.sourcetype: Str(quotes) - -> host.name: Str(workshop-instance) - -> os.type: Str(linux) - -> otelcol.service.mode: Str(agent) - ``` - -{{% /tab %}} -{{% /tabs %}} - -For both the `agent` and `gateway` confirm that `SeverityText` and `SeverityNumber` in the `LogRecord` is now defined with the severity `level` from the log body. Confirm that the JSON fields from the body can be accessed as top-level log `Attributes`: - -{{% tabs %}} -{{% tab title="New Debug Output" %}} - -```text - -SeverityText: WARN -SeverityNumber: Warn(13) -Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"}) -Attributes: - -> log.file.path: Str(quotes.log) - -> level: Str(WARN) - -> message: Str(Your focus determines your reality.) - -> movie: Str(SW) - -> timestamp: Str(2025-03-07 11:17:26) - -``` - -{{% /tab %}} -{{% tab title="Original Debug Output" %}} - -```text - -SeverityText: -SeverityNumber: Unspecified(0) -Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"}) -Attributes: - -> log.file.path: Str(quotes.log) - -``` - -{{% /tab %}} -{{% /tabs %}} - -**Check file output**: In the new `gateway-logs.out` file verify the data has been transformed: - -{{% tabs %}} -{{% tab title="jq Query" %}} - -```bash -jq '[.resourceLogs[].scopeLogs[].logRecords[] | {severityText, severityNumber, body: .body.stringValue}]' gateway-logs.out -``` - -{{% /tabs %}} -{{% tab title="Example Output" %}} - -```json -[ - { - "severityText": "DEBUG", - "severityNumber": 5, - "body": "{\"level\":\"DEBUG\",\"message\":\"All we have to decide is what to do with the time that is given us.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}" - }, - { - "severityText": "WARN", - "severityNumber": 13, - "body": "{\"level\":\"WARN\",\"message\":\"The Force will be with you. Always.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}" - }, - { - "severityText": "ERROR", - "severityNumber": 17, - "body": "{\"level\":\"ERROR\",\"message\":\"One does not simply walk into Mordor.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}" - }, - { - "severityText": "DEBUG", - "severityNumber": 5, - "body": "{\"level\":\"DEBUG\",\"message\":\"Do or do not, there is no try.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}" - } -] -[ - { - "severityText": "ERROR", - "severityNumber": 17, - "body": "{\"level\":\"ERROR\",\"message\":\"There is some good in this world, and it's worth fighting for.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}" - } -] -``` - -{{% /tab %}} -{{% /tabs %}} - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/_index.md deleted file mode 100644 index 88809ff7b3..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/7-transform-data/_index.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -title: 7. Transform Data -linkTitle: 7. Transform Data -time: 10 minutes -weight: 9 ---- - -The [**Transform Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md) lets you modify telemetry data—logs, metrics, and traces—as it flows through the pipeline. Using the **OpenTelemetry Transformation Language (OTTL)**, you can filter, enrich, and transform data on the fly without touching your application code. - -In this exercise we’ll update `agent.yaml` to include a **Transform Processor** that will: - -- **Filter** log resource attributes. -- **Parse** JSON structured log data into attributes. -- **Set** log severity levels based on the log message body. - -You may have noticed that in previous logs, fields like `SeverityText` and `SeverityNumber` were undefined. This is typical of the `filelog` receiver. However, the severity is embedded within the log body: - -```text - -SeverityText: -SeverityNumber: Unspecified(0) -Body: Str(2025-01-31 15:49:29 [WARN] - Do or do not, there is no try.) - -``` - -Logs often contain structured data encoded as JSON within the log body. Extracting these fields into attributes allows for better indexing, filtering, and querying. Instead of manually parsing JSON in downstream systems, OTTL enables automatic transformation at the telemetry pipeline level. - -{{% notice title="Exercise" style="green" icon="running" %}} - -- Inside the `[WORKSHOP]` directory, create a new subdirectory named `7-transform-data`. -- Next, copy `*.yaml` from the `6-sensitve-data` directory into `7-transform-data`. - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `[WORKSHOP]/7-transform-data` directory.** - -Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/8-1-connector.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/8-1-connector.md deleted file mode 100644 index a59135171b..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/8-1-connector.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: 8.1 Configure the Routing Connector -linkTitle: 8.1 Routing Configuration -weight: 1 ---- - -In this exercise, you will configure the **Routing Connector** in the `gateway.yaml` file. This setup enables the `gateway` to route traces based on the `deployment.environment` attribute in the spans you send. By implementing this, you can process and handle traces differently depending on their attributes. - -{{% notice title="Exercise" style="green" icon="running" %}} - -In OpenTelemetry configuration files, `connectors` have their own dedicated section, similar to receivers and processors. - -**Add the `routing` connector**: -In the **Gateway terminal** window, edit `gateway.yaml` and uncomment the `#connectors:` section. Then, add the following below the `connectors:` section: - -```yaml -connectors: - routing: - default_pipelines: [traces/standard] # Default pipeline if no rule matches - error_mode: ignore # Ignore errors in routing - table: # Define routing rules - # Routes spans to a target pipeline if the resourceSpan attribute matches the rule - - statement: route() where attributes["deployment.environment"] == "security-applications" - pipelines: [traces/security] # Target pipeline -``` - -The rules above apply to traces, but this approach also applies to `metrics` and `logs`, allowing them to be routed based on attributes in `resourceMetrics` or `resourceLogs`. - -**Configure `file:` exporters**: The `routing` connector requires separate targets for routing. In the `exporters:` section, add two file exporters, `file/traces/security` and `file/traces/standard`, to ensure data is directed correctly. - -```yaml - file/traces/standard: # Exporter for regular traces - path: "./gateway-traces-standard.out" # Path for saving trace data - append: false # Overwrite the file each time - file/traces/security: # Exporter for security traces - path: "./gateway-traces-security.out" # Path for saving trace data - append: false # Overwrite the file each time -``` - -{{% /notice %}} - -With the `routing` configuration complete, the next step is to configure the `pipelines` to apply these routing rules. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/8-2-pipelines.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/8-2-pipelines.md deleted file mode 100644 index 377e568878..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/8-2-pipelines.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -title: 8.2 Configuring the Pipelines -linkTitle: 8.2 Pipeline Configuration -weight: 2 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Update the `traces` pipeline to use routing**: - -1. To enable `routing`, update the original `traces:` pipeline by using `routing` as the only exporter. This ensures all span data is sent through the routing connector for evaluation. -2. Remove all processors and replace it with an empty array (`[]`). These are now defined in the `traces/standard` and `traces/security` pipelines. - - ```yaml - pipelines: - traces: # Original traces pipeline - receivers: - - otlp # OTLP Receiver - processors: [] - exporters: - - routing # Routing Connector - ``` - -**Add both the `standard` and `security` traces pipelines**: - -1. **Configure the Security pipeline**: This pipeline will handle all spans that match the routing rule for `security`. -This uses `routing` as its receiver. Place it below the existing `traces:` pipeline: - - ```yaml - traces/security: # New Security Traces/Spans Pipeline - receivers: - - routing # Receive data from the routing connector - processors: - - memory_limiter # Memory Limiter Processor - - resource/add_mode # Adds collector mode metadata - - batch - exporters: - - debug # Debug Exporter - - file/traces/security # File Exporter for spans matching rule - ``` - -2. **Add the Standard pipeline**: This pipeline processes all spans that do not match the routing rule. -This pipeline is also using `routing` as its receiver. Add this below the `traces/security` one: - - ```yaml - traces/standard: # Default pipeline for unmatched spans - receivers: - - routing # Receive data from the routing connector - processors: - - memory_limiter # Memory Limiter Processor - - resource/add_mode # Adds collector mode metadata - - batch - exporters: - - debug # Debug exporter - - file/traces/standard # File exporter for unmatched spans - ``` - -{{% /notice %}} - -Validate the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `traces:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(   otlp   
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(memory_limiter
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(resource
fa:fa-microchip
add_mode):::processor - PRO5(batch
fa:fa-microchip):::processor - PRO6(batch
fa:fa-microchip):::processor - EXP1(  debug  
fa:fa-upload):::exporter - EXP2(  file  
fa:fa-upload
traces):::exporter - EXP3(  debug  
fa:fa-upload):::exporter - EXP4(  file  
fa:fa-upload
traces):::exporter - ROUTE1( routing 
fa:fa-route):::con-export - ROUTE2( routing 
fa:fa-route):::con-receive - ROUTE3( routing 
fa:fa-route):::con-receive - %% Links - subID1:::sub-traces - subID2:::sub-traces - subID3:::sub-traces - subgraph " " - direction LR - subgraph subID1["`**Traces**`"] - REC1 --> ROUTE1 - end - subgraph subID2[**Traces/standard**] - ROUTE1 --> ROUTE2 - ROUTE2 --> PRO1 - PRO1 --> PRO3 - PRO3 --> PRO5 - PRO5 --> EXP1 - PRO5 --> EXP2 - end - subgraph subID3[**Traces/security**] - ROUTE1 --> ROUTE3 - ROUTE3 --> PRO2 - PRO2 --> PRO4 - PRO4 --> PRO6 - PRO6 --> EXP3 - PRO6 --> EXP4 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/8-3-test-routing.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/8-3-test-routing.md deleted file mode 100644 index a2b4f53c12..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/8-3-test-routing.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: 8.3 Test Routing Connector -linkTitle: 8.3 Test Routing Connector -weight: 3 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -In this section, we will test the `routing` rule configured for the **Gateway**. The expected result is that the`span` generated by the `loadgen` will be sent to the `gateway-traces-security.out` file. - -**Start the Gateway**: In your **Gateway terminal** window start the `gateway`. - -```bash -../otelcol --config gateway.yaml -``` - -**Start the Agent**: In your **Agent terminal** window start the `agent`. - -```bash -../otelcol --config agent.yaml -``` - -**Send a Regular Span**: In the **Spans terminal** window send a regular span using the `loadgen`: - -```bash -../loadgen -count 1 -``` - -Both the `agent` and `gateway` will display debug information. The gateway will also generate a new `gateway-traces-standard.out` file, as this is now the designated destination for regular spans. - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -If you check `gateway-traces-standard.out`, it will contain the `span` sent by `loadgen`. You will also see an empty `gateway-traces-security.out` file, as the routing configuration creates output files immediately, even if no matching spans have been processed yet. -{{% /notice %}} - -**Send a Security Span**: In the **Spans terminal** window send a security span using the `security` flag: - -```bash -../loadgen -security -count 1 -``` - -Again, both the `agent` and `gateway` should display debug information, including the span you just sent. This time, the `gateway` will write a line to the `gateway-traces-security.out` file, which is designated for spans where the `deployment.environment` resource attribute matches `"security-applications"`. -The `gateway-traces-standard.out` should be unchanged. - -{{% tabs %}} -{{% tab title="Validate resource attribute matches" %}} - -```bash -jq -c '.resourceSpans[] as $resource | $resource.scopeSpans[].spans[] | {spanId: .spanId, deploymentEnvironment: ($resource.resource.attributes[] | select(.key == "deployment.environment") | .value.stringValue)}' gateway-traces-security.out -``` - -{{% /tabs %}} -{{% tab title="Output" %}} - -```json -{"spanId":"cb799e92e26d5782","deploymentEnvironment":"security-applications"} -``` - -{{% /tab %}} -{{% /tabs %}} - -You can repeat this scenario multiple times, and each trace will be written to its corresponding output file. - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} - -## Conclusion - -In this section, we successfully tested the routing connector in the gateway by sending different spans and verifying their destinations. - -- **Regular spans** were correctly routed to `gateway-traces-standard.out`, confirming that spans without a matching `deployment.environment` attribute follow the default pipeline. - -- **Security-related spans** were routed to `gateway-traces-security.out`, demonstrating that the routing rule based on `"deployment.environment": "security-applications"` works as expected. - -By inspecting the output files, we confirmed that the OpenTelemetry Collector *correctly evaluates span attributes and routes them to the appropriate destinations*. This validates that routing rules can effectively separate and direct telemetry data for different use cases. - -You can now extend this approach by defining additional routing rules to further categorize spans, metrics, and logs based on different attributes. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/_index.md deleted file mode 100644 index f969d77bb8..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/8-routing-data/_index.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: 8. Routing Data -linkTitle: 8. Routing Data -time: 10 minutes -weight: 10 ---- - -The [**Routing Connector**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/routingconnector) in OpenTelemetry is a powerful feature that allows you to direct data (`traces`, `metrics`, or `logs`) to different pipelines based on specific criteria. This is especially useful in scenarios where you want to apply different processing or exporting logic to subsets of your telemetry data. - -For example, you might want to send *production* data to one exporter while directing *test* or *development* data to another. Similarly, you could route certain spans based on their attributes, such as service name, environment, or span name, to apply custom processing or storage logic. - -{{% notice title="Exercise" style="green" icon="running" %}} - -- Inside the `[WORKSHOP]` directory, create a new subdirectory named `8-routing-data`. -- Next, copy `*.yaml` from the `7-transform-data` directory into `8-routing-data`. -- Change **all** terminal windows to the `[WORKSHOP]/8-routing-data` directory. - -Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /notice %}} - -Next, we will configure the routing connector and the respective pipelines. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/9-1-count-test.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/9-1-count-test.md deleted file mode 100644 index 0315a81f44..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/9-1-count-test.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: 9.1 Testing the Count Connector -linkTitle: 9.1 Test Count Connector -weight: 1 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Gateway**: -In the **Gateway terminal** window navigate to the `[WORKSHOP]/9-sum-count` directory and run: - -```bash { title="Start the Gateway" } -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: -In the **Agent terminal** window navigate to the `[WORKSHOP]/9-sum-count` directory and run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Send 12 Logs lines with the Loadgen**: -In the **Spans terminal** window navigate to the `[WORKSHOP]/9-sum-count` directory. -Send 12 log lines, they should be read in two intervals. Do this with the following `loadgen` command: - -```bash { title="Loadgen" } -../loadgen -logs -json -count 12 -``` - -Both the `agent` and `gateway` will display debug information, showing they are processing data. Wait until the loadgen completes. - -**Verify metrics have been generated** -As the logs are processed, the **Agent** generates metrics and forwards them to the **Gateway**, which then writes them to `gateway-metrics.out`. - -To check if the metrics `logs.full.count`, `logs.sw.count`, `logs.lotr.count`, and `logs.error.count` are present in the output, run the following **jq** query: - -{{% tabs %}} -{{% tab title="jq query command" %}} - -```bash -jq '.resourceMetrics[].scopeMetrics[].metrics[] - | select(.name == "logs.full.count" or .name == "logs.sw.count" or .name == "logs.lotr.count" or .name == "logs.error.count") - | {name: .name, value: (.sum.dataPoints[0].asInt // "-")}' gateway-metrics.out -``` - -{{% /tab %}} -{{% tab title="jq example output" %}} - -```json -{ - "name": "logs.sw.count", - "value": "2" -} -{ - "name": "logs.lotr.count", - "value": "2" -} -{ - "name": "logs.full.count", - "value": "4" -} -{ - "name": "logs.error.count", - "value": "2" -} -{ - "name": "logs.error.count", - "value": "1" -} -{ - "name": "logs.sw.count", - "value": "2" -} -{ - "name": "logs.lotr.count", - "value": "6" -} -{ - "name": "logs.full.count", - "value": "8" -} -``` - -{{% /tab %}} -{{% /tabs %}} -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -Note: the `logs.full.count` normally is equal to `logs.sw.count` + `logs.lotr.count`, while the `logs.error.count` will be a random number. -{{% /notice %}} - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/9-2-sum.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/9-2-sum.md deleted file mode 100644 index ff13f37511..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/9-2-sum.md +++ /dev/null @@ -1,158 +0,0 @@ ---- -title: Create metrics with Sum Connector -linkTitle: 9.2 Sum Connector -time: 10 minutes -weight: 2 ---- - -In this section, we’ll explore how the [**Sum Connector**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/sumconnector) can extract values from spans and convert them into metrics. - -We’ll specifically use the credit card charges from our base spans and leverage the Sum Connector to retrieve the total charges as a metric. - -The connector can be used to collect (**sum**) attribute values from spans, span events, metrics, data points, and log records. It captures each individual value, transforms it into a metric, and passes it along. However, it’s the **backend’s** job to use these metrics and their attributes for calculations and further processing. - -{{% notice title="Exercise" style="green" icon="running" %}} - -Switch to your **Agent terminal** window and open the `agent.yaml` file in your editor. - -- **Add the Sum Connector** -Include the Sum Connector in the connectors section of your configuration and define the metrics counters: - -```yaml - sum: - spans: - user.card-charge: - source_attribute: payment.amount - conditions: - - attributes["payment.amount"] != "NULL" - attributes: - - key: user.name - -``` - -{{% /notice %}} - -In the example above, we check for the `payment.amount` attribute in spans. If it has a valid value, the **Sum** connector generates a metric called `user.card-charge` and includes the `user.name` as an attribute. This enables the backend to track and display a user’s total charges over an extended period, such as a billing cycle. - -In the pipeline configuration below, the connector exporter is added to the traces section, while the connector receiver is added to the metrics section. - -{{% notice title="Exercise" style="green" icon="running" %}} - -- **Configure the Count Connector in the pipelines** - -```yaml - pipelines: - traces: - receivers: - - otlp - processors: - - memory_limiter - - attributes/update # Update, hash, and remove attributes - - redaction/redact # Redact sensitive fields using regex - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - file - - otlphttp - - sum # Sum connector which aggregates payment.amount from spans and sends to metrics pipeline - metrics: - receivers: - - sum # Receives metrics from the sum exporter in the traces pipeline - - count # Receives count metric from logs count exporter in logs pipeline. - - otlp - #- hostmetrics # Host Metrics Receiver - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - otlphttp - logs: - receivers: - - otlp - - filelog/quotes - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - transform/logs # Transform logs processor - - batch - exporters: - - count # Count Connector that exports count as a metric to metrics pipeline. - - debug - - otlphttp -``` - -- **Validate** the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `traces` and `metrics:` sections of your pipelines will look like this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(otlp
fa:fa-download
):::receiver - REC3(otlp
fa:fa-download
):::receiver - PRO1(memory_limiter
fa:fa-microchip
):::processor - PRO2(memory_limiter
fa:fa-microchip
):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(resource
fa:fa-microchip
add_mode):::processor - PRO5(batch
fa:fa-microchip
):::processor - PRO6(batch
fa:fa-microchip
):::processor - PRO7(resourcedetection
fa:fa-microchip
):::processor - PRO8(resourcedetection
fa:fa-microchip
):::processor - - PROA(attributes
fa:fa-microchip
redact):::processor - PROB(redaction
fa:fa-microchip
update):::processor - EXP1(  debug  
fa:fa-upload
):::exporter - EXP2(  file  
fa:fa-upload
):::exporter - EXP3(  debug  
fa:fa-upload
):::exporter - EXP4(  otlphttp  
fa:fa-upload
):::exporter - EXP5(  otlphttp  
fa:fa-upload
):::exporter - ROUTE1( sum 
fa:fa-route
):::con-export - ROUTE2( count 
fa:fa-route
):::con-receive - ROUTE3( sum 
fa:fa-route
):::con-receive - - %% Links - subID1:::sub-traces - subID2:::sub-metrics - subgraph " " - direction LR - subgraph subID1["`**Traces**`"] - direction LR - REC1 --> PRO1 - PRO1 --> PROA - PROA --> PROB - PROB --> PRO7 - PRO7 --> PRO3 - PRO3 --> PRO5 - PRO5 --> EXP1 - PRO5 --> EXP2 - PRO5 --> EXP5 - PRO5 --> ROUTE1 - end - - subgraph subID2["`**Metrics**`"] - direction LR - ROUTE1 --> ROUTE3 - ROUTE3 --> PRO2 - ROUTE2 --> PRO2 - REC3 --> PRO2 - PRO2 --> PRO8 - PRO8 --> PRO4 - PRO4 --> PRO6 - PRO6 --> EXP3 - PRO6 --> EXP4 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3; -classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3; -classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3; -``` - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/9-3-sum-test.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/9-3-sum-test.md deleted file mode 100644 index bd507a84b0..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/9-3-sum-test.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -title: 9.3 Testing the Count Connector -linkTitle: 9.3 Test Sum Connector -weight: 3 ---- - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Start the Gateway**: -In the **Gateway terminal** window navigate to the `[WORKSHOP]/9-sum-count` directory and run: - -```bash { title="Start the Gateway" } -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: -In the **Agent terminal** window navigate to the `[WORKSHOP]/9-sum-count` directory and run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Start the Loadgen**: -In the **Spans terminal** window navigate to the `[WORKSHOP]/9-sum-count` directory. Send 8 spans with the following `loadgen` command: - -```bash { title="Loadgen" } -../loadgen -count 8 -``` - -Both the `agent` and `gateway` will display debug information, showing they are processing data. Wait until the loadgen completes. - -**Verify that metrics** -While processing the span, the **Agent**, has generated metrics and passed them on to the **Gateway**. The **Gateway** has written them into `gateway-metrics.out`. - -To confirm the presence of `user.card-charge`and verify each one has an attribute `user.name` in the metrics output, run the following jq query: - -{{% tabs %}} -{{% tab title="jq query command" %}} - -```bash - -jq -r '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "user.card-charge") | .sum.dataPoints[] | "\(.attributes[] | select(.key == "user.name").value.stringValue)\t\(.asDouble)"' gateway-metrics.out | while IFS=$'\t' read -r name charge; do - printf "%-20s %s\n" "$name" "$charge" -done -``` - -{{% /tab %}} -{{% tab title="jq example output" %}} - -```text -George Lucas 67.49 -Frodo Baggins 87.14 -Thorin Oakenshield 90.98 -Luke Skywalker 51.37 -Luke Skywalker 65.56 -Thorin Oakenshield 67.5 -Thorin Oakenshield 66.66 -Peter Jackson 94.39 -``` - -{{% /tab %}} -{{% /tabs %}} - -> [!IMPORTANT] -> Stop the `agent` and the `gateway` processes by pressing `Ctrl-C` in their respective terminals. - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/_index.md deleted file mode 100644 index 42c8b1d7d6..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/9-sum-count/_index.md +++ /dev/null @@ -1,191 +0,0 @@ ---- -title: Create metrics with Count Connector -linkTitle: 9. Count & Sum Connector -time: 10 minutes -weight: 11 ---- -In this section, we'll explore how to use the [**Count Connector**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/countconnector) to extract attribute values from logs and convert them into meaningful metrics. - -Specifically, we'll use the Count Connector to track the number of "Star Wars" and "Lord of the Rings" quotes appearing in our logs, turning them into measurable data points. - -{{% notice title="Exercise" style="green" icon="running" %}} - -- Inside the `[WORKSHOP]` directory, create a new subdirectory named `9-sum-count`. -- Next, copy `*.yaml` from the `8-routing-data` directory into `9-sum-count`. -- Change **all** terminal windows to the `[WORKSHOP]/9-sum-count` directory. - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -- **Update the agent.yaml** to change the frequency that we read logs. -Find the `filelog/quotes` receiver in the `agent.yaml` and add a `poll_interval` attribute: - -```yaml - filelog/quotes: # Receiver Type/Name - poll_interval: 10s # Only read every ten seconds -``` - -{{% /notice %}} - -The reason for the delay is that the Count Connector in the OpenTelemetry Collector counts logs only within each processing interval. This means that every time the data is read, the count resets to zero for the next interval. With the default `Filelog reciever` interval of 200ms, it reads every line the loadgen writes, giving us counts of 1. With this interval we make sure we have multiple entries to count. - -The Collector can maintain a running count for each read interval by omitting conditions, as shown below. However, it’s best practice to let your backend handle running counts since it can track them over a longer time period. - -{{% notice title="Exercise" style="green" icon="running" %}} - -- **Add the Count Connector** - -Include the Count Connector in the connector's section of your configuration and define the metrics counters we want to use: - -```yaml -connectors: - count: - logs: - logs.full.count: - description: "Running count of all logs read in interval" - logs.sw.count: - description: "StarWarsCount" - conditions: - - attributes["movie"] == "SW" - logs.lotr.count: - description: "LOTRCount" - conditions: - - attributes["movie"] == "LOTR" - logs.error.count: - description: "ErrorCount" - conditions: - - attributes["level"] == "ERROR" -``` - -- **Explanation of the Metrics Counters** - - - `logs.full.count`: Tracks the total number of logs processed during each read interval. - Since this metric has no filtering conditions, every log that passes through the system is included in the count. - - `logs.sw.count` Counts logs that contain a quote from a Star Wars movie. - - `logs.lotr.count`: Counts logs that contain a quote from a Lord of the Rings movie. - - `logs.error.count`: Represents a real-world scenario by counting logs with a severity level of ERROR for the read interval. - -- **Configure the Count Connector in the pipelines** -In the pipeline configuration below, the connector exporter is added to the `logs` section, while the connector receiver is added to the `metrics` section. - -```yaml - pipelines: - traces: - receivers: - - otlp - processors: - - memory_limiter - - attributes/update # Update, hash, and remove attributes - - redaction/redact # Redact sensitive fields using regex - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - file - - otlphttp - metrics: - receivers: - - count # Count Connector that receives count metric from logs count exporter in logs pipeline. - - otlp - #- hostmetrics # Host Metrics Receiver - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - otlphttp - logs: - receivers: - - otlp - - filelog/quotes - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - transform/logs # Transform logs processor - - batch - exporters: - - count # Count Connector that exports count as a metric to metrics pipeline. - - debug - - otlphttp -``` - -{{% /notice %}} - -We count logs based on their attributes. If your log data is stored in the log body instead of attributes, you’ll need to use a `Transform` processor in your pipeline to extract key/value pairs and add them as attributes. - -In this workshop, we’ve already added `merge_maps(attributes, cache, "upsert")` in the `07-transform` section. This ensures that all relevant data is included in the log attributes for processing. - -When selecting fields to create attributes from, be mindful—adding all fields indiscriminately is generally not ideal for production environments. Instead, choose only the fields that are truly necessary to avoid unnecessary data clutter. - -{{% notice title="Exercise" style="green" icon="running" %}} - -- **Validate** the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `logs` and `metrics:` sections of your pipelines will look like this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(otlp
fa:fa-download):::receiver - REC2(filelog
fa:fa-download
quotes):::receiver - REC3(otlp
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(memory_limiter
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(resource
fa:fa-microchip
add_mode):::processor - PRO5(batch
fa:fa-microchip):::processor - PRO6(batch
fa:fa-microchip):::processor - PRO7(resourcedetection
fa:fa-microchip):::processor - PRO8(resourcedetection
fa:fa-microchip):::processor - PRO9(transfrom
fa:fa-microchip
logs):::processor - EXP1(  debug  
fa:fa-upload):::exporter - EXP2(  otlphttp  
fa:fa-upload):::exporter - EXP3(  debug  
fa:fa-upload):::exporter - EXP4(  otlphttp  
fa:fa-upload):::exporter - ROUTE1( count 
fa:fa-route):::con-export - ROUTE2( count 
fa:fa-route):::con-receive - - %% Links - subID1:::sub-logs - subID2:::sub-metrics - subgraph " " - direction LR - subgraph subID1[**Logs**] - direction LR - REC1 --> PRO1 - REC2 --> PRO1 - PRO1 --> PRO7 - PRO7 --> PRO3 - PRO3 --> PRO9 - PRO9 --> PRO5 - PRO5 --> ROUTE1 - PRO5 --> EXP1 - PRO5 --> EXP2 - end - - subgraph subID2["`**Metrics**`"] - direction LR - ROUTE1 --> ROUTE2 - ROUTE2 --> PRO2 - REC3 --> PRO2 - PRO2 --> PRO8 - PRO8 --> PRO4 - PRO4 --> PRO6 - PRO6 --> EXP3 - PRO6 --> EXP4 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3; -classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3; -``` - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/99-end/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/99-end/_index.md deleted file mode 100644 index 15be8e510a..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/99-end/_index.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Wrap-up -linkTitle: 10. Wrap-up -weight: 12 ---- - -![Well done](../images/welldone.png) diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/_index.md deleted file mode 100644 index c25747e55c..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/_index.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Advanced Collector Configuration -description: Practice setting up the OpenTelemetry Collector configuration from scratch and go though several advanced configuration scenarios's. -weight: 2 -archetype: chapter -authors: ["Robert Castley", "Charity Anderson", "Pieter Hagen", "Geoff Higginbottom"] -time: 90 minutes -hidden: true ---- - -The goal of this workshop is to help you gain confidence in creating and modifying OpenTelemetry Collector configuration files. You’ll start with a minimal `agent.yaml` file and progressively build it out to handle several advanced, real-world scenarios. - -A key focus of this workshop is learning how to configure the OpenTelemetry Collector to store telemetry data locally, rather than sending it to a third-party vendor backend. This approach not only simplifies debugging and troubleshooting but is also ideal for testing and development environments where you want to avoid sending data to production systems. - -To make the most of this workshop, you should have: - -- A basic understanding of the OpenTelemetry Collector and its configuration file structure. -- Proficiency in editing YAML files. - -Everything in this workshop is designed to run locally, ensuring a hands-on and accessible learning experience. Let’s dive in and start building! - -## Workshop Overview - -During this workshop, we will cover the following topics: - -- **Setting up the agent locally**: Add metadata, and introduce the debug and file exporters. -- **Configuring a gateway**: Route traffic from the agent to the gateway. -- **Configuring the FileLog receiver**: Collect log data from various log files. -- **Enhancing agent resilience**: Basic configurations for fault tolerance. -- **Configuring processors**: - - Filter out noise by dropping specific spans (e.g., health checks). - - Remove unnecessary tags, and handle sensitive data. - - Transform data using OTTL (OpenTelemetry Transformation Language) in the pipeline before exporting. -- **Configuring Connectors**: - - Route data to different endpoints based on the values received. - - Convert log and span data to metrics. - -By the end of this workshop, you'll be familiar with configuring the OpenTelemetry Collector for a variety of real-world use cases. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/images/welldone.png b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/images/welldone.png deleted file mode 100644 index b041ff09bf..0000000000 Binary files a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/images/welldone.png and /dev/null differ diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/prerequisites.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/prerequisites.md deleted file mode 100644 index 5c92008a34..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector-old/prerequisites.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Pre-requisites -weight: 2.1 -archetype: chapter -time: 90 minutes ---- - -### Prerequisites - -- Proficiency in editing YAML files using `vi`, `vim`, `nano`, or your preferred text editor. -- Supported Environments: - - Splunk Workshop Instance (preferred) - - Apple Mac (Apple Silicon). Installation of `jq` is required - [**https://jqlang.org/download/**](https://jqlang.org/download/) - -{{% notice title="Exercise" style="green" icon="running" %}} - -**Create a workshop directory**: In your environment create a new directory (e.g., `advanced-otel-workshop`). We will refer to this directory as `[WORKSHOP]` for the remainder of the workshop. - -**Download workshop binaries**: Change into your `[WORKSHOP]` directory and download the OpenTelemetry Collector and Load Generator binaries: - -{{% tabs %}} -{{% tab title="Splunk Workshop Instance" %}} - -```bash -curl -L https://github.com/signalfx/splunk-otel-collector/releases/download/v{{< legacy-otel-version >}}/otelcol_linux_amd64 -o otelcol && \ -curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 -o loadgen -``` - -**Update file permissions**: Once downloaded, update the file permissions to make both executable: - -```bash -chmod +x otelcol loadgen && \ -./otelcol -v && \ -./loadgen --help -``` - -{{% /tab %}} -{{% tab title="Apple Silicon" %}} - -```bash -curl -L https://github.com/signalfx/splunk-otel-collector/releases/download/v{{< legacy-otel-version >}}/otelcol_darwin_arm64 -o otelcol && \ -curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-arm64 -o loadgen -``` - -{{% notice style="warning" title="macOS Users" icon="desktop" %}} -Before running the binaries on macOS, you need to remove the quarantine attribute that macOS applies to downloaded files. This step ensures they can execute without restrictions. - -Run the following command in your terminal: - -```bash { title="Remove Quarantine Attribute"} -xattr -dr com.apple.quarantine otelcol && \ -xattr -dr com.apple.quarantine loadgen -``` - -**Update file permissions**: Once downloaded, update the file permissions to make both executable: - -```bash -chmod +x otelcol loadgen && \ -./otelcol -v && \ -./loadgen --help -``` - -{{% /notice %}} - -{{% /tab %}} -{{% /tabs %}} - -```text { title="Initial Directory Structure" } -[WORKSHOP] -├── otelcol # OpenTelemetry Collector binary -└── loadgen # Load Generator binary -``` - - -{{% /notice %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-1-gateway.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-1-gateway.md deleted file mode 100644 index 2c135af713..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-1-gateway.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -title: 1.1 Verify Gateway Configuration -linkTitle: 1.1 Gateway Configuration -weight: 1 ---- - -The **OpenTelemetry Gateway** serves as a central hub for receiving, processing, and exporting telemetry data. It sits between your telemetry sources (such as applications and services) and your observability backends like Splunk Observability Cloud. - -By centralizing telemetry traffic, the gateway enables advanced features such as data filtering, enrichment, transformation, and routing to one or more destinations. It helps reduce the burden on individual services by offloading telemetry processing and ensures consistent, standardized data across distributed systems. - -This makes your observability pipeline easier to manage, scale, and analyze—especially in complex, multi-service environments. - -{{% exercise title="Inspect the gateway configuration" %}} - -Open or create your second terminal window and name it **Gateway**. Navigate to the first exercise directory `[WORKSHOP]/1-agent-gateway` -then check the contents of the `gateway.yaml` file. - -This file outlines the core structure of the OpenTelemetry Collector as deployed in **Gateway** mode. - -{{% /exercise %}} - -## Understanding the Gateway Configuration - -Let’s explore the `gateway.yaml` file that defines how the OpenTelemetry Collector is configured in **Gateway** mode during this workshop. This **Gateway** is responsible for receiving telemetry from the **Agent**, then processing and exporting it for inspection or forwarding. - -* **OTLP Receiver (Custom Port)** - - ```yaml - receivers: - otlp: - protocols: - http: - endpoint: "0.0.0.0:5318" - ``` - - The port `5318` matches the `otlphttp` exporter in the **Agent** configuration, ensuring that all telemetry data sent by the **Agent** is accepted by the **Gateway**. - - > [!NOTE] - > This separation of ports avoids conflicts and keeps responsibilities clear between agent and gateway roles. - -* **File Exporters** - - The **Gateway** uses three file exporters to output telemetry data to local files. These exporters are defined as: - - ```yaml - exporters: # List of exporters - debug: # Debug exporter - verbosity: detailed # Enable detailed debug output - file/traces: # Exporter Type/Name - path: "./gateway-traces.out" # Path for OTLP JSON output for traces - append: false # Overwrite the file each time - file/metrics: # Exporter Type/Name - path: "./gateway-metrics.out" # Path for OTLP JSON output for metrics - append: false # Overwrite the file each time - file/logs: # Exporter Type/Name - path: "./gateway-logs.out" # Path for OTLP JSON output for logs - append: false # Overwrite the file each time - ``` - - Each exporter writes a specific signal type to its corresponding file. - - These files are created once the gateway is started and will be populated with real telemetry as the agent sends data. You can monitor these files in real time to observe the flow of telemetry through your pipeline. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-2-send-metrics.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-2-send-metrics.md deleted file mode 100644 index f824d71bfe..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-2-send-metrics.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -title: 1.2 Validate & Test Configuration -linkTitle: 1.2 Validate & Test Configuration -weight: 3 ---- - -Now, we can start the **Gateway** and the **Agent**, which is configured to automatically send **Host Metrics** at startup. We do this to verify that data is properly routed from the **Agent** to the **Gateway**. - -{{% exercise title="Start the Gateway and Agent" %}} - -**Gateway**: In the **Gateway terminal** window, run the following command to start the **Gateway**: - -```bash {title="Start the Gateway"} -../otelcol --config=gateway.yaml -``` - -If everything is configured correctly, the collector will start and state `Everything is ready. Begin running and processing data.` in the output, similar to the following: - -```text -2025-06-09T09:22:11.944+0100 info service@v0.126.0/service.go:289 Everything is ready. Begin running and processing data. {"resource": {}} -``` - -Once the **Gateway** is running, it will listen for incoming data on port `5318` and export the received data to the following files: - -* `gateway-traces.out` -* `gateway-metrics.out` -* `gateway-logs.out` - -**Start the Agent**: In the **Agent terminal** window start the agent with the agent configuration: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Verify CPU Metrics**: - -1. Check that when the **Agent** starts, it immediately starts sending **CPU** metrics. -2. Both the **Agent** and the **Gateway** will display this activity in their debug output. The output should resemble the following snippet: - -```text - -NumberDataPoints #31 -Data point attributes: - -> cpu: Str(cpu3) - -> state: Str(wait) -StartTimestamp: 2025-07-07 16:49:42 +0000 UTC -Timestamp: 2025-07-09 09:36:21.190226459 +0000 UTC -Value: 77.380000 - {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"} -``` - -At this stage, the **Agent** continues to collect **CPU** metrics once per hour or upon each restart and sends them to the gateway. The **Gateway** processes these metrics and exports them to a file named `gateway-metrics.out`. This file stores the exported metrics as part of the pipeline service. - -**Verify Data arrived at Gateway**: To confirm that CPU metrics, specifically for `cpu0`, have successfully reached the gateway, we’ll inspect the `gateway-metrics.out` file using the `jq` command. - -The following command filters and extracts the `system.cpu.time` metric, focusing on `cpu0`. It displays the metric’s state (e.g., `user`, `system`, `idle`, `interrupt`) along with the corresponding values. - -Open or create your third terminal window and name it **Tests**. Run the command below in the **Tests terminal** to check the `system.cpu.time` metric: - -{{% tabs %}} -{{% tab title="Check CPU Metrics" %}} - -```bash -jq '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "system.cpu.time") | .sum.dataPoints[] | select(.attributes[0].value.stringValue == "cpu0") | {cpu: .attributes[0].value.stringValue, state: .attributes[1].value.stringValue, value: .asDouble}' gateway-metrics.out -``` - -{{% /tab %}} -{{% tab title="Example Output" %}} - -```json -{ - "cpu": "cpu0", - "state": "user", - "value": 123407.02 -} -{ - "cpu": "cpu0", - "state": "system", - "value": 64866.6 -} -{ - "cpu": "cpu0", - "state": "idle", - "value": 216427.87 -} -{ - "cpu": "cpu0", - "state": "interrupt", - "value": 0 -} -``` - -{{% /tab %}} -{{% /tabs %}} - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-3-send-traces.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-3-send-traces.md deleted file mode 100644 index fed2c2c4c8..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-3-send-traces.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -title: 1.3 Send traces from the Agent to the Gateway -linkTitle: 1.3 Send Traces -weight: 4 -hidden: true -draft: true ---- - -{{% exercise title="Send a test trace" %}} - -**Send a Test Trace**: - -1. Validate **Agent** and **Gateway** are still running. -2. In the **Loadgen terminal** window, run the following command to send 5 spans and validate the output of the **Agent** and **Gateway** debug logs: - -{{% tabs %}} -{{% tab title="Start the Load Generator" %}} - -```bash -../loadgen -count 5 -``` - -{{% /tab %}} -{{% tab title="Agent/Gateway Debug Output" %}} - -```text -2025-03-06T11:49:00.456Z info Traces {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1} -2025-03-06T11:49:00.456Z info ResourceSpans #0 -Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1 -Resource attributes: - -> service.name: Str(cinema-service) - -> deployment.environment: Str(production) - -> host.name: Str(workshop-instance) - -> os.type: Str(linux) - -> otelcol.service.mode: Str(agent) -ScopeSpans #0 -ScopeSpans SchemaURL: -InstrumentationScope cinema.library 1.0.0 -InstrumentationScope attributes: - -> fintest.scope.attribute: Str(Starwars, LOTR) -Span #0 - Trace ID : 97fb4e5b13400b5689e3306da7cff077 - Parent ID : - ID : 413358465e5b4f15 - Name : /movie-validator - Kind : Server - Start time : 2025-03-06 11:49:00.431915 +0000 UTC - End time : 2025-03-06 11:49:01.431915 +0000 UTC - Status code : Ok - Status message : Success -Attributes: - -> user.name: Str(George Lucas) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - -> payment.amount: Double(87.01) - {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"} -``` - -{{% /tab %}} -{{% /tabs %}} - -**Verify the Gateway has handled the spans**: Once the gateway processes incoming spans, it writes the trace data to a file named `gateway-traces.out`. To confirm that the spans have been successfully handled, you can inspect this file. - -In your **Tests terminal**, use the `jq` command to extract and display key details about each span, such as its `spanId` and its position in the file. Also, we can extract the attributes that the **Hostmetrics Receiver** added to the spans. - -{{% tabs %}} -{{% tab title="Inspect the Gateway Trace File" %}} - -```bash -jq -c '.resourceSpans[] as $resource | $resource.scopeSpans[].spans[] | "Span \(input_line_number) found with spanId \(.spanId), hostname \($resource.resource.attributes[] | select(.key == "host.name") | .value.stringValue), os \($resource.resource.attributes[] | select(.key == "os.type") | .value.stringValue)"' ./gateway-traces.out -``` - -{{% /tab %}} -{{% tab title="Example Output" %}} - -```text -"Span 1 found with spanId d71fe6316276f97d, hostname workshop-instance, os linux" -"Span 2 found with spanId e8d19489232f8c2a, hostname workshop-instance, os linux" -"Span 3 found with spanId 9dfaf22857a6bd05, hostname workshop-instance, os linux" -"Span 4 found with spanId c7f544a4b5fef5fc, hostname workshop-instance, os linux" -"Span 5 found with spanId 30bb49261315969d, hostname workshop-instance, os linux" -``` - -{{% /tab %}} -{{% /tabs %}} - - - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-4-send-logs.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-4-send-logs.md deleted file mode 100644 index ce2875e6de..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/1-4-send-logs.md +++ /dev/null @@ -1,154 +0,0 @@ ---- -title: 1.4 Send Logs -linkTitle: 1.4 Send Logs -weight: 5 -hidden: true -draft: true ---- - -{{% exercise title="Send logs through the pipeline" %}} - -**Start the log load generator:** In the **Loadgen terminal** window, run the following command to start the `loadgen`: - -```bash -../loadgen -logs -``` - -A continuous stream of log data from the `quotes.log` will be in the **Agent** and **Gateway** debug logs: - -```text { title="Agent/Gateway Debug Output" } -Timestamp: 1970-01-01 00:00:00 +0000 UTC -SeverityText: -SeverityNumber: Unspecified(0) -Body: Str(2025-03-06 15:18:32 [ERROR] - There is some good in this world, and it's worth fighting for. LOTR) -Attributes: - -> log.file.path: Str(quotes.log) -Trace ID: -Span ID: -Flags: 0 -LogRecord #1 -``` - -**Stop the `loadgen`**: In the **Loadgen terminal** window, stop the `loadgen` using `Ctrl-C`. - -**Verify the gateway**: Check if the **Gateway** has written a `./gateway-logs.out` file. - -At this point, your directory structure will appear as follows: - -```text { title="Updated Directory Structure" } -. -├── agent.out -├── agent.yaml -├── gateway-logs.out # Output from the logs pipeline -├── gateway-metrics.out # Output from the metrics pipeline -├── gateway-traces.out # Output from the traces pipeline -├── gateway.yaml -└── quotes.log # File containing Random log lines -``` - -**Examine a log line**: In `gateway-logs.out` compare a log line with the snippet below. Verify that the log entry includes the same attributes as we have seen in metrics and traces data previously: - -{{% tabs %}} -{{% tab title="cat /gateway-logs.out" %}} - -```json -{"resourceLogs":[{"resource":{"attributes":[{"key":"com.splunk.source","value":{"stringValue":"./quotes.log"}},{"key":"com.splunk.sourcetype","value":{"stringValue":"quotes"}},{"key":"host.name","value":{"stringValue":"workshop-instance"}},{"key":"os.type","value":{"stringValue":"linux"}},{"key":"otelcol.service.mode","value":{"stringValue":"gateway"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1741274312475540000","body":{"stringValue":"2025-03-06 15:18:32 [DEBUG] - All we have to decide is what to do with the time that is given us. LOTR"},"attributes":[{"key":"log.file.path","value":{"stringValue":"quotes.log"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1741274312475560000","body":{"stringValue":"2025-03-06 15:18:32 [DEBUG] - Your focus determines your reality. SW"},"attributes":[{"key":"log.file.path","value":{"stringValue":"quotes.log"}}],"traceId":"","spanId":""}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.6.1"}]} -``` - -{{% /tab %}} -{{% tab title="cat ./gateway-logs.out | jq" %}} - -```json -{ - "resourceLogs": [ - { - "resource": { - "attributes": [ - { - "key": "com.splunk.source", - "value": { - "stringValue": "./quotes.log" - } - }, - { - "key": "com.splunk.sourcetype", - "value": { - "stringValue": "quotes" - } - }, - { - "key": "host.name", - "value": { - "stringValue": "workshop-instance" - } - }, - { - "key": "os.type", - "value": { - "stringValue": "linux" - } - }, - { - "key": "otelcol.service.mode", - "value": { - "stringValue": "gateway" - } - } - ] - }, - "scopeLogs": [ - { - "scope": {}, - "logRecords": [ - { - "observedTimeUnixNano": "1741274312475540000", - "body": { - "stringValue": "2025-03-06 15:18:32 [DEBUG] - All we have to decide is what to do with the time that is given us. LOTR" - }, - "attributes": [ - { - "key": "log.file.path", - "value": { - "stringValue": "quotes.log" - } - } - ], - "traceId": "", - "spanId": "" - }, - { - "observedTimeUnixNano": "1741274312475560000", - "body": { - "stringValue": "2025-03-06 15:18:32 [DEBUG] - Your focus determines your reality. SW" - }, - "attributes": [ - { - "key": "log.file.path", - "value": { - "stringValue": "quotes.log" - } - } - ], - "traceId": "", - "spanId": "" - } - ] - } - ], - "schemaUrl": "https://opentelemetry.io/schemas/1.6.1" - } - ] -} -``` - -{{% /tab %}} -{{% /tabs %}} - -You may also have noticed that every log line contains empty placeholders for `"traceId":""` and `"spanId":""`. -The FileLog receiver will populate these fields only if they are not already present in the log line. -For example, if the log line is generated by an application instrumented with an OpenTelemetry instrumentation library, these fields will already be included and will not be overwritten. - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/_index.md deleted file mode 100644 index 64a3d8de95..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent-gateway/_index.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: 1. Verify Agent Configuration -linkTitle: 1. Agent Configuration -time: 15 minutes -weight: 3 ---- -Welcome! In this section, we’ll begin with a fully functional OpenTelemetry setup that includes both an **Agent** and a **Gateway**. - -We’ll start by quickly reviewing their configuration files to get familiar with the overall structure and to highlight key sections that control the telemetry pipeline. - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -Throughout the workshop, you’ll work with multiple terminal windows. To keep things organized, give each terminal a unique name or color. This will help you easily recognize and switch between them during the exercises. - -We will refer to these terminals as: **Agent**, **Gateway**, **Loadgen**, and **Test**. -{{% /notice %}} - -{{% exercise title="Verify the agent files" %}} - -1. Create your first terminal window and name it **Agent**. Navigate to the directory for the first exercise `[WORKSHOP]/1-agent-gateway` and verify that the required files have been generated: - - ```bash - cd 1-agent-gateway - ls -l - ``` - -2. You should see the following files in the directory. If not, re-run the `setup-workshop.sh` script as described in the **Pre-requisites** section: - - ```text { title="Directory Structure" } - . - ├── agent.yaml - └── gateway.yaml - ``` - -{{% /exercise %}} - -## Understanding the Agent configuration - -Let’s review the key components of the `agent.yaml` file used in this workshop. We’ve made some important additions to support metrics, traces, and logs. - -### Receivers - -The `receivers` section defines how the **Agent** ingests telemetry data. In this setup, three types of receivers have been configured: - -* **Host Metrics Receiver** - - ```yaml - hostmetrics: # Host Metrics Receiver - collection_interval: 3600s # Collection Interval (1hr) - scrapers: - cpu: # CPU Scraper - ``` - - Collects CPU usage from the local system every hour. We’ll use this to generate example metric data. - -* **OTLP Receiver (HTTP protocol)** - - ```yaml - otlp: # OTLP Receiver - protocols: - http: # Configure HTTP protocol - endpoint: "0.0.0.0:4318" # Endpoint to bind to - ``` - - Enables the agent to receive metrics, traces, and logs over HTTP on port `4318`. This is used to send data to the collector in future exercises. - -* **FileLog Receiver** - - ```yaml - filelog/quotes: # Receiver Type/Name - include: ./quotes.log # The file to read log data from - include_file_path: true # Include file path in the log data - include_file_name: false # Exclude file name from the log data - resource: # Add custom resource attributes - com.splunk.source: ./quotes.log # Source of the log data - com.splunk.sourcetype: quotes # Source type of the log data - ``` - - Enables the agent to tail a local log file (`quotes.log`) and convert it to structured log events, enriched with metadata such as `source` and `sourceType`. - -### Exporters - -* **Debug Exporter** - - ```yaml - debug: # Exporter Type - verbosity: detailed # Enabled detailed debug output - ``` - -* **OTLPHTTP Exporter** - - ```yaml - otlphttp: # Exporter Type - endpoint: "http://localhost:5318" # Gateway OTLP endpoint - ``` - - The `debug` exporter sends data to the console for visibility and debugging during the workshop while the `otlphttp` exporter forwards all telemetry to the local **Gateway** instance. - - **This dual-export strategy ensures you can see the raw data locally while also sending it downstream for further processing and export.** diff --git a/content/en/conf/3-obs1184/1-agent/1-1-agent.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-1-agent.md similarity index 100% rename from content/en/conf/3-obs1184/1-agent/1-1-agent.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-1-agent.md diff --git a/content/en/conf/3-obs1184/1-agent/1-2-send-metrics.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-2-send-metrics.md similarity index 100% rename from content/en/conf/3-obs1184/1-agent/1-2-send-metrics.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-2-send-metrics.md diff --git a/content/en/conf/3-obs1184/1-agent/1-3-send-traces.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-3-send-traces.md similarity index 100% rename from content/en/conf/3-obs1184/1-agent/1-3-send-traces.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-3-send-traces.md diff --git a/content/en/conf/3-obs1184/1-agent/1-4-send-logs.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-4-send-logs.md similarity index 100% rename from content/en/conf/3-obs1184/1-agent/1-4-send-logs.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-4-send-logs.md diff --git a/content/en/conf/3-obs1184/1-agent/1-5-cloud-validation.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-5-cloud-validation.md similarity index 100% rename from content/en/conf/3-obs1184/1-agent/1-5-cloud-validation.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-5-cloud-validation.md diff --git a/content/en/conf/3-obs1184/1-agent/1-6-config-builder.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-6-config-builder.md similarity index 96% rename from content/en/conf/3-obs1184/1-agent/1-6-config-builder.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-6-config-builder.md index fc2843f9c2..e51673f584 100644 --- a/content/en/conf/3-obs1184/1-agent/1-6-config-builder.md +++ b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/1-6-config-builder.md @@ -22,7 +22,7 @@ your computer. {{% tab title="Splunk Show instance" %}} Open the -[OBS1184 example Collector configuration](https://github.com/splunk/observability-workshop/blob/main/workshop/ninja/obs1184/agent_config.yaml). +[OBS1184 example Collector configuration](https://github.com/splunk/observability-workshop/blob/main/workshop/ninja/advanced-otel/agent_config.yaml). On GitHub, select **Download raw file** and save `agent_config.yaml` on the computer running your browser. @@ -47,7 +47,7 @@ variable references; `workshop-env.sh` can contain your access token. {{% expand title="How this Collector configuration works" %}} The Collector has eight pipelines for three signal types. Six come from the -version `0.157.0` default Collector configuration in the Splunk Distribution of +version `0.161.0` default Collector configuration in the Splunk Distribution of the OpenTelemetry Collector. The two pipelines ending in `/workshop` are for this workshop: diff --git a/content/en/conf/3-obs1184/1-agent/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/_index.md similarity index 100% rename from content/en/conf/3-obs1184/1-agent/_index.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/1-agent/_index.md diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-1-configuration.md deleted file mode 100644 index a232044920..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-1-configuration.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -title: 2.1 File Storage Configuration -linkTitle: 2.1 Configuration -weight: 1 ---- - -In this exercise, we will update the `extensions:` section of the `agent.yaml` file. This section is part of the OpenTelemetry configuration YAML and defines optional components that enhance or modify the OpenTelemetry Collector’s behavior. - -While these components do not process telemetry data directly, they provide valuable capabilities and services to improve the Collector’s functionality. - -{{% exercise title="Add file storage to the Agent" %}} - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `2-building-resilience` directory and run the `clear` command.** - -Your directory structure will look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -**Update the `agent.yaml`**: In the **Agent terminal** window, add the `file_storage` extension under the existing `health_check` extension: - -```yaml - file_storage/checkpoint: # Extension Type/Name - directory: "./checkpoint-dir" # Define directory - create_directory: true # Create directory - timeout: 1s # Timeout for file operations - compaction: # Compaction settings - on_start: true # Start compaction at Collector startup - # Define compaction directory - directory: "./checkpoint-dir/tmp" - max_transaction_size: 65536 # Max. size limit before compaction occurs -``` - -**Add `file_storage` to the exporter**: Modify the `otlphttp` exporter to configure retry and queuing mechanisms, ensuring data is retained and resent if failures occur. Add the following under the `endpoint: "http://localhost:5318"` and make sure the indentation matches `endpoint`: - -```yaml - retry_on_failure: - enabled: true # Enable retry on failure - sending_queue: # - enabled: true # Enable sending queue - num_consumers: 10 # No. of consumers - queue_size: 10000 # Max. queue size - storage: file_storage/checkpoint # File storage extension -``` - -**Update the `services` section**: Add the `file_storage/checkpoint` extension to the existing `extensions:` section and the configuration needs to look like this: - -```yaml -service: - extensions: - - health_check - - file_storage/checkpoint # Enabled extensions for this collector -``` - -**Update the `metrics` pipeline**: For this exercise we are going to comment out the `hostmetrics` receiver from the Metric pipeline to reduce debug and log noise, again the configuration needs to look like this: - -```yaml - metrics: - receivers: - # - hostmetrics # Hostmetric reciever (cpu only) - - otlp -``` - -{{% /exercise %}} - -Validate the **Agent** configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `metrics:` section of your pipelines will look similar to this: - -{{% mermaid %}} -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(resourcedetection
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - EXP1( debug 
fa:fa-upload):::exporter - EXP2(otlphttp
fa:fa-upload):::exporter - EXP3( file 
fa:fa-upload):::exporter - %% Links - subID1:::sub-metrics - subgraph " " - subgraph subID1["`**Metrics**`"] - direction LR - REC1 --> PRO1 - PRO1 --> PRO2 - PRO2 --> PRO3 - PRO3 --> EXP1 - PRO3 --> EXP3 - PRO3 --> EXP2 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3; -{{% /mermaid %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-2-test-environment.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-2-test-environment.md deleted file mode 100644 index 32ec43ca00..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-2-test-environment.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: 2.2 Setup environment for Resilience Testing -linkTitle: 2.2 Setup environment -weight: 2 ---- - -Next, we will configure our environment to be ready for testing the **File Storage** configuration. - -{{% exercise title="Set up the resilience test" %}} - -**Start the Gateway**: In the **Gateway terminal** window run: - -```bash { title="Start the Gateway" } -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: In the **Agent terminal** window run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Send five test spans**: In the **Loadgen terminal** window run: - -```bash { title="Start Load Generator" } -../loadgen -count 5 -``` - -Both the **Agent** and **Gateway** should display debug logs, and the **Gateway** should create a `./gateway-traces.out` file. - -If everything functions correctly, we can proceed with testing system resilience. -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-3-failure.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-3-failure.md deleted file mode 100644 index cf3ee4f3d2..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-3-failure.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -title: 2.3 Simulate Failure -linkTitle: 2.3 Simulate Failure -weight: 3 ---- - -To assess the **Agent's** resilience, we'll simulate a temporary **Gateway** outage and observe how the **Agent** handles it: - -{{% exercise title="Simulate a Gateway outage" %}} - -**Simulate a network failure**: In the **Gateway terminal** stop the **Gateway** with `Ctrl-C` and wait until the gateway console shows that it has stopped. The **Agent** will continue running, but it will not be able to send data to the gateway. The output in the **Gateway terminal** should look similar to this: - -```text -2025-07-09T10:22:37.941Z info service@v0.126.0/service.go:345 Shutdown complete. {"resource": {}} -``` - -**Send traces**: In the **Loadgen terminal** window send five more traces using the `loadgen`. - -```bash { title="Start Load Generator" } -../loadgen -count 5 -``` - -Notice that the agent’s retry mechanism is activated as it continuously attempts to resend the data. In the agent’s console output, you will see repeated messages similar to the following: - -```text -2025-01-28T14:22:47.020+0100 info internal/retry_sender.go:126 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:5318/v1/traces\": dial tcp 127.0.0.1:5318: connect: connection refused", "interval": "9.471474933s"} -``` - -**Stop the Agent**: In the **Agent terminal** window, use `Ctrl-C` to stop the agent. Wait until the agent’s console confirms it has stopped: - -```text -2025-07-09T10:25:59.344Z info service@v0.126.0/service.go:345 Shutdown complete. {"resource": {}} -``` - -{{% /exercise %}} - -> [!IMPORTANT] -> When you stop the agent, any metrics, traces, or logs held in memory for retry will be lost. However, because we have configured the FileStorage Extension, all telemetry that has not yet been accepted by the target endpoint are safely checkpointed on disk. -> -> Stopping the agent is a crucial step to clearly demonstrate how the system recovers when the agent is restarted. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-4-recovery.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-4-recovery.md deleted file mode 100644 index 2f8a100849..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/2-4-recovery.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -title: 2.4 Recovery -linkTitle: 2.4 Recovery -weight: 4 ---- - -In this exercise, we’ll test how the **OpenTelemetry Collector** recovers from a network outage by restarting the **Gateway** collector. When the **Gateway** becomes available again, the **Agent** will resume sending data from its last check-pointed state, ensuring no data loss. - -{{% exercise title="Restart and verify recovery" %}} - -**Restart the Gateway**: In the **Gateway terminal** window run: - -```bash {title="Start the Gateway"} -../otelcol --config=gateway.yaml -``` - -**Restart the Agent**: In the **Agent terminal** window run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -> After the **Agent** is up and running, the **File_Storage** extension will detect buffered data in the checkpoint folder. It will start to dequeue the stored spans from the last checkpoint folder, ensuring no data is lost. - -**Verify the Agent Debug output:** Note that the **Agent** debug output does **NOT** change and still shows the following line indicating no new data is being exported: - - ```text - 2025-07-11T08:31:58.176Z info service@v0.126.0/service.go:289 Everything is ready. Begin running and processing data. {"resource": {}} - ``` - -**Watch the Gateway Debug output** -You should see from the **Gateway** debug screen, it has started receiving the previously missed traces without requiring any additional action on your part e.g.: - - ```txt -Attributes: - -> user.name: Str(Luke Skywalker) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - -> payment.amount: Double(75.75) - {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces"} - ``` - -**Check the `gateway-traces.out` file:** Using `jq`, count the number of traces in the recreated `gateway-traces.out`. It should match the number you send when the **Gateway** was down. - -{{% tabs %}} -{{% tab title="Check Gateway Traces Out File" %}} - -```bash -jq '.resourceSpans | length | "\(.) resourceSpans found"' gateway-traces.out -``` - -{{% /tab %}} - -{{% tab title="Example output" %}} - -```text -"5 resourceSpans found" -``` - -{{% /tab %}} -{{% /tabs %}} - -{{% /exercise %}} - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. - -### Conclusion - -This exercise demonstrated how to enhance the resilience of the OpenTelemetry Collector by configuring the `file_storage` extension, enabling retry mechanisms for the `otlp` exporter, and using a file-backed queue for temporary data storage. - -By implementing file-based check-pointing and queue persistence, you ensure the telemetry pipeline can gracefully recover from temporary interruptions, making it a more robust and reliable for production environments. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/_index.md deleted file mode 100644 index ca3f264608..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-building-resilience/_index.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: 2. Building In Resilience -linkTitle: 2. Building Resilience -time: 10 minutes -weight: 4 ---- - -The OpenTelemetry Collector’s [**FileStorage Extension**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/19bc7d6ee854c0c1b5c97d8d348e5b9d1199e8aa/extension/storage/filestorage/README.md) is a critical component for building a more resilient telemetry pipeline. It enables the Collector to reliably checkpoint in-flight data, manage retries efficiently, and gracefully handle temporary failures without losing valuable telemetry. - -With FileStorage enabled, the Collector can persist intermediate states to disk, ensuring that your traces, metrics, and logs are not lost during network disruptions, backend outages, or Collector restarts. This means that even if your network connection drops or your backend becomes temporarily unavailable, the Collector will continue to receive and buffer telemetry, resuming delivery seamlessly once connectivity is restored. - -By integrating the FileStorage Extension into your pipeline, you can strengthen the durability of your observability stack and maintain high-quality telemetry ingestion, even in environments where connectivity may be unreliable. -{{% notice note %}} - -This solution will work for metrics as long as the connection downtime is brief, up to 15 minutes. If the downtime exceeds this, Splunk Observability Cloud might drop data to make sure no data-point is out of order. - -For logs, there are plans to implement a full enterprise-ready solution in one of the upcoming Splunk OpenTelemetry Collector releases. - -{{% /notice %}} diff --git a/content/en/conf/3-obs1184/2-dropping-spans/2-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-dropping-spans/2-1-configuration.md similarity index 99% rename from content/en/conf/3-obs1184/2-dropping-spans/2-1-configuration.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-dropping-spans/2-1-configuration.md index 607ec47f8d..d41608f30f 100644 --- a/content/en/conf/3-obs1184/2-dropping-spans/2-1-configuration.md +++ b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-dropping-spans/2-1-configuration.md @@ -108,7 +108,7 @@ service: processors: - memory_limiter - filter/health - - resourcedetection + - resource_detection - resource/add_mode - batch exporters: diff --git a/content/en/conf/3-obs1184/2-dropping-spans/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-dropping-spans/_index.md similarity index 100% rename from content/en/conf/3-obs1184/2-dropping-spans/_index.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/2-dropping-spans/_index.md diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-dropping-spans/3-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-dropping-spans/3-1-configuration.md deleted file mode 100644 index 3b49a2b798..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-dropping-spans/3-1-configuration.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -title: 3.1 Configuration -linkTitle: 3.1 Configuration -weight: 1 ---- - -{{% exercise title="Add a `filter` processor" %}} - -Switch to your **Gateway terminal** window and open the `gateway.yaml` file. Update the `processors` section with the following configuration: - -1. **Add a `filter` processor**: - Configure the gateway to exclude spans with the name `/_healthz`. The `error_mode: ignore` directive ensures that any errors encountered during filtering are ignored, allowing the pipeline to continue running smoothly. The `traces` section defines the filtering rules, specifically targeting spans named `/_healthz` for exclusion. - - ```yaml - filter/health: # Defines a filter processor - error_mode: ignore # Ignore errors - traces: # Filtering rules for traces - span: # Exclude spans named "/_healthz" - - 'name == "/_healthz"' - ``` - -2. **Add the `filter` processor to the `traces` pipeline**: - Include the `filter/health` processor in the `traces` pipeline. For optimal performance, place the filter as early as possible—right after the `memory_limiter` and before the `batch` processor. Here’s how the configuration should look: - - ```yaml - traces: - receivers: - - otlp - processors: - - memory_limiter - - filter/health # Filters data based on rules - - resource/add_mode - - batch - exporters: - - debug - - file/traces - ``` - -This setup ensures that health check related spans (`/_healthz`) are filtered out early in the pipeline, reducing unnecessary noise in your telemetry data. - -{{% /exercise %}} - -Validate the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `traces:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(filter
fa:fa-microchip
health):::processor - PRO5(batch
fa:fa-microchip):::processor - EXP1( debug 
fa:fa-upload):::exporter - EXP2(  file  
fa:fa-upload
traces):::exporter - %% Links - subID1:::sub-traces - subgraph " " - subgraph subID1["`**Traces**`"] - direction LR - REC1 --> PRO1 - PRO1 --> PRO4 - PRO4 --> PRO3 - PRO3 --> PRO5 - PRO5 --> EXP1 - PRO5 --> EXP2 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-dropping-spans/3-2-test-filter.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-dropping-spans/3-2-test-filter.md deleted file mode 100644 index 45e89f5443..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-dropping-spans/3-2-test-filter.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -title: 3.2 Test Filter Processor -linkTitle: 3.2 Test Filter Processor -weight: 2 ---- - -To test your configuration, you'll need to generate some trace data that includes a span named `"/_healthz"`. - -{{% exercise title="Verify spans are dropped" %}} - -**Start the Gateway**: In your **Gateway terminal** window start the **Gateway**. - -```bash -../otelcol --config ./gateway.yaml -``` - -**Start the Agent**: In your **Agent terminal** window start the **Agent**. - -```bash -../otelcol --config ./agent.yaml -``` - -**Start the Loadgen**: In the **Loadgen terminal** window, execute the following command to start the load generator with health check spans enabled: - -```bash -../loadgen -health -count 5 -``` - - The debug output in the **Agent terminal** will show `_healthz` spans: - - ```text - InstrumentationScope healthz 1.0.0 -Span #0 - Trace ID : 0cce8759b5921c8f40b346b2f6e2f4b6 - Parent ID : - ID : bc32bd0e4ddcb174 - Name : /_healthz - Kind : Server - Start time : 2025-07-11 08:47:50.938703979 +0000 UTC - End time : 2025-07-11 08:47:51.938704091 +0000 UTC - Status code : Ok - Status message : Success -``` - -They will not be present in the **Gateway** debug as they are dropped by the filter processor that was configured earlier. - -**Verify `agent.out`**: Using `jq`, in the **Test terminal**, confirm the name of the spans received by the **Agent**: - -{{% tabs %}} -{{% tab title="Check spans in agent.out" %}} - -```bash -jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./agent.out -``` - -{{% /tab %}} -{{% tab title="Example output" %}} - -```text -"Span 1 found with name /movie-validator" -"Span 2 found with name /_healthz" -"Span 3 found with name /movie-validator" -"Span 4 found with name /_healthz" -"Span 5 found with name /movie-validator" -"Span 6 found with name /_healthz" -"Span 7 found with name /movie-validator" -"Span 8 found with name /_healthz" -"Span 9 found with name /movie-validator" -"Span 10 found with name /_healthz" -``` - -{{% /tab %}} -{{% /tabs %}} - -**Check the Gateway Debug output**: Using `jq` confirm the name of the spans received by the **Gateway**: - -{{% tabs %}} -{{% tab title="Check spans in gateway-traces.out" %}} - -```bash -jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./gateway-traces.out -``` - -{{% /tab %}} -{{% tab title="Example output" %}} - -The `gateway-metrics.out` file will not contain any spans named `/_healthz`. - -```text -"Span 1 found with name /movie-validator" -"Span 2 found with name /movie-validator" -"Span 3 found with name /movie-validator" -"Span 4 found with name /movie-validator" -"Span 5 found with name /movie-validator" -``` - -{{% /tab %}} -{{% /tabs %}} - -{{% /exercise %}} - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} - -To ensure optimal performance with the Filter processor, thoroughly understand your incoming data format and rigorously test your configuration. **Use the most specific filtering criteria possible** to minimize the risk of inadvertently dropping important data. - -This configuration can be extended to filter spans based on various attributes, tags, or custom criteria, enhancing the OpenTelemetry Collector's flexibility and efficiency for your specific observability requirements. -{{% /notice %}} - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. - - diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-dropping-spans/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-dropping-spans/_index.md deleted file mode 100644 index 75193d0756..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-dropping-spans/_index.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: 3. Dropping Spans -linkTitle: 3. Dropping Spans -time: 5 minutes -weight: 5 ---- - -In this section, we will explore how to use the [**Filter Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md) to selectively drop spans based on certain conditions. - -Specifically, we will drop traces based on the span name, which is commonly used to filter out unwanted spans such as health checks or internal communication traces. In this case, we will be filtering out spans that contain `"/_healthz"`, typically associated with health check requests and usually are quite "**noisy**". - -{{% exercise title="Set up the `3-dropping-spans` directory" %}} - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `3-dropping-spans` directory and run the `clear` command.** - -Copy `*.yaml` from the `2-building-resilience` directory into `3-dropping-spans`. Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /exercise %}} - -Next, we will configure the **filter processor** and the respective pipelines. diff --git a/content/en/conf/3-obs1184/3-sensitive-data/3-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-sensitive-data/3-1-configuration.md similarity index 98% rename from content/en/conf/3-obs1184/3-sensitive-data/3-1-configuration.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-sensitive-data/3-1-configuration.md index dcfa79577c..5be9a2d31a 100644 --- a/content/en/conf/3-obs1184/3-sensitive-data/3-1-configuration.md +++ b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-sensitive-data/3-1-configuration.md @@ -111,7 +111,7 @@ Select **Pipelines** and select the pencil-shaped **Edit** icon for `traces`. Select **+** beside **processors** to add `attributes`, then repeat to add `redaction`. Keep every existing receiver, processor, and exporter. Use the drag handles to place both new processors after `filter/health` and before -`resourcedetection`, then select **Edit**. +`resource_detection`, then select **Edit**. This order first removes spans you do not plan to keep, then protects the sensitive values in the remaining spans. Resource detection and batching run @@ -144,7 +144,7 @@ service: - filter/health - attributes - redaction - - resourcedetection + - resource_detection - resource/add_mode - batch exporters: diff --git a/content/en/conf/3-obs1184/3-sensitive-data/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-sensitive-data/_index.md similarity index 100% rename from content/en/conf/3-obs1184/3-sensitive-data/_index.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/3-sensitive-data/_index.md diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/4-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/4-1-configuration.md deleted file mode 100644 index d0e86ce2c6..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/4-1-configuration.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -title: 4.1 Configuration -linkTitle: 4.1 Configuration -weight: 1 ---- - -In this step, we'll modify `agent.yaml` to include the `attributes` and `redaction` processors. These processors will help ensure that sensitive data within span attributes is properly handled before being logged or exported. - -Previously, you may have noticed that some span attributes displayed in the console contained personal and sensitive data. We'll now configure the necessary processors to filter out and redact this information effectively. - -```text -Attributes: - -> user.name: Str(George Lucas) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.account_password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - {"kind": "exporter", "data_type": "traces", "name": "debug"} -``` - -{{% exercise title="Add attributes and redaction processors" %}} - -Switch to your **Agent terminal** window and open the `agent.yaml` file in your editor. We’ll add two processors to enhance the security and privacy of your telemetry data. - -**1. Add an `attributes` Processor**: The [**Attributes Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor) allows you to modify span attributes (tags) by updating, deleting, or hashing their values. This is particularly useful for obfuscating sensitive information before it is exported. - -In this step, we’ll: - -1. **Update** the `user.phone_number` attribute to a static value `("UNKNOWN NUMBER")`. -2. **Hash** the `user.email` attribute to ensure the original email is not exposed. -3. **Delete** the `user.password` attribute to remove it entirely from the span. - -```yaml - attributes/update: - actions: # Actions - - key: user.phone_number # Target key - action: update # Update action - value: "UNKNOWN NUMBER" # New value - - key: user.email # Target key - action: hash # Hash the email value - - key: user.password # Target key - action: delete # Delete the password - ``` - -**2. Add a `redaction` Processor**: The [**Redaction Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/redactionprocessor) detects and redacts sensitive data in span attributes based on predefined patterns, such as credit card numbers or other personally identifiable information (PII). - -In this step: - -- We set `allow_all_keys: true` to ensure all attributes are processed (if set to `false`, only explicitly allowed keys are retained). - -- We define `blocked_values` with regular expressions to detect and redact **Visa** and **MasterCard** credit card numbers. - -- The `summary: debug` option logs detailed information about the redaction process for debugging purposes. - -```yaml - redaction/redact: - allow_all_keys: true # If false, only allowed keys will be retained - blocked_values: # List of regex patterns to block - - '\b4[0-9]{3}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b' # Visa - - '\b5[1-5][0-9]{2}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b' # MasterCard - summary: debug # Show debug details about redaction -``` - -**Update the `traces` Pipeline**: Integrate both processors into the `traces` pipeline. Make sure that you comment out the redaction processor at first (we will enable it later in a separate exercise). Your configuration should look like this: - -```yaml - traces: - receivers: - - otlp - processors: - - memory_limiter - - attributes/update # Update, hash, and remove attributes - #- redaction/redact # Redact sensitive fields using regex - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - file - - otlphttp -``` - -{{% /exercise %}} - -Validate the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `traces:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRML(memory_limiter
fa:fa-microchip):::processor - PRRD(resourcedetection
fa:fa-microchip):::processor - PRRS(resource
fa:fa-microchip
add_mode):::processor - PRUP(attributes
fa:fa-microchip
update):::processor - EXP1(otlphttp
fa:fa-upload):::exporter - EXP2(  debug  
fa:fa-upload):::exporter - EXP3(file
fa:fa-upload):::exporter - - %% Links - subID1:::sub-traces - subgraph " " - subgraph subID1["`**Traces**`"] - direction LR - REC1 --> PRML - PRML --> PRUP - PRUP --> PRRD - PRRD --> PRRS - PRRS --> EXP2 - PRRS --> EXP3 - PRRS --> EXP1 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/4-2-test-delete-tag.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/4-2-test-delete-tag.md deleted file mode 100644 index e8b25f25e3..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/4-2-test-delete-tag.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -title: 4.2 Test Attribute Processor -linkTitle: 4.2 Test Attribute Processor -weight: 2 ---- - -In this exercise, we will **delete** the `user.account_password`, **update** the `user.phone_number` **attribute** and **hash** the `user.email` in the span data before it is exported by the **Agent**. - -{{% exercise title="Test the attributes processor" %}} - -**Start the Gateway**: In your **Gateway terminal** window start the **Gateway**. - -```bash -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: In your **Agent terminal** window start the **Agent**. - -```bash -../otelcol --config=agent.yaml -``` - -**Start the Load Generator**: In the **Loadgen terminal** window start the `loadgen`: - -```bash -../loadgen -count 1 -``` - -**Check the debug output**: For both the **Agent** and **Gateway** confirm that `user.account_password` has been removed, and both `user.phone_number` & `user.email` have been updated: - -{{% tabs %}} -{{% tab title="New Debug Output" %}} - - ```text - -> user.name: Str(George Lucas) - -> user.phone_number: Str(UNKNOWN NUMBER) - -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287) - -> payment.amount: Double(51.71) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - ``` - -{{% /tab %}} -{{% tab title="Original Debug Output" %}} - - ```text - -> user.name: Str(George Lucas) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - -> payment.amount: Double(95.22) - ``` - -{{% /tab %}} -{{% /tabs %}} - -**Check file output**: Using `jq` validate that `user.account_password` has been removed, and `user.phone_number` & `user.email` have been updated in `gateway-taces.out`: - -{{% tabs %}} -{{% tab title="Validate attribute changes" %}} - -```bash -jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.password" or .key == "user.phone_number" or .key == "user.email") | {key: .key, value: .value.stringValue}' ./gateway-traces.out -``` - -{{% /tabs %}} -{{% tab title="Output" %}} - -Notice that the `user.account_password` has been removed, and the `user.phone_number` & `user.email` have been updated: - -```json -{ - "key": "user.phone_number", - "value": "UNKNOWN NUMBER" -} -{ - "key": "user.email", - "value": "62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287" -} -``` - -{{% /tab %}} -{{% /tabs %}} - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/4-3-test-redaction.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/4-3-test-redaction.md deleted file mode 100644 index e26ba4593b..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/4-3-test-redaction.md +++ /dev/null @@ -1,140 +0,0 @@ ---- -title: 4.3 Test Redaction Processor -linkTitle: 4.3 Test Redaction Processor -weight: 3 ---- - -The `redaction` processor gives precise control over which attributes and values are **permitted** or **removed** from telemetry data. - -In this exercise, we will **redact** the `user.visa` & `user.mastercard` values in the span data before it is exported by the **Agent**. -{{% exercise title="Test the redaction processor" %}} - -**Start the Gateway**: In your **Gateway terminal** window start the **Gateway**. - -```bash -../otelcol --config=gateway.yaml -``` - -**Enable the `redaction/redact` processor**: In the **Agent terminal** window, edit `agent.yaml` and remove the `#` we inserted in the previous exercise. - -```yaml - traces: - receivers: - - otlp - processors: - - memory_limiter - - attributes/update # Update, hash, and remove attributes - - redaction/redact # Redact sensitive fields using regex - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - file - - otlphttp -``` - -**Start the Agent**: In your **Agent terminal** window start the **Agent**. - -```bash -../otelcol --config=agent.yaml -``` - -**Start the Load Generator**: In the **Loadgen terminal** window start the `loadgen`: - -```bash -../loadgen -count 1 -``` - -**Check the debug output**: For both the **Agent** and **Gateway** confirm the values for `user.visa` & `user.mastercard` have been updated. Notice `user.amex` attribute value was **NOT** redacted because a matching regex pattern was not added to `blocked_values` - -{{% tabs %}} -{{% tab title="New Debug Output" %}} - - ```text - -> user.name: Str(George Lucas) - -> user.phone_number: Str(UNKNOWN NUMBER) - -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287) - -> payment.amount: Double(69.71) - -> user.visa: Str(****) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(****) - -> redaction.masked.keys: Str(user.mastercard,user.visa) - -> redaction.masked.count: Int(2) - ``` - -{{% /tab %}} -{{% tab title="Original Debug Output" %}} - - ```text - -> user.name: Str(George Lucas) - -> user.phone_number: Str(+1555-867-5309) - -> user.email: Str(george@deathstar.email) - -> user.password: Str(LOTR>StarWars1-2-3) - -> user.visa: Str(4111 1111 1111 1111) - -> user.amex: Str(3782 822463 10005) - -> user.mastercard: Str(5555 5555 5555 4444) - -> payment.amount: Double(65.54) - ``` - -{{% /tab %}} -{{% /tabs %}} - -{{% notice note %}} -By including `summary:debug` in the redaction processor, the debug output will include summary information about which matching key values were redacted, along with the count of values that were masked. - -```text - -> redaction.masked.keys: Str(user.mastercard,user.visa) - -> redaction.masked.count: Int(2) - ``` - -{{% /notice %}} - -**Check file output**: Using `jq` verify that `user.visa` & `user.mastercard` have been updated in the `gateway-traces.out`. - -{{% tabs %}} -{{% tab title="Validate attribute changes" %}} - -```bash -jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.visa" or .key == "user.mastercard" or .key == "user.amex") | {key: .key, value: .value.stringValue}' ./gateway-traces.out -``` - -{{% /tabs %}} -{{% tab title="Output" %}} - -Notice that `user.amex` has not been redacted because a matching regex pattern was not added to `blocked_values`: - -```json -{ - "key": "user.visa", - "value": "****" -} -{ - "key": "user.amex", - "value": "3782 822463 10005" -} -{ - "key": "user.mastercard", - "value": "****" -} -``` - -{{% /tab %}} -{{% /tabs %}} - -These are just a couple of examples of how `attributes` and `redaction` processors can be configured to protect sensitive data. - -{{% /exercise %}} - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. - - diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/_index.md deleted file mode 100644 index f89b9e132b..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-sensitive-data/_index.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: 4. Redacting Sensitive Data -linkTitle: 4. Sensitive Data -time: 10 minutes -weight: 6 ---- - -In this section, you'll learn how to configure the OpenTelemetry Collector to remove specific tags and redact sensitive data from telemetry spans. This is crucial for protecting sensitive information such as credit card numbers, personal data, or other security-related details that must be anonymized before being processed or exported. - -We'll walk through configuring key processors in the OpenTelemetry Collector, including: - -- **[Attributes Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/README.md)**: Modifies or removes specific span attributes. -- [**Redaction Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/redactionprocessor/README.md): Ensures sensitive data is sanitized before being stored or transmitted. - -{{% exercise title="Set up the `4-sensitive-data` directory" %}} - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `4-sensitive-data` directory and run the `clear` command.** - -Copy `*.yaml` from the `3-dropping-spans` directory into `4-sensitive-data`. Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /exercise %}} diff --git a/content/en/conf/3-obs1184/4-transform-data/4-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-transform-data/4-1-configuration.md similarity index 95% rename from content/en/conf/3-obs1184/4-transform-data/4-1-configuration.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-transform-data/4-1-configuration.md index 633833eb2b..68bc81351f 100644 --- a/content/en/conf/3-obs1184/4-transform-data/4-1-configuration.md +++ b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-transform-data/4-1-configuration.md @@ -123,18 +123,18 @@ Select **Pipelines** and select the pencil-shaped **Edit** icon for `logs/workshop`. Select **+** beside **processors** and add `transform`. Keep every existing receiver, processor, and exporter. Use the drag handle to place `transform` -after `resourcedetection`, then select **Edit**. +after `resource_detection`, then select **Edit**. Open **Collector YAML** and confirm: - `transform` contains one resource context and one log context. - `transform` appears exactly once in the processors list under `service.pipelines.logs/workshop`. -- It appears after `resourcedetection`, allowing `host.name` to be detected +- It appears after `resource_detection`, allowing `host.name` to be detected before the resource allowlist is applied. - `filter/health`, `attributes`, and `redaction` remain connected to `traces`. -The position after `resourcedetection` is intentional. If `transform` ran +The position after `resource_detection` is intentional. If `transform` ran first, `host.name` might not exist yet and therefore could not survive the resource allowlist. `resource/add_mode` runs afterward and adds `otelcol.service.mode=agent`, so the final log resource contains both the @@ -151,7 +151,7 @@ service: - file_log/quotes processors: - memory_limiter - - resourcedetection + - resource_detection - transform - resource/add_mode exporters: @@ -160,7 +160,7 @@ service: ``` Review **Collector YAML** and resolve any errors shown. If `transform` is not -after `resourcedetection`, return to the Pipeline editor and use the drag +after `resource_detection`, return to the Pipeline editor and use the drag handle to move it. The resource allowlist must run after host metadata is detected. diff --git a/content/en/conf/3-obs1184/4-transform-data/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-transform-data/_index.md similarity index 100% rename from content/en/conf/3-obs1184/4-transform-data/_index.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/4-transform-data/_index.md diff --git a/content/en/conf/3-obs1184/5-deploy-and-validate/5-1-deploy-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-deploy-and-validate/5-1-deploy-configuration.md similarity index 93% rename from content/en/conf/3-obs1184/5-deploy-and-validate/5-1-deploy-configuration.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-deploy-and-validate/5-1-deploy-configuration.md index c026f6757c..8881d103bd 100644 --- a/content/en/conf/3-obs1184/5-deploy-and-validate/5-1-deploy-configuration.md +++ b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-deploy-and-validate/5-1-deploy-configuration.md @@ -17,7 +17,7 @@ validation easier to follow. In **Collector YAML**, confirm: - `filter/health`, `attributes`, and `redaction` are connected to `traces`. -- `transform` is connected to `logs/workshop` after `resourcedetection`. +- `transform` is connected to `logs/workshop` after `resource_detection`. - All eight imported pipelines are present: `traces`, `metrics`, `metrics/internal`, `logs/signalfx`, `logs`, `logs/entities`, `metrics/workshop`, and `logs/workshop`. @@ -68,7 +68,7 @@ In the **Command terminal** on the Splunk Show instance, copy the completed configuration into the Collector folder: ```bash -cp ~/workshop/ninja/obs1184/agent_config.solution.yaml \ +cp ~/workshop/ninja/advanced-otel/agent_config.solution.yaml \ ~/advanced-otel-workshop/1-agent/agent_config.yaml ``` @@ -81,9 +81,9 @@ YAML into the SSH session. {{% notice title="Recovery copy" style="info" %}} If you need a new copy of the completed configuration, Splunk Show attendees can copy -`~/workshop/ninja/obs1184/agent_config.solution.yaml` again. Attendees running +`~/workshop/ninja/advanced-otel/agent_config.solution.yaml` again. Attendees running the Collector on the same computer as the browser can download -[agent_config.solution.yaml](https://github.com/splunk/observability-workshop/blob/main/workshop/ninja/obs1184/agent_config.solution.yaml) +[agent_config.solution.yaml](https://github.com/splunk/observability-workshop/blob/main/workshop/ninja/advanced-otel/agent_config.solution.yaml) and copy it to `[WORKSHOP]/1-agent/agent_config.yaml`. {{% /notice %}} diff --git a/content/en/conf/3-obs1184/5-deploy-and-validate/5-2-validate-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-deploy-and-validate/5-2-validate-configuration.md similarity index 100% rename from content/en/conf/3-obs1184/5-deploy-and-validate/5-2-validate-configuration.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-deploy-and-validate/5-2-validate-configuration.md diff --git a/content/en/conf/3-obs1184/5-deploy-and-validate/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-deploy-and-validate/_index.md similarity index 100% rename from content/en/conf/3-obs1184/5-deploy-and-validate/_index.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-deploy-and-validate/_index.md diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/5-1-configuration.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/5-1-configuration.md deleted file mode 100644 index c68b94f294..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/5-1-configuration.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: 5.1 Configuration -linkTitle: 5.1 Configuration -weight: 1 ---- - -{{% exercise title="Add a `transform` processor" %}} -**Add a `transform` processor**: Switch to your **Gateway terminal** window and edit the `gateway.yaml` and add the following `transform` processor: - -```yaml - transform/logs: # Processor Type/Name - log_statements: # Log Processing Statements - - context: resource # Log Context - statements: # List of attribute keys to keep - - keep_keys(attributes, ["com.splunk.sourcetype", "host.name", "otelcol.service.mode"]) -``` - -By using the `-context: resource` key we are targeting the `resourceLog` attributes of logs. - -This configuration ensures that only the relevant resource attributes (`com.splunk.sourcetype`, `host.name`, `otelcol.service.mode`) are retained, improving log efficiency and reducing unnecessary metadata. - -**Adding a Context Block for Log Severity Mapping**: To properly set the `severity_text` and `severity_number` fields of a log record, we add a `log` context block within `log_statements`. This configuration extracts the `level` value from the log body, maps it to `severity_text`, and assigns the corresponding `severity_number` based on the log level: - -```yaml - - context: log # Log Context - statements: # Transform Statements Array - - set(cache, ParseJSON(body)) where IsMatch(body, "^\\{") # Parse JSON log body into a cache object - - flatten(cache, "") # Flatten nested JSON structure - - merge_maps(attributes, cache, "upsert") # Merge cache into attributes, updating existing keys - - set(severity_text, attributes["level"]) # Set severity_text from the "level" attribute - - set(severity_number, 1) where severity_text == "TRACE" # Map severity_text to severity_number - - set(severity_number, 5) where severity_text == "DEBUG" - - set(severity_number, 9) where severity_text == "INFO" - - set(severity_number, 13) where severity_text == "WARN" - - set(severity_number, 17) where severity_text == "ERROR" - - set(severity_number, 21) where severity_text == "FATAL" -``` - -The `merge_maps` function is used to combine two maps (dictionaries) into one. In this case, it merges the `cache` object (containing parsed JSON data from the log body) into the `attributes` map. - -- **Parameters**: - - `attributes`: The target map where the data will be merged. - - `cache`: The source map containing the parsed JSON data. - - `"upsert"`: This mode ensures that if a key already exists in the `attributes` map, its value will be updated with the value from `cache`. If the key does not exist, it will be inserted. - -This step is crucial because it ensures that all relevant fields from the log body (e.g., `level`, `message`, etc.) are added to the `attributes` map, making them available for further processing or exporting. - -**Summary of Key Transformations**: - -- **Parse JSON**: Extracts structured data from the log body. -- **Flatten JSON**: Converts nested JSON objects into a flat structure. -- **Merge Attributes**: Integrates extracted data into log attributes. -- **Map Severity Text**: Assigns severity_text from the log’s level attribute. -- **Assign Severity Numbers**: Converts severity levels into standardized numerical values. - -> [!IMPORTANT] -> You should have a **single** `transform` processor containing two context blocks: one whose context is for `resource` and one whose context is for `log`. - -This configuration ensures that log severity is correctly extracted, standardized, and structured for efficient processing. - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -This method of mapping all JSON fields to top-level attributes should only be used for **testing and debugging OTTL**. It will result in high cardinality in a production scenario. -{{% /notice %}} - -**Update the `logs` pipeline**: Add the `transform/logs:` processor into the `logs:` pipeline so your configuration looks like this: - -```yaml - logs: # Logs pipeline - receivers: - - otlp # OTLP receiver - processors: # Processors for logs - - memory_limiter - - resource/add_mode - - transform/logs - - batch - exporters: - - debug # Debug exporter - - file/logs -``` - -{{% /exercise %}} - -Validate the agent configuration using [**https://otelbin.io**](https://otelbin.io/). For reference, the `logs:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(  otlp  
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(transform
fa:fa-microchip
logs):::processor - PRO5(batch
fa:fa-microchip):::processor - EXP1(file
fa:fa-upload
logs):::exporter - EXP2(  debug  
fa:fa-upload):::exporter - %% Links - subID1:::sub-logs - subgraph " " - subgraph subID1["`**Logs**`"] - direction LR - REC1 --> PRO1 - PRO1 --> PRO3 - PRO3 --> PRO4 - PRO4 --> PRO5 - PRO5 --> EXP2 - PRO5 --> EXP1 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/5-2-setup.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/5-2-setup.md deleted file mode 100644 index 217efa90c4..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/5-2-setup.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: 5.2 Setup Environment -linkTitle: 5.2 Setup Environment -weight: 2 ---- - -{{% exercise title="Set up the transform test" %}} - -**Start the Gateway**: In the **Gateway terminal** run: - -```bash { title="Start the Gateway" } -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: In the **Agent terminal** run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Start the Load Generator**: In the **Loadgen terminal** window, execute the following command to start the load generator with **JSON enabled**: - -```bash { title="Log Generator" } -../loadgen -logs -json -count 5 -``` - -The `loadgen` will write 5 log lines to `./quotes.log` in JSON format. - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/5-3-test-transform.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/5-3-test-transform.md deleted file mode 100644 index 3d9ededa90..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/5-3-test-transform.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: 5.3 Test Transform Processor -linkTitle: 5.3 Test Transform Processor -weight: 3 ---- - -This test verifies that the `com.splunk/source` and `os.type` metadata have been **removed** from the log resource attributes before being exported by the **Agent**. Additionally, the test ensures that: - -1. The log body is parsed to extract severity information. - - `SeverityText` and `SeverityNumber` are set on the `LogRecord`. -2. JSON fields from the log body are promoted to log `attributes`. - -This ensures proper metadata filtering, severity mapping, and structured log enrichment before exporting. - -{{% exercise title="Verify the attributes were removed" %}} - -**Check the debug output**: For both the **Agent** and **Gateway** confirm that `com.splunk/source` and `os.type` have been removed: - -{{% tabs %}} -{{% tab title="Gateway Debug Output" %}} - - ```text -Resource attributes: - -> com.splunk.sourcetype: Str(quotes) - -> host.name: Str(workshop-instance) - -> otelcol.service.mode: Str(agent) - ``` - -{{% /tab %}} -{{% tab title="Agent Debug Output" %}} - - ```text -Resource attributes: - -> com.splunk.source: Str(./quotes.log) - -> com.splunk.sourcetype: Str(quotes) - -> host.name: Str(workshop-instance) - -> os.type: Str(linux) - -> otelcol.service.mode: Str(agent) - ``` - -{{% /tab %}} -{{% /tabs %}} - -For both the **Agent** and **Gateway** confirm that `SeverityText` and `SeverityNumber` in the `LogRecord` is now defined with the severity `level` from the log body. Confirm that the JSON fields from the body can be accessed as top-level log `Attributes`: - -{{% tabs %}} -{{% tab title="Gateway Debug Output" %}} - -```text - -SeverityText: WARN -SeverityNumber: Warn(13) -Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"}) -Attributes: - -> log.file.path: Str(quotes.log) - -> level: Str(WARN) - -> message: Str(Your focus determines your reality.) - -> movie: Str(SW) - -> timestamp: Str(2025-03-07 11:17:26) - -``` - -{{% /tab %}} -{{% tab title="Agemt Debug Output" %}} - -```text - -SeverityText: -SeverityNumber: Unspecified(0) -Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"}) -Attributes: - -> log.file.path: Str(quotes.log) - -``` - -{{% /tab %}} -{{% /tabs %}} - -**Check file output**: In the new `gateway-logs.out` file verify the data has been transformed: - -{{% tabs %}} -{{% tab title="jq Query" %}} - -```bash -jq '[.resourceLogs[].scopeLogs[].logRecords[] | {severityText, severityNumber, body: .body.stringValue}]' gateway-logs.out -``` - -{{% /tabs %}} -{{% tab title="Example Output" %}} - -```json -[ - { - "severityText": "DEBUG", - "severityNumber": 5, - "body": "{\"level\":\"DEBUG\",\"message\":\"All we have to decide is what to do with the time that is given us.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}" - }, - { - "severityText": "WARN", - "severityNumber": 13, - "body": "{\"level\":\"WARN\",\"message\":\"The Force will be with you. Always.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}" - }, - { - "severityText": "ERROR", - "severityNumber": 17, - "body": "{\"level\":\"ERROR\",\"message\":\"One does not simply walk into Mordor.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}" - }, - { - "severityText": "DEBUG", - "severityNumber": 5, - "body": "{\"level\":\"DEBUG\",\"message\":\"Do or do not, there is no try.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}" - } -] -[ - { - "severityText": "ERROR", - "severityNumber": 17, - "body": "{\"level\":\"ERROR\",\"message\":\"There is some good in this world, and it's worth fighting for.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}" - } -] -``` - -{{% /tab %}} -{{% /tabs %}} - -{{% /exercise %}} - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/_index.md deleted file mode 100644 index 9d272f6173..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/5-transform-data/_index.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -title: 5. Transform Data -linkTitle: 5. Transform Data -time: 10 minutes -weight: 7 ---- - -The [**Transform Processor**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md) lets you modify telemetry data—logs, metrics, and traces—as it flows through the pipeline. Using the **OpenTelemetry Transformation Language (OTTL)**, you can filter, enrich, and transform data on the fly without touching your application code. - -In this exercise we’ll update `gateway.yaml` to include a **Transform Processor** that will: - -- **Filter** log resource attributes. -- **Parse** JSON structured log data into attributes. -- **Set** log severity levels based on the log message body. - -You may have noticed that in previous logs, fields like `SeverityText` and `SeverityNumber` were undefined. This is typical of the `filelog` receiver. However, the severity is embedded within the log body e.g.: - -```text -SeverityText: -SeverityNumber: Unspecified(0) -Body: Str(2025-01-31 15:49:29 [WARN] - Do or do not, there is no try.) -``` - -Logs often contain structured data encoded as JSON within the log body. Extracting these fields into attributes allows for better indexing, filtering, and querying. Instead of manually parsing JSON in downstream systems, OTTL enables automatic transformation at the telemetry pipeline level. - -{{% exercise title="Set up the `5-transform-data` directory" %}} - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `5-transform-data` directory and run the `clear` command.** - -Copy `*.yaml` from the `4-sensitve-data` directory into `5-transform-data`. Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/6-1-connector.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/6-1-connector.md deleted file mode 100644 index 518aed3631..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/6-1-connector.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -title: 6.1 Configure the Routing Connector -linkTitle: 6.1 Routing Configuration -weight: 1 ---- - -In this exercise, you will configure the **Routing Connector** in the `gateway.yaml`. The Routing Connector can route metrics, traces, and logs based on any attributes, we will focus exclusively on trace routing based on the `deployment.environment` attribute (though any span/log/metirc attribute can be used). - -{{% exercise title="Configure the routing connector" %}} - -**Add new `file` exporters**: The `routing` connector requires different targets for routing. In the **Gateway terminal** create two new file exporters, `file/traces/route1-regular` and `file/traces/route2-security`, to ensure data is directed correctly in the `exporters` section of the `gateway.yaml`: - -```yaml - file/traces/route1-regular: # Exporter for regular traces - path: "./gateway-traces-route1-regular.out" # Path for saving trace data - append: false # Overwrite the file each time - file/traces/route2-security: # Exporter for security traces - path: "./gateway-traces-route2-security.out" # Path for saving trace data - append: false # Overwrite the file each time -``` - -**Enable Routing** by adding the `routing` connector. In OpenTelemetry configuration files, `connectors` have their own dedicated section, similar to receivers and processors. - -Find and uncomment the `#connectors:` section. Then, add the following below the `connectors:` section: - -```yaml - routing: - default_pipelines: [traces/route1-regular] # Default pipeline if no rule matches - error_mode: ignore # Ignore errors in routing - table: # Define routing rules - # Routes spans to a target pipeline if the resourceSpan attribute matches the rule - - statement: route() where attributes["deployment.environment"] == "security-applications" - pipelines: [traces/route2-security] # Security target pipeline -``` - -{{% /exercise %}} - -The default pipeline in the configuration file works at a Catch all. It will be the routing target for any data (spans in our case) that do not match a rule in the routing rules table, In this table you find the pipeline that is the target for any span that matches the following rule: `["deployment.environment"] == "security-applications"` - -With the `routing` configuration complete, the next step is to configure the `pipelines` to apply these routing rules. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/6-2-pipelines.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/6-2-pipelines.md deleted file mode 100644 index 50207da95d..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/6-2-pipelines.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: 6.2 Configuring the Pipelines -linkTitle: 6.2 Pipeline Configuration -weight: 2 ---- - -{{% exercise title="Wire up the routing pipelines" %}} - -**Update the original `traces` pipeline to use routing**: - -1. To enable `routing`, update the original `traces` pipeline to use `routing` as the only exporter. This ensures all span data is sent through the **Routing Connector** for evaluation and then onwards to connected pipelines. Also, remove **all** processors and replace it with an empty array (`[]`) as this will now behandeld in the `traces/route1-regular` and `traces/route2-security` pipelines, allowing for custom behaviour for each route. Your `traces:` configuration should look like this: - - ```yaml - traces: # Traces pipeline - receivers: - - otlp # OTLP receiver - processors: [] # Processors for traces - exporters: - - routing - ``` - -**Add both the `route1-regular` and `route2-security` traces pipelines** below the existing `traces` pipeline: - -1. **Configure Route1-regular pipeline**: This pipeline will handle all spans that have **no match** in the routing table in the connector. -Notice this uses `routing` as its only receiver and will recieve data thought its `connection` from the original traces pipeline. - - ```yaml - traces/route1-regular: # Default pipeline for unmatched spans - receivers: - - routing # Receive data from the routing connector - processors: - - memory_limiter # Memory Limiter Processor - - resource/add_mode # Adds collector mode metadata - - batch - exporters: - - debug # Debug Exporter - - file/traces/route1-regular # File Exporter for unmatched spans - ``` - -2. **Add the route2-security pipeline**: This pipeline processes all spans that do match our rule `"[deployment.environment"] == "security-applications"` in the the routing rule. This pipeline is also using `routing` as its receiver. Add this pipline below the `traces/route1-regular` one. - - ```yaml - traces/route2-security: # Default pipeline for unmatched spans - receivers: - - routing # Receive data from the routing connector - processors: - - memory_limiter # Memory Limiter Processor - - resource/add_mode # Adds collector mode metadata - - batch - exporters: - - debug # Debug exporter - - file/traces/route2-security # File exporter for unmatched spans - ``` - -{{% /exercise %}} - -Validate the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `traces:` section of your pipelines will look similar to this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(   otlp   
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(memory_limiter
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(resource
fa:fa-microchip
add_mode):::processor - PRO5(batch
fa:fa-microchip):::processor - PRO6(batch
fa:fa-microchip):::processor - EXP1(  debug  
fa:fa-upload):::exporter - EXP2(  file  
fa:fa-upload
traces):::exporter - EXP3(  debug  
fa:fa-upload):::exporter - EXP4(  file  
fa:fa-upload
traces):::exporter - ROUTE1( routing 
fa:fa-route):::con-export - ROUTE2( routing 
fa:fa-route):::con-receive - ROUTE3( routing 
fa:fa-route):::con-receive - %% Links - subID1:::sub-traces - subID2:::sub-traces - subID3:::sub-traces - subgraph " " - direction LR - subgraph subID1["`**Traces**`"] - REC1 --> ROUTE1 - end - subgraph subID2["`**Traces/route2-security**`"] - ROUTE1 --> ROUTE2 - ROUTE2 --> PRO1 - PRO1 --> PRO3 - PRO3 --> PRO5 - PRO5 --> EXP1 - PRO5 --> EXP2 - end - subgraph subID3["`**Traces/route1-regular**`"] - ROUTE1 --> ROUTE3 - ROUTE3 --> PRO2 - PRO2 --> PRO4 - PRO4 --> PRO6 - PRO6 --> EXP3 - PRO6 --> EXP4 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3; -``` diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/6-3-test-routing.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/6-3-test-routing.md deleted file mode 100644 index 6f30f6776b..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/6-3-test-routing.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -title: 6.3 Test Routing Connector -linkTitle: 6.3 Test Routing Connector -weight: 3 ---- - -{{% exercise title="Verify spans route to the security file" %}} - -In this section, we will test the `routing` rule configured for the **Gateway**. The expected result is that a `span` generated by the `loadgen` that match the `"[deployment.environment"] == "security-applications"` rule will be sent to the `gateway-traces-route2-security.out` file. - -**Start the Gateway**: In your **Gateway terminal** window start the **Gateway**. - -```bash -../otelcol --config gateway.yaml -``` - -**Start the Agent**: In your **Agent terminal** window start the **Agent**. - -```bash -../otelcol --config agent.yaml -``` - -**Send a Regular Span**: In the **Loadgen terminal** window send a regular span using the `loadgen`: - -```bash -../loadgen -count 1 -``` - -Both the **Agent** and **Gateway** will display debug information. The gateway will also generate a new `gateway-traces-route1-regular.out` file, as this is now the designated destination for regular spans. - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -If you check `gateway-traces-route1-regular.out`, it will contain the `span` sent by `loadgen`. You will also see an empty `gateway-traces-route2-security..out` file, as the routing configuration creates output files immediately, even if no matching spans have been processed yet. -{{% /notice %}} - -**Send a Security Span**: In the **Loadgen terminal** window send a security span using the `security` flag: - -```bash -../loadgen -security -count 1 -``` - -Again, both the **Agent** and **Gateway** should display debug information, including the span you just sent. This time, the **Gateway** will write a line to the `gateway-traces-route2-security.out` file, which is designated for spans where the `deployment.environment` resource attribute matches `"security-applications"`. - -{{% tabs %}} -{{% tab title="Validate resource attribute matches" %}} - -```bash -jq -c '.resourceSpans[] as $resource | $resource.scopeSpans[].spans[] | {spanId: .spanId, deploymentEnvironment: ($resource.resource.attributes[] | select(.key == "deployment.environment") | .value.stringValue)}' gateway-traces-route2-security.out -``` - -{{% /tabs %}} -{{% tab title="Output" %}} - -```json -{"spanId":"cb799e92e26d5782","deploymentEnvironment":"security-applications"} -``` - -{{% /tab %}} -{{% /tabs %}} - -You can repeat this scenario multiple times, and each trace will be written to its corresponding output file. - -{{% /exercise %}} - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. - -## Conclusion - -In this section, we successfully tested the routing connector in the gateway by sending different spans and verifying their destinations. - -- **Regular spans** were correctly routed to `gateway-traces-route1-regular.out`, confirming that spans without a matching `deployment.environment` attribute follow the default pipeline. - -- **Security-related spans** were routed to `gateway-traces-route2-security.out`, demonstrating that the routing rule based on `"deployment.environment": "security-applications"` works as expected. - -By inspecting the output files, we confirmed that the OpenTelemetry Collector *correctly evaluates span attributes and routes them to the appropriate destinations*. This validates that routing rules can effectively separate and direct telemetry data for different use cases. - -You can now extend this approach by defining additional routing rules to further categorize spans, metrics, and logs based on different attributes. diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/_index.md deleted file mode 100644 index 7247e9dedf..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-routing-data/_index.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: 6. Routing Data -linkTitle: 6. Routing Data -time: 10 minutes -weight: 8 ---- - -The [**Routing Connector**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/routingconnector) in OpenTelemetry is a powerful feature that allows you to direct data (`traces`, `metrics`, or `logs`) to different pipelines/destinations based on specific criteria. This is especially useful in scenarios where you want to apply different processing or exporting logic to subsets of your telemetry data. - -For example, you might want to send *production* data to one exporter while directing *test* or *development* data to another. Similarly, you could route certain spans based on their attributes, such as service name, environment, or span name, to apply custom processing or storage logic. - -{{% exercise title="Set up the `6-routing-data` directory" %}} - -> [!IMPORTANT] -> **Change *ALL* terminal windows to the `6-routing-data` directory and run the `clear` command.** - -Copy `*.yaml` from the `5-transform-data` directory into `6-routing-data`. Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -{{% /exercise %}} - -Next, we will configure the routing connector and the respective pipelines. diff --git a/content/en/conf/3-obs1184/6-wrap-up/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-wrap-up/_index.md similarity index 100% rename from content/en/conf/3-obs1184/6-wrap-up/_index.md rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/6-wrap-up/_index.md diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/7-1-count-test.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/7-1-count-test.md deleted file mode 100644 index 87ebb9c775..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/7-1-count-test.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: 7.1 Testing the Count Connector -linkTitle: 7.1 Test Count Connector -weight: 1 ---- - -{{% exercise title="Test the Count Connector" %}} - -**Start the Gateway**: -In the **Gateway terminal** window run: - -```bash { title="Start the Gateway" } -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: -In the **Agent terminal** window run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Send 12 Logs lines with the Loadgen**: -In the **Spans terminal** window send 12 log lines, they should be read in two intervals. Do this with the following `loadgen` command: - -```bash { title="Loadgen" } -../loadgen -logs -json -count 12 -``` - -Both the **Agent** and **Gateway** will display debug information, showing they are processing data. Wait until the `loadgen` completes. - -**Verify metrics have been generated** -As the logs are processed, the **Agent** generates metrics and forwards them to the **Gateway**, which then writes them to `gateway-metrics.out`. - -To check if the metrics `logs.full.count`, `logs.sw.count`, `logs.lotr.count`, and `logs.error.count` are present in the output, run the following **jq** query: - -{{% tabs %}} -{{% tab title="jq query command" %}} - -```bash -jq '.resourceMetrics[].scopeMetrics[].metrics[] - | select(.name == "logs.full.count" or .name == "logs.sw.count" or .name == "logs.lotr.count" or .name == "logs.error.count") - | {name: .name, value: (.sum.dataPoints[0].asInt // "-")}' gateway-metrics.out -``` - -{{% /tab %}} -{{% tab title="jq example output" %}} - -```json -{ - "name": "logs.sw.count", - "value": "2" -} -{ - "name": "logs.lotr.count", - "value": "2" -} -{ - "name": "logs.full.count", - "value": "4" -} -{ - "name": "logs.error.count", - "value": "2" -} -{ - "name": "logs.error.count", - "value": "1" -} -{ - "name": "logs.sw.count", - "value": "2" -} -{ - "name": "logs.lotr.count", - "value": "6" -} -{ - "name": "logs.full.count", - "value": "8" -} -``` - -{{% /tab %}} -{{% /tabs %}} -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -Note: the `logs.full.count` normally is equal to `logs.sw.count` + `logs.lotr.count`, while the `logs.error.count` will be a random number. -{{% /notice %}} - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/7-2-sum.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/7-2-sum.md deleted file mode 100644 index c8ff96f69a..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/7-2-sum.md +++ /dev/null @@ -1,159 +0,0 @@ ---- -title: 7.2 Create metrics with Sum Connector -linkTitle: 7.2 Sum Connector -time: 10 minutes -weight: 2 ---- - -In this section, we’ll explore how the [**Sum Connector**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/sumconnector) can extract values from spans and convert them into metrics. - -We’ll specifically use the credit card charges from our base spans and leverage the Sum Connector to retrieve the total charges as a metric. - -The connector can be used to collect (**sum**) attribute values from spans, span events, metrics, data points, and log records. It captures each individual value, transforms it into a metric, and passes it along. However, it’s the **backend’s** job to use these metrics and their attributes for calculations and further processing. - -{{% exercise title="Add the Sum Connector" %}} - -Switch to your **Agent terminal** window and open the `agent.yaml` file in your editor. - -- **Add the Sum Connector** -Include the Sum Connector in the connectors section of your configuration and define the metrics counters: - -```yaml - sum: - spans: - user.card-charge: - source_attribute: payment.amount - conditions: - - attributes["payment.amount"] != "NULL" - attributes: - - key: user.name - -``` - -{{% /exercise %}} - -In the example above, we check for the `payment.amount` attribute in spans. If it has a valid value, the **Sum** connector generates a metric called `user.card-charge` and includes the `user.name` as an attribute. This enables the backend to track and display a user’s total charges over an extended period, such as a billing cycle. - -In the pipeline configuration below, the connector exporter is added to the traces section, while the connector receiver is added to the metrics section. - -{{% exercise title="Wire the connector into the pipelines" %}} - -- **Configure the Count Connector in the pipelines** - -```yaml - pipelines: - traces: - receivers: - - otlp - processors: - - memory_limiter - - attributes/update # Update, hash, and remove attributes - - redaction/redact # Redact sensitive fields using regex - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - file - - otlphttp - - sum # Sum connector which aggregates payment.amount from spans and sends to metrics pipeline - metrics: - receivers: - - sum # Receives metrics from the sum exporter in the traces pipeline - - count # Receives count metric from logs count exporter in logs pipeline. - - otlp - #- hostmetrics # Host Metrics Receiver - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - otlphttp - logs: - receivers: - - otlp - - filelog/quotes - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - transform/logs # Transform logs processor - - batch - exporters: - - count # Count Connector that exports count as a metric to metrics pipeline. - - debug - - otlphttp -``` - -- **Validate** the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `traces` and `metrics:` sections of your pipelines will look like this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(otlp
fa:fa-download
):::receiver - REC3(otlp
fa:fa-download
):::receiver - PRO1(memory_limiter
fa:fa-microchip
):::processor - PRO2(memory_limiter
fa:fa-microchip
):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(resource
fa:fa-microchip
add_mode):::processor - PRO5(batch
fa:fa-microchip
):::processor - PRO6(batch
fa:fa-microchip
):::processor - PRO7(resourcedetection
fa:fa-microchip
):::processor - PRO8(resourcedetection
fa:fa-microchip
):::processor - - PROA(attributes
fa:fa-microchip
redact):::processor - PROB(redaction
fa:fa-microchip
update):::processor - EXP1(  debug  
fa:fa-upload
):::exporter - EXP2(  file  
fa:fa-upload
):::exporter - EXP3(  debug  
fa:fa-upload
):::exporter - EXP4(  otlphttp  
fa:fa-upload
):::exporter - EXP5(  otlphttp  
fa:fa-upload
):::exporter - ROUTE1( sum 
fa:fa-route
):::con-export - ROUTE2( count 
fa:fa-route
):::con-receive - ROUTE3( sum 
fa:fa-route
):::con-receive - - %% Links - subgraph wrapper[" "] - direction LR - subgraph subID1["`**Traces**`"] - direction LR - REC1 --> PRO1 - PRO1 --> PROA - PROA --> PROB - PROB --> PRO7 - PRO7 --> PRO3 - PRO3 --> PRO5 - PRO5 --> EXP1 - PRO5 --> EXP2 - PRO5 --> EXP5 - PRO5 --> ROUTE1 - end - - subgraph subID2["`**Metrics**`"] - direction LR - ROUTE1 --> ROUTE3 - ROUTE3 --> PRO2 - ROUTE2 --> PRO2 - REC3 --> PRO2 - PRO2 --> PRO8 - PRO8 --> PRO4 - PRO4 --> PRO6 - PRO6 --> EXP3 - PRO6 --> EXP4 - end - end - class subID1 sub-traces - class subID2 sub-metrics - -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-logs stroke:#34d399,stroke-width:1px,color:#34d399,stroke-dasharray: 3 3; -classDef sub-traces stroke:#fbbf24,stroke-width:1px,color:#fbbf24,stroke-dasharray: 3 3; -classDef sub-metrics stroke:#38bdf8,stroke-width:1px,color:#38bdf8,stroke-dasharray: 3 3; -``` - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/7-3-sum-test.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/7-3-sum-test.md deleted file mode 100644 index bd0d2d6be4..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/7-3-sum-test.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -title: 7.3 Testing the Count Connector -linkTitle: 7.3 Test Sum Connector -weight: 3 ---- - -{{% exercise title="Test the Sum Connector" %}} - -**Start the Gateway**: -In the **Gateway terminal** window run: - -```bash { title="Start the Gateway" } -../otelcol --config=gateway.yaml -``` - -**Start the Agent**: -In the **Agent terminal** window run: - -```bash { title="Start the Agent" } -../otelcol --config=agent.yaml -``` - -**Start the Loadgen**: -In the **Spans terminal** window send 8 spans with the following `loadgen` command: - -```bash { title="Loadgen" } -../loadgen -count 8 -``` - -Both the **Agent** and **Gateway** will display debug information, showing they are processing data. Wait until the loadgen completes. - -**Verify the metrics**: -While processing the span, the **Agent**, has generated metrics and passed them on to the **Gateway**. The **Gateway** has written them into `gateway-metrics.out`. - -To confirm the presence of `user.card-charge`and verify each one has an attribute `user.name` in the metrics output, run the following jq query: - -{{% tabs %}} -{{% tab title="jq query command" %}} - -```bash - -jq -r '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "user.card-charge") | .sum.dataPoints[] | "\(.attributes[] | select(.key == "user.name").value.stringValue)\t\(.asDouble)"' gateway-metrics.out | while IFS=$'\t' read -r name charge; do - printf "%-20s %s\n" "$name" "$charge" -done -``` - -{{% /tab %}} -{{% tab title="jq example output" %}} - -```text -George Lucas 67.49 -Frodo Baggins 87.14 -Thorin Oakenshield 90.98 -Luke Skywalker 51.37 -Luke Skywalker 65.56 -Thorin Oakenshield 67.5 -Thorin Oakenshield 66.66 -Peter Jackson 94.39 -``` - -{{% /tab %}} -{{% /tabs %}} - -> [!IMPORTANT] -> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals. - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/_index.md deleted file mode 100644 index af4a689175..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/7-sum-count/_index.md +++ /dev/null @@ -1,192 +0,0 @@ ---- -title: 7. Create metrics with Count Connector -linkTitle: 7. Count & Sum Connector -time: 10 minutes -weight: 9 ---- -In this section, we'll explore how to use the [**Count Connector**](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/countconnector) to extract attribute values from logs and convert them into meaningful metrics. - -Specifically, we'll use the Count Connector to track the number of "Star Wars" and "Lord of the Rings" quotes appearing in our logs, turning them into measurable data points. - -{{% exercise title="Set up the `7-sum-count` directory" %}} - -> [!IMPORTANT] -> **Change _ALL_ terminal windows to the `7-sum-count` directory and run the `clear` command.** - -Copy `*.yaml` from the `6-routing-data` directory into `7-sum-count`. Your updated directory structure will now look like this: - -```text { title="Updated Directory Structure" } -. -├── agent.yaml -└── gateway.yaml -``` - -- **Update the agent.yaml** to change the frequency that we read logs. -Find the `filelog/quotes` receiver in the `agent.yaml` and add a `poll_interval` attribute: - -```yaml - filelog/quotes: # Receiver Type/Name - poll_interval: 10s # Only read every ten seconds -``` - -{{% /exercise %}} - -The reason for the delay is that the Count Connector in the OpenTelemetry Collector counts logs only within each processing interval. This means that every time the data is read, the count resets to zero for the next interval. With the default `Filelog reciever` interval of 200ms, it reads every line the loadgen writes, giving us counts of 1. With this interval we make sure we have multiple entries to count. - -The Collector can maintain a running count for each read interval by omitting conditions, as shown below. However, it’s best practice to let your backend handle running counts since it can track them over a longer time period. - -{{% exercise title="Add the Count Connector" %}} - -- **Add the Count Connector** - -Include the Count Connector in the connector's section of your configuration and define the metrics counters we want to use: - -```yaml -connectors: - count: - logs: - logs.full.count: - description: "Running count of all logs read in interval" - logs.sw.count: - description: "StarWarsCount" - conditions: - - attributes["movie"] == "SW" - logs.lotr.count: - description: "LOTRCount" - conditions: - - attributes["movie"] == "LOTR" - logs.error.count: - description: "ErrorCount" - conditions: - - attributes["level"] == "ERROR" -``` - -- **Explanation of the Metrics Counters** - - - `logs.full.count`: Tracks the total number of logs processed during each read interval. - Since this metric has no filtering conditions, every log that passes through the system is included in the count. - - `logs.sw.count` Counts logs that contain a quote from a Star Wars movie. - - `logs.lotr.count`: Counts logs that contain a quote from a Lord of the Rings movie. - - `logs.error.count`: Represents a real-world scenario by counting logs with a severity level of ERROR for the read interval. - -- **Configure the Count Connector in the pipelines** -In the pipeline configuration below, the connector exporter is added to the `logs` section, while the connector receiver is added to the `metrics` section. - -```yaml - pipelines: - traces: - receivers: - - otlp - processors: - - memory_limiter - - attributes/update # Update, hash, and remove attributes - - redaction/redact # Redact sensitive fields using regex - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - file - - otlphttp - metrics: - receivers: - - count # Count Connector that receives count metric from logs count exporter in logs pipeline. - - otlp - #- hostmetrics # Host Metrics Receiver - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - batch - exporters: - - debug - - otlphttp - logs: - receivers: - - otlp - - filelog/quotes - processors: - - memory_limiter - - resourcedetection - - resource/add_mode - - transform/logs # Transform logs processor - - batch - exporters: - - count # Count Connector that exports count as a metric to metrics pipeline. - - debug - - otlphttp -``` - -{{% /exercise %}} - -We count logs based on their attributes. If your log data is stored in the log body instead of attributes, you’ll need to use a `Transform` processor in your pipeline to extract key/value pairs and add them as attributes. - -In this workshop, we’ve already added `merge_maps(attributes, cache, "upsert")` in the `05-transform-data` section. This ensures that all relevant data is included in the log attributes for processing. - -When selecting fields to create attributes from, be mindful—adding all fields indiscriminately is generally not ideal for production environments. Instead, choose only the fields that are truly necessary to avoid unnecessary data clutter. - -{{% exercise title="Validate the agent configuration" %}} - -- **Validate** the agent configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `logs` and `metrics:` sections of your pipelines will look like this: - -```mermaid -%%{init:{"fontFamily":"monospace"}}%% -graph LR - %% Nodes - REC1(otlp
fa:fa-download):::receiver - REC2(filelog
fa:fa-download
quotes):::receiver - REC3(otlp
fa:fa-download):::receiver - PRO1(memory_limiter
fa:fa-microchip):::processor - PRO2(memory_limiter
fa:fa-microchip):::processor - PRO3(resource
fa:fa-microchip
add_mode):::processor - PRO4(resource
fa:fa-microchip
add_mode):::processor - PRO5(batch
fa:fa-microchip):::processor - PRO6(batch
fa:fa-microchip):::processor - PRO7(resourcedetection
fa:fa-microchip):::processor - PRO8(resourcedetection
fa:fa-microchip):::processor - PRO9(transfrom
fa:fa-microchip
logs):::processor - EXP1(  debug  
fa:fa-upload):::exporter - EXP2(  otlphttp  
fa:fa-upload):::exporter - EXP3(  debug  
fa:fa-upload):::exporter - EXP4(  otlphttp  
fa:fa-upload):::exporter - ROUTE1( count 
fa:fa-route):::con-export - ROUTE2( count 
fa:fa-route):::con-receive - - %% Links - subID1:::sub-logs - subID2:::sub-metrics - subgraph " " - direction LR - subgraph subID1[**Logs**] - direction LR - REC1 --> PRO1 - REC2 --> PRO1 - PRO1 --> PRO7 - PRO7 --> PRO3 - PRO3 --> PRO9 - PRO9 --> PRO5 - PRO5 --> ROUTE1 - PRO5 --> EXP1 - PRO5 --> EXP2 - end - - subgraph subID2["`**Metrics**`"] - direction LR - ROUTE1 --> ROUTE2 - ROUTE2 --> PRO2 - REC3 --> PRO2 - PRO2 --> PRO8 - PRO8 --> PRO4 - PRO4 --> PRO6 - PRO6 --> EXP3 - PRO6 --> EXP4 - end - end -classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff; -classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff; -classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff; -classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3; -classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3; -``` - -{{% /exercise %}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/8-wrap-up/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/8-wrap-up/_index.md deleted file mode 100644 index 6fe2b2077f..0000000000 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/8-wrap-up/_index.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: 8. Wrap-up -linkTitle: 8. Wrap-up -weight: 10 ---- - -![Well done](../images/welldone.png) diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/_index.md index 63c00ee4d5..6c726c79c0 100644 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/_index.md +++ b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/_index.md @@ -1,35 +1,31 @@ --- title: Advanced OpenTelemetry Collector -description: Practice setting up the OpenTelemetry Collector configuration from scratch and go though several advanced configuration scenarios's. +description: Practice reviewing and modifying a host-installed OpenTelemetry Collector configuration. weight: 2 type: chapter -authors: ["Robert Castley", "Charity Anderson", "Pieter Hagen", "Geoff Higginbottom"] -time: 75 minutes +authors: ["Robert Castley", "Charity Anderson", "Pieter Hagen", "Geoff Higginbottom", "Kyle Wang", "Antoine Toulme"] +time: 55 minutes +aliases: + - /conf/3-obs1184/ --- -The goal of this workshop is to help you gain confidence in creating and modifying OpenTelemetry Collector configuration files. You’ll start with a minimal `agent.yaml` and `gateway.yaml` file and progressively build them out to handle several advanced, real-world scenarios. +In this workshop, you run one Splunk Distribution of the OpenTelemetry +Collector in **agent mode** on a Linux host or Apple silicon Mac. The Collector +receives sample traces, reads sample logs, and collects host metrics. You then +use OTel Collector Config Builder to improve the data before it leaves the +Collector. -A key focus of this workshop is learning how to configure the OpenTelemetry Collector to store telemetry data locally, rather than sending it to a third-party vendor backend. This approach not only simplifies debugging and troubleshooting but is also ideal for testing and development environments where you want to avoid sending data to production systems. +## Workshop overview -To make the most of this workshop, you should have: +During this workshop, you will: -- A basic understanding of the OpenTelemetry Collector and its configuration file structure. -- Proficiency in editing YAML files. +- Run version `0.161.0` of the Collector as one Collector process. +- Generate traces and logs, and collect host metrics. +- Check all three signals locally and, when available, in Splunk Observability + Cloud. +- Upload `agent_config.yaml` and inspect it in OTel Collector Config Builder. +- Filter noisy spans, protect sensitive attributes, and transform logs. +- Download the completed configuration, apply it to the Collector, and verify the + results. -Everything in this workshop is designed to run locally, ensuring a hands-on and accessible learning experience. Let’s dive in and start building! - -## Workshop Overview - -During this workshop, we will cover the following topics: - -- **Setting up the agent and gateway locally**: Test that metrics, traces, and logs go via the agent to the gateway. -- **Enhancing agent resilience**: Basic configurations for fault tolerance. -- **Configuring processors**: - - Filter out noise by dropping specific spans (e.g., health checks). - - Remove unnecessary tags, and handle sensitive data. - - Transform data using OTTL (OpenTelemetry Transformation Language) in the pipeline before exporting. -- **Configuring Connectors**: - - Route data to different endpoints based on the values received. - - -By the end of this workshop, you'll be familiar with configuring the OpenTelemetry Collector for a variety of real-world use cases. +Chapter 6 includes more ways to continue learning after the workshop. diff --git a/content/en/conf/3-obs1184/images/cloud-validation-active-hosts-navigator.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-active-hosts-navigator.webp similarity index 100% rename from content/en/conf/3-obs1184/images/cloud-validation-active-hosts-navigator.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-active-hosts-navigator.webp diff --git a/content/en/conf/3-obs1184/images/cloud-validation-host-navigator.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-host-navigator.webp similarity index 100% rename from content/en/conf/3-obs1184/images/cloud-validation-host-navigator.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-host-navigator.webp diff --git a/content/en/conf/3-obs1184/images/cloud-validation-host-search.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-host-search.webp similarity index 100% rename from content/en/conf/3-obs1184/images/cloud-validation-host-search.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-host-search.webp diff --git a/content/en/conf/3-obs1184/images/cloud-validation-trace-analyzer.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-trace-analyzer.webp similarity index 100% rename from content/en/conf/3-obs1184/images/cloud-validation-trace-analyzer.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-trace-analyzer.webp diff --git a/content/en/conf/3-obs1184/images/cloud-validation-trace-span-properties.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-trace-span-properties.webp similarity index 100% rename from content/en/conf/3-obs1184/images/cloud-validation-trace-span-properties.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/cloud-validation-trace-span-properties.webp diff --git a/content/en/conf/3-obs1184/images/config-builder-component-type.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-component-type.webp similarity index 100% rename from content/en/conf/3-obs1184/images/config-builder-component-type.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-component-type.webp diff --git a/content/en/conf/3-obs1184/images/config-builder-filter-component.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-filter-component.webp similarity index 100% rename from content/en/conf/3-obs1184/images/config-builder-filter-component.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-filter-component.webp diff --git a/content/en/conf/3-obs1184/images/config-builder-filter-health-preview.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-filter-health-preview.webp similarity index 100% rename from content/en/conf/3-obs1184/images/config-builder-filter-health-preview.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-filter-health-preview.webp diff --git a/content/en/conf/3-obs1184/images/config-builder-filter-trace-conditions.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-filter-trace-conditions.webp similarity index 100% rename from content/en/conf/3-obs1184/images/config-builder-filter-trace-conditions.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-filter-trace-conditions.webp diff --git a/content/en/conf/3-obs1184/images/config-builder-redaction-options.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-redaction-options.webp similarity index 100% rename from content/en/conf/3-obs1184/images/config-builder-redaction-options.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-redaction-options.webp diff --git a/content/en/conf/3-obs1184/images/config-builder-traces-attributes-redaction.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-traces-attributes-redaction.webp similarity index 100% rename from content/en/conf/3-obs1184/images/config-builder-traces-attributes-redaction.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-traces-attributes-redaction.webp diff --git a/content/en/conf/3-obs1184/images/config-builder-transform-empty-log-statements.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-transform-empty-log-statements.webp similarity index 100% rename from content/en/conf/3-obs1184/images/config-builder-transform-empty-log-statements.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-transform-empty-log-statements.webp diff --git a/content/en/conf/3-obs1184/images/config-builder-transform-options-contexts.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-transform-options-contexts.webp similarity index 100% rename from content/en/conf/3-obs1184/images/config-builder-transform-options-contexts.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-transform-options-contexts.webp diff --git a/content/en/conf/3-obs1184/images/config-builder-transform-preview.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-transform-preview.webp similarity index 100% rename from content/en/conf/3-obs1184/images/config-builder-transform-preview.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/config-builder-transform-preview.webp diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/welldone.png b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/welldone.png deleted file mode 100644 index b041ff09bf..0000000000 Binary files a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/welldone.png and /dev/null differ diff --git a/content/en/conf/3-obs1184/images/welldone.webp b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/welldone.webp similarity index 100% rename from content/en/conf/3-obs1184/images/welldone.webp rename to content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/images/welldone.webp diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/prerequisites.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/prerequisites.md index 36b1533a7c..13ffaa4757 100644 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/prerequisites.md +++ b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/2-advanced-collector/prerequisites.md @@ -1,175 +1,170 @@ --- -title: Pre-requisites +title: Prerequisites weight: 2.1 archetype: chapter time: 5 minutes --- -## Prerequisites +## Before you begin -- Proficiency in editing YAML files using `vi`, `vim`, `nano`, or your preferred text editor. -- Supported Environments: - - A provided Splunk Workshop Instance (preferred). Outbound access to port `2222` is required for `ssh` access. - - Apple Mac (Apple Silicon). Installation of `jq` is required - [**https://jqlang.org/download/**](https://jqlang.org/download/) +1. Pair up with other attendees (optional). -{{% exercise title="Create the workshop directory" %}} +2. Open Splunk Observability Cloud. Choose one of these options: -{{< step "Initial Setup" "1" >}} + - Sign in to the Observability Workshop organization provided with your + Splunk Show instance. + - Register for a free Splunk Observability Cloud organization and sign in. -In your environment create a new directory and change into it: + **Registration:** [Register for Splunk Observability Cloud Free](https://www.splunk.com/en_us/download/observability-cloud-free-edition.html) -``` bash -mkdir advanced-otel-workshop && \ -cd advanced-otel-workshop -``` +3. Choose one supported execution path. -We will refer to this directory as `[WORKSHOP]` for the remainder of the workshop. + The workshop requires `bash`, `curl`, `jq`, a text editor, outbound HTTPS, + and free local ports `2222`, `4318`, and `13133`. The Splunk Show path also + requires `ssh` on your local computer. The `scp` command is optional and is + needed only if you choose to transfer files manually. -{{% notice title="Remove any existing OpenTelemetry Collectors" style="warning" %}} -If you have completed the Splunk IM workshop, please ensure you have deleted the collector running in Kubernetes before continuing. This can be done by running the following command: + - **Splunk Show instance:** Open a terminal on your computer and connect to + the Splunk Show instance with the supplied SSH command. Keep the supplied + SSH command and password handy because you use them in each new terminal. + For example, enter: -``` bash -helm delete splunk-otel-collector -``` + ```bash + ssh -p 2222 splunk@127.0.0.1 + ``` + + At the prompt, enter the provided password. + - **Linux laptop or Apple silicon Mac:** Use Terminal locally. Linux systems + can use an `x86_64`/`amd64` or `arm64`/`aarch64` processor. + +{{% notice title="Windows and Intel-based Mac computers" style="warning" %}} +Connect to a Splunk Show instance to participate in this workshop remotely. +Contact a facilitator if you need help. +{{% /notice %}} -The EC2 instance in that case may also run some services that can interfere with this workshop , so run the following command to make sure they are stopped if present: +{{% exercise title="Set up the workshop" %}} -``` bash -kubectl delete ~/workshop/apm/deployment.yaml +{{< step "Create a folder" "1" >}} + +```bash +mkdir -p ~/advanced-otel-workshop +cd ~/advanced-otel-workshop ``` -{{% /notice %}} +The remaining pages refer to this folder as `[WORKSHOP]`. + {{< /step >}} -{{< step "Download workshop binaries" "2" >}} +{{< step "Get the three workshop files" "2" >}} -Change into your `[WORKSHOP]` directory and download the OpenTelemetry Collector, Load Generator binaries and setup script: +Select the tab for the computer that runs the Collector. {{% tabs %}} -{{% tab title="Splunk Workshop Instance" %}} +{{% tab title="Splunk Show instance" %}} + +{{% notice title="Keep your SSH details handy" style="info" %}} +The SSH command and password for your Splunk Show instance are provided by +email or by the workshop facilitator. Keep them in a convenient, secure place. +You must use the SSH command and password each time you open a new terminal and +connect to the instance. +{{% /notice %}} + +Connect to the Splunk Show instance with the supplied SSH command, then run: ```bash -curl -L https://github.com/signalfx/splunk-otel-collector/releases/download/v{{< legacy-otel-version >}}/otelcol_linux_amd64 -o otelcol && \ -curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 -o loadgen && \ -curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/setup-workshop.sh -o setup-workshop.sh && \ +cd ~/advanced-otel-workshop +curl -fL https://github.com/signalfx/splunk-otel-collector/releases/download/v0.161.0/otelcol_linux_amd64 -o otelcol +curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 -o loadgen +curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/setup-workshop.sh -o setup-workshop.sh chmod +x setup-workshop.sh ``` +Keep the supplied SSH details available for each new terminal. The guided +workshop does not require `scp`. + {{% /tab %}} -{{% tab title="Apple Silicon" %}} +{{% tab title="Linux x86_64" %}} + +Use this tab for an `x86_64` or `amd64` Linux laptop. ```bash -curl -L https://github.com/signalfx/splunk-otel-collector/releases/download/v{{< legacy-otel-version >}}/otelcol_darwin_arm64 -o otelcol && \ -curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-arm64 -o loadgen && \ -curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/setup-workshop.sh -o setup-workshop.sh && \ +curl -fL https://github.com/signalfx/splunk-otel-collector/releases/download/v0.161.0/otelcol_linux_amd64 -o otelcol +curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 -o loadgen +curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/setup-workshop.sh -o setup-workshop.sh chmod +x setup-workshop.sh ``` {{% /tab %}} -{{% /tabs %}} +{{% tab title="Linux ARM64" %}} -{{< /step >}} - -{{< step "Run the setup" "3" >}} -Run the `setup-workshop.sh` script which will configure the correct permissions and also create the initial configurations for the **Agent** and the **Gateway**: - -{{% tabs %}} -{{% tab title="Setup Workshop" %}} +Use this tab when `uname -m` reports `arm64` or `aarch64`. ```bash -./setup-workshop.sh +curl -fL https://github.com/signalfx/splunk-otel-collector/releases/download/v0.161.0/otelcol_linux_arm64 -o otelcol +curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-arm64 -o loadgen +curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/setup-workshop.sh -o setup-workshop.sh +chmod +x setup-workshop.sh ``` {{% /tab %}} -{{% tab title="Verify Setup" %}} +{{% tab title="Apple silicon" %}} -```text -███████╗██████╗ ██╗ ██╗ ██╗███╗ ██╗██╗ ██╗ ██╗ -██╔════╝██╔══██╗██║ ██║ ██║████╗ ██║██║ ██╔╝ ╚██╗ -███████╗██████╔╝██║ ██║ ██║██╔██╗ ██║█████╔╝ ╚██╗ -╚════██║██╔═══╝ ██║ ██║ ██║██║╚██╗██║██╔═██╗ ██╔╝ -███████║██║ ███████╗╚██████╔╝██║ ╚████║██║ ██╗ ██╔╝ -╚══════╝╚═╝ ╚══════╝ ╚═════╝ ╚═╝ ╚═══╝╚═╝ ╚═╝ ╚═╝ - -Welcome to the Splunk Advanced OpenTelemetry Workshop! -====================================================== - -macOS detected. Removing quarantine attributes... -otelcol version v0.126.0 -Usage: loadgen [OPTIONS] -Options: - -base Send base traces (enabled by default) - -health Send health traces - -security Send security traces - -logs Enable logging of random quotes to quotes.log - -json Output logs in JSON format (only applicable with -logs) - -count Number of traces or logs to send (default: infinite) - -h, --help Display this help message - -Example: - loadgen -health -security -count 10 Send 10 health and security traces - loadgen -logs -json -count 5 Write 5 random quotes in JSON format to quotes.log -Creating workshop directories... -✓ Created subdirectories: - ├── 1-agent-gateway - ├── 2-building-resilience - ├── 3-dropping-spans - ├── 4-sensitive-data - ├── 5-transform-data - ├── 6-routing-data - └── 7-sum-count - -Creating configuration files for 1-agent-gateway... -Creating OpenTelemetry Collector agent configuration file: 1-agent-gateway/agent.yaml -✓ Configuration file created successfully: 1-agent-gateway/agent.yaml -✓ File size: 4355 bytes - -Creating OpenTelemetry Collector gateway configuration file: 1-agent-gateway/gateway.yaml -✓ Configuration file created successfully: 1-agent-gateway/gateway.yaml -✓ File size: 3376 bytes - -✓ Completed configuration files for 1-agent-gateway - -Creating configuration files for 2-building-resilience... -Creating OpenTelemetry Collector agent configuration file: 2-building-resilience/agent.yaml -✓ Configuration file created successfully: 2-building-resilience/agent.yaml -✓ File size: 4355 bytes - -Creating OpenTelemetry Collector gateway configuration file: 2-building-resilience/gateway.yaml -✓ Configuration file created successfully: 2-building-resilience/gateway.yaml -✓ File size: 3376 bytes - -✓ Completed configuration files for 2-building-resilience - -Workshop environment setup complete! -Configuration files created in the following directories: - 1-agent-gateway/ - ├── agent.yaml - └── gateway.yaml - 2-building-resilience/ - ├── agent.yaml - └── gateway.yaml +```bash +curl -fL https://github.com/signalfx/splunk-otel-collector/releases/download/v0.161.0/otelcol_darwin_arm64 -o otelcol +curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-arm64 -o loadgen +curl -fL https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/setup-workshop.sh -o setup-workshop.sh +chmod +x setup-workshop.sh ``` {{% /tab %}} {{% /tabs %}} -```text { title="Initial Directory Structure" } +{{< /step >}} + +{{< step "Run setup" "3" >}} + +```bash +./setup-workshop.sh +``` + +At the cloud-export prompt, press **Enter** to send metrics and traces to +Splunk Observability Cloud. To keep all workshop data local, enter `n`. +The script then asks for your realm and access token. On a Splunk Show +instance, the supplied realm appears as the default, and an access token is +already available. Press **Enter** to use each supplied value, or enter a +replacement, such as the realm and access token for your Splunk Observability +Cloud Free organization. Token characters are hidden while you type. + +Setup creates one Collector configuration: + +```text [WORKSHOP] -├── 1-agent-gateway -├── 2-building-resilience -├── 3-dropping-spans -├── 4-sensitive-data -├── 5-transform-data -├── 6-routing-data -├── 7-sum-count +├── 1-agent +│ └── agent_config.yaml ├── loadgen ├── otelcol -└── setup-workshop.sh +├── setup-workshop.sh +└── workshop-env.sh ``` +You learn about the components and pipelines in `agent_config.yaml` in Step +1.6. + +{{% expand title="Optional: find your realm and access token when using your own organization" %}} + +Skip this section when you are using a Splunk Show instance. + +If you use your own Splunk Observability Cloud organization: + +- Find the realm in the organization URL. For example, a URL containing `us1` + uses the `us1` realm. +- Go to **Settings > Access Tokens**. Create a token or use an existing token + that has ingest authorization. + +{{% /expand %}} + {{< /step >}} {{% /exercise %}} -{{< checkpoint "Workshop environment is ready — onto **Chapter 1: Agent Configuration**." >}} +{{< checkpoint "One Collector configuration is ready." >}} diff --git a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/_index.md b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/_index.md index c932c9d70e..fee23f3022 100644 --- a/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/_index.md +++ b/content/en/ninja-workshops/foundations/2-opentelemetry-collector-workshops/_index.md @@ -1,8 +1,8 @@ --- title: OpenTelemetry Collector weight: 2 -time: 2 hours 15 minutes -description: Two-part deep dive into the OpenTelemetry Collector — start with the receiver/processor/exporter fundamentals, then build agent and gateway configurations from scratch through real-world scenarios. +time: 1 hour 55 minutes +description: Two-part deep dive into the OpenTelemetry Collector — start with the receiver/processor/exporter fundamentals, then review and improve a host-installed agent configuration through real-world scenarios. aliases: - /ninja-workshops/3-opentelemetry-collector-workshops/ layout: "hero" diff --git a/content/en/ninja-workshops/foundations/_index.md b/content/en/ninja-workshops/foundations/_index.md index d1bc49ff04..4add66995a 100644 --- a/content/en/ninja-workshops/foundations/_index.md +++ b/content/en/ninja-workshops/foundations/_index.md @@ -4,6 +4,6 @@ hero_title: Foundations layout: "hero" weight: 1 description: Baseline OpenTelemetry, dashboards, and the day-to-day troubleshooting workflow. -time: 4 hours 45 minutes +time: 4 hours 25 minutes --- diff --git a/content/en/splunk4rookies/o11y-rookies-26/pathways/ai-monitoring/_index.md b/content/en/splunk4rookies/o11y-rookies-26/pathways/ai-monitoring/_index.md index 2dbb42809f..86b981aa23 100644 --- a/content/en/splunk4rookies/o11y-rookies-26/pathways/ai-monitoring/_index.md +++ b/content/en/splunk4rookies/o11y-rookies-26/pathways/ai-monitoring/_index.md @@ -8,7 +8,7 @@ description: Follow the AI path to use AI-assisted workflows and application tel Complete these modules in the recommended order: -{{< cards >}} +{{< cards target="_blank" >}} {{< card title="Let's Introduce Our Friend" href="/splunk4rookies/o11y-rookies-26/modules/2-ai/" show-time="true" >}} Use Splunk AI capabilities to reduce noise, surface likely causes, and accelerate investigations. {{< /card >}} diff --git a/content/en/splunk4rookies/o11y-rookies-26/pathways/digital-experience-monitoring/_index.md b/content/en/splunk4rookies/o11y-rookies-26/pathways/digital-experience-monitoring/_index.md index e5c44ad575..65787525e5 100644 --- a/content/en/splunk4rookies/o11y-rookies-26/pathways/digital-experience-monitoring/_index.md +++ b/content/en/splunk4rookies/o11y-rookies-26/pathways/digital-experience-monitoring/_index.md @@ -7,7 +7,7 @@ description: Follow the digital experience path to understand real user behaviou Complete these modules in the recommended order: -{{< cards >}} +{{< cards target="_blank" >}} {{< card title="Beat social media to the issue" href="/splunk4rookies/o11y-rookies-26/modules/5-dem/" show-time="true" >}} Combine Real User Monitoring and Synthetics to identify and prevent poor customer experiences. {{< /card >}} diff --git a/content/en/splunk4rookies/o11y-rookies-26/pathways/infrastructure-monitoring/_index.md b/content/en/splunk4rookies/o11y-rookies-26/pathways/infrastructure-monitoring/_index.md index c6270be948..a1f5314104 100644 --- a/content/en/splunk4rookies/o11y-rookies-26/pathways/infrastructure-monitoring/_index.md +++ b/content/en/splunk4rookies/o11y-rookies-26/pathways/infrastructure-monitoring/_index.md @@ -8,7 +8,7 @@ description: Follow the infrastructure path to investigate resources, logs, and Complete these modules in the recommended order: -{{< cards >}} +{{< cards target="_blank" >}} {{< card title="Neighbours are a pain" href="/splunk4rookies/o11y-rookies-26/modules/4-im/" show-time="true" >}} Find a noisy neighbour using Splunk Infrastructure Monitoring. {{< /card >}} diff --git a/go.mod b/go.mod index 918e150f94..55ff572aa3 100644 --- a/go.mod +++ b/go.mod @@ -2,6 +2,6 @@ module github.com/splunk/observability-workshops go 1.26.3 -require github.com/splunk/hugo-theme-splunk-workshop v0.13.20 // indirect +require github.com/splunk/hugo-theme-splunk-workshop v0.13.21 // indirect //replace github.com/splunk/hugo-theme-splunk-workshop => /Users/rcastley/Documents/GitHub/hugo-theme-splunk-workshop diff --git a/go.sum b/go.sum index ae8a316765..9afc3266c6 100644 --- a/go.sum +++ b/go.sum @@ -1,2 +1,2 @@ -github.com/splunk/hugo-theme-splunk-workshop v0.13.20 h1:ZwYN5w38I0CB/X2uVtNhytUbWF4GrH1fHGzdIIt1q7g= -github.com/splunk/hugo-theme-splunk-workshop v0.13.20/go.mod h1:W41sA3uf5oTWqIUA1aUY4M0C/JEaMa0gV3CqVDs0ReA= +github.com/splunk/hugo-theme-splunk-workshop v0.13.21 h1:KSOurrRWtoOdf+DxBi0dxp0RYWyGHs9DSEO+yYEiqiM= +github.com/splunk/hugo-theme-splunk-workshop v0.13.21/go.mod h1:W41sA3uf5oTWqIUA1aUY4M0C/JEaMa0gV3CqVDs0ReA= diff --git a/workshop/ninja/obs1184/agent_config.solution.yaml b/workshop/ninja/advanced-otel/agent_config.solution.yaml similarity index 89% rename from workshop/ninja/obs1184/agent_config.solution.yaml rename to workshop/ninja/advanced-otel/agent_config.solution.yaml index 2ee40e65a7..30fd21fc90 100644 --- a/workshop/ninja/obs1184/agent_config.solution.yaml +++ b/workshop/ninja/advanced-otel/agent_config.solution.yaml @@ -95,7 +95,7 @@ receivers: type: processlist zipkin: endpoint: "${SPLUNK_LISTEN_INTERFACE}:9411" - # Used by setup-workshop-conf2026.sh only when cloud export is disabled. + # Used by setup-workshop.sh only when cloud export is disabled. nop: processors: @@ -125,9 +125,14 @@ processors: - '\b4[0-9]{3}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b' - '\b5[1-5][0-9]{2}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b' summary: debug - resourcedetection: + resource_detection: detectors: [gcp, ecs, ec2, azure, system] override: true + transform/limit_histogram_buckets: + metric_statements: + - context: datapoint + statements: + - merge_histogram_buckets(32, method="limit_buckets") resource/add_mode: attributes: - action: upsert @@ -213,33 +218,33 @@ service: pipelines: traces: receivers: [jaeger, otlp, zipkin] - processors: [memory_limiter, filter/health, attributes, redaction, resourcedetection, resource/add_mode, batch] + processors: [memory_limiter, filter/health, attributes, redaction, resource_detection, resource/add_mode, batch] exporters: [debug, file/traces, otlp_http] metrics: receivers: [host_metrics, otlp] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, transform/limit_histogram_buckets, batch, resource_detection] exporters: [signalfx] metrics/internal: receivers: [prometheus/internal] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, batch, resource_detection] exporters: [signalfx] logs/signalfx: receivers: [smartagent/processlist] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, batch, resource_detection] exporters: [signalfx] logs: receivers: [fluent_forward, otlp] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, batch, resource_detection] exporters: [splunk_hec, splunk_hec/profiling] logs/entities: receivers: [nop] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, batch, resource_detection] exporters: [otlp_http/entities] metrics/workshop: receivers: [host_metrics/workshop] - processors: [memory_limiter, resourcedetection, resource/add_mode] + processors: [memory_limiter, resource_detection, resource/add_mode] exporters: [debug, file/metrics] logs/workshop: receivers: [otlp, file_log/quotes] - processors: [memory_limiter, resourcedetection, transform, resource/add_mode] + processors: [memory_limiter, resource_detection, transform, resource/add_mode] exporters: [debug, file/logs] diff --git a/workshop/ninja/obs1184/agent_config.yaml b/workshop/ninja/advanced-otel/agent_config.yaml similarity index 83% rename from workshop/ninja/obs1184/agent_config.yaml rename to workshop/ninja/advanced-otel/agent_config.yaml index 56c75e162a..7c34a71533 100644 --- a/workshop/ninja/obs1184/agent_config.yaml +++ b/workshop/ninja/advanced-otel/agent_config.yaml @@ -1,7 +1,7 @@ # Collector configuration for the .conf26 Advanced Collector workshop. # # The default components and six default service pipelines are based on the -# Splunk Distribution of the OpenTelemetry Collector v0.157.0 configuration. +# Splunk Distribution of the OpenTelemetry Collector v0.161.0 configuration. # Gateway-only alternatives are intentionally omitted. The workshop adds local # file/debug exporters plus metrics/workshop and logs/workshop pipelines. @@ -110,9 +110,16 @@ processors: memory_limiter: check_interval: 2s limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB} - resourcedetection: + resource_detection: detectors: [gcp, ecs, ec2, azure, system] override: true + # Limit explicit histograms to at most 32 buckets before export through + # the signalfx exporter. Added in the v0.161.0 default metrics pipeline. + transform/limit_histogram_buckets: + metric_statements: + - context: datapoint + statements: + - merge_histogram_buckets(32, method="limit_buckets") # Workshop-only marker used in local and cloud validation. resource/add_mode: attributes: @@ -163,7 +170,7 @@ exporters: file/logs: path: ./agent-logs.out append: false - # Used by setup-workshop-conf2026.sh only when cloud export is disabled. + # Used by setup-workshop.sh only when cloud export is disabled. nop: service: @@ -186,37 +193,37 @@ service: # Default pipeline, augmented with workshop debug/file output and mode marker. traces: receivers: [jaeger, otlp, zipkin] - processors: [memory_limiter, resourcedetection, resource/add_mode, batch] + processors: [memory_limiter, resource_detection, resource/add_mode, batch] exporters: [debug, file/traces, otlp_http] - # Default Splunk pipelines from v0.157.0. + # Default Splunk pipelines from v0.161.0. metrics: receivers: [host_metrics, otlp] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, transform/limit_histogram_buckets, batch, resource_detection] exporters: [signalfx] metrics/internal: receivers: [prometheus/internal] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, batch, resource_detection] exporters: [signalfx] logs/signalfx: receivers: [smartagent/processlist] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, batch, resource_detection] exporters: [signalfx] logs/entities: receivers: [nop] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, batch, resource_detection] exporters: [otlp_http/entities] - # Default Splunk logs pipeline from v0.157.0. Workshop-generated file logs + # Default Splunk logs pipeline from v0.161.0. Workshop-generated file logs # use logs/workshop instead, so this pipeline receives no live-lab traffic. logs: receivers: [fluent_forward, otlp] - processors: [memory_limiter, batch, resourcedetection] + processors: [memory_limiter, batch, resource_detection] exporters: [splunk_hec, splunk_hec/profiling] # Workshop-only pipelines for bounded local validation. metrics/workshop: receivers: [host_metrics/workshop] - processors: [memory_limiter, resourcedetection, resource/add_mode] + processors: [memory_limiter, resource_detection, resource/add_mode] exporters: [debug, file/metrics] logs/workshop: receivers: [otlp, file_log/quotes] - processors: [memory_limiter, resourcedetection, resource/add_mode] + processors: [memory_limiter, resource_detection, resource/add_mode] exporters: [debug, file/logs] diff --git a/workshop/ninja/obs1184/loadgen/README.md b/workshop/ninja/advanced-otel/loadgen/README.md similarity index 100% rename from workshop/ninja/obs1184/loadgen/README.md rename to workshop/ninja/advanced-otel/loadgen/README.md diff --git a/workshop/ninja/advanced-otel/loadgen/build.sh b/workshop/ninja/advanced-otel/loadgen/build.sh old mode 100644 new mode 100755 diff --git a/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-amd64 b/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-amd64 deleted file mode 100755 index 4c8a1e338b..0000000000 Binary files a/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-amd64 and /dev/null differ diff --git a/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-arm64 b/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-arm64 index fde50b0e23..7942d8f5f6 100755 Binary files a/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-arm64 and b/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-arm64 differ diff --git a/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 b/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 index 330c69d148..a9808fe306 100755 Binary files a/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 and b/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 differ diff --git a/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-arm64 b/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-arm64 index a9456610d5..849bbe7945 100755 Binary files a/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-arm64 and b/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-arm64 differ diff --git a/workshop/ninja/advanced-otel/loadgen/loadgen.go b/workshop/ninja/advanced-otel/loadgen/loadgen.go index 2a4c9b8e97..6a07732782 100644 --- a/workshop/ninja/advanced-otel/loadgen/loadgen.go +++ b/workshop/ninja/advanced-otel/loadgen/loadgen.go @@ -17,6 +17,9 @@ import ( // Global flag to ensure the first trace always uses "George Lucas" var firstAttempt = true +// previewOnly prints the original OTLP/HTTP JSON payload without sending it. +var previewOnly bool + // getRandomUserName selects "George Lucas" for the first trace, then randomizes func getRandomUserName() string { userNames := []string{ @@ -173,8 +176,10 @@ func sendBaseTrace(traceID, spanID string, startTime, endTime int64) { }, } - if err := sendJSON("http://localhost:4318/v1/traces", spanJSON); err != nil { + if err := sendJSON("http://127.0.0.1:4318/v1/traces", spanJSON); err != nil { log.Printf("Failed to send base trace: %v", err) + } else if previewOnly { + fmt.Println("Base trace preview shown above; nothing was sent.") } else { fmt.Printf("\nBase trace sent with traceId: %s and spanId: %s\n", traceID, spanID) } @@ -236,8 +241,10 @@ func sendSecurityTrace(traceID, spanID string, startTime, endTime int64) { }, } - if err := sendJSON("http://localhost:4318/v1/traces", securityJSON); err != nil { + if err := sendJSON("http://127.0.0.1:4318/v1/traces", securityJSON); err != nil { log.Printf("Failed to send security trace: %v", err) + } else if previewOnly { + fmt.Println("Security trace preview shown above; nothing was sent.") } else { fmt.Printf("\nSecurity trace sent with traceId: %s and spanId: %s\n", traceID, spanID) } @@ -290,15 +297,101 @@ func sendHealthTrace(traceID, spanID string, startTime, endTime int64) { }, } - if err := sendJSON("http://localhost:4318/v1/traces", healthJSON); err != nil { + if err := sendJSON("http://127.0.0.1:4318/v1/traces", healthJSON); err != nil { log.Printf("Failed to send health trace: %v", err) + } else if previewOnly { + fmt.Println("Health trace preview shown above; nothing was sent.") } else { fmt.Printf("\nHealth trace sent with traceId: %s and spanId: %s\n", traceID, spanID) } } +// sendCorrelatedLog sends one OTLP log record with the same trace and span IDs +// as a base trace. This makes the optional Splunk Platform and Log Observer +// Connect exercise demonstrably correlated instead of merely colocated. +func sendCorrelatedLog(traceID, spanID string, timestamp int64) { + quote, movie := getRandomQuote() + logJSON := map[string]interface{}{ + "resourceLogs": []interface{}{ + map[string]interface{}{ + "resource": map[string]interface{}{ + "attributes": []interface{}{ + map[string]interface{}{ + "key": "service.name", + "value": map[string]interface{}{ + "stringValue": "cinema-service", + }, + }, + map[string]interface{}{ + "key": "deployment.environment", + "value": map[string]interface{}{ + "stringValue": "production", + }, + }, + }, + }, + "scopeLogs": []interface{}{ + map[string]interface{}{ + "scope": map[string]interface{}{ + "name": "cinema.library", + "version": "1.0.0", + }, + "logRecords": []interface{}{ + map[string]interface{}{ + "timeUnixNano": fmt.Sprintf("%d", timestamp), + "severityNumber": 9, + "severityText": "INFO", + "body": map[string]interface{}{ + "stringValue": quote, + }, + "attributes": []interface{}{ + map[string]interface{}{ + "key": "movie", + "value": map[string]interface{}{ + "stringValue": movie, + }, + }, + map[string]interface{}{ + "key": "workshop.signal", + "value": map[string]interface{}{ + "stringValue": "correlated-log", + }, + }, + }, + "traceId": traceID, + "spanId": spanID, + "flags": 1, + }, + }, + }, + }, + }, + }, + } + + if err := sendJSON("http://127.0.0.1:4318/v1/logs", logJSON); err != nil { + log.Printf("Failed to send correlated log: %v", err) + } else if previewOnly { + fmt.Println("Correlated log preview shown above; nothing was sent.") + } else { + fmt.Printf("Correlated log sent with traceId: %s and spanId: %s\n", traceID, spanID) + } +} + // Helper function to send JSON data via HTTP POST func sendJSON(url string, data interface{}) error { + if previewOnly { + var formatted bytes.Buffer + encoder := json.NewEncoder(&formatted) + encoder.SetIndent("", " ") + encoder.SetEscapeHTML(false) + if err := encoder.Encode(data); err != nil { + return fmt.Errorf("failed to format JSON preview: %w", err) + } + fmt.Printf("\nOriginal OTLP payload for %s:\n%s", url, formatted.String()) + return nil + } + jsonData, err := json.Marshal(data) if err != nil { return fmt.Errorf("failed to marshal JSON data: %w", err) @@ -379,7 +472,11 @@ func generateLogEntry(jsonOutput bool) string { // Function to write logs to a file func writeLogs(jsonOutput bool, count int) { logFile := "quotes.log" - fmt.Printf("Writing logs to %s. Press Ctrl+C to stop.\n", logFile) + if count == 0 { + fmt.Printf("Writing logs to %s. Press Ctrl+C to stop.\n", logFile) + } else { + fmt.Printf("Writing %d logs to %s.\n", count, logFile) + } // If count is 0, run infinitely if count == 0 { @@ -417,12 +514,17 @@ Options: -security Send security traces -logs Enable logging of random quotes to quotes.log -json Output logs in JSON format (only applicable with -logs) + -correlated Send an OTLP log correlated to each base trace + -preview Print original OTLP JSON payloads without sending them -count Number of traces or logs to send (default: infinite) -h, --help Display this help message Example: loadgen -health -security -count 10 Send 10 health and security traces - loadgen -logs -json -count 5 Write 5 random quotes in JSON format to quotes.log`) + loadgen -preview -health -count 1 Show one original app span and health span without sending + loadgen -preview -count 1 Show one original app span without sending + loadgen -logs -json -count 5 Write 5 random quotes in JSON format to quotes.log + loadgen -correlated -count 5 Send 5 traces with correlated OTLP logs`) } func main() { @@ -432,6 +534,8 @@ func main() { securityFlag := flag.Bool("security", false, "Send security traces") logsFlag := flag.Bool("logs", false, "Enable logging of random quotes to quotes.log") jsonFlag := flag.Bool("json", false, "Output logs in JSON format (only applicable with -logs)") + correlatedFlag := flag.Bool("correlated", false, "Send an OTLP log correlated to each base trace") + previewFlag := flag.Bool("preview", false, "Print original OTLP JSON payloads without sending them") countFlag := flag.Int("count", 0, "Number of traces or logs to send (default: infinite)") helpFlag := flag.Bool("h", false, "Display help message") helpFlagLong := flag.Bool("help", false, "Display help message") @@ -444,12 +548,23 @@ func main() { os.Exit(0) } - // Start logging if -logs flag is provided + previewOnly = *previewFlag + + // File-only log generation does not need the trace loop. Run it directly so + // a finite -count exits as soon as the requested lines have been written. + if *logsFlag && !*correlatedFlag { + writeLogs(*jsonFlag, *countFlag) + return + } if *logsFlag { - go writeLogs(*jsonFlag, *countFlag) // Run logs in a separate goroutine + go writeLogs(*jsonFlag, *countFlag) } - fmt.Println("Sending traces. Use Ctrl-C to stop.") + if previewOnly { + fmt.Println("Preview mode: original OTLP payloads are printed but not sent.") + } else { + fmt.Println("Sending traces. Use Ctrl-C to stop.") + } for i := 0; *countFlag == 0 || i < *countFlag; i++ { traceID := generateTraceID() @@ -457,20 +572,30 @@ func main() { currentTime := getCurrentTime() endTime := currentTime + int64(time.Second) - if *baseFlag && !*logsFlag { + if *baseFlag && (!*logsFlag || *correlatedFlag) { sendBaseTrace(traceID, spanID, currentTime, endTime) } + if *correlatedFlag { + sendCorrelatedLog(traceID, spanID, getCurrentTime()) + } + if *healthFlag { - time.Sleep(2 * time.Second) + if !previewOnly { + time.Sleep(2 * time.Second) + } sendHealthTrace(traceID, generateSpanID(), getCurrentTime(), getCurrentTime()+int64(time.Second)) } if *securityFlag { - time.Sleep(2 * time.Second) + if !previewOnly { + time.Sleep(2 * time.Second) + } sendSecurityTrace(traceID, generateSpanID(), getCurrentTime(), getCurrentTime()+int64(time.Second)) } - time.Sleep(2 * time.Second) + if !previewOnly { + time.Sleep(2 * time.Second) + } } } diff --git a/workshop/ninja/advanced-otel/setup-workshop.sh b/workshop/ninja/advanced-otel/setup-workshop.sh index d2b2c50dc5..0799a69bc7 100755 --- a/workshop/ninja/advanced-otel/setup-workshop.sh +++ b/workshop/ninja/advanced-otel/setup-workshop.sh @@ -1,301 +1,209 @@ -#!/bin/bash - -# Splunk Workshop Setup Script -# This script displays the Splunk ASCII art and creates workshop directories - -echo "" -echo "███████╗██████╗ ██╗ ██╗ ██╗███╗ ██╗██╗ ██╗ ██╗" -echo "██╔════╝██╔══██╗██║ ██║ ██║████╗ ██║██║ ██╔╝ ╚██╗" -echo "███████╗██████╔╝██║ ██║ ██║██╔██╗ ██║█████╔╝ ╚██╗" -echo "╚════██║██╔═══╝ ██║ ██║ ██║██║╚██╗██║██╔═██╗ ██╔╝" -echo "███████║██║ ███████╗╚██████╔╝██║ ╚████║██║ ██╗ ██╔╝" -echo "╚══════╝╚═╝ ╚══════╝ ╚═════╝ ╚═╝ ╚═══╝╚═╝ ╚═╝ ╚═╝" -echo "" -echo "Welcome to the Splunk Advanced OpenTelemetry Workshop!" -echo "======================================================" -echo "" - -# Function to run common commands -run_commands() { - chmod +x otelcol loadgen && \ - ./otelcol -v && \ - ./loadgen --help +#!/usr/bin/env bash +set -euo pipefail + +setup_failed() { + exit_code=$? + trap - ERR + echo >&2 + echo "Setup did not complete. Review the error above, then rerun the script." >&2 + exit "${exit_code}" } +trap setup_failed ERR + +collector_version="0.161.0" +repo_owner="splunk" +repo_name="observability-workshop" +repo_ref="main" +asset_path="workshop/ninja/advanced-otel" +workshop_root="${PWD}" +agent_dir="${workshop_root}/1-agent" +config_path="${agent_dir}/agent_config.yaml" +environment_path="${workshop_root}/workshop-env.sh" + +echo +echo "Splunk Advanced OpenTelemetry Workshop .conf26" +echo "================================================" + +for command_name in curl jq uname sed; do + if ! command -v "${command_name}" >/dev/null 2>&1; then + echo "Required command not found: ${command_name}" >&2 + exit 1 + fi +done -# Platform-specific setup -case "$OSTYPE" in - darwin*) - echo "macOS detected. Removing quarantine attributes..." - xattr -dr com.apple.quarantine otelcol loadgen 2>/dev/null - run_commands - ;; - linux-gnu*) - echo "Linux detected." - run_commands - ;; - *) - echo "Unsupported platform ($OSTYPE). This script only works on macOS and Linux." +case "$(uname -s)" in + Darwin) + if [[ "$(uname -m)" != "arm64" ]]; then + echo "This workshop supports Apple silicon Macs, not Intel-based Macs." >&2 + exit 1 + fi + xattr -dr com.apple.quarantine otelcol loadgen 2>/dev/null || true + echo "Apple silicon Mac detected." + ;; + Linux) + case "$(uname -m)" in + x86_64|amd64|aarch64|arm64) ;; + *) + echo "Unsupported Linux CPU architecture: $(uname -m)" >&2 exit 1 ;; + esac + echo "Linux $(uname -m) detected." + ;; + *) + echo "Unsupported platform. Use Linux or an Apple silicon Mac." >&2 + exit 1 + ;; esac -# Create workshop directories -echo "Creating workshop directories..." - -# Common workshop subdirectories -mkdir -p 1-agent-gateway -mkdir -p 2-building-resilience -mkdir -p 3-dropping-spans -mkdir -p 4-sensitive-data -mkdir -p 5-transform-data -mkdir -p 6-routing-data -mkdir -p 7-sum-count - -echo "✓ Created subdirectories:" -echo " ├── 1-agent-gateway" -echo " ├── 2-building-resilience" -echo " ├── 3-dropping-spans" -echo " ├── 4-sensitive-data" -echo " ├── 5-transform-data" -echo " ├── 6-routing-data" -echo " └── 7-sum-count" -echo "" - -# Function to create agent.yaml configuration -create_agent_config() { - local config_file="$1" - echo "Creating OpenTelemetry Collector agent configuration file: ${config_file}" - - cat > "${config_file}" << 'EOF' -########################### This section holds all the -## Configuration section ## configurations that can be -########################### used in this OpenTelemetry Collector -extensions: # Array of Extensions - health_check: # Configures the health check extension - endpoint: 0.0.0.0:13133 # Endpoint to collect health check data - -receivers: # Array of Receivers - hostmetrics: # Receiver Type - collection_interval: 3600s # Scrape metrics every hour - scrapers: # Array of hostmetric scrapers - cpu: # Scraper for cpu metrics - otlp: # Receiver Type - protocols: # list of Protocols used - http: # This wil enable the HTTP Protocol - endpoint: "0.0.0.0:4318" # Endpoint for incoming telemetry data - filelog/quotes: # Receiver Type/Name - include: ./quotes.log # The file to read log data from - include_file_path: true # Include file path in the log data - include_file_name: false # Exclude file name from the log data - resource: # Add custom resource attributes - com.splunk.source: ./quotes.log # Source of the log data - com.splunk.sourcetype: quotes # Source type of the log data - -exporters: # Array of Exporters - debug: # Exporter Type - verbosity: detailed # Enabled detailed debug output - otlphttp: # Exporter Type - endpoint: "http://localhost:5318" # Gateway OTLP endpoint - file: # Exporter Type - path: "./agent.out" # Save path (OTLP JSON) - append: false # Overwrite the file each time -processors: # Array of Processors - memory_limiter: # Limits memory usage by Collectors pipeline - check_interval: 2s # Interval to check memory usage - limit_mib: 512 # Memory limit in MiB - resourcedetection: # Processor Type - detectors: [system] # Detect system resource information - override: true # Overwrites existing attributes - resource/add_mode: # Processor Type/Name - attributes: # Array of attributes and modifications - - action: insert # Action is to insert a key - key: otelcol.service.mode # Key name - value: "agent" # Key value - -########################### This section controls what -### Activation Section ### configurations will be used -########################### by this OpenTelemetry Collector -service: # Services configured for this Collector - extensions: # Enabled extensions - - health_check - pipelines: # Array of configured pipelines - traces: - receivers: - - otlp - processors: - - memory_limiter # Memory Limiter processor - - resourcedetection # Adds system attributes to the data - - resource/add_mode # Adds collector mode metadata - exporters: - - debug - - file - - otlphttp - metrics: - receivers: - - hostmetrics # Hostmetric reciever (cpu only) - - otlp - processors: - - memory_limiter # Memory Limiter processor - - resourcedetection # Adds system attributes to the data - - resource/add_mode # Adds collector mode metadata - exporters: - - debug - - file - - otlphttp - logs: - receivers: - - otlp - - filelog/quotes # Filelog Receiver - processors: - - memory_limiter # Memory Limiter processor - - resourcedetection # Adds system attributes to the data - - resource/add_mode # Adds collector mode metadata - exporters: - - debug - - file - - otlphttp -EOF - - # Check if the file was created successfully - if [ $? -eq 0 ]; then - echo "✓ Configuration file created successfully: ${config_file}" - echo "✓ File size: $(wc -c < "${config_file}") bytes" - echo "" - return 0 - else - echo "✗ Error: Failed to create configuration file: ${config_file}" - return 1 - fi -} - -# Function to create gateway.yaml configuration -create_gateway_config() { - local config_file="$1" - echo "Creating OpenTelemetry Collector gateway configuration file: ${config_file}" - - cat > "${config_file}" << 'EOF' -########################### This section holds all the -## Configuration section ## configurations that can be -########################### used in this OpenTelemetry Collector -extensions: # List of extensions - health_check: # Health check extension - endpoint: 0.0.0.0:14133 # Custom port to avoid conflicts - -receivers: - otlp: # OTLP receiver - protocols: - http: # HTTP protocol - endpoint: "0.0.0.0:5318" # Custom port to avoid conflicts - include_metadata: true # Required for token pass-through - -exporters: # List of exporters - debug: # Debug exporter - verbosity: detailed # Enable detailed debug output - file/traces: # Exporter Type/Name - path: "./gateway-traces.out" # Path for OTLP JSON output for traces - append: false # Overwrite the file each time - file/metrics: # Exporter Type/Name - path: "./gateway-metrics.out" # Path for OTLP JSON output for metrics - append: false # Overwrite the file each time - file/logs: # Exporter Type/Name - path: "./gateway-logs.out" # Path for OTLP JSON output for logs - append: false # Overwrite the file each time - -processors: # List of processors - memory_limiter: # Limits memory usage - check_interval: 2s # Memory check interval - limit_mib: 512 # Memory limit in MiB - batch: # Batches data before exporting - metadata_keys: # Groups data by token - - X-SF-Token - resource/add_mode: # Adds metadata - attributes: - - action: upsert # Inserts or updates a key - key: otelcol.service.mode # Key name - value: "gateway" # Key value - -# Connectors -#connectors: # leave this commented out; we will uncomment in an upcoming exercise - -########################### -### Activation Section ### -########################### -service: # Service configuration - telemetry: - metrics: - level: none # Disable metrics - extensions: [health_check] # Enabled extensions - pipelines: # Configured pipelines - traces: # Traces pipeline - receivers: - - otlp # OTLP receiver - processors: # Processors for traces - - memory_limiter - - resource/add_mode - - batch - exporters: - - debug # Debug exporter - - file/traces - metrics: # Metrics pipeline - receivers: - - otlp # OTLP receiver - processors: # Processors for metrics - - memory_limiter - - resource/add_mode - - batch - exporters: - - debug # Debug exporter - - file/metrics - logs: # Logs pipeline - receivers: - - otlp # OTLP receiver - processors: # Processors for logs - - memory_limiter - - resource/add_mode - - batch - exporters: - - debug # Debug exporter - - file/logs -EOF - - # Check if the file was created successfully - if [ $? -eq 0 ]; then - echo "✓ Configuration file created successfully: ${config_file}" - echo "✓ File size: $(wc -c < "${config_file}") bytes" - echo "" - return 0 - else - echo "✗ Error: Failed to create configuration file: ${config_file}" - return 1 - fi -} - -# Create configuration files for multiple directories -DIRECTORIES=("1-agent-gateway" "2-building-resilience") - -for dir in "${DIRECTORIES[@]}"; do - echo "Creating configuration files for ${dir}..." - - # Create agent.yaml - if ! create_agent_config "${dir}/agent.yaml"; then - echo "✗ Failed to create agent.yaml in ${dir}" - exit 1 - fi - - # Create gateway.yaml - if ! create_gateway_config "${dir}/gateway.yaml"; then - echo "✗ Failed to create gateway.yaml in ${dir}" - exit 1 - fi - - echo "✓ Completed configuration files for ${dir}" - echo "" -done +if [[ ! -f otelcol || ! -f loadgen ]]; then + echo "otelcol and loadgen must be in the current directory." >&2 + exit 1 +fi + +chmod +x otelcol loadgen + +if installed_version="$(./otelcol --version 2>&1)"; then + : +else + echo "The Collector binary does not run on $(uname -s)/$(uname -m)." >&2 + exit 1 +fi + +if [[ "${installed_version}" != *"${collector_version}"* ]]; then + echo "Expected Collector ${collector_version}, but found: ${installed_version}" >&2 + exit 1 +fi + +if ! loadgen_help="$(./loadgen --help 2>&1)"; then + echo "The load generator does not run on $(uname -s)/$(uname -m)." >&2 + exit 1 +fi +if [[ "${loadgen_help}" != *"-preview"* ]]; then + echo "This workshop requires the OBS1184 load generator with -preview support." >&2 + echo "Download the loadgen binary again by following the Prerequisites." >&2 + exit 1 +fi +unset loadgen_help + +cloud_setting="${CONF2026_CLOUD_ENABLED:-}" +cloud_prompt="Y/n" +case "${cloud_setting}" in + n|N|no|NO|No|false|FALSE|False|0) + cloud_prompt="y/N" + ;; +esac +read -r -p "Send metrics and traces to Splunk Observability Cloud? [${cloud_prompt}]: " cloud_input +if [[ -n "${cloud_input}" ]]; then + cloud_setting="${cloud_input}" +elif [[ -z "${cloud_setting}" ]]; then + cloud_setting="y" +fi +unset cloud_input cloud_prompt + +case "${cloud_setting}" in + ""|y|Y|yes|YES|Yes|true|TRUE|True|1) + cloud_enabled=true + ;; + n|N|no|NO|No|false|FALSE|False|0) + cloud_enabled=false + ;; + *) + echo "Enter y or n." >&2 + exit 1 + ;; +esac -echo "Workshop environment setup complete!" -echo "Configuration files created in the following directories:" -for dir in "${DIRECTORIES[@]}"; do - echo " ${dir}/" - echo " ├── agent.yaml" - echo " └── gateway.yaml" -done \ No newline at end of file +realm="${REALM:-}" +splunk_access_token="${SPLUNK_ACCESS_TOKEN:-${ACCESS_TOKEN:-}}" +splunk_api_url="${SPLUNK_API_URL:-}" +splunk_ingest_url="${SPLUNK_INGEST_URL:-}" +splunk_hec_token="${SPLUNK_HEC_TOKEN:-not-configured}" +splunk_hec_url="${SPLUNK_HEC_URL:-https://127.0.0.1:8088/services/collector}" +splunk_listen_interface="${SPLUNK_LISTEN_INTERFACE:-127.0.0.1}" +splunk_memory_limit_mib="${SPLUNK_MEMORY_LIMIT_MIB:-512}" + +if [[ "${cloud_enabled}" == "true" ]]; then + if [[ -n "${realm}" ]]; then + read -r -p "Splunk Observability Cloud realm (for example us1) [${realm}]: " realm_input + else + read -r -p "Splunk Observability Cloud realm (for example us1): " realm_input + fi + if [[ -n "${realm_input}" ]]; then + realm="${realm_input}" + fi + unset realm_input + if [[ -z "${realm}" ]]; then + echo "A realm is required for cloud export." >&2 + exit 1 + fi + + if [[ -n "${splunk_access_token}" ]]; then + read -r -s -p "Splunk Observability Cloud access token (press Enter to use the token already provided): " token_input + else + read -r -s -p "Splunk Observability Cloud access token with ingest authorization: " token_input + fi + echo + if [[ -n "${token_input}" ]]; then + splunk_access_token="${token_input}" + fi + unset token_input + if [[ -z "${splunk_access_token}" ]]; then + echo "An access token with ingest authorization is required for cloud export." >&2 + exit 1 + fi + + splunk_api_url="${splunk_api_url:-https://api.${realm}.observability.splunkcloud.com}" + splunk_ingest_url="${splunk_ingest_url:-https://ingest.${realm}.observability.splunkcloud.com}" +else + realm="" + splunk_access_token="not-configured" + splunk_api_url="http://127.0.0.1:18089" + splunk_ingest_url="http://127.0.0.1:18089" +fi + +mkdir -p "${agent_dir}" +config_url="https://github.com/${repo_owner}/${repo_name}/raw/refs/heads/${repo_ref}/${asset_path}/agent_config.yaml" +curl -fL --retry 3 "${config_url}" -o "${config_path}" + +# Keep all eight pipelines in one configuration. Without cloud export, replace +# active cloud destinations with nop while the workshop pipelines continue +# local debug and file validation. +if [[ "${cloud_enabled}" == "false" ]]; then + sed -i.bak \ + -e 's/exporters: \[debug, file\/traces, otlp_http\]/exporters: [debug, file\/traces]/' \ + -e 's/exporters: \[signalfx\]/exporters: [nop]/' \ + -e 's/exporters: \[otlp_http\/entities\]/exporters: [nop]/' \ + -e 's/extensions: \[headers_setter, health_check, http_forwarder, http_forwarder\/opamp_splunk_o11y, opamp\/splunk_o11y, zpages\]/extensions: [health_check, zpages]/' \ + "${config_path}" + rm -f "${config_path}.bak" +fi + +# Remove the obsolete overlay if setup is rerun in an earlier workshop folder. +rm -f "${agent_dir}/agent_config.local.yaml" + +{ + printf 'export REALM=%q\n' "${realm}" + printf 'export ACCESS_TOKEN=%q\n' "${splunk_access_token}" + printf 'export SPLUNK_ACCESS_TOKEN=%q\n' "${splunk_access_token}" + printf 'export SPLUNK_API_URL=%q\n' "${splunk_api_url}" + printf 'export SPLUNK_INGEST_URL=%q\n' "${splunk_ingest_url}" + printf 'export SPLUNK_HEC_TOKEN=%q\n' "${splunk_hec_token}" + printf 'export SPLUNK_HEC_URL=%q\n' "${splunk_hec_url}" + printf 'export SPLUNK_LISTEN_INTERFACE=%q\n' "${splunk_listen_interface}" + printf 'export SPLUNK_MEMORY_LIMIT_MIB=%q\n' "${splunk_memory_limit_mib}" + printf 'export CONF2026_CLOUD_ENABLED=%q\n' "${cloud_enabled}" +} > "${environment_path}" +chmod 600 "${environment_path}" +unset splunk_access_token splunk_hec_token +trap - ERR + +echo +echo "Workshop setup complete." +echo "Collector: ${installed_version}" +echo "Cloud export: ${cloud_enabled}" +echo +echo "Start the Collector:" +echo " cd 1-agent" +echo " source ../workshop-env.sh" +echo " ../otelcol --config=agent_config.yaml" diff --git a/workshop/ninja/obs1184/loadgen/build.sh b/workshop/ninja/obs1184/loadgen/build.sh deleted file mode 100755 index 42a897ab02..0000000000 --- a/workshop/ninja/obs1184/loadgen/build.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/bash - -# Name of the Go application -APP_NAME="loadgen" - -# Output directory for compiled binaries -OUTPUT_DIR="build" - -# Supported workshop platforms. Intel-based Macs use a Splunk Show instance. -PLATFORMS=( - "darwin/arm64" - "linux/amd64" - "linux/arm64" -) - -# Create the output directory if it doesn't exist -mkdir -p "$OUTPUT_DIR" - -# Function to compile for a specific platform -compile() { - local GOOS=$1 - local GOARCH=$2 - local OUTPUT_FILE="$OUTPUT_DIR/$APP_NAME-$GOOS-$GOARCH" - - echo "Compiling for $GOOS/$GOARCH..." - env CGO_ENABLED=0 GO111MODULE=off GOOS="$GOOS" GOARCH="$GOARCH" \ - go build -trimpath -ldflags="-s -w" -o "$OUTPUT_FILE" . - if [ $? -ne 0 ]; then - echo "Failed to compile for $GOOS/$GOARCH" - exit 1 - fi - echo "Compiled successfully: $OUTPUT_FILE" -} - -# Loop through all platforms and compile -for PLATFORM in "${PLATFORMS[@]}"; do - GOOS=${PLATFORM%/*} - GOARCH=${PLATFORM#*/} - compile "$GOOS" "$GOARCH" -done - -echo "All builds completed successfully. Output files are in the '$OUTPUT_DIR' directory." diff --git a/workshop/ninja/obs1184/loadgen/build/loadgen-darwin-arm64 b/workshop/ninja/obs1184/loadgen/build/loadgen-darwin-arm64 deleted file mode 100755 index 7942d8f5f6..0000000000 Binary files a/workshop/ninja/obs1184/loadgen/build/loadgen-darwin-arm64 and /dev/null differ diff --git a/workshop/ninja/obs1184/loadgen/build/loadgen-linux-amd64 b/workshop/ninja/obs1184/loadgen/build/loadgen-linux-amd64 deleted file mode 100755 index a9808fe306..0000000000 Binary files a/workshop/ninja/obs1184/loadgen/build/loadgen-linux-amd64 and /dev/null differ diff --git a/workshop/ninja/obs1184/loadgen/build/loadgen-linux-arm64 b/workshop/ninja/obs1184/loadgen/build/loadgen-linux-arm64 deleted file mode 100755 index 849bbe7945..0000000000 Binary files a/workshop/ninja/obs1184/loadgen/build/loadgen-linux-arm64 and /dev/null differ diff --git a/workshop/ninja/obs1184/loadgen/loadgen.go b/workshop/ninja/obs1184/loadgen/loadgen.go deleted file mode 100644 index 6a07732782..0000000000 --- a/workshop/ninja/obs1184/loadgen/loadgen.go +++ /dev/null @@ -1,601 +0,0 @@ -package main - -import ( - "bytes" - "encoding/hex" - "encoding/json" - "flag" - "fmt" - "io" - "log" - "math/rand" - "net/http" - "os" - "time" -) - -// Global flag to ensure the first trace always uses "George Lucas" -var firstAttempt = true - -// previewOnly prints the original OTLP/HTTP JSON payload without sending it. -var previewOnly bool - -// getRandomUserName selects "George Lucas" for the first trace, then randomizes -func getRandomUserName() string { - userNames := []string{ - "George Lucas", - "Darth Vader", - "Luke Skywalker", - "Frodo Baggins", - "Peter Jackson", - "Thorin Oakenshield", - } - - if firstAttempt { - firstAttempt = false // Ensure future calls are random - return "George Lucas" - } - return userNames[rand.Intn(len(userNames)-1)+1] // Skip "George Lucas" after the first attempt -} - -// Function to generate a random trace ID -func generateTraceID() string { - return randomHex(16) -} - -// Function to generate a random span ID -func generateSpanID() string { - return randomHex(8) -} - -// GenerateRandomPayment generates a random payment amount between 50 and 100 -func generateRandomPayment() float64 { - rand.Seed(time.Now().UnixNano()) - amount := 50.00 + rand.Float64()*(100.00-50.00) // Random float between 50.00 and 100.00 - return float64(int(amount*100)) / 100 // Round to two decimal places -} - -// Helper function to generate a random hex string of a given length -func randomHex(length int) string { - b := make([]byte, length) - _, err := rand.Read(b) - if err != nil { - log.Fatalf("Failed to generate random bytes: %v", err) - } - return hex.EncodeToString(b) -} - -// Function to get the current timestamp in nanoseconds -func getCurrentTime() int64 { - return time.Now().UnixNano() -} - -// Function to send a base trace -func sendBaseTrace(traceID, spanID string, startTime, endTime int64) { - randomUser := getRandomUserName() // Ensure first attempt is "George Lucas", then randomize - spanJSON := map[string]interface{}{ - "resourceSpans": []interface{}{ - map[string]interface{}{ - "resource": map[string]interface{}{ - "attributes": []interface{}{ - map[string]interface{}{ - "key": "service.name", - "value": map[string]interface{}{ - "stringValue": "cinema-service", - }, - }, - map[string]interface{}{ - "key": "deployment.environment", - "value": map[string]interface{}{ - "stringValue": "production", - }, - }, - }, - }, - "scopeSpans": []interface{}{ - map[string]interface{}{ - "scope": map[string]interface{}{ - "name": "cinema.library", - "version": "1.0.0", - "attributes": []interface{}{ - map[string]interface{}{ - "key": "fintest.scope.attribute", - "value": map[string]interface{}{ - "stringValue": "Starwars, LOTR", - }, - }, - }, - }, - "spans": []interface{}{ - map[string]interface{}{ - "traceId": traceID, - "spanId": spanID, - "name": "/movie-validator", - "startTimeUnixNano": fmt.Sprintf("%d", startTime), - "endTimeUnixNano": fmt.Sprintf("%d", endTime), - "kind": 2, - "status": map[string]interface{}{ - "code": 1, - "message": "Success", - }, - "attributes": []interface{}{ - map[string]interface{}{ - "key": "user.name", - "value": map[string]interface{}{ - "stringValue": randomUser, // Uses "George Lucas" first, then random - }, - }, - map[string]interface{}{ - "key": "user.phone_number", - "value": map[string]interface{}{ - "stringValue": "+1555-867-5309", - }, - }, - map[string]interface{}{ - "key": "user.email", - "value": map[string]interface{}{ - "stringValue": "george@deathstar.email", - }, - }, - map[string]interface{}{ - "key": "user.password", - "value": map[string]interface{}{ - "stringValue": "LOTR>StarWars1-2-3", - }, - }, - map[string]interface{}{ - "key": "user.visa", - "value": map[string]interface{}{ - "stringValue": "4111 1111 1111 1111", - }, - }, - map[string]interface{}{ - "key": "user.amex", - "value": map[string]interface{}{ - "stringValue": "3782 822463 10005", - }, - }, - map[string]interface{}{ - "key": "user.mastercard", - "value": map[string]interface{}{ - "stringValue": "5555 5555 5555 4444", - }, - }, - map[string]interface{}{ - "key": "payment.amount", - "value": map[string]interface{}{ - "doubleValue": generateRandomPayment(), - }, - }, - }, - }, - }, - }, - }, - }, - }, - } - - if err := sendJSON("http://127.0.0.1:4318/v1/traces", spanJSON); err != nil { - log.Printf("Failed to send base trace: %v", err) - } else if previewOnly { - fmt.Println("Base trace preview shown above; nothing was sent.") - } else { - fmt.Printf("\nBase trace sent with traceId: %s and spanId: %s\n", traceID, spanID) - } -} - -// Function to send a security trace -func sendSecurityTrace(traceID, spanID string, startTime, endTime int64) { - securityJSON := map[string]interface{}{ - "resourceSpans": []interface{}{ - map[string]interface{}{ - "resource": map[string]interface{}{ - "attributes": []interface{}{ - map[string]interface{}{ - "key": "service.name", - "value": map[string]interface{}{ - "stringValue": "password-check", - }, - }, - map[string]interface{}{ - "key": "deployment.environment", - "value": map[string]interface{}{ - "stringValue": "security-applications", - }, - }, - }, - }, - "scopeSpans": []interface{}{ - map[string]interface{}{ - "scope": map[string]interface{}{ - "name": "movie.library", - "version": "1.0.0", - }, - "spans": []interface{}{ - map[string]interface{}{ - "traceId": traceID, - "spanId": spanID, - "parentSpanId": generateSpanID(), - "name": "password-validation", - "startTimeUnixNano": fmt.Sprintf("%d", startTime), - "endTimeUnixNano": fmt.Sprintf("%d", endTime), - "kind": 2, - "status": map[string]interface{}{ - "code": 1, - "message": "Success", - }, - "attributes": []interface{}{ - map[string]interface{}{ - "key": "user.name", - "value": map[string]interface{}{ - "stringValue": "George Lucas", - }, - }, - }, - }, - }, - }, - }, - }, - }, - } - - if err := sendJSON("http://127.0.0.1:4318/v1/traces", securityJSON); err != nil { - log.Printf("Failed to send security trace: %v", err) - } else if previewOnly { - fmt.Println("Security trace preview shown above; nothing was sent.") - } else { - fmt.Printf("\nSecurity trace sent with traceId: %s and spanId: %s\n", traceID, spanID) - } -} - -// Function to send a health trace -func sendHealthTrace(traceID, spanID string, startTime, endTime int64) { - healthJSON := map[string]interface{}{ - "resourceSpans": []interface{}{ - map[string]interface{}{ - "resource": map[string]interface{}{ - "attributes": []interface{}{ - map[string]interface{}{ - "key": "service.name", - "value": map[string]interface{}{ - "stringValue": "frontend-service", - }, - }, - map[string]interface{}{ - "key": "deployment.environment", - "value": map[string]interface{}{ - "stringValue": "production", - }, - }, - }, - }, - "scopeSpans": []interface{}{ - map[string]interface{}{ - "scope": map[string]interface{}{ - "name": "healthz", - "version": "1.0.0", - }, - "spans": []interface{}{ - map[string]interface{}{ - "traceId": traceID, - "spanId": spanID, - "name": "/_healthz", - "startTimeUnixNano": fmt.Sprintf("%d", startTime), - "endTimeUnixNano": fmt.Sprintf("%d", endTime), - "kind": 2, - "status": map[string]interface{}{ - "code": 1, - "message": "Success", - }, - }, - }, - }, - }, - }, - }, - } - - if err := sendJSON("http://127.0.0.1:4318/v1/traces", healthJSON); err != nil { - log.Printf("Failed to send health trace: %v", err) - } else if previewOnly { - fmt.Println("Health trace preview shown above; nothing was sent.") - } else { - fmt.Printf("\nHealth trace sent with traceId: %s and spanId: %s\n", traceID, spanID) - } -} - -// sendCorrelatedLog sends one OTLP log record with the same trace and span IDs -// as a base trace. This makes the optional Splunk Platform and Log Observer -// Connect exercise demonstrably correlated instead of merely colocated. -func sendCorrelatedLog(traceID, spanID string, timestamp int64) { - quote, movie := getRandomQuote() - logJSON := map[string]interface{}{ - "resourceLogs": []interface{}{ - map[string]interface{}{ - "resource": map[string]interface{}{ - "attributes": []interface{}{ - map[string]interface{}{ - "key": "service.name", - "value": map[string]interface{}{ - "stringValue": "cinema-service", - }, - }, - map[string]interface{}{ - "key": "deployment.environment", - "value": map[string]interface{}{ - "stringValue": "production", - }, - }, - }, - }, - "scopeLogs": []interface{}{ - map[string]interface{}{ - "scope": map[string]interface{}{ - "name": "cinema.library", - "version": "1.0.0", - }, - "logRecords": []interface{}{ - map[string]interface{}{ - "timeUnixNano": fmt.Sprintf("%d", timestamp), - "severityNumber": 9, - "severityText": "INFO", - "body": map[string]interface{}{ - "stringValue": quote, - }, - "attributes": []interface{}{ - map[string]interface{}{ - "key": "movie", - "value": map[string]interface{}{ - "stringValue": movie, - }, - }, - map[string]interface{}{ - "key": "workshop.signal", - "value": map[string]interface{}{ - "stringValue": "correlated-log", - }, - }, - }, - "traceId": traceID, - "spanId": spanID, - "flags": 1, - }, - }, - }, - }, - }, - }, - } - - if err := sendJSON("http://127.0.0.1:4318/v1/logs", logJSON); err != nil { - log.Printf("Failed to send correlated log: %v", err) - } else if previewOnly { - fmt.Println("Correlated log preview shown above; nothing was sent.") - } else { - fmt.Printf("Correlated log sent with traceId: %s and spanId: %s\n", traceID, spanID) - } -} - -// Helper function to send JSON data via HTTP POST -func sendJSON(url string, data interface{}) error { - if previewOnly { - var formatted bytes.Buffer - encoder := json.NewEncoder(&formatted) - encoder.SetIndent("", " ") - encoder.SetEscapeHTML(false) - if err := encoder.Encode(data); err != nil { - return fmt.Errorf("failed to format JSON preview: %w", err) - } - fmt.Printf("\nOriginal OTLP payload for %s:\n%s", url, formatted.String()) - return nil - } - - jsonData, err := json.Marshal(data) - if err != nil { - return fmt.Errorf("failed to marshal JSON data: %w", err) - } - - resp, err := http.Post(url, "application/json", bytes.NewBuffer(jsonData)) - if err != nil { - return fmt.Errorf("HTTP POST request failed: %w", err) - } - defer resp.Body.Close() - - if resp.StatusCode < 200 || resp.StatusCode >= 300 { - return fmt.Errorf("received non-2xx HTTP status code: %d", resp.StatusCode) - } - - body, err := io.ReadAll(resp.Body) - if err != nil { - return fmt.Errorf("failed to read response body: %w", err) - } - fmt.Println("Response:", string(body)) - - return nil -} - -// Function to generate a random quote -func getRandomQuote() (string, string) { - lotrQuotes := []string{ - "One does not simply walk into Mordor.", - "Even the smallest person can change the course of the future.", - "All we have to decide is what to do with the time that is given us.", - "There is some good in this world, and it's worth fighting for.", - "Not all those who wander are lost.", - "There's some good in this world, Mr. Frodo … and it's worth fighting for.", - "I wish the Ring had never come to me. I wish none of this had happened.", - } - - starWarsQuotes := []string{ - "Do or do not, there is no try.", - "The Force will be with you. Always.", - "I find your lack of faith disturbing.", - "In my experience, there is no such thing as luck.", - "Help me, Obi-Wan Kenobi. You're my only hope.", - "May the Force be with you.", - "Your focus determines your reality.", - } - - if rand.Intn(100) < 66 { - return lotrQuotes[rand.Intn(len(lotrQuotes))], "LOTR" - } - return starWarsQuotes[rand.Intn(len(starWarsQuotes))], "SW" -} - -// Function to generate a random log level -func getRandomLogLevel() string { - logLevels := []string{"INFO", "WARN", "ERROR", "DEBUG"} - return logLevels[rand.Intn(len(logLevels))] -} - -// Function to generate a log entry -func generateLogEntry(jsonOutput bool) string { - timestamp := time.Now().Format("2006-01-02 15:04:05") - level := getRandomLogLevel() - quote, movie := getRandomQuote() - - if jsonOutput { - logEntry := map[string]string{ - "timestamp": timestamp, - "level": level, - "message": quote, - "movie": movie, - } - jsonData, _ := json.Marshal(logEntry) - return string(jsonData) - } - return fmt.Sprintf("%s [%s] - %s %s", timestamp, level, quote, movie) -} - -// Function to write logs to a file -func writeLogs(jsonOutput bool, count int) { - logFile := "quotes.log" - if count == 0 { - fmt.Printf("Writing logs to %s. Press Ctrl+C to stop.\n", logFile) - } else { - fmt.Printf("Writing %d logs to %s.\n", count, logFile) - } - - // If count is 0, run infinitely - if count == 0 { - for { - logEntry := generateLogEntry(jsonOutput) - file, err := os.OpenFile(logFile, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) - if err != nil { - log.Fatalf("Failed to open log file: %v", err) - } - file.WriteString(logEntry + "\n") - file.Close() - time.Sleep(100 * time.Millisecond) - } - } else { - // Otherwise, run for the specified count - for i := 0; i < count; i++ { - logEntry := generateLogEntry(jsonOutput) - file, err := os.OpenFile(logFile, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) - if err != nil { - log.Fatalf("Failed to open log file: %v", err) - } - file.WriteString(logEntry + "\n") - file.Close() - time.Sleep(100 * time.Millisecond) - } - } -} - -// Display usage instructions -func printHelp() { - fmt.Println(`Usage: loadgen [OPTIONS] -Options: - -base Send base traces (enabled by default) - -health Send health traces - -security Send security traces - -logs Enable logging of random quotes to quotes.log - -json Output logs in JSON format (only applicable with -logs) - -correlated Send an OTLP log correlated to each base trace - -preview Print original OTLP JSON payloads without sending them - -count Number of traces or logs to send (default: infinite) - -h, --help Display this help message - -Example: - loadgen -health -security -count 10 Send 10 health and security traces - loadgen -preview -health -count 1 Show one original app span and health span without sending - loadgen -preview -count 1 Show one original app span without sending - loadgen -logs -json -count 5 Write 5 random quotes in JSON format to quotes.log - loadgen -correlated -count 5 Send 5 traces with correlated OTLP logs`) -} - -func main() { - // Define flags - baseFlag := flag.Bool("base", true, "Send base traces") - healthFlag := flag.Bool("health", false, "Send health traces") - securityFlag := flag.Bool("security", false, "Send security traces") - logsFlag := flag.Bool("logs", false, "Enable logging of random quotes to quotes.log") - jsonFlag := flag.Bool("json", false, "Output logs in JSON format (only applicable with -logs)") - correlatedFlag := flag.Bool("correlated", false, "Send an OTLP log correlated to each base trace") - previewFlag := flag.Bool("preview", false, "Print original OTLP JSON payloads without sending them") - countFlag := flag.Int("count", 0, "Number of traces or logs to send (default: infinite)") - helpFlag := flag.Bool("h", false, "Display help message") - helpFlagLong := flag.Bool("help", false, "Display help message") - - flag.Parse() - - // Display help and exit if -h or --help is provided - if *helpFlag || *helpFlagLong { - printHelp() - os.Exit(0) - } - - previewOnly = *previewFlag - - // File-only log generation does not need the trace loop. Run it directly so - // a finite -count exits as soon as the requested lines have been written. - if *logsFlag && !*correlatedFlag { - writeLogs(*jsonFlag, *countFlag) - return - } - if *logsFlag { - go writeLogs(*jsonFlag, *countFlag) - } - - if previewOnly { - fmt.Println("Preview mode: original OTLP payloads are printed but not sent.") - } else { - fmt.Println("Sending traces. Use Ctrl-C to stop.") - } - - for i := 0; *countFlag == 0 || i < *countFlag; i++ { - traceID := generateTraceID() - spanID := generateSpanID() - currentTime := getCurrentTime() - endTime := currentTime + int64(time.Second) - - if *baseFlag && (!*logsFlag || *correlatedFlag) { - sendBaseTrace(traceID, spanID, currentTime, endTime) - } - - if *correlatedFlag { - sendCorrelatedLog(traceID, spanID, getCurrentTime()) - } - - if *healthFlag { - if !previewOnly { - time.Sleep(2 * time.Second) - } - sendHealthTrace(traceID, generateSpanID(), getCurrentTime(), getCurrentTime()+int64(time.Second)) - } - - if *securityFlag { - if !previewOnly { - time.Sleep(2 * time.Second) - } - sendSecurityTrace(traceID, generateSpanID(), getCurrentTime(), getCurrentTime()+int64(time.Second)) - } - - if !previewOnly { - time.Sleep(2 * time.Second) - } - } -} diff --git a/workshop/ninja/obs1184/setup-workshop-conf2026.sh b/workshop/ninja/obs1184/setup-workshop-conf2026.sh deleted file mode 100755 index 2edaf7c5b2..0000000000 --- a/workshop/ninja/obs1184/setup-workshop-conf2026.sh +++ /dev/null @@ -1,209 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -setup_failed() { - exit_code=$? - trap - ERR - echo >&2 - echo "Setup did not complete. Review the error above, then rerun the script." >&2 - exit "${exit_code}" -} -trap setup_failed ERR - -collector_version="0.157.0" -repo_owner="splunk" -repo_name="observability-workshop" -repo_ref="main" -asset_path="workshop/ninja/obs1184" -workshop_root="${PWD}" -agent_dir="${workshop_root}/1-agent" -config_path="${agent_dir}/agent_config.yaml" -environment_path="${workshop_root}/workshop-env.sh" - -echo -echo "Splunk Advanced OpenTelemetry Workshop .conf26" -echo "================================================" - -for command_name in curl jq uname sed; do - if ! command -v "${command_name}" >/dev/null 2>&1; then - echo "Required command not found: ${command_name}" >&2 - exit 1 - fi -done - -case "$(uname -s)" in - Darwin) - if [[ "$(uname -m)" != "arm64" ]]; then - echo "This workshop supports Apple silicon Macs, not Intel-based Macs." >&2 - exit 1 - fi - xattr -dr com.apple.quarantine otelcol loadgen 2>/dev/null || true - echo "Apple silicon Mac detected." - ;; - Linux) - case "$(uname -m)" in - x86_64|amd64|aarch64|arm64) ;; - *) - echo "Unsupported Linux CPU architecture: $(uname -m)" >&2 - exit 1 - ;; - esac - echo "Linux $(uname -m) detected." - ;; - *) - echo "Unsupported platform. Use Linux or an Apple silicon Mac." >&2 - exit 1 - ;; -esac - -if [[ ! -f otelcol || ! -f loadgen ]]; then - echo "otelcol and loadgen must be in the current directory." >&2 - exit 1 -fi - -chmod +x otelcol loadgen - -if installed_version="$(./otelcol --version 2>&1)"; then - : -else - echo "The Collector binary does not run on $(uname -s)/$(uname -m)." >&2 - exit 1 -fi - -if [[ "${installed_version}" != *"${collector_version}"* ]]; then - echo "Expected Collector ${collector_version}, but found: ${installed_version}" >&2 - exit 1 -fi - -if ! loadgen_help="$(./loadgen --help 2>&1)"; then - echo "The load generator does not run on $(uname -s)/$(uname -m)." >&2 - exit 1 -fi -if [[ "${loadgen_help}" != *"-preview"* ]]; then - echo "This workshop requires the OBS1184 load generator with -preview support." >&2 - echo "Download the loadgen binary again by following the Prerequisites." >&2 - exit 1 -fi -unset loadgen_help - -cloud_setting="${CONF2026_CLOUD_ENABLED:-}" -cloud_prompt="Y/n" -case "${cloud_setting}" in - n|N|no|NO|No|false|FALSE|False|0) - cloud_prompt="y/N" - ;; -esac -read -r -p "Send metrics and traces to Splunk Observability Cloud? [${cloud_prompt}]: " cloud_input -if [[ -n "${cloud_input}" ]]; then - cloud_setting="${cloud_input}" -elif [[ -z "${cloud_setting}" ]]; then - cloud_setting="y" -fi -unset cloud_input cloud_prompt - -case "${cloud_setting}" in - ""|y|Y|yes|YES|Yes|true|TRUE|True|1) - cloud_enabled=true - ;; - n|N|no|NO|No|false|FALSE|False|0) - cloud_enabled=false - ;; - *) - echo "Enter y or n." >&2 - exit 1 - ;; -esac - -realm="${REALM:-}" -splunk_access_token="${SPLUNK_ACCESS_TOKEN:-${ACCESS_TOKEN:-}}" -splunk_api_url="${SPLUNK_API_URL:-}" -splunk_ingest_url="${SPLUNK_INGEST_URL:-}" -splunk_hec_token="${SPLUNK_HEC_TOKEN:-not-configured}" -splunk_hec_url="${SPLUNK_HEC_URL:-https://127.0.0.1:8088/services/collector}" -splunk_listen_interface="${SPLUNK_LISTEN_INTERFACE:-127.0.0.1}" -splunk_memory_limit_mib="${SPLUNK_MEMORY_LIMIT_MIB:-512}" - -if [[ "${cloud_enabled}" == "true" ]]; then - if [[ -n "${realm}" ]]; then - read -r -p "Splunk Observability Cloud realm (for example us1) [${realm}]: " realm_input - else - read -r -p "Splunk Observability Cloud realm (for example us1): " realm_input - fi - if [[ -n "${realm_input}" ]]; then - realm="${realm_input}" - fi - unset realm_input - if [[ -z "${realm}" ]]; then - echo "A realm is required for cloud export." >&2 - exit 1 - fi - - if [[ -n "${splunk_access_token}" ]]; then - read -r -s -p "Splunk Observability Cloud access token (press Enter to use the token already provided): " token_input - else - read -r -s -p "Splunk Observability Cloud access token with ingest authorization: " token_input - fi - echo - if [[ -n "${token_input}" ]]; then - splunk_access_token="${token_input}" - fi - unset token_input - if [[ -z "${splunk_access_token}" ]]; then - echo "An access token with ingest authorization is required for cloud export." >&2 - exit 1 - fi - - splunk_api_url="${splunk_api_url:-https://api.${realm}.observability.splunkcloud.com}" - splunk_ingest_url="${splunk_ingest_url:-https://ingest.${realm}.observability.splunkcloud.com}" -else - realm="" - splunk_access_token="not-configured" - splunk_api_url="http://127.0.0.1:18089" - splunk_ingest_url="http://127.0.0.1:18089" -fi - -mkdir -p "${agent_dir}" -config_url="https://github.com/${repo_owner}/${repo_name}/raw/refs/heads/${repo_ref}/${asset_path}/agent_config.yaml" -curl -fL --retry 3 "${config_url}" -o "${config_path}" - -# Keep all eight pipelines in one configuration. Without cloud export, replace -# active cloud destinations with nop while the workshop pipelines continue -# local debug and file validation. -if [[ "${cloud_enabled}" == "false" ]]; then - sed -i.bak \ - -e 's/exporters: \[debug, file\/traces, otlp_http\]/exporters: [debug, file\/traces]/' \ - -e 's/exporters: \[signalfx\]/exporters: [nop]/' \ - -e 's/exporters: \[otlp_http\/entities\]/exporters: [nop]/' \ - -e 's/extensions: \[headers_setter, health_check, http_forwarder, http_forwarder\/opamp_splunk_o11y, opamp\/splunk_o11y, zpages\]/extensions: [health_check, zpages]/' \ - "${config_path}" - rm -f "${config_path}.bak" -fi - -# Remove the obsolete overlay if setup is rerun in an earlier workshop folder. -rm -f "${agent_dir}/agent_config.local.yaml" - -{ - printf 'export REALM=%q\n' "${realm}" - printf 'export ACCESS_TOKEN=%q\n' "${splunk_access_token}" - printf 'export SPLUNK_ACCESS_TOKEN=%q\n' "${splunk_access_token}" - printf 'export SPLUNK_API_URL=%q\n' "${splunk_api_url}" - printf 'export SPLUNK_INGEST_URL=%q\n' "${splunk_ingest_url}" - printf 'export SPLUNK_HEC_TOKEN=%q\n' "${splunk_hec_token}" - printf 'export SPLUNK_HEC_URL=%q\n' "${splunk_hec_url}" - printf 'export SPLUNK_LISTEN_INTERFACE=%q\n' "${splunk_listen_interface}" - printf 'export SPLUNK_MEMORY_LIMIT_MIB=%q\n' "${splunk_memory_limit_mib}" - printf 'export CONF2026_CLOUD_ENABLED=%q\n' "${cloud_enabled}" -} > "${environment_path}" -chmod 600 "${environment_path}" -unset splunk_access_token splunk_hec_token -trap - ERR - -echo -echo "Workshop setup complete." -echo "Collector: ${installed_version}" -echo "Cloud export: ${cloud_enabled}" -echo -echo "Start the Collector:" -echo " cd 1-agent" -echo " source ../workshop-env.sh" -echo " ../otelcol --config=agent_config.yaml"