Prepare the mocked backend to receive agentless FFE traffic by cbeauchesne · Pull Request #8186 · DataDog/system-tests

cbeauchesne · 2026-10-26T17:56:25Z

Motivation

First step to remove the datadog_sidecar and datadog_direct interfaces: FFE agentless traffic will go to the mocked backend, and the sender will be identified by its API key.

Changes

interfaces.backend_v2 methods accept with_api_key, to only get requests sent with this key
serverless-init uses its own fake API key
the mocked backend serves either HTTP (default) or HTTPS: MockedBackendContainer(tls=True) serves the mock intake hostnames (*.mock-intake.invalid) on 443, with a committed certificate signed by the mitmproxy CA

No behavior change yet: no scenario uses tls=True.

🤖 Generated with Claude Code

…erverless-init Each sender uses its own API key, so with_api_key tells where a request comes from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ainer Agentless senders only speak HTTPS to the hostnames derived from DD_SITE. The mocked backend container gets network aliases for them, and serves them over TLS with a certificate signed by the mitmproxy CA, sharing its routes and records with the HTTP port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-26T17:56:48Z

CODEOWNERS have been resolved as:

utils/mocked_backend/tls/README.md                                      @DataDog/system-tests-core
utils/mocked_backend/tls/mocked-backend.pem                             @DataDog/system-tests-core
tests/test_the_test/test_mock_backend_v2.py                             @DataDog/system-tests-reviewers
utils/_context/containers.py                                            @DataDog/system-tests-core
utils/interfaces/_backend_v2.py                                         @DataDog/system-tests-core
utils/mocked_backend/__main__.py                                        @DataDog/system-tests-core
utils/mocked_backend/_backend.py                                        @DataDog/system-tests-core
utils/mocked_backend/_backend_ffe.py                                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/mocked_backend/_client.py                                         @DataDog/system-tests-core
utils/mocked_backend/_request_handler.py                                @DataDog/system-tests-core

datadog-datadog-us1-prod · 2026-10-26T18:03:47Z

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 22 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.1-zts 1 — ❌ 2 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌

tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language.test_expression_language_access_variables[apache-mod-7.1-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7f60fa62b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7f60fa62b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language object at 0x7f60f073c650&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language object at 0x7f60f073c650&gt;

    def test_expression_language_access_variables(self):
&gt;       self._assert(expected_response=200)

...

❌

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM.test_log_method_snapshot[apache-mod-7.1-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7ff24ce2b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7ff24ce2b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7ff2440bd400&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7ff2440bd400&gt;

    def test_log_method_snapshot(self):
&gt;       self._assert()

...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.2-zts 1 — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions.test_method_parameter_capture_expressions[apache-mod-7.2-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7fcc6882b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7fcc6882b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7fcc59f75670&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7fcc59f75670&gt;

    def test_method_parameter_capture_expressions(self):
&gt;       self._assert()

...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.3-zts 1 — ❌ 2 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌

tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language.test_expression_language_access_variables[apache-mod-7.3-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7fb45482b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7fb45482b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language object at 0x7fb445eba5d0&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language object at 0x7fb445eba5d0&gt;

    def test_expression_language_access_variables(self):
&gt;       self._assert(expected_response=200)

...

❌

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM.test_log_method_snapshot[apache-mod-7.3-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7fdb4f42b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7fdb4f42b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7fdb449cca40&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7fdb449cca40&gt;

    def test_log_method_snapshot(self):
&gt;       self._assert()

...

View all 22 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 0a12ffb | Docs | View more details | Give us feedback!}

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

Failed TLS initialization leaves the HTTP listener and background thread running, potentially blocking subsequent startup attempts.

1 open issue

Clean up HTTP server when TLS initialization fails · New

What changed in this PR

Prepares the mocked backend to receive agentless feature-flag evaluation traffic and identify senders by API key, without redirecting traffic yet.

Changes:

Adds optional API-key filtering to backend interface methods.
Adds HTTPS support for mock intake hostnames using the mitmproxy CA.
Gives serverless-init a distinct fake API key and adds filtering and TLS tests.

File	Description
`utils/mocked_backend/_tls.py`	Adds certificate generation and a shared-backend HTTPS listener.
`utils/mocked_backend/_backend.py`	Integrates optional TLS startup and shutdown.
`utils/mocked_backend/__main__.py`	Exposes TLS configuration through CLI options.
`utils/interfaces/_backend_v2.py`	Adds API-key filtering to retrieval methods.
`utils/_context/containers.py`	Configures intake aliases, HTTPS, and a distinct serverless-init key.
`tests/test_the_test/test_mock_backend_v2.py`	Tests API-key filtering, HTTPS recording, and hostname validation.

_{Review effort: Balanced}

Instead of generating one at startup. A test checks it covers the container hostnames, and is not about to expire. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

No scenario needs both: the container gets a tls option (off by default), and scenarios will turn it on when they move their HTTPS senders to the mocked backend. The TLS listener is gone; the certificate also covers the container name and localhost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne · 2026-10-27T00:28:45Z

@codex review

chatgpt-codex-connector · 2026-10-27T00:28:58Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-27T14:02:32.774787Z	`d312c79`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot

🟡 Changes recommended

Unresolved TLS handshake blocking, bridge-network alias incompatibility, and HTTP-only URL helpers prevent reliable use of the new HTTPS mode.

3 open issues

Use user-defined network for TLS container aliases · New
Disable TLS handshake on accept thread · New
Propagate HTTPS scheme to backend URLs · New

1 resolved since last review

Clean up HTTP server when TLS initialization fails

_{Review effort: Balanced}

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An idle connection no longer blocks the other requests, nor close(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…2-with-api-key

cbeauchesne · 2026-10-27T09:54:47Z

@codex review

Copilot

🟡 Changes recommended

The committed certificate does not cover host.docker.internal, so verified HTTPS connections to the server’s advertised host-based URLs fail.

1 open issue

Add host.docker.internal to certificate SANs · New

3 resolved since last review

_{Review effort: Balanced}

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d312c796bb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

cbeauchesne and others added 2 commits October 26, 2026 18:48

Add with_api_key to backend_v2 methods, and a dedicated API key for s…

a1c6613

…erverless-init Each sender uses its own API key, so with_api_key tells where a request comes from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne requested a balanced review from Copilot October 26, 2026 18:04

Copilot started reviewing on behalf of cbeauchesne October 26, 2026 18:05

View session

Rename MOCK_INTAKE_SITE to MOCK_INVALID_INTAKE_SITE

d5bdde5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI reviewed Oct 26, 2026

View reviewed changes

Comment thread utils/mocked_backend/_backend.py Outdated

cbeauchesne and others added 4 commits October 26, 2026 19:13

Use a committed certificate for the mocked backend HTTPS listener

b5d08f8

Instead of generating one at startup. A test checks it covers the container hostnames, and is not about to expire. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Rename MOCK_INVALID_INTAKE_SITE back to MOCK_INTAKE_SITE

34da82a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Release the mocked backend port if its certificate cannot be loaded

4c3833b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne requested a balanced review from Copilot October 27, 2026 00:28

Copilot started reviewing on behalf of cbeauchesne October 27, 2026 00:28

View session

Copilot AI reviewed Oct 27, 2026

View reviewed changes

Comment thread utils/_context/containers.py

Comment thread utils/mocked_backend/_backend.py Outdated

Comment thread utils/mocked_backend/_backend.py

cbeauchesne and others added 4 commits October 27, 2026 10:32

Note that network aliases are not supported on GitLab's bridge network

bbd2a08

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Run the mocked backend TLS handshake in the request thread

12bb1b3

An idle connection no longer blocks the other requests, nor close(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Advertise the mocked backend URLs with its scheme

5984109

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into cbeauchesne/backend-v…

c83c4dd

…2-with-api-key

cbeauchesne requested a balanced review from Copilot October 27, 2026 09:54

Copilot started reviewing on behalf of cbeauchesne October 27, 2026 09:54

View session

Copilot AI reviewed Oct 27, 2026

View reviewed changes

Comment thread utils/mocked_backend/tls/mocked-backend.pem

cbeauchesne and others added 2 commits October 27, 2026 11:06

Document that HTTPS is meant for the mocked backend container

22a0121

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge branch 'main' into cbeauchesne/backend-v2-with-api-key

d312c79

cbeauchesne marked this pull request as ready for review October 27, 2026 13:56

cbeauchesne requested review from a team as code owners October 27, 2026 13:56

cbeauchesne requested a review from a team as a code owner October 27, 2026 13:56

nccatoni approved these changes Oct 27, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Oct 27, 2026

View reviewed changes

Comment thread utils/_context/containers.py

cbeauchesne added 2 commits October 27, 2026 18:05

Merge branch 'main' into cbeauchesne/backend-v2-with-api-key

f5b6905

remove dead code

0a12ffb

Navigation Menu

Repository navigation

Prepare the mocked backend to receive agentless FFE traffic - #8186

Prepare the mocked backend to receive agentless FFE traffic#8186
cbeauchesne wants to merge 15 commits into
mainfrom
cbeauchesne/backend-v2-with-api-key

cbeauchesne commented Oct 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

datadog-datadog-us1-prod Bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

cbeauchesne commented Oct 27, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbeauchesne commented Oct 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cbeauchesne commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

github-actions Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-us1-prod Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

cbeauchesne commented Oct 27, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbeauchesne commented Oct 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cbeauchesne commented Oct 26, 2026 •

edited

Loading

github-actions Bot commented Oct 26, 2026 •

edited

Loading

datadog-datadog-us1-prod Bot commented Oct 26, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 27, 2026 •

edited

Loading