[debugger] Test that updating one APM_TRACING config keeps the others applied by dudikeleti · Pull Request #7908 · DataDog/system-tests

dudikeleti · 2026-10-01T07:40:26Z

Motivation

Remote config lists every active config in client_configs, but a library only processes the files whose hash changed. With several APM_TRACING configs active, updating one of them must keep the others applied.

dd-trace-dotnet dropped the unchanged configs in that case, so a product enabled by another config turned off (fixed in DataDog/dd-trace-dotnet#9344). From the code of the other libraries:

Java, Node.js and Ruby keep every active config and merge them all on each update.
Python merges only the configs received in that poll, but its handlers ignore a missing setting, so dynamic instrumentation stays on there too.

Changes

New Test_Debugger_InProduct_Enablement_Multiconfig_Partial_Update:
1. A wildcard config enables dynamic instrumentation and a weblog/system-tests config sets code_origin_enabled: true. Probes must emit.
2. Only the service config changes (code_origin_enabled: false). The wildcard config is resent unchanged, at the same path. A new probe must still emit.
It uses rc.tracer_rc_state so the unchanged config keeps its path and hash. The existing APM_TRACING helpers create new paths on every call.
The log probe template moves to module level so both classes share it.
Manifests follow the weblogs of the in-product DI test. Each library is gated on its first release with APM_TRACING multiconfig support: Java 1.54.0, Node.js 5.83.0, PHP 1.20.0, Python 3.15.0, Ruby 2.42.0. Go is missing_feature. .NET is bug (DEBUG-4637) below 3.57.0, the first release with the fix ([Debugger] DEBUG-4637 Fix DI remote enablement dd-trace-dotnet#9344 merged after 3.56.0 was released).
.NET weblog: mark DebuggerController.LogProbe as NoInlining, the same one-line change as [debugger] Enable .NET in-product enablement tests from 3.57.0 #7876. ASP.NET Core MVC calls actions through a compiled expression tree, and the JIT inlines the 12-byte LogProbe into it on the first /debugger/log request. The in-product DI test that runs before this one calls /debugger/log while DI is off, so after remote enablement the tracer ReJITs LogProbe but requests keep running the inlined copy, and the line probe never emits (run 40476449992). Scenarios that enable DI at startup install the probe before the first call. The attribute goes on the existing [Consumes] line, so no line number in the file moves. Inlined methods are a separate DI limitation that will get its own test.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Tests, manifest, weblog are modified -> you'll need a review from system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.
- Framework is modified, or non obvious usage of it -> get a review from system-tests-core (slack)

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

github-actions · 2026-10-01T07:40:52Z

CODEOWNERS have been resolved as:

manifests/dotnet.yml                                                    @DataDog/system-tests-reviewers
manifests/golang.yml                                                    @DataDog/system-tests-reviewers
manifests/java.yml                                                      @DataDog/system-tests-reviewers
manifests/nodejs.yml                                                    @DataDog/system-tests-reviewers
manifests/php.yml                                                       @DataDog/system-tests-reviewers
manifests/python.yml                                                    @DataDog/system-tests-reviewers
manifests/ruby.yml                                                      @DataDog/system-tests-reviewers
tests/debugger/test_debugger_inproduct_enablement.py                    @DataDog/debugger
tests/debugger/utils.py                                                 @DataDog/debugger
utils/build/docker/dotnet/weblog/Controllers/DebuggerController.cs      @DataDog/system-tests-reviewers

… applied When only one APM_TRACING config changes, the unchanged configs are still listed in client_configs and must stay applied. The new test enables dynamic instrumentation from a wildcard config, then changes a single setting of a service config, and expects probes to keep emitting. It is enabled for all libraries on purpose, to see which ones drop the unchanged config.

datadog-datadog-prod-us1-2 · 2026-10-01T07:53:32Z

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 9 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.2-zts 1 — ❌ 2 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌

tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language.test_expression_language_access_variables[apache-mod-7.2-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7f8b9222b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7f8b9222b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language object at 0x7f8b7b986870&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language object at 0x7f8b7b986870&gt;

    def test_expression_language_access_variables(self):
&gt;       self._assert(expected_response=200)

...

❌

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM.test_log_method_snapshot[apache-mod-7.2-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7ffa4062b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7ffa4062b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7ffa3877c590&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7ffa3877c590&gt;

    def test_log_method_snapshot(self):
&gt;       self._assert()

...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-8.0-zts 1 — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM.test_log_method_snapshot[apache-mod-8.0-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7fdecc42b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7fdecc42b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7fdec474cb30&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7fdec474cb30&gt;

    def test_log_method_snapshot(self):
&gt;       self._assert()

...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-8.1-zts 1 — ❌ 2 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌

tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language.test_expression_language_access_variables[apache-mod-8.1-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7f18d1c2b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7f18d1c2b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language object at 0x7f18bb143b90&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_expression_language.Test_Debugger_Expression_Language object at 0x7f18bb143b90&gt;

    def test_expression_language_access_variables(self):
&gt;       self._assert(expected_response=200)

...

❌

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM.test_log_method_snapshot[apache-mod-8.1-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status&#39; = &lt;built-in method join of str object at 0x7f50af02b720&gt;([&#39;Probes did not reach INSTALLED status&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7f50af02b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status&#39;] = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7f50a46a78f0&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7f50a46a78f0&gt;

    def test_log_method_snapshot(self):
&gt;       self._assert()

...

View all 9 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 98153c0 | Docs | View more details | Give us feedback!}

Follow the weblogs of the in-product DI test and gate each library on the first release with APM_TRACING multiconfig support. .NET is a known bug below 3.55.0 (DEBUG-4637).

…-update

Co-authored-by: Cursor <cursoragent@cursor.com>

## Summary of changes - Dynamic Instrumentation (DI) enabled only from remote configuration now actually starts when `DD_DYNAMIC_INSTRUMENTATION_ENABLED` is unset. - An explicit `false` from either the env var or remote configuration keeps DI off. - When several APM_TRACING configs are active, their DI, Exception Replay and Code Origin settings are now merged by priority, like the other settings. - Updating one APM_TRACING config no longer drops the other active configs that did not change. - Tests: one theory replaces the separate DI enabled/disabled unit tests, new merger test for the debugger settings, updated `CombineApmTracingConfiguration` tests. `DebuggerManagerDynamicTests` now waits for DI to finish initializing, not just for the instance to be constructed, and waits for log entries before the final memory snapshot. ## Reason for change - **Remote-only enablement:** `DebuggerManager` created the `DynamicInstrumentation` instance, recorded it as initialized and reported DI as enabled in telemetry. However, `Initialize()` only checked the env var, so the instance never subscribed to `LIVE_DEBUGGING` and never started its uploaders or probe status poller. Probes created in the UI were never applied. The integration test missed this because it waited for "Initializing Dynamic Instrumentation", which the constructor logs even if DI never starts. - **Several configs (org/env-level enablement):** `ApmTracingConfigMerger` had its own copy of the field merge without the three debugger fields, so merging two or more configs returned them as null. DI is off unless a config says `true`, so it turned off. - **Updating one config:** remote configuration only delivers files whose content changed; unchanged files are still listed in `client_configs` but not resent. `DynamicConfigurationManager.CombineApmTracingConfiguration` cleared all active configs on every update, so changing one config (for example, turning Code Origin off for a service) dropped a wildcard config that enabled DI, and DI turned off. ## Implementation details - `Initialize()` now checks `DynamicInstrumentationCanBeEnabled && (DynamicSettings.DynamicInstrumentationEnabled ?? DynamicInstrumentationEnabled)`: - an env `false` sets `CanBeEnabled` to false, so DI stays off; - a remote value overrides the env value when present, so a remote `false` turns DI off even if the env var is `true`; - with no remote value, the env value decides. - `DebuggerManager` already applies the same precedence before calling `Initialize()`. - `ApmTracingConfigMerger` now uses `ApmTracingConfig.MergeLibConfigs` (now `internal`) instead of its own incomplete copy, so one method merges every field. - `CombineApmTracingConfiguration` keeps active configs until they are removed explicitly. `RcmSubscriptionManager` already reports every applied config that is missing from `client_configs` as removed, and only forgets an applied config when it reports it, so the active set stays in sync. - A setting that no active config sets behaves as before: DI turns off, while Exception Replay and Code Origin keep their current state (only an explicit `false` disables them). ## Test coverage - `dotnet test tracer/test/Datadog.Trace.Tests/Datadog.Trace.Tests.csproj -f net10.0 --filter "FullyQualifiedName~Datadog.Trace.Tests.Configuration.DynamicConfigurationManagerTests|FullyQualifiedName~Datadog.Trace.Tests.Configuration.ApmTracingConfigMergerTests|FullyQualifiedName~Datadog.Trace.Tests.Debugger.DynamicInstrumentationTests"` on 40ba240: 124 passed. This includes: - the six rows of `DynamicInstrumentationEnablement_ServicesCalledOnlyWhenEnabled`: env on; remote on with env unset (the remote-only case); both unset; env off; env off with remote on; env on with remote off; - `MergeConfigurations_DebuggerSettings_CorrectPriorityMerging`: a wildcard config with DI, Exception Replay and Code Origin on, plus a service+env config with Exception Replay off, merges to Exception Replay off and DI and Code Origin on; - `CombineApmTracingConfiguration_WhenNewConfigIsProvided_KeepsUnchangedConfigs`: updating one config keeps the other active config. - `DebuggerManagerDynamicTests` (integration) not run locally. ## Other details - System tests: DataDog/system-tests#7876 enables the .NET in-product enablement tests from 3.55.0. DataDog/system-tests#7908 adds a test where only one of several APM_TRACING configs changes; Java, Node.js, Python and Ruby already pass it. - Keeping every active config and merging them on each update matches Java, Node.js and Ruby.

…-update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

DataDog/dd-trace-dotnet#9344 merged after 3.56.0 was released, so the first release with the fix is 3.57.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chatgpt-codex-connector · 2026-10-27T09:01:15Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-27T09:07:39.367148Z	`9d3a17f`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d3a17f193

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ASP.NET Core MVC calls actions through a compiled expression tree, and the JIT inlines the 12-byte LogProbe into it on the first /debugger/log request. Earlier in-product enablement tests call /debugger/log while DI is off, so after remote enablement the tracer ReJITs LogProbe but requests keep running the inlined copy, and the line probe never emits. Mark LogProbe NoInlining on its existing attribute line so no line number in the file moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… test Address review feedback on the new partial update test: - The PHP /debugger/init request now lives in BaseDebuggerTest.resolve_php_probes, used by send_rc_probes, send_rc_apm_tracing_and_probes and the new test, so the test has no library branch. - wait_for_all_probes also returns when a probe reports ERROR, so the test now collects the probe diagnostics and requires both probes to have reached EMITTING. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne · 2026-10-27T14:03:52Z

rebasing to fix the java issue

…-update

dudikeleti force-pushed the dudik/debugger-inproduct-multiconfig-partial-update branch from dcce257 to eaefea2 Compare October 1, 2026 07:42

[debugger] Declare the multiconfig partial update test per library

28a3f79

Follow the weblogs of the in-product DI test and gate each library on the first release with APM_TRACING multiconfig support. .NET is a known bug below 3.55.0 (DEBUG-4637).

dudikeleti changed the title ~~[debugger] Test that updating one APM_TRACING config keeps the others applied~~ [dotnet@dudik/fix-di-remote-enablement] [debugger] Test that updating one APM_TRACING config keeps the others applied Oct 1, 2026

dudikeleti mentioned this pull request Oct 1, 2026

[Debugger] DEBUG-4637 Fix DI remote enablement DataDog/dd-trace-dotnet#9344

Merged

Merge branch 'main' into dudik/debugger-inproduct-multiconfig-partial…

6ff46f2

…-update

dudikeleti changed the title ~~[dotnet@dudik/fix-di-remote-enablement] [debugger] Test that updating one APM_TRACING config keeps the others applied~~ [debugger] Test that updating one APM_TRACING config keeps the others applied Oct 14, 2026

Bump .NET multiconfig partial update fix version to 3.56.0

f650173

Co-authored-by: Cursor <cursoragent@cursor.com>

dudikeleti and others added 2 commits October 27, 2026 09:49

Merge branch 'main' into dudik/debugger-inproduct-multiconfig-partial…

fb27f24

…-update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Gate .NET multiconfig partial update test on 3.57.0

9d3a17f

DataDog/dd-trace-dotnet#9344 merged after 3.56.0 was released, so the first release with the fix is 3.57.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

dudikeleti marked this pull request as ready for review October 27, 2026 09:00

dudikeleti requested review from a team as code owners October 27, 2026 09:00

chatgpt-codex-connector Bot reviewed Oct 27, 2026

View reviewed changes

Comment thread tests/debugger/test_debugger_inproduct_enablement.py Outdated

Comment thread tests/debugger/test_debugger_inproduct_enablement.py

Comment thread tests/debugger/test_debugger_inproduct_enablement.py

dudikeleti mentioned this pull request Oct 27, 2026

[debugger] Enable .NET in-product enablement tests from 3.57.0 #7876

Open

watson approved these changes Oct 27, 2026

View reviewed changes

dudikeleti enabled auto-merge (squash) October 27, 2026 10:55

Merge branch 'main' into dudik/debugger-inproduct-multiconfig-partial…

98153c0

…-update

Navigation Menu

Repository navigation

[debugger] Test that updating one APM_TRACING config keeps the others applied - #7908

[debugger] Test that updating one APM_TRACING config keeps the others applied#7908
dudikeleti wants to merge 9 commits into
mainfrom
dudik/debugger-inproduct-multiconfig-partial-update

dudikeleti commented Oct 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 1, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Oct 1, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbeauchesne commented Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dudikeleti commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Uh oh!

github-actions Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Oct 1, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbeauchesne commented Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dudikeleti commented Oct 1, 2026 •

edited

Loading

github-actions Bot commented Oct 1, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Oct 1, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

chatgpt-codex-connector Bot commented Oct 27, 2026 •

edited

Loading