[debugger] Enable .NET in-product enablement tests from 3.57.0 by dudikeleti · Pull Request #7876 · DataDog/system-tests

dudikeleti · 2026-10-01T06:21:33Z

Motivation

In-product enablement of Dynamic Instrumentation, and Exception Replay with several APM_TRACING configs, did not work in dd-trace-dotnet (DEBUG-4637). The fix, DataDog/dd-trace-dotnet#9344, merged after 3.56.0 was released, so it ships in 3.57.0.

Changes

manifests/dotnet.yml: enable Test_Debugger_InProduct_Enablement_Dynamic_Instrumentation and test_inproduct_enablement_exception_replay_apm_multiconfig from 3.57.0. Older versions stay marked bug (DEBUG-4637).
.NET weblog: mark DebuggerController.LogProbe as NoInlining. ASP.NET Core MVC calls actions through a compiled expression tree, and the JIT inlines the 12-byte LogProbe into it on the first /debugger/log request. The in-product DI test calls /debugger/log while DI is off, so after remote enablement the tracer ReJITs LogProbe but requests keep running the inlined copy, and the line probe never emits (run 40476449992). Scenarios that enable DI at startup install the probe before the first call. The attribute goes on the existing [Consumes] line, so no line number in the file moves. Inlined methods are a separate DI limitation that will get its own test.

Test_Debugger_InProduct_Enablement_Code_Origin doesn't depend on #9344. It's enabled from 3.56.0 in #8123, so this PR doesn't change it.

Notes:

dd-trace-dotnet master includes #9344, so the dev job in this PR's CI runs these tests with the fix.
test_inproduct_enablement_exception_replay_apm_multiconfig also needs the .NET tracer to send timestamp at the root of the snapshot payload, as the log probes RFC defines it (.NET only sets debugger.snapshot.timestamp today). Without it, the start_time filter in tests/debugger/utils.py can't drop the simple exception snapshot captured by test_inproduct_enablement_exception_replay, and the multiconfig setup matches it while Exception Replay is off. The fix is [Debugger] Add root timestamp to snapshot payloads dd-trace-dotnet#9532. Until it is in the dev build, this test fails on dev. Merge this PR after #9532, and before 3.57.0 is cut, or move the multiconfig gate to the release that ships it. [debugger] Require the root timestamp on debugger snapshot payloads #8197 makes the agent schema require the root timestamp.
[debugger] Test that updating one APM_TRACING config keeps the others applied #7908 makes the same one-line weblog change.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Tests, manifest, weblog are modified -> you'll need a review from system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.
- Framework is modified, or non obvious usage of it -> get a review from system-tests-core (slack)

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

github-actions · 2026-10-01T06:22:04Z

CODEOWNERS have been resolved as:

manifests/dotnet.yml                                                    @DataDog/system-tests-reviewers
utils/build/docker/dotnet/weblog/Controllers/DebuggerController.cs      @DataDog/system-tests-reviewers

datadog-official · 2026-10-01T06:31:25Z

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 3 Pipeline jobs failed

Testing the test | System Tests (dotnet, dev) / End-to-end #2 / poc 2 — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions
❌ tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Exception_Replay.test_inproduct_enablement_exception_replay_apm_multicon... from system_tests_suite
AssertionError: Expected snapshots to not be emitting when exception replay was disabled
assert False
 &#43;  where False = &lt;tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Exception_Replay object at 0x7fc661519550&gt;.er_initial_enabled

self = &lt;tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Exception_Replay object at 0x7fc661519550&gt;

    @slow
    def test_inproduct_enablement_exception_replay_apm_multiconfig(self):
        self.assert_rc_state_not_error()
        self.assert_all_weblog_responses_ok(expected_code=500)
...
Testing the test | System Tests (dotnet, dev) / End-to-end #2 / uds 2 — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions
❌ tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Exception_Replay.test_inproduct_enablement_exception_replay_apm_multicon... from system_tests_suite
AssertionError: Expected snapshots to not be emitting when exception replay was disabled
assert False
 &#43;  where False = &lt;tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Exception_Replay object at 0x7fdf4c5c2510&gt;.er_initial_enabled

self = &lt;tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Exception_Replay object at 0x7fdf4c5c2510&gt;

    @slow
    def test_inproduct_enablement_exception_replay_apm_multiconfig(self):
        self.assert_rc_state_not_error()
        self.assert_all_weblog_responses_ok(expected_code=500)
...
Testing the test | all-jobs-are-green

View more details · View in GitHub Actions

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 314be5e | Docs | View more details | Give us feedback!}

DEBUG-4637 is fixed in dd-trace-dotnet 3.55.0 (DataDog/dd-trace-dotnet#9344). .NET has code origin on by default since 3.45.0, so the code origin test now expects it enabled at startup, as for Node.js.

…ement

Co-authored-by: Cursor <cursoragent@cursor.com>

## Summary of changes - Dynamic Instrumentation (DI) enabled only from remote configuration now actually starts when `DD_DYNAMIC_INSTRUMENTATION_ENABLED` is unset. - An explicit `false` from either the env var or remote configuration keeps DI off. - When several APM_TRACING configs are active, their DI, Exception Replay and Code Origin settings are now merged by priority, like the other settings. - Updating one APM_TRACING config no longer drops the other active configs that did not change. - Tests: one theory replaces the separate DI enabled/disabled unit tests, new merger test for the debugger settings, updated `CombineApmTracingConfiguration` tests. `DebuggerManagerDynamicTests` now waits for DI to finish initializing, not just for the instance to be constructed, and waits for log entries before the final memory snapshot. ## Reason for change - **Remote-only enablement:** `DebuggerManager` created the `DynamicInstrumentation` instance, recorded it as initialized and reported DI as enabled in telemetry. However, `Initialize()` only checked the env var, so the instance never subscribed to `LIVE_DEBUGGING` and never started its uploaders or probe status poller. Probes created in the UI were never applied. The integration test missed this because it waited for "Initializing Dynamic Instrumentation", which the constructor logs even if DI never starts. - **Several configs (org/env-level enablement):** `ApmTracingConfigMerger` had its own copy of the field merge without the three debugger fields, so merging two or more configs returned them as null. DI is off unless a config says `true`, so it turned off. - **Updating one config:** remote configuration only delivers files whose content changed; unchanged files are still listed in `client_configs` but not resent. `DynamicConfigurationManager.CombineApmTracingConfiguration` cleared all active configs on every update, so changing one config (for example, turning Code Origin off for a service) dropped a wildcard config that enabled DI, and DI turned off. ## Implementation details - `Initialize()` now checks `DynamicInstrumentationCanBeEnabled && (DynamicSettings.DynamicInstrumentationEnabled ?? DynamicInstrumentationEnabled)`: - an env `false` sets `CanBeEnabled` to false, so DI stays off; - a remote value overrides the env value when present, so a remote `false` turns DI off even if the env var is `true`; - with no remote value, the env value decides. - `DebuggerManager` already applies the same precedence before calling `Initialize()`. - `ApmTracingConfigMerger` now uses `ApmTracingConfig.MergeLibConfigs` (now `internal`) instead of its own incomplete copy, so one method merges every field. - `CombineApmTracingConfiguration` keeps active configs until they are removed explicitly. `RcmSubscriptionManager` already reports every applied config that is missing from `client_configs` as removed, and only forgets an applied config when it reports it, so the active set stays in sync. - A setting that no active config sets behaves as before: DI turns off, while Exception Replay and Code Origin keep their current state (only an explicit `false` disables them). ## Test coverage - `dotnet test tracer/test/Datadog.Trace.Tests/Datadog.Trace.Tests.csproj -f net10.0 --filter "FullyQualifiedName~Datadog.Trace.Tests.Configuration.DynamicConfigurationManagerTests|FullyQualifiedName~Datadog.Trace.Tests.Configuration.ApmTracingConfigMergerTests|FullyQualifiedName~Datadog.Trace.Tests.Debugger.DynamicInstrumentationTests"` on 40ba240: 124 passed. This includes: - the six rows of `DynamicInstrumentationEnablement_ServicesCalledOnlyWhenEnabled`: env on; remote on with env unset (the remote-only case); both unset; env off; env off with remote on; env on with remote off; - `MergeConfigurations_DebuggerSettings_CorrectPriorityMerging`: a wildcard config with DI, Exception Replay and Code Origin on, plus a service+env config with Exception Replay off, merges to Exception Replay off and DI and Code Origin on; - `CombineApmTracingConfiguration_WhenNewConfigIsProvided_KeepsUnchangedConfigs`: updating one config keeps the other active config. - `DebuggerManagerDynamicTests` (integration) not run locally. ## Other details - System tests: DataDog/system-tests#7876 enables the .NET in-product enablement tests from 3.55.0. DataDog/system-tests#7908 adds a test where only one of several APM_TRACING configs changes; Java, Node.js, Python and Ruby already pass it. - Keeping every active config and merging them on each update matches Java, Node.js and Ruby.

DataDog/dd-trace-dotnet#9344 merged after 3.56.0 was released, so the first release with the fix is 3.57.0. Code origin enablement doesn't depend on #9344 and is enabled from 3.56.0 in #8123, so this PR no longer touches it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chatgpt-codex-connector · 2026-10-27T09:00:50Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-27T09:02:42.189435Z	`36c443f`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…as a bug The start_time filter only read the top-level timestamp of a snapshot payload. .NET only sets it in the snapshot itself, so the filter never applied, and the apm_multiconfig setup matched the simple exception snapshot captured by the previous test while exception replay was off. Fall back to the snapshot timestamp. Test_Debugger_InProduct_Enablement_Dynamic_Instrumentation stays bug (DEBUG-4637): with DataDog/dd-trace-dotnet#9344 in the dev build, remote enablement installs the probe, but it never emits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…from 3.57.0 ASP.NET Core MVC calls actions through a compiled expression tree, and the JIT inlines the 12-byte LogProbe into it on the first /debugger/log request. The in-product DI test calls /debugger/log while DI is off, so after remote enablement the tracer ReJITs LogProbe but requests keep running the inlined copy, and the line probe never emits. Scenarios that enable DI at startup install the probe before the first call. Mark LogProbe NoInlining on its existing attribute line so no line number in the file moves, and declare the DI test bug (DEBUG-4637) only below 3.57.0, the first release with DataDog/dd-trace-dotnet#9344. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

watson · 2026-10-27T10:44:37Z

+def _get_content_timestamp(content: dict) -> int | None:
+    """Return the timestamp of a snapshot payload.
+
+    Some tracers set it at the top level of the payload, .NET only sets it in the snapshot.


If .NET is the only SDK not adding a timestamp at the root of the payload, why don't we align .NET with the other SDKs so we include that timestamp there as well?

I'm not sure if the debugger intake is clever enough to take the timestamp inside the snapshot and use it as the official timestamp for event. Bit if not, it means that the ingested event will get a timestamp based on when it is received at the intake instead of when it is generated by the SDK.

Agreed. The log probes RFC defines timestamp at the root of the intake payload as well as in debugger.snapshot. .NET only writes the latter. Of the libraries that run these in-product Exception Replay tests (Java, Python, .NET), only .NET is missing it, so I removed the fallback and will add the root timestamp in dd-trace-dotnet.

(Node.js also omits it (send.js), but it doesn’t run these tests yet (missing_feature), and will need the same change when it does.)

I'll ensure that the Node.js SDK is updated as well.

DataDog/dd-trace-js#10939

The log probes RFC defines the intake payload with a root timestamp as well as debugger.snapshot.timestamp. Of the libraries that run the in-product Exception Replay tests, only .NET omits the root one, so fix the tracer instead of working around it here. Until the .NET build sends it, test_inproduct_enablement_exception_replay_apm_multiconfig matches the snapshot left over from the previous test on .NET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

## Summary of changes - Dynamic Instrumentation and Exception Replay snapshot payloads now send `timestamp` at the root, in epoch milliseconds, with the same value as `debugger.snapshot.timestamp`. - `DebuggerSnapshotCreator` reads the clock once per snapshot and writes that value both in `debugger.snapshot` and at the root (`AddGeneralInfo`). - Debugger integration-test approvals now include the scrubbed root `timestamp`, and a new unit test checks that both fields are present and equal. ## Reason for change - The "Log probes configuration" RFC defines the snapshot payload with `timestamp` both at the root and in `debugger.snapshot`, in epoch milliseconds with the same value. Java, Python, Ruby and PHP send both. .NET only sent `debugger.snapshot.timestamp`. - Snapshots go through the logs intake as log entries, and the logs pipeline takes the event date from the root attribute. Without it, the snapshot is likely dated when the intake receives it rather than when it was taken. - system-tests drops snapshots sent before a test started (the `start_time` filter in `tests/debugger/utils.py`) using the root `timestamp`. On .NET that filter never applies, so `Test_Debugger_InProduct_Enablement_Exception_Replay::test_inproduct_enablement_exception_replay_apm_multiconfig` matches a `simple exception` snapshot left over from the previous test. In [system-tests run 40476449992](https://github.com/DataDog/system-tests/actions/runs/40476449992) (poc `dev` job), the payloads in `logs_debugger_inproduct_enablement/interfaces/agent/*__api_v2_debugger.json` have the top-level keys `dd.span_id, dd.trace_id, ddsource, debugger, debugger.product, debugger.type, logger, message, process_tags, service` and no `timestamp`. DataDog/system-tests#7876 depends on this change. ## Implementation details - Every DI snapshot (method and line probes, with and without `captureSnapshot`) and every Exception Replay snapshot is finalized through `FinalizeSnapshot` (`EndSnapshot`, then `AddGeneralInfo`). `ExceptionReplaySnapshotCreator.EndSnapshot` calls the base method. So one change covers all of them. - The value is cached on the creator (`GetTimestampMs()`), so the clock is read once and the two fields always match. `debugger.snapshot.timestamp` keeps its value, its units and the point where it's taken (`EndSnapshot`, `DateTimeOffset.Now.ToUnixTimeMilliseconds()`). - No new allocations: one extra `long` property per payload. ## Test coverage - New unit test `DebuggerSnapshotCreatorTests.Timestamp_IsWrittenAtRootAndInSnapshotWithSameValue` covers Dynamic Instrumentation with and without a full snapshot, and Exception Replay. It checks that both fields are integers, are equal, and fall within the time of the call. - Snapshot approvals updated: 101 files under `tracer/test/Datadog.Trace.Debugger.IntegrationTests/Approvals/snapshots` (209 payloads). Each payload only gains `"timestamp": "ScrubbedValue"` at the root. The `Approver` already scrubs `timestamp` at every level, so no scrubber change was needed. - Ran locally on Windows x64: - `dotnet test tracer/test/Datadog.Trace.Tests -c Release -f net10.0 --filter "FullyQualifiedName~Datadog.Trace.Tests.Debugger"`: 880 passed, 2 skipped. - Same filter with `-f net48`: 844 passed, 29 skipped. - `tracer\build.cmd BuildAndRunDebuggerIntegrationTests --framework net8.0 --TargetPlatform x64 --DebugType portable --Optimize true --filter "FullyQualifiedName~Datadog.Trace.Debugger.IntegrationTests.ProbesTests"`: 129 passed, 46 skipped, 0 failed. This exercised 93 of the 95 probe test types whose approvals changed. `FullSnapshotWithConditionError` and `NestedRefStructLockLocalTest` didn't run on net8.0. - Not run locally: other target frameworks, unoptimized builds, Linux, and the Exception Replay integration tests (`AspNetCore5ExceptionReplay.TestExceptionReplay` is skipped on master). ## Other details - About 50 older snapshot approvals were left as they are. They already lack `debugger.type` and `debugger.product`, which every current payload has, because their tests are skipped (for example `AspNetCore5ExceptionReplay.TestExceptionReplay`), expect no snapshots, or no longer exist. They were stale before this change. - Release: `master` is on 3.57.0, which isn't released yet. I expect this to make 3.57.0 if it merges before the release is cut. DataDog/system-tests#7876 enables the multiconfig Exception Replay test from 3.57.0 and needs this fix in the `dev` build first. - Node.js has the same gap (dd-trace-js `send.js`). It isn't part of this PR.

…ement

dudikeleti changed the title ~~[dotnet@dudik/fix-di-remote-enablement] Enable debugger in-product enablement tests from 3.55.0~~ [dotnet@dudik/fix-di-remote-enablement] [debugger] Enable .NET in-product enablement tests from 3.55.0 Oct 1, 2026

[dotnet] Enable debugger in-product enablement tests from 3.55.0

27c9e80

DEBUG-4637 is fixed in dd-trace-dotnet 3.55.0 (DataDog/dd-trace-dotnet#9344). .NET has code origin on by default since 3.45.0, so the code origin test now expects it enabled at startup, as for Node.js.

dudikeleti force-pushed the dudik/dotnet-enable-debugger-inproduct-enablement branch from 7c401d5 to 27c9e80 Compare October 1, 2026 07:42

dudikeleti mentioned this pull request Oct 1, 2026

[Debugger] DEBUG-4637 Fix DI remote enablement DataDog/dd-trace-dotnet#9344

Merged

Merge branch 'main' into dudik/dotnet-enable-debugger-inproduct-enabl…

4fe2966

…ement

dudikeleti changed the title ~~[dotnet@dudik/fix-di-remote-enablement] [debugger] Enable .NET in-product enablement tests from 3.55.0~~ [debugger] Enable .NET in-product enablement tests from 3.55.0 Oct 14, 2026

Bump .NET in-product enablement fix version to 3.56.0

043ef9e

Co-authored-by: Cursor <cursoragent@cursor.com>

dudikeleti changed the title ~~[debugger] Enable .NET in-product enablement tests from 3.55.0~~ [dotnet@dudik/fix-di-remote-enablement] [debugger] Enable .NET in-product enablement tests from 3.56.0 Oct 14, 2026

dudikeleti mentioned this pull request Oct 21, 2026

[debugger] Enable .NET in-product Code Origin enablement test from 3.56.0 #8123

Merged

dudikeleti changed the title ~~[dotnet@dudik/fix-di-remote-enablement] [debugger] Enable .NET in-product enablement tests from 3.56.0~~ [debugger] Enable .NET in-product enablement tests from 3.56.0 Oct 21, 2026

dudikeleti changed the title ~~[debugger] Enable .NET in-product enablement tests from 3.56.0~~ [debugger] Enable .NET in-product enablement tests from 3.57.0 Oct 27, 2026

dudikeleti marked this pull request as ready for review October 27, 2026 09:00

dudikeleti requested a review from a team as a code owner October 27, 2026 09:00

dudikeleti requested a review from a team as a code owner October 27, 2026 09:26

dudikeleti changed the title ~~[debugger] Enable .NET in-product enablement tests from 3.57.0~~ [debugger] Enable .NET in-product Exception Replay multiconfig test from 3.57.0 Oct 27, 2026

dudikeleti changed the title ~~[debugger] Enable .NET in-product Exception Replay multiconfig test from 3.57.0~~ [debugger] Enable .NET in-product enablement tests from 3.57.0 Oct 27, 2026

dudikeleti mentioned this pull request Oct 27, 2026

[debugger] Test that updating one APM_TRACING config keeps the others applied #7908

Open

watson reviewed Oct 27, 2026

View reviewed changes

This was referenced Oct 27, 2026

[Debugger] Add root timestamp to snapshot payloads DataDog/dd-trace-dotnet#9532

Merged

[debugger] Require the root timestamp on debugger snapshot payloads #8197

Draft

Merge branch 'main' into dudik/dotnet-enable-debugger-inproduct-enabl…

314be5e

…ement

Navigation Menu

Repository navigation

[debugger] Enable .NET in-product enablement tests from 3.57.0 - #7876

[debugger] Enable .NET in-product enablement tests from 3.57.0#7876
dudikeleti wants to merge 8 commits into
mainfrom
dudik/dotnet-enable-debugger-inproduct-enablement

dudikeleti commented Oct 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 1, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Oct 1, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026 •

edited

Loading

Uh oh!

watson Oct 27, 2026

Uh oh!

dudikeleti Oct 27, 2026

Uh oh!

watson Oct 27, 2026

Uh oh!

watson Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dudikeleti commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Uh oh!

github-actions Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Oct 1, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

watson Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

dudikeleti Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

watson Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

watson Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dudikeleti commented Oct 1, 2026 •

edited

Loading

github-actions Bot commented Oct 1, 2026 •

edited

Loading

datadog-official Bot commented Oct 1, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

chatgpt-codex-connector Bot commented Oct 27, 2026 •

edited

Loading