Repository navigation
Conversation
The evaluate response can carry an extra_tags object so AI Guard policies can enrich traces without a tracer release per tag (APPSEC-70724). Scalars must land as ai_guard span tags under the key exactly as returned, and structured values in the ai_guard meta struct; the backend owns the ai_guard prefix, so tracers neither add it nor filter by namespace. A malformed payload must be ignored without changing the evaluation. Cassettes are hand-crafted because the backend does not return the field yet; gen_extra_tags_cassettes.py generates them together with the scenario sidecar and is checked by format.sh. Enabled for python flask-poc (dd-trace-py#21145), missing_feature for every other tracer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
|
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 8 Pipeline jobs failed
ℹ️ InfoNo other issues found (see more)❄️ No new flaky tests detected Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 77ac7f5 | Docs | View more details | Give us feedback! |
avara1986
marked this pull request as ready for review
October 22, 2026 09:41
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
florentinl
approved these changes
Oct 22, 2026
A tracer without extra_tags support also ignores a malformed payload, so Test_ExtraTagsFailSafe XPASSes under missing_feature (verified on Node.js) and the easy-win bot would wrongly declare the feature supported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Span metrics decode as float64, so a tracer that stores a boolean extra tag as a metric reports 1.0/0.0, which the type guard rejected. Reuse the redacted-tag check, which accepts exactly 1/0 from metrics, for both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cbeauchesne
approved these changes
Oct 22, 2026
The backend contract for APPSEC-70724 changed: the evaluate response
carries two separate attributes, both lists of {key, value} objects,
since dotted keys such as ai_guard.evaluate_id are not valid JSON:API
member names.
- extra_tags: string values, set as ai_guard span tags under the key
exactly as returned
- extra_metastruct: any value, added to the ai_guard meta_struct under
the key exactly as returned
- the evaluation ID is sent in both, as ai_guard.evaluate_id and
evaluate_id, matching data.id
Regenerate the hand-written cassettes for the list shape, with a stable
UUID as data.id per scenario, and rewrite Test_ExtraTags and
Test_ExtraTagsFailSafe accordingly. The fail-safe tests now also assert
that the well-formed part of the response is still applied, so they no
longer pass vacuously and the easy-win skip entries are dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ull skips The backend now sends the evaluation ID only as the ai_guard.evaluate_id span tag; extra_metastruct is empty for now (ddoghq/dd-source#139566). - EVALUATE_ID mirrors today's response: ai_guard.evaluate_id in extra_tags, equal to data.id, and no extra_metastruct. The test also asserts the ID never reaches the ai_guard meta_struct. - ON_DENY no longer sends evaluate_id in extra_metastruct. - EXTRA_METASTRUCT uses the illustrative backend example (objects, arrays of objects, a scalar). - MALFORMED_ENTRIES adds a boolean extra_tags value and a null extra_metastruct value, and checks that a non-string or empty key is never stringified or kept. Regenerated with utils/scripts/gen_extra_tags_cassettes.py; cassette names are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Jira: APPSEC-70724, AI Guard: Enrich traces with custom context from AI Guard policies.
AI Guard policies can add their own context to traces without a tracer release per tag. The evaluate response carries two optional attributes, both lists of
{key, value}objects (dotted keys such asai_guard.evaluate_idare not valid JSON:API member names, so they cannot be map keys):What the backend sends today (the evaluation ID, as a span tag only;
extra_metastructis empty for now):Illustrative
extra_metastructshape tracers must implement (examples only, not sent today):extra_tags: string values, each set as a tag on theai_guardspan under the key exactly as returned. Non-string values are skipped.extra_metastruct: any non-null value (object, array or scalar), each added to theai_guardmeta_struct under the key exactly as returned.ai_guard.prefix and the namespace. Tracers don't add the prefix and don't filter keys.Implementations:
Changes
Test_ExtraTags:extra_tagsentries land as string tags on the spanextra_metastructentries (objects, arrays, scalars) land in the meta_structai_guard.evaluate_idtag, matchesdata.id, and never reaches the meta_structai_guard.namespace (no prefix added, no filtering)Test_ExtraTagsFailSafe: the action, the span (no error) and the response stay unchanged, while every well-formed part is still applied:extra_tagssent as an object (the first draft's map shape) is skipped,extra_metastructstill landsextra_metastructsent as an object is skipped,extra_tagsstill landsextra_tagsvalue including a boolean,nullextra_metastructvalue) are skipped one by one, and a bad key is never stringified or keptutils/scripts/gen_extra_tags_cassettes.pygenerates them together withtests/ai_guard/extra_tags_scenarios.json, which also records what the tracer must apply for each scenario. It reuses the request hashing ofgen_redaction_cassettes.pyand refuses to overwrite another test's cassette.format.shchecks it for drift, the same way it checks the redaction fixtures.flask-poc: enabled atv4.17.0-dev.missing_feature (APPSEC-70724).Testing
Ran the
AI_GUARDscenario limited toTest_ExtraTagsandTest_ExtraTagsFailSafe:flask-pocwith dd-trace-pymain(4.17.0rc1, fd1721f1, includes the #21252 merge): all 8 new tests pass.express5with the dd-trace-js#10865 branch (c3f32458), forced with-Fsince the manifest still marks itmissing_feature: all 8 new tests pass, and the wholeAI_GUARDscenario (61 tests) passes.mainwith the map contract from #21145 only (ed54859b): all 8 new tests fail, the fail-safe ones included, so none passes vacuously.DataDog/dd-trace-py#21252 is merged, so the Python dev CI runs these tests against the list contract. The Node.js manifest gets flipped once dd-trace-js#10865 is released.
Workflow
system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.system-tests-core(slack)🚀 Once your PR is reviewed and the CI green, you can merge it!
🛟 #apm-shared-testing 🛟
🤖 Generated with Claude Code