Skip to content

Fix fixture discovery and assertions for serial-ID metadata tests - #7821

Merged
hhan2024 merged 0 commit into
ffe-serial-id-metadata-system-testfrom
codex/ffe-serial-id-metadata-corrections
Sep 25, 2026
Merged

hhan2024 merged 0 commit into
ffe-serial-id-metadata-system-testfrom
codex/ffe-serial-id-metadata-corrections

Conversation

@leoromanovsky

Copy link
Copy Markdown
Contributor

Motivation

The serial-ID metadata tests in #7820 introduce a UFC configuration file that the existing dynamic-evaluation loader mistakes for test cases. The new suite also runs for Rust without FFE support, and its absent-ID assertion accepts failed evaluations.

Changes

  • Exclude serial-id-metadata-flags.json from evaluation-case discovery while preserving its UFC configuration format.
  • Mark the new suite missing_feature in the Rust manifest.
  • Require the configured control value, no error code, and a non-error reason before accepting absent serial-ID metadata.
  • Apply Ruff formatting and the docstring lint correction.

Decisions

This draft targets ffe-serial-id-metadata-system-test, the branch for #7820, so the corrections can be reviewed and merged into that PR. It follows the existing fixture-exclusion convention and neighboring tests' evaluation-success checks. Multiple-split coverage remains a separate follow-up.

Validation

  • ./format.sh passed, including mypy, Ruff, YAML/manifest checks, shellcheck, and Node.js linters.
  • ./run.sh TEST_THE_TEST tests/test_the_test/test_manifest.py: 32 passed.
  • Focused probes preserved all 23 existing evaluation-case fixtures and confirmed all five new tests are disabled for all nine languages.
  • Executed the absent-ID test body with SDK/RC calls stubbed: reproduced the original false pass, then verified two valid responses pass and seven invalid responses fail after the correction.
  • Docker-backed SDK parametric tests were not run.

@github-actions

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

manifests/rust.yml                                                      @DataDog/system-tests-reviewers
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_ffe/test_serial_id_metadata.py                    @DataDog/feature-flagging-and-experimentation-sdk

@leoromanovsky
leoromanovsky marked this pull request as ready for review September 25, 2026 01:02
@leoromanovsky
leoromanovsky requested review from a team as code owners September 25, 2026 01:02
@leoromanovsky
leoromanovsky requested review from btthomas, hhan2024 and sameerank and removed request for a team, btthomas and sameerank September 25, 2026 01:02
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-25T01:04:39.308950Z cee928a Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@hhan2024 hhan2024 closed this Sep 25, 2026
@hhan2024
hhan2024 force-pushed the codex/ffe-serial-id-metadata-corrections branch from 55d147d to cee928a Compare September 25, 2026 20:36
@hhan2024
hhan2024 merged commit cee928a into ffe-serial-id-metadata-system-test Sep 25, 2026
435 checks passed
@hhan2024
hhan2024 deleted the codex/ffe-serial-id-metadata-corrections branch September 25, 2026 20:36
@datadog-official

datadog-official Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Pipelines  Tests

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 4 Pipeline jobs failed

Testing the test | lint / lint — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

DataDog/system-tests | setup

View more details · View in GitLab

DataDog/system-tests | ssi_tests

View more details · View in GitLab

View all 4 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 55d147d | Docs | View more details | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants