Repository navigation
test(ffe): consume canonical fixture snapshot - #8139
Open
leoromanovsky wants to merge 10 commits into
Open
leoromanovsky wants to merge 10 commits into
leoromanovsky wants to merge 10 commits into
Conversation
Replace local fixture copies with the shared corpus, preserve extracted metadata and span inputs, and add a reviewed snapshot update workflow. Environment: Datadog workspace
Contributor
|
|
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 16 Pipeline jobs failed
ℹ️ InfoNo other issues found (see more)❄️ No new flaky tests detected Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 4a9a868 | Docs | View more details | Give us feedback! |
Expose Node.js error codes and Ruby resolution details to canonical assertions. Convert Ruby numeric defaults to Float before calling the OpenFeature client. Environment: Datadog workspace
leoromanovsky
marked this pull request as ready for review
October 21, 2026 20:19
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e3ad6aae6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
cbeauchesne
approved these changes
Oct 22, 2026
Keep the unified FFE mock backend loading the canonical fixture snapshot and preserve updater ownership. Environment: Datadog workspace
typotter
approved these changes
Oct 22, 2026
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: removed
and removed
mergequeue-status: waiting
labels
Co-authored-by: Codex GPT-5 <noreply@localhost>
Preserve the canonical FFE fixture path through the mock backend container refactor and retain the merged platform applicability filters. Environment: Datadog workspace
Download dd-openfeature.jar from the selected Java snapshot branch so FFE tests do not pair the development agent with an incompatible release provider. Cover default and custom branches, Java Lambda, and missing providers. Environment: Datadog workspace
sameerank
approved these changes
Oct 26, 2026
Preserve canonical fixture discovery and platform filters. Keep the development OpenFeature provider out of Java Lambda builds, cover both loader paths, and document unasserted split-feature metadata.
leoromanovsky
enabled auto-merge (squash)
October 27, 2026 17:31
The 0.5.4 release moved back to OpenTelemetry 0.32, which lacks the compression fallback and scope schema URL behavior present in 0.33. Preserve the existing minimum versions and declare only the released 0.5.4 unsupported; keep the 0.5.4 development build enabled. Environment: Datadog workspace
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: waiting
and removed
mergequeue-status: removed
labels
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Use the canonical FFE fixtures from
DataDog/ffe-system-test-datain system-tests. The existing local fixture copies are maintained separately.Changes
caa6d1a, with provenance inSOURCE.md.Float.Fixture integration options
Known conformance mismatches
The snapshot carries upstream
_skip/_includeannotations, which the runner honors per SDK. These exclude Python object/array opt-out inputs and PHP disabled/empty flags and missing targeting keys while their known conformance gaps remain. The canonical expected results are unchanged; dd-trace-py#21172 tracks the Python fix.The split-feature suite now compares canonical
result.featureswith the public hook callback and actual exposure/evaluation payloads, including false, zero, empty strings, first-valid-key selection, and allocation isolation. It also checks selected metadata and suppressed exposure delivery fordoLog=false. Java is activated from the validated1.69.0-SNAPSHOTbuild; other SDKs retain baseline evaluation coverage and skip the slower destination test until their hook capture is implemented. FFLSDK-274 records the implementation and remaining SDK adapters.Rust 0.5.4 reverted to OpenTelemetry 0.32 in dd-trace-rs#343. Its manifest now records the seven affected compression-fallback and scope-schema limitations for that release; the development build remains enabled.
Validation: formatting/type/import/manifest checks pass; 44 harness and manifest tests pass. Six Java parametric tests pass against
1.69.0-SNAPSHOT+e7584c7bca, covering all four canonical metadata cases plus RC and baseline evaluation. This validates callbacks and payload delivery to a local test agent; hosted CI remains separate.Workflow
system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.system-tests-core(slack)🚀 Once your PR is reviewed and the CI green, you can merge it!
🛟 #apm-shared-testing 🛟
FFE reviewer selection
Reviewers: @greghuels, @sameerank
Why: Global fallback — This PR moves FFE evaluation tests onto the canonical ffe-system-test-data snapshot. Global reviewers were selected because no project or SME pod matched.
🤖 Automated selection.
Selection details
Initiative: global / global
Why this match: This PR moves FFE evaluation tests onto the canonical ffe-system-test-data snapshot. It checks in the canonical evaluation cases and UFC config, and removes the old local copies. The dynamic-evaluation, configuration-source, serial-ID, span-enrichment and mocked-backend tests now read from that snapshot. It also adds a weekly updater workflow and script. The Node.js and Ruby parametric servers now forward evaluation reason, error code and metadata so the tests can check those conformance expectations. None of this fits a catalog project. It isn't bandits. It isn't EVP telemetry: the only change to tests/ffe/test_flag_eval_evp.py is a comment. And forwarding details to the test harness isn't onboarding diagnostics. The catalog has no fallback initiative, so this routes to global.
The PR author was excluded. Existing reviewers and teams were preserved.
Debug or fix a problem
Workflow run · Bits Code session
dry_run=false. Unchanged completed runs are skipped. If the receipt saysneeds_reconciliationor a write outcome is uncertain, ask the workflow owner to reconcile GitHub and the receipt before retrying; do not delete the lock blindly.Evidence and run details
✅ Complete — reviewer coverage verified. The workflow checked pending requests or submitted reviews on the evaluated head.
Generated asynchronously by the FFE reviewer-routing workflow.
Changed paths:
tests/parametric/test_ffe/ffe-system-test-data/SOURCE.md,tests/parametric/test_ffe/ffe-system-test-data/ufc-config.json,tests/parametric/test_ffe/test_dynamic_evaluation.py,tests/parametric/test_ffe/utils.py,utils/scripts/update_ffe_fixtures.py,.github/workflows/update-ffe-fixtures.yml,utils/build/docker/nodejs/parametric/server.js,utils/build/docker/ruby/parametric/server.rb,utils/mocked_backend/_backend_ffe.pyEvaluated head:
51867b13eca51097c73d418ee3b15b4b328d9101Registry revision:
revision:1946aa2a-530e-487a-84a7-83aff44a1089Pinned skill source:
d214cfc65ae154b97d39af5cf32caf48aaec9bf0Instructions: workflow-injected snapshot (no plugin loaded)
Instruction snapshot SHA-256:
f23b2a88c13c0da90e2dd58246a14f257af91e2195c555039bd555b52e6f3516