Skip to content

test(ffe): add direct dependent flags coverage [nodejs@typo/dependent-flags-evaluator-system-tests] - #8190

Open
typotter wants to merge 5 commits into
mainfrom
typo/dependent-flags-direct-system-tests
Open

test(ffe): add direct dependent flags coverage [nodejs@typo/dependent-flags-evaluator-system-tests]#8190
typotter wants to merge 5 commits into
mainfrom
typo/dependent-flags-direct-system-tests

Conversation

@typotter

@typotter typotter commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation

Run dependent-flags evaluation coverage directly in the existing parametric FFE suite against the latest JavaScript evaluator implementation through a Node Tracer SDK artifact.

Changes

  • add an explicitly named test_ffe_dependent_flags method beside the existing dynamic-evaluation test
  • keep its UFC and cases isolated under tests/parametric/test_ffe/dependent-flags/
  • assert the expected value, reason, and errorCode fields provided by each case, while rejecting unexpected errors from successful evaluations
  • activate the method for Node v7.0.0-pre; retain missing_feature (FFL-1694) for every other server SDK
  • leave existing root fixture discovery and test_ffe_flag_evaluation unchanged

The fixtures are an exact copy of evaluation-suites/dependent-flags from DataDog/ffe-system-test-data commit f20ab80 (the merge of PR #45).

Tested implementation provenance

  • evaluator: DataDog/openfeature-js-client#437 at df51d65f0d89dc192c9c96cf7ac000dcf9ac1f43
  • packaged libraries: @datadog/flagging-core@3.3.0 and @datadog/openfeature-node-server@2.6.0
  • Node Tracer SDK artifact: DataDog/dd-trace-js@b93162f72178fa756d0ee1bee64143c2b7cb00c8
  • CI target branch: DataDog/dd-trace-js#typo/dependent-flags-evaluator-system-tests

Telemetry remains outside this system-test assertion. Exposure and flag-evaluation telemetry are covered in the SDK repository.

Validation

  • evaluator core: 1197 passed, 6 skipped
  • evaluator Node provider: 560 passed
  • Node Tracer OpenFeature suite: 682 passed, 2 assertion failures reflecting changed missing-dependency telemetry
  • ./format.sh
  • focused system-test execution through the pushed Node Tracer artifact: 1 passed, covering all 9 dependent-graph cases

The purpose of this PR run is to produce system-test results from the latest evaluator implementation through the Node Tracer SDK artifact; green results are not required to establish that provenance.

@typotter Codex GPT-5
test(ffe): add direct dependent flag coverage
11eba69 
Run isolated dependent-flag fixtures through the existing parametric FFE evaluation path while keeping every SDK disabled until its supporting release is known.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@github-actions

github-actions Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/dependent-flags/cases/dependent-flags.json    @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_ffe/dependent-flags/ufc-config.json               @DataDog/feature-flagging-and-experimentation-sdk
manifests/dotnet.yml                                                    @DataDog/system-tests-reviewers
manifests/golang.yml                                                    @DataDog/system-tests-reviewers
manifests/java.yml                                                      @DataDog/system-tests-reviewers
manifests/nodejs.yml                                                    @DataDog/system-tests-reviewers
manifests/php.yml                                                       @DataDog/system-tests-reviewers
manifests/python.yml                                                    @DataDog/system-tests-reviewers
manifests/ruby.yml                                                      @DataDog/system-tests-reviewers
manifests/rust.yml                                                      @DataDog/system-tests-reviewers
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Oct 27, 2026 •
edited by datadog-prod-us1-4 Bot
Loading

Copy link
Copy Markdown
Contributor

Pipelines  Tests

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 4 Pipeline jobs failed

Testing the test | System Tests (rust, prod) / parametric / parametric (1) — ❌ 7 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.parametric.otel_env_vars.test_otel_exporter_otlp_compression.Test_OTEL_EXPORTER_OTLP_COMPRESSION.test_empty[empty, parametric-rust] from system_tests_suite 
ValueError: Number (1) of metrics not available from test agent, got 0

self = &lt;tests.parametric.otel_env_vars.test_otel_exporter_otlp_compression.Test_OTEL_EXPORTER_OTLP_COMPRESSION object at 0x7f5dfc97e060&gt;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f5dfba4ecc0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f5deecabbf0&gt;

    @pytest.mark.parametrize(&#34;compression_env&#34;, [pytest.param({&#34;OTEL_EXPORTER_OTLP_COMPRESSION&#34;: &#34;&#34;}, id=&#34;empty&#34;)])
    def test_empty(self, test_agent: TestAgentAPI, test_library: APMLibrary) -&gt; None:
&gt;       _assert_compression(test_agent, test_library, &#34;none&#34;)

...
❌ tests.parametric.otel_env_vars.test_otel_exporter_otlp_compression.Test_OTEL_EXPORTER_OTLP_COMPRESSION.test_invalid[invalid, parametric-rust] from system_tests_suite 
ValueError: Number (1) of metrics not available from test agent, got 0

self = &lt;tests.parametric.otel_env_vars.test_otel_exporter_otlp_compression.Test_OTEL_EXPORTER_OTLP_COMPRESSION object at 0x7f5dfc97e3c0&gt;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f5dfba4ecc0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f5deec0e7b0&gt;

    @pytest.mark.parametrize(
        &#34;compression_env&#34;, [pytest.param({&#34;OTEL_EXPORTER_OTLP_COMPRESSION&#34;: &#34;not-a-compression&#34;}, id=&#34;invalid&#34;)]
    )
    def test_invalid(self, test_agent: TestAgentAPI, test_library: APMLibrary) -&gt; None:
...
❌ tests.parametric.otel_env_vars.test_otel_exporter_otlp_logs_compression.Test_OTEL_EXPORTER_OTLP_LOGS_COMPRESSION.test_empty[empty, parametric-rust] from system_tests_suite 
ValueError: Number 1 of logs not available from test agent, got 0

self = &lt;tests.parametric.otel_env_vars.test_otel_exporter_otlp_logs_compression.Test_OTEL_EXPORTER_OTLP_LOGS_COMPRESSION object at 0x7ffbbdf5ca10&gt;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7ffb8b421ca0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7ffb80f9ec00&gt;

    @pytest.mark.parametrize(&#34;compression_env&#34;, [pytest.param({&#34;OTEL_EXPORTER_OTLP_LOGS_COMPRESSION&#34;: &#34;&#34;}, id=&#34;empty&#34;)])
    def test_empty(self, test_agent: TestAgentAPI, test_library: APMLibrary) -&gt; None:
&gt;       _assert_compression(test_agent, test_library, &#34;none&#34;)

...
↳ and 4 more — View all
Testing the test | System Tests (nodejs, dev) / parametric / parametric (2) — ❌ 8 tests failed

View more details · View in GitHub Actions

❌ tests.parametric.test_ffe.test_rules_configuration.Test_FFE_Rules_Configuration.test_cancel_before_request[parametric-nodejs] from system_tests_suite 
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:4500/ffe/fetch-rules-configuration

self = &lt;tests.parametric.test_ffe.test_rules_configuration.Test_FFE_Rules_Configuration object at 0x7ff41bc33170&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7ff40e621970&gt;
mocked_backend = &lt;utils.mocked_backend._backend.MockedBackendServer object at 0x7ff40e6bf2c0&gt;

    def test_cancel_before_request(self, test_library: APMLibrary, mocked_backend: MockedBackendServer) -&gt; None:
        with test_library:
&gt;           result = test_library.ffe_fetch_rules_configuration({**CLIENT_OPTIONS, &#34;abort&#34;: True})

...
❌ tests.parametric.test_ffe.test_rules_configuration.Test_FFE_Rules_Configuration.test_cancel_in_flight[body, parametric-nodejs] from system_tests_suite 
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:4502/ffe/fetch-rules-configuration

self = &lt;tests.parametric.test_ffe.test_rules_configuration.Test_FFE_Rules_Configuration object at 0x7f0316219a90&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f0308ca47d0&gt;
mocked_backend = &lt;utils.mocked_backend._backend.MockedBackendServer object at 0x7f0308c337d0&gt;
response = &#39;delayed_body&#39;

    @pytest.mark.parametrize(&#34;response&#34;, [&#34;delayed&#34;, &#34;delayed_body&#34;], ids=[&#34;headers&#34;, &#34;body&#34;])
    def test_cancel_in_flight(
        self, test_library: APMLibrary, mocked_backend: MockedBackendServer, response: str
...
❌ tests.parametric.test_ffe.test_rules_configuration.Test_FFE_Rules_Configuration.test_cancel_in_flight[headers, parametric-nodejs] from system_tests_suite 
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:4504/ffe/fetch-rules-configuration

self = &lt;tests.parametric.test_ffe.test_rules_configuration.Test_FFE_Rules_Configuration object at 0x7fb486bbd580&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fb4794da6f0&gt;
mocked_backend = &lt;utils.mocked_backend._backend.MockedBackendServer object at 0x7fb4794bb020&gt;
response = &#39;delayed&#39;

    @pytest.mark.parametrize(&#34;response&#34;, [&#34;delayed&#34;, &#34;delayed_body&#34;], ids=[&#34;headers&#34;, &#34;body&#34;])
    def test_cancel_in_flight(
        self, test_library: APMLibrary, mocked_backend: MockedBackendServer, response: str
...
↳ and 5 more — View all
Testing the test | Fail if target branch is specified

View more details · View in GitHub Actions

View all 4 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6071bb2 | Docs | View more details | Give us feedback!

@typotter Codex GPT-5
test(nodejs): activate dependent flags coverage
09f6b46 
Enable the direct dependent-flags method for the 7.0.0-pre SDK integration branch carrying the pinned WIP evaluator bundle.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@typotter typotter changed the title test(ffe): add direct dependent flags coverage [nodejs@dependent-flags-variant-key-node-integration] test(ffe): add direct dependent flags coverage [nodejs@typo/dependent-flags-evaluator-system-tests] Oct 27, 2026
@typotter typotter changed the title test(ffe): add direct dependent flags coverage [nodejs@typo/dependent-flags-evaluator-system-tests] test(ffe): add direct dependent flags coverage Oct 27, 2026
@typotter
Merge remote-tracking branch 'origin/main' into typo/dependent-flags-…
ebbcce6 
…direct-system-tests

# Conflicts:
#	manifests/golang.yml
#	manifests/java.yml
#	manifests/nodejs.yml
#	manifests/php.yml
#	manifests/python.yml
#	manifests/ruby.yml
@typotter
typotter marked this pull request as ready for review October 27, 2026 21:02
@typotter
typotter requested review from a team as code owners October 27, 2026 21:02
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T21:05:05.713756Z ebbcce6 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@typotter
typotter requested a review from hhan2024 October 27, 2026 21:03
@typotter typotter changed the title test(ffe): add direct dependent flags coverage test(ffe): add direct dependent flags coverage [nodejs@typo/dependent-flags-evaluator-system-tests] Oct 27, 2026
@typotter
test(ffe): activate dependent flags for node
6071bb2 
Run against the Node tracer artifact branch containing openfeature-js-client PR #437 at df51d65f0d89dc192c9c96cf7ac000dcf9ac1f43.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hhan2024 hhan2024 Awaiting requested review from hhan2024

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@typotter