unflake a System Test: 'test_remote_sampling_rules_retention'

[vlad-scherbich]

https://datadoghq.atlassian.net/browse/PROF-13796

Motivation

test_remote_sampling_rules_retention fails intermittently in CI. Observed flaking in a dd-trace-py PR that only changes CI workflow config (zero relation to test logic). See failed job.

The root cause is a race condition in set_and_wait_rc: its wait signals (telemetry event + RC ACKNOWLEDGED state) are not config-version-specific, so stale events from a prior RC config can satisfy them before the new sampling rules are actually active in the library.

Changes

Adds a retry loop around the first trace assertion in test_remote_sampling_rules_retention to tolerate the brief propagation window between RC acknowledgment and actual rule application. This is consistent with how other tests in the same file handle similar timing sensitivity (e.g., get_sampled_trace).

Testing

$ ./run.sh PARAMETRIC -L python -k "retention"
...
============================================================================= test context ==============================================================================
Scenario: PARAMETRIC
Logs folder: ./logs_parametric
Library: python@4.4.0
========================================================================== test session starts ==========================================================================
gw0 [1] / gw1 [1] / gw2 [1] / gw3 [1] / gw4 [1] / gw5 [1] / gw6 [1] / gw7 [1] / gw8 [1] / gw9 [1] / gw10 [1] / gw11 [1] / gw12 [1] / gw13 [1] / gw14 [1] / gw15 [1]
.                                                                                                                                                                 [100%]
--------------------------- generated xml file: /Users/vlad.scherbich/go/src/github.com/DataDog/system-tests/logs_parametric/reportJunit.xml ----------------------------
========================================================================== 1 passed in 32.80s ===========================================================================
COMP-LR7JK0FKW1:system-tests vlad.scherbich$ 

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/parametric/test_dynamic_configuration.py                          @DataDog/system-tests-core @DataDog/apm-sdk-capabilities

[KowalskiThomas]

If I understand correctly, we try up to 30 times (waiting 0.1 seconds between each attempt) to see a span whose metadata shows that the new sampling rules were correctly received and applied?

[vlad-scherbich]

Exactly!

[cbeauchesne]

Would it be possible to improve the wait_rc condition to wait for the precise new config ?

[vlad-scherbich]

@cbeauchesne , I like this idea.
set_and_wait_rc is just a helper function used only in test_dynamic_configuration.py at 20 call sites.

I'll open a separate PR for the proposed fix, which would be to clear the session before setting the new RC config.

[vlad-scherbich]

@cbeauchesne , new PR implementing your suggestion: #6349

Lots of unrelated system test fails, do I just re-run them until they pass?

[datadog-datadog-prod-us1]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

❄️ 706 New flaky tests detected

tests.appsec.test_alpha.Test_Basic.test_headers[chi] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_alpha.Test_Basic object at 0x7f57d96e87d0>

    def test_headers(self):
        """Via server.request.headers.no_cookies"""
        # Note: we do not check the returned key_path nor rule_id for the alpha version
        address = "server.request.headers.no_cookies"
        pattern = "../"
>       interfaces.library.assert_waf_attack(self.r_headers_1, pattern=pattern, address=address)
...

tests.appsec.test_alpha.Test_Basic.test_headers[echo] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_alpha.Test_Basic object at 0x7fada1c89370>

    def test_headers(self):
        """Via server.request.headers.no_cookies"""
        # Note: we do not check the returned key_path nor rule_id for the alpha version
        address = "server.request.headers.no_cookies"
        pattern = "../"
>       interfaces.library.assert_waf_attack(self.r_headers_1, pattern=pattern, address=address)
...

tests.appsec.test_alpha.Test_Basic.test_headers[gin] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_alpha.Test_Basic object at 0x7fb8cc2d0320>

    def test_headers(self):
        """Via server.request.headers.no_cookies"""
        # Note: we do not check the returned key_path nor rule_id for the alpha version
        address = "server.request.headers.no_cookies"
        pattern = "../"
>       interfaces.library.assert_waf_attack(self.r_headers_1, pattern=pattern, address=address)
...

View all

ℹ️ Info

🧪 All tests passed

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9f8ebf4 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}