Skip to content

Add APPSEC_OTEL_API scenarios for M1 topology (APPSEC-68836) - #7582

Draft
RamyElkest wants to merge 3 commits into
mainfrom
ramy.elkest/appsec-otel-api-M1
Draft

Add APPSEC_OTEL_API scenarios for M1 topology (APPSEC-68836)#7582
RamyElkest wants to merge 3 commits into
mainfrom
ramy.elkest/appsec-otel-api-M1

Conversation

@RamyElkest

Copy link
Copy Markdown
Contributor

Summary

Validates M1 topology from the AAP for OTel Customers strategic plan: DD Tracer as drop-in OTel SDK replacement.

M1 tests the instrumentation layer — the dd-tracer implements the OTel API (), so customers can use standard in their code while getting full AppSec. This is different from M2 (PR #7578) which tests the transport layer (OTLP export).

Closes APPSEC-68836

What's New

Two Scenarios

Scenario Ruleset Purpose
APPSEC_OTEL_API Blocking rules (31) WAF blocking, fingerprinting, trace tagging, security response IDs
APPSEC_OTEL_API_DEFAULT_RULES Default ruleset (199) Alpha detection, obfuscator, header collection, payment events

Both use: DD_TRACE_OTEL_ENABLED=true + include_agent=True + rc_api_enabled=True + appsec_enabled=True

Key Difference from M2 (PR #7578)

Aspect M2 (PR #7578) M1 (this PR)
Layer Transport (OTLP export) Instrumentation (OTel API)
Traces via OTLP protocol DD agent native protocol
OTLP adapter Required (format conversion) Not needed (native format)
New weblog No (existing flask-poc) No (existing flask-poc)
Customer code change None (just env vars) None (just dependency swap)

No new weblog Dockerfile needed — the existing flask-poc weblog already imports and uses opentelemetry.trace, opentelemetry.baggage, and opentelemetry.propagate. With DD_TRACE_OTEL_ENABLED=true, dd-tracer implements these APIs.

Test Results

Scenario Passed Failed XFailed
APPSEC_OTEL_API 77 0 4
APPSEC_OTEL_API_DEFAULT_RULES 26 0 0
Total 103 0 4

Plus 2 smoke tests (Threats + UserEvents) passing in each scenario.

CI Integration

  • Scenarios added to .github/workflows/run-end-to-end.yml
  • Part of tracer_release group — runs on every tracer release PR
  • Dynamic selection — triggered when test files with @scenarios.appsec_otel_api are modified

M1: DD Tracer as drop-in OTel SDK replacement (instrumentation layer).
Uses standard DD agent for transport (native protocol), OTel API backed by dd-tracer.

Two new scenarios:
- APPSEC_OTEL_API: blocking ruleset, tests WAF blocking + detection
- APPSEC_OTEL_API_DEFAULT_RULES: default ruleset, tests alpha/obfuscator/payment

No new weblog needed — existing flask-poc weblog already uses OTel API
with DD_TRACE_OTEL_ENABLED=true. No OTLP adapter needed — traces go
via DD agent in native Datadog format.

Test results: 42 + 26 + 35 + 2 = 105 passed, 0 failed, 4 xfailed.

CI: scenarios added to .github/workflows/run-end-to-end.yml
@RamyElkest
RamyElkest requested review from a team as code owners August 25, 2026 15:45
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/appsec/smoke_tests/test_otel_api.py                               @DataDog/asm-libraries @DataDog/system-tests-core
utils/build/docker/python/flask-poc-otel-api.Dockerfile                 @DataDog/system-tests-reviewers
utils/build/docker/python/flask-poc-otel-api/app.py                     @DataDog/system-tests-reviewers
utils/build/docker/python/flask-poc-otel-api/app.sh                     @DataDog/system-tests-reviewers
utils/build/docker/python/flask-poc-otel-api/weblog_metadata.yml        @DataDog/system-tests-reviewers
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
tests/appsec/api_security/test_apisec_sampling.py                       @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/api_security/test_apisecurity_telemetry.py                 @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/api_security/test_endpoint_discovery.py                    @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/api_security/test_endpoint_fallback.py                     @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/api_security/test_endpoints.py                             @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/api_security/test_schemas.py                               @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/api_security/test_schemas_auth.py                          @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/rasp/test_api10.py                                         @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/rasp/test_cmdi.py                                          @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/rasp/test_lfi.py                                           @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/rasp/test_shi.py                                           @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/rasp/test_sqli.py                                          @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/rasp/test_ssrf.py                                          @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_alpha.py                                              @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_automated_payment_events.py                           @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_blocking_addresses.py                                 @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_conf.py                                               @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_fingerprinting.py                                     @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_ip_blocking_full_denylist.py                          @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_rate_limiter.py                                       @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_remote_config_rule_changes.py                         @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_reports.py                                            @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_request_blocking.py                                   @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_runtime_activation.py                                 @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_span_tags_headers.py                                  @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_suspicious_attacker_blocking.py                       @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_trace_tagging.py                                      @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_traces.py                                             @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_user_blocking_full_denylist.py                        @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_versions.py                                           @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/waf/test_blocking.py                                       @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/waf/test_blocking_security_response_id.py                  @DataDog/asm-libraries @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0696d415d2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

appsec_otel_api = DdTraceEndToEndScenario(
"APPSEC_OTEL_API",
weblog_env={
"DD_TRACE_OTEL_ENABLED": "true",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Exercise the OTel API in the new scenarios

Setting DD_TRACE_OTEL_ENABLED does not make these AppSec checks exercise the drop-in OTel API: the new smoke classes call /waf and /login (tests/appsec/smoke_tests/utils.py:122-124,332-335), and the Python handlers use normal framework instrumentation and explicit Datadog AppSec helpers (utils/build/docker/python/flask/app.py:399-409,1547-1559); the actual OTel API calls are confined to separate /otel_drop_in_* handlers starting at line 1941. Consequently, a tracer whose OTel bridge is absent or broken can still pass both scenarios, so the reported M1 experiment validates ordinary Datadog instrumentation rather than AppSec through the OTel API; generate the tested request/span through an OTel API endpoint or instrument the request path with the OTel API.

Useful? React with 👍 / 👎.

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Aug 25, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 3 Pipeline jobs failed

Testing the test | Test the test — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

1 failed test. AssertionError: assert not True at tests/test_the_test/test_ci_orchestrator.py:161.

Testing the test | lint / lint — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

Unused import 'utils.context' in tests/appsec/smoke_tests/test_otel_api.py:11

Testing the test | all-jobs-are-green

View more details · View in GitHub Actions

Multiple CI checks have failed including 'lint' and 'Test the test'.

📋 Copy fix prompt
CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Before you start, set up the Datadog software-delivery tooling so you can
query the CI data yourself:

1. Check whether you already have the Datadog software-delivery MCP tools
   (e.g. a `search_datadog_ci_pipeline_events` tool) and the `unblock-pr` skill.
2. If either is missing, STOP and ask me for permission before installing
   anything. Do not install or run anything until I have said yes.
3. Only with my explicit approval, set up the Datadog software-delivery MCP
   server and skills by following:
     https://docs.datadoghq.com/getting_started/software_delivery_mcp_tools/
   then restart so the skill is picked up.
4. If I decline, skip all of the above and work from the context below alone.

Then run /unblock-pr — it will pull the CI data itself. The job context below is what we already know.

If /unblock-pr is not available — because I declined the setup above, or it did not install — work from the context below instead.

Datadog has already classified this failure as caused by changes in this PR.
Take that as given and work the fix:

1. Locate the change. Diff this branch against its base and find the change
   that produces this error. Explain the mechanism, don't just name a file:
     git fetch origin && git diff $(git merge-base origin/main HEAD)...HEAD
2. Reproduce it locally. Run the failing job's command or test before
   proposing anything.
3. Propose the smallest fix that addresses the root cause — not a workaround,
   not a broadened assertion, not a disabled or skipped test.
4. Re-run the same command to confirm, and say exactly what you ran.
5. If the failure turns out to be intermittent rather than deterministic, say
   so plainly instead of "fixing" it — that is a flaky test, and patching it
   hides the problem.

If the right move is to re-run the job rather than change code, use the job
link in the context below. For GitHub Actions: `gh run rerun <run-id> --failed`,
where the run ID is the number after `/runs/` in that URL (not the trailing
number, which is the job ID).

Branch: ramy.elkest/appsec-otel-api-M1

Testing the test | Test the test
Commit: 632399a69d0979d3c50228d20455db37f8cc19d0
Error (code / test):
1 failed test. AssertionError: assert not True at tests/test_the_test/test_ci_orchestrator.py:161.
CI job: https://github.com/DataDog/system-tests/actions/runs/32908095298/job/97996432998

Testing the test | lint / lint
Commit: 632399a69d0979d3c50228d20455db37f8cc19d0
Error (code / quality):
Unused import 'utils.context' in tests/appsec/smoke_tests/test_otel_api.py:11
CI job: https://github.com/DataDog/system-tests/actions/runs/32908095298/job/97996433157

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 632399a | Docs | View more details | Give us feedback!

…r to M1 (APPSEC-68836)

5 new OTel API scenarios covering all required AppSec capabilities:
- APPSEC_OTEL_API_SECURITY: API Security schema discovery + sampling
- APPSEC_OTEL_API_RASP: RASP detection (cmdi, lfi, sqli, ssrf, shi, api10)
- APPSEC_OTEL_API_RUNTIME_ACTIVATION: RC-driven AppSec on/off + rule changes
- APPSEC_OTEL_API_BLOCKING_FULL_DENYLIST: RC-driven IP/user blocking
- APPSEC_OTEL_API_RATE_LIMITER: AppSec rate limiter

101 new test classes decorated across 21 files.
All smoke tests pass: 1 passed per scenario for API Sec, RASP, RC, blocking, rate limiter.

This completes the 'supported' definition from the strategic plan:
WAF + ATO + API Security all validated via OTel API.
@RamyElkest
RamyElkest marked this pull request as draft August 25, 2026 22:24
…APPSEC-68836)

New weblog variant that creates spans via opentelemetry.trace.start_span()
instead of dd-trace's Flask auto-instrumentation. This validates M1:
AppSec works when the OTel API is the instrumentation layer.

Key endpoints (/waf, /rasp/*, /login, /identify, /headers) are wrapped with
otel_tracer.start_as_current_span() so spans are created through the OTel API.

Uses ddtrace-run to load dd-tracer with DD_TRACE_OTEL_ENABLED=true,
so dd-tracer implements the OTel API.

First test passes (XPASS — needs manifest entries for new weblog variant).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant