Skip to content

Add /llm endpoint to flask-poc weblog#7127

Draft
christophe-papazian wants to merge 10 commits into
mainfrom
christophe-papazian/flask-llm-endpoint
Draft

Add /llm endpoint to flask-poc weblog#7127
christophe-papazian wants to merge 10 commits into
mainfrom
christophe-papazian/flask-llm-endpoint

Conversation

@christophe-papazian

@christophe-papazian christophe-papazian commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

APPSEC-68345

Adds the GET /llm endpoint to the flask-poc Python weblog, enabling system tests for LLM OpenAI instrumentation.

Covers 6 operations (3 sync via OpenAI(), 3 async via AsyncOpenAI()). Legacy openai v0 operations are not included as they are incompatible with openai==2.41.1.

Also adds openai==2.41.1 to requirements-flask-poc.txt.

⚠️ This PR should be merged after DataDog/dd-trace-py#18562

@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
tests/appsec/api_security/test_endpoints.py                             @DataDog/asm-libraries @DataDog/system-tests-core
utils/build/docker/python/docker-bake.hcl                               @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/flask-poc.Dockerfile                          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/flask-poc.base.Dockerfile                     @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/flask/app.py                                  @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/flask/requirements-flask-poc.txt              @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/flask/requirements-uwsgi-poc.txt              @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/uds-flask.Dockerfile                          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/uwsgi-poc.Dockerfile                          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/uwsgi-poc.base.Dockerfile                     @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core

@datadog-official

datadog-official Bot commented Jun 11, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 87 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.0 1   View in Datadog   GitHub Actions

🧪 1 Test failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[apache-mod-7.0] from system_tests_suite   View in Datadog (Fix with Cursor)
AssertionError: Expected metric for non-existent flag 'non-existent-eval-metric-flag', found none. All: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7f4d18408cb0>

    def test_ffe_eval_config_exists_flag_missing(self):
        """Test that missing flag (with config loaded) produces error.type=flag_not_found."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
    
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.0-zts 1   View in Datadog   GitHub Actions

🧪 1 Test failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[apache-mod-7.0-zts] from system_tests_suite   View in Datadog (Fix with Cursor)
AssertionError: Expected metric for non-existent flag 'non-existent-eval-metric-flag', found none. All: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7f6504629760>

    def test_ffe_eval_config_exists_flag_missing(self):
        """Test that missing flag (with config loaded) produces error.type=flag_not_found."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
    
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.1 1   View in Datadog   GitHub Actions

🧪 1 Test failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[apache-mod-7.1] from system_tests_suite   View in Datadog (Fix with Cursor)
AssertionError: Expected metric for non-existent flag 'non-existent-eval-metric-flag', found none. All: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7f2c247e4bc0>

    def test_ffe_eval_config_exists_flag_missing(self):
        """Test that missing flag (with config loaded) produces error.type=flag_not_found."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
    
...

View all 87 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6b7d29f | Docs | Datadog PR Page | Give us feedback!

christophe-papazian and others added 6 commits June 12, 2026 10:28
Implements the GET /llm endpoint (model + operation query params) for the
flask-poc weblog, covering the 6 openai-v2 operations (3 sync, 3 async).
Adds openai==2.41.1 to requirements-flask-poc.txt.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Add /llm endpoint to flask weblog (sync + async OpenAI operations)
- Fix setup_root_has_no_llm_tags method name in test_endpoints.py
- Update python.yml manifest: enable tests for flask-poc v4.12.0rc1,
  mark legacy operations as irrelevant (openai >= 1.0 removed them),
  mark other weblogs as irrelevant (framework-independent feature)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@christophe-papazian christophe-papazian force-pushed the christophe-papazian/flask-llm-endpoint branch from 35898c8 to 2084bdb Compare June 12, 2026 08:28
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
christophe-papazian and others added 3 commits June 12, 2026 10:57
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
All weblogs sharing flask/app.py (flask-poc, uds-flask, uwsgi-poc)
now have openai available.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/dd-trace-py that referenced this pull request Jun 15, 2026
)

APPSEC-68345

## Description

Emit `server.business_logic.llm.event` to the WAF when AppSec is enabled and an OpenAI `chat.completions`, `completions`, or `responses` call is made during a web request. Enables WAF rules to tag LLM-backed endpoints in the API Endpoint Catalog.

- New `ddtrace/appsec/_contrib/openai/` listener dispatches the WAF address with provider and model on each call
- `_constants.py` registers `LLM_EVENT` as a persistent WAF address
- `patch.py` fixes raw-response streaming early-return path to still fire AppSec `.before` events; adds `completions` endpoint dispatch

## Testing

`tests/contrib/openai/test_openai_appsec_llm_events.py` covers sync/async, chat/completions/responses, Azure OpenAI (`engine` kwarg), and streaming paths.

Also validated with DataDog/system-tests#7127

## Risks

No-op when AppSec is off or outside a request context. WAF deduplicates persistent addresses so only the first LLM call per request is surfaced (by design).

Co-authored-by: christophe.papazian <christophe.papazian@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant