Skip to content

conformance testing: add granular violation suppressions#89

Open
lmolkova wants to merge 4 commits into
open-telemetry:mainfrom
lmolkova:conformance-exclusion
Open

conformance testing: add granular violation suppressions#89
lmolkova wants to merge 4 commits into
open-telemetry:mainfrom
lmolkova:conformance-exclusion

Conversation

@lmolkova
Copy link
Copy Markdown
Member

some instrumentations can't populate certain non-required attributes. Add mechanism allowing to suppress specific violations without turning checks off for everyone.

@lmolkova lmolkova requested a review from a team as a code owner May 23, 2026 19:31
Copilot AI review requested due to automatic review settings May 23, 2026 19:31
@lmolkova lmolkova added the Skip Changelog PR does not require a changelog entry label May 23, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a mechanism to declare and suppress specific, known Weaver live-check violations on a per-conformance-scenario basis, enabling tighter default validation while allowing instrumentations with unavoidable gaps to remain conformant.

Changes:

  • Introduces ExpectedViolation and Scenario.expected_violations, and adds violation reconciliation in run_conformance.
  • Tightens the span validation policy to expect server.address for more operations.
  • Updates LangChain conformance scenarios to suppress the known server.address missing-attribute violation.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
util/opentelemetry-test-util-genai/src/opentelemetry/test_util_genai/conformance.py Adds ExpectedViolation + enforcement logic so scenarios can declare and suppress specific known violations while still failing on unexpected/new ones.
policies/genai_span_validation.rego Makes server.address an expected attribute across additional operation sets, producing stable advice IDs for suppression.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/workflow.py Declares an expected Weaver violation for missing server.address in the workflow scenario.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/inference.py Declares an expected Weaver violation for missing server.address in the inference scenario.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/agent.py Declares an expected Weaver violation for missing server.address in the agent scenario.

Comment thread policies/genai_span_validation.rego
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Skip Changelog PR does not require a changelog entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants