Skip to content

Isolate webhook telemetry sampling tests from concurrent tracing providers #2667

Open

Description

@niemyjski

The required API check for #2629 failed in the unchanged, already-merged WebHooksJobTests.ProcessQueueEntryAsync_GeneralWebHook_DeliversWithSafeTelemetry(sampled: false): expected 0 client spans, observed 5. Failing shard. The test file is identical to main at 0f2bda22b.

Local reproduction on .NET/ASP.NET Core 10.0.12 with the repository's OpenTelemetry 1.19.x packages:

  • Both isolated theory cases pass without retries (2/2).
  • Add the following temporary provider at the start of that test's Arrange section, before its existing provider. The unsampled case fails deterministically with expected 0, actual 5; the sampled case passes. All earlier request-header/cookie assertions still pass.
using var competingTracing = Sdk.CreateTracerProviderBuilder()
    .SetSampler(new AlwaysOnSampler())
    .AddHttpClientInstrumentation()
    .Build();
dotnet build tests/Exceptionless.Tests --no-restore
dotnet tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll --filter-method '*GeneralWebHook*' --minimum-expected-tests 1

This establishes that a competing process-wide provider is sufficient to produce the CI signature; it does not identify which hosted test/provider was active at that instant. Program/ApmExtensions and AssistantDiagnosticsTests register HTTP tracing providers, and the CI log shows API hosts starting around the failure.

Implement the smallest deterministic isolation of sampling/provider lifetimes (consider an isolated nonparallel collection or process boundary after checking fixture disposal). Preserve all span counts, URL-redaction, cookie, redirect, baggage, and propagation assertions. Do not add sleeps, acceptance retries, or skip/relax the flaky-test gate. Verify both sampler cases repeatedly with retries disabled and run the affected complete API shard. Keep any production webhook behavior change separately justified.

The temporary provider was removed from #2629; the forwarding PR does not change this test. A later green rerun alone does not close this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions