Skip to content

fix(tests): reset OTel tracer provider to None to stop cross-test recursion - #336

Merged
morgan-wowk merged 1 commit into
masterfrom
fix-otel-tracer-provider-reset-leak
Aug 18, 2026
Merged

fix(tests): reset OTel tracer provider to None to stop cross-test recursion#336
morgan-wowk merged 1 commit into
masterfrom
fix-otel-tracer-provider-reset-leak

Conversation

@morgan-wowk

@morgan-wowk morgan-wowk commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Problem

Running the full tests/ suite produced 6 failures that were not reproducible when the affected files were run in isolation:

FAILED tests/test_container_execution_refresh_retries.py::...::test_missing_workload_is_counted_and_terminalizes
FAILED tests/test_container_execution_refresh_retries.py::...::test_retriable_errors_below_budget_leave_execution_running
FAILED tests/test_container_execution_refresh_retries.py::...::test_successful_refresh_resets_the_budget
FAILED tests/test_container_execution_refresh_retries.py::...::test_retry_moves_execution_to_the_back_of_the_queue
FAILED tests/test_container_execution_refresh_retries.py::...::test_failing_execution_does_not_hold_up_the_queue
FAILED tests/test_orchestrator_failed_log_upload.py::test_failed_execution_survives_log_upload_failure

The failures were a symptom (SYSTEM_ERROR instead of PENDING, mocks never called, timestamps not advancing), with a RecursionError: maximum recursion depth exceeded buried in the output.

Root cause

The reset_otel_providers autouse fixture in tests/instrumentation/opentelemetry/conftest.py reset the global tracer provider to a fresh ProxyTracerProvider():

trace._TRACER_PROVIDER = trace.ProxyTracerProvider()   # ← self-referential

OpenTelemetry's ProxyTracerProvider.get_tracer() delegates to the module-global trace._TRACER_PROVIDER. Once that global is a ProxyTracerProvider, the delegation points back at itself, so start_as_current_span recurses until RecursionError.

Because the fixture mutates OTel's process-global state, the broken provider leaked out of the opentelemetry/ test package into the rest of the suite (default collection order runs it before test_container_execution_refresh_retries). Later tests that emit spans via orchestrator_tracing.operation_span / execution_tracing.emit_execution_trace hit the RecursionError; the orchestrator caught it as a generic processing error and drove container executions to SYSTEM_ERROR, which is what the assertions actually tripped on.

Fix

Reset to None, matching the meter-provider reset on the very next line:

trace._TRACER_PROVIDER = None

With None, get_tracer_provider() falls back to OTel's internal _PROXY_TRACER_PROVIDER and ProxyTracer resolves to the no-op tracer — no self-delegation, no recursion.

Verification

  • Before: 6 failed, 462 passed (full suite)
  • After: 468 passed (full suite, pytest tests/)
  • The previously-failing files still pass in isolation, and the opentelemetry/ suite that owns the fixture continues to pass.

Downstream

This is beneficial to downstream applications with their own suite of tests against OTEL.

…ursion

The reset_otel_providers autouse fixture reset the global tracer
provider to a fresh ProxyTracerProvider() after each test. A
ProxyTracerProvider delegates get_tracer() to the module-global
trace._TRACER_PROVIDER; once that global *is* a ProxyTracerProvider,
the delegation points back at itself and start_as_current_span recurses
until RecursionError.

Because the fixture mutates OTel's process-global state, the broken
provider leaked out of the opentelemetry test package into the rest of
the suite. Later tests that emit spans (orchestrator_tracing,
execution_tracing) hit the RecursionError inside operation_span; the
orchestrator swallowed it as a processing error, so container
executions were driven to SYSTEM_ERROR instead of their expected
states. This surfaced as 6 unrelated-looking failures in
test_container_execution_refresh_retries and
test_orchestrator_failed_log_upload only when the full suite ran.

Reset to None (matching the meter-provider reset on the next line) so
get_tracer_provider() falls back to the internal proxy and
ProxyTracer resolves to the no-op tracer. Full suite: 468 passed.
@morgan-wowk
morgan-wowk requested a review from a team August 18, 2026 00:17
@morgan-wowk
morgan-wowk merged commit 0abbe3a into master Aug 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants