OpenTelemetry context detach errors when BaseAgent wrapper exits inner agent's run_async() early

[mmphego]

Describe the bug

When a BaseAgent subclass wraps another agent and exits early from iterating over self.main_agent.run_async(ctx) (via return or break inside async for), multiple Failed to detach context errors are logged by OpenTelemetry. This happens because base_agent.run_async() wraps the generator yield in tracer.start_as_current_span(), and when GeneratorExit propagates through the generator chain, the OTel context tokens cannot be detached — they were created in a different contextvars.Context.

google-adk version: 1.27.2
Python version: 3.13
opentelemetry-api version: 1.33.0

This appears to be the same root cause as #423 and #1670 (both closed), but with a distinct and reproducible trigger path.

To Reproduce

Create a BaseAgent subclass that wraps another agent and conditionally exits early:

from google.adk.agents import BaseAgent, LlmAgent
from google.adk.events.event import Event
from typing import AsyncGenerator
from typing_extensions import override

class WrapperAgent(BaseAgent):
    main_agent: BaseAgent

    @override
    async def _run_async_impl(self, ctx) -> AsyncGenerator[Event, None]:
        async for event in self.main_agent.run_async(ctx):
            if should_block(event):
                yield create_redirect_event(ctx)
                return  # <-- triggers GeneratorExit on inner generator chain
            yield event

When should_block() returns True and the wrapper does return inside the async for loop, the inner generator (main_agent.run_async(ctx)) is abandoned. Python sends GeneratorExit into the generator chain, which unwinds through base_agent.run_async():

base_agent.run_async()
  → tracer.start_as_current_span('invoke_agent ...')  # attaches OTel context token
    → Aclosing(self._run_async_impl(ctx))
      → async for event in agen:
        → yield event  # GeneratorExit thrown here
    # __exit__ tries to detach token → ValueError

Error output

ERROR:opentelemetry.context:Failed to detach context
Traceback (most recent call last):
  File ".../opentelemetry/trace/__init__.py", line 589, in use_span
    yield span
  File ".../opentelemetry/sdk/trace/__init__.py", line 1105, in start_as_current_span
    yield span
  File ".../google/adk/agents/base_agent.py", line 304, in run_async
    yield event
GeneratorExit

During handling of the above exception, another exception occurred:

  File ".../opentelemetry/context/__init__.py", line 155, in detach
    _RUNTIME_CONTEXT.detach(token)
  File ".../opentelemetry/context/contextvars_context.py", line 53, in detach
    self._current_context.reset(token)
ValueError: <Token ...> was created in a different Context

This repeats for each nesting level in the agent/tracing hierarchy (4 errors for a typical pipeline).

Root cause analysis

base_agent.run_async() at line 294 uses a synchronous context manager (tracer.start_as_current_span()) wrapping an async generator's yield point:

with tracer.start_as_current_span(f'invoke_agent {self.name}') as span:
    # ...
    async with Aclosing(self._run_async_impl(ctx)) as agen:
        async for event in agen:
            yield event  # <-- GeneratorExit enters here

The start_as_current_span context manager calls context.attach() on __enter__ (storing a contextvars.Token) and context.detach(token) on __exit__. When GeneratorExit is thrown into the generator, __exit__ fires, but the ContextVar.reset(token) fails because the token was created in a different contextvars.Context (the async context has shifted between generator suspension and cleanup).

Workaround

Application-level: Suppress the OTel context logger:

logging.getLogger("opentelemetry.context").setLevel(logging.CRITICAL)

Wrapper agent level: Use break instead of return inside the async for loop, and yield the redirect event after the loop exits. This ensures the inner generator's cleanup (including OTel span detach) completes before the wrapper yields:

@override
async def _run_async_impl(self, ctx) -> AsyncGenerator[Event, None]:
    redirect_event = None
    async for event in self.main_agent.run_async(ctx):
        if should_block(event):
            redirect_event = create_redirect_event(ctx)
            break  # inner generator closed here, in same async context
        yield event
    if redirect_event is not None:
        yield redirect_event

Note: The break workaround may or may not resolve the issue depending on whether Python's async for cleanup after break preserves the contextvars.Context. In our testing, it reduces but may not fully eliminate the errors.

Suggested fix

Consider wrapping the tracing in base_agent.run_async() with GeneratorExit-safe cleanup:

async def run_async(self, parent_context):
    with tracer.start_as_current_span(f'invoke_agent {self.name}') as span:
        ctx = self._create_invocation_context(parent_context)
        tracing.trace_agent_invocation(span, self, ctx)
        try:
            if event := await self._handle_before_agent_callback(ctx):
                yield event
            if ctx.end_invocation:
                return
            async with Aclosing(self._run_async_impl(ctx)) as agen:
                async for event in agen:
                    yield event
            if ctx.end_invocation:
                return
            if event := await self._handle_after_agent_callback(ctx):
                yield event
        except GeneratorExit:
            # Suppress OTel context detach errors during generator cleanup
            span.end()
            return

Or use an async-generator-safe tracing wrapper that doesn't rely on contextvars.Token for cleanup.

Related issues

• Bug: Failed to detach context #423 (closed) — same error, different trigger (parallel tool downloads)
• ERROR:opentelemetry.context:Failed to detach context #1670 (closed) — same error, SSE streaming context
• Context Detachment Errors with Human-in-the-Loop Tool Confirmation #3607 (closed) — same error, Human-in-the-Loop tool confirmation
• Support for Disabling or Extending Internal OpenTelemetry Tracing in Google ADK (Python) #2792 — request to disable/extend internal OTel tracing

[alexmojaki]

The suggested fix doesn't work:

from opentelemetry.sdk.trace import TracerProvider
import asyncio
from opentelemetry.trace import get_current_span

tracer_provider = TracerProvider()
tracer = tracer_provider.get_tracer('my_tracer')


async def run_async():
    with tracer.start_as_current_span(f'invoke_agent') as span:
        try:
            yield 1
            yield 2
        except GeneratorExit:
            raise Exception("This doesn't actually get triggered when the loop breaks")
            span.end()
            return


async def main():
    async for value in run_async():
        print(value)
        break

    print(get_current_span())


asyncio.run(main())

The exception is still logged, and also the span remains active:

Failed to detach context
Traceback (most recent call last):
  File "/Users/alex/work/logfire/.venv/lib/python3.12/site-packages/opentelemetry/trace/__init__.py", line 589, in use_span
    yield span
  File "/Users/alex/work/logfire/.venv/lib/python3.12/site-packages/opentelemetry/sdk/trace/__init__.py", line 1105, in start_as_current_span
    yield span
  File "/Users/alex/Library/Application Support/JetBrains/PyCharm2025.3/scratches/scratch_2774.py", line 10, in run_async
    yield 1
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alex/work/logfire/.venv/lib/python3.12/site-packages/opentelemetry/context/__init__.py", line 155, in detach
    _RUNTIME_CONTEXT.detach(token)
  File "/Users/alex/work/logfire/.venv/lib/python3.12/site-packages/opentelemetry/context/contextvars_context.py", line 53, in detach
    self._current_context.reset(token)
ValueError: <Token var=<ContextVar name='current_context' default={} at 0x10d912070> at 0x10dc7c180> was created in a different Context
1
_Span(name="invoke_agent", context=SpanContext(trace_id=0x8500018b8cdc046f8cd55c3405cf026f, span_id=0x6529f990576b69fe, trace_flags=0x01, trace_state=[], is_remote=False))

See https://logfire.pydantic.dev/docs/reference/advanced/generators/

The fact is you can't wrap generators in spans like this. This means the async for event in runner.run_async interface itself has to change into one where the user first opens a context manager, so that they can reliably signal when they're done.

[surajksharma07]

The root cause is that start_as_current_span spans the yield point inside run_async, so the OTel context token gets created in one contextvars.Context — but when the generator is abandoned early, the cleanup fires in a different one, which is what triggers the error.

One workaround worth trying: in base_agent.py, replace the with tracer.start_as_current_span(...) block with explicit tracer.start_span() + otel_context.attach().
Then in the finally clause, briefly push the opentelemetry.context logger level above CRITICAL before calling otel_context.detach(token), restore it afterwards, and call span.end().
That should suppress the ValueError from surfacing without leaking the span.

Could you give this a shot on your side and see if it consistently clears the errors in your pipeline?
If it holds up across both the early-exit and normal-completion paths, that'd be a solid foundation for a PR — just want to make sure it actually works in your setup before taking it further.

[alexmojaki]

No, the whole point is that the detach fails, hence the log. If you have to suppress the log, the error is still happening. The span still leaks.

from opentelemetry.sdk.trace import TracerProvider
import asyncio
from opentelemetry.trace import get_current_span, set_span_in_context
from opentelemetry import context

tracer_provider = TracerProvider()
tracer = tracer_provider.get_tracer('my_tracer')


async def run_async():
    with tracer.start_span(f'invoke_agent') as span:
        token = context.attach(set_span_in_context(span))
        try:
            yield 1
            yield 2
        finally:
            print("Detaching context")
            context.detach(token)


async def main():
    async for value in run_async():
        print(value)
        break
    print(get_current_span())


asyncio.run(main())

Failed to detach context
Traceback (most recent call last):
...
ValueError: <Token var=<ContextVar name='current_context' default={} at 0x1163da6b0> at 0x11676ff80> was created in a different Context
1
_Span(name="invoke_agent", context=SpanContext(trace_id=0xac52ae30130dcade960b8a26d26fe95d, span_id=0x3d0461f9ae583b45, trace_flags=0x01, trace_state=[], is_remote=False))
Detaching context

Note print(get_current_span()) printing _Span(name="invoke_agent", showing the leakage. Detaching context only gets printed after because the finally isn't triggered by the loop ending, it's triggered by the generator being garbage collected. That happens at an unreliable time in a different context, hence the error log.

If you do this:

async def main():
    g = run_async()
    async for value in g:
        print(value)
        break
    await g.aclose()
    print(get_current_span())

then it works correctly:

1
Detaching context
NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))

Note the change in order of the lines as well as the NonRecordingSpan. And then you can just go back to start_as_current_span without needing any special stuff in run_async.

[surajksharma07]

You're right — suppressing the log was just hiding the symptom while the span continued to leak underneath, which is the real problem.
And your point about the finally running at GC time in the wrong async context is exactly why my suggestion didn't hold up.

The explicit aclose() approach you showed is the correct fix — it forces the generator cleanup to happen in the right context, so the detach succeeds and the span closes cleanly.

Looks like mmphego has already filed PR #4919 with a fix — will take a look to make sure it covers both the early-exit path and normal completion correctly.

[surajksharma07]

@jawoszek Please have a look into it once.