feat(otel): add ExecutionOtelPlugin with Workflow-rooted trace hierarchy#576
Draft
SilanHe wants to merge 16 commits into
Draft
feat(otel): add ExecutionOtelPlugin with Workflow-rooted trace hierarchy#576SilanHe wants to merge 16 commits into
SilanHe wants to merge 16 commits into
Conversation
Port aws-durable-execution-sdk-js#729 (ExecutionOtelPlugin) to Python. Adds a self-contained execution-view plugin producing the deterministic span hierarchy Workflow -> Invocation -> Operation -> Attempt, stitching one trace across every Lambda invocation of a durable execution. The Workflow span is the root (empty context, deterministic span ID from the execution ARN) and is exported only on a terminal invocation status; non-terminal statuses drop it. Operations are parented under the Workflow span and linked to the current invocation span. Shared infrastructure is extracted so both plugins reuse it: - execution_plugin_config.py: config + exporter dataclasses - provider.py: create_tracer_provider (explicit/global/auto OTLP) - instrumentations.py: register_standalone_instrumentations - deterministic_id_generator.py: derive_workflow_span_id() pyproject gains an optional "instrumentation" extra for the auto-configured provider path. Tests: 22 new (test_execution_plugin.py, test_provider.py); full otel suite passes.
Package is in Alpha, so the bare OtelPlugin name and its backward-compat alias are removed. InvocationOtelPlugin is now the concrete class, matching the JS SDK naming (aws-durable-execution-sdk-js#729). Updated all references across source, tests, examples, and README; deduped the package __all__/imports. All otel unit tests pass.
SilanHe
force-pushed
the
feat/execution-otel-plugin
branch
from
July 23, 2026 17:47
eb1e0b5 to
d477917
Compare
Collapse now-shorter call lines and drop a trailing blank line after the OtelPlugin -> InvocationOtelPlugin rename. No behavior change; hatch fmt --check is clean across all packages.
The types env now installs opentelemetry-instrumentation-botocore and -urllib3 so mypy resolves the optional instrumentation imports instead of reporting import-not-found.
ProviderResult.tracer_provider was typed as bare object, so mypy rejected get_tracer() and the register_standalone_instrumentations call. Type it as the OTel API TracerProvider (the common type of all three resolution paths), align the register parameter, and annotate the urllib3 kwargs dict as dict[str, Any].
install_log_filter and OtelContextLogFilter were typed to accept only InvocationOtelPlugin, but ExecutionOtelPlugin (a sibling class) is also passed. Both only need get_current_span_context(), so accept a structural _SpanContextProvider Protocol instead of a concrete class. This also drops the TYPE_CHECKING import cycle on the plugin module. mypy now passes with no errors.
Contributor
Author
The module holds InvocationOtelPlugin, so name it to match (mirrors execution_plugin.py). Updated the four importers; the core SDK aws_durable_execution_sdk_python.plugin module is untouched.
Previously ExecutionOtelPlugin skipped the invocation span when using the default tracer provider. Per aws-durable-execution-sdk-js#756, now always create it: in default-provider mode it is parented to the ambient Lambda invocation span (ADOT/auto-instrumentation) and carries only durable correlation attributes. _build_invocation_links falls through to the invocation span when no ambient context is present. Adds two default-mode tests.
Rename module execution_plugin_config.py to otel_plugin_config.py and class ExecutionOtelPluginConfig to OtelPluginConfig, so the name reflects that the config is shared by both ExecutionOtelPlugin and InvocationOtelPlugin. Updated all importers and references.
ExecutionOtelPlugin defensively ended still-open operation spans in on_invocation_end. Those belong to suspended (PENDING/RETRYING) operations, so ending them exported a misleading completed span. Match JS PR #756: drop the references without ending them (_reset_state clears the map); operation spans are ended only by on_operation_end. Adds a test that an unfinished operation span is not exported.
Aligns ExecutionOtelPlugin's private helper name with the equivalent _start_span on InvocationOtelPlugin. No behavior change.
Aligns ExecutionOtelPlugin's operation/attempt attribute key with InvocationOtelPlugin and JS PR #756: durable.operation.attempt becomes durable.attempt.number, matching the sibling durable.attempt.outcome.
SilanHe
force-pushed
the
feat/execution-otel-plugin
branch
from
July 23, 2026 21:48
9665a14 to
f295602
Compare
Match the module rename plugin.py -> invocation_plugin.py; these tests cover InvocationOtelPlugin. No content changes.
Strip 'JS Req <n>', 'JS PR #<n>', and 'Requirements <n>' annotations from comments and docstrings across the otel package. Comment-only; no behavior change.
SilanHe
commented
Jul 23, 2026
Comment on lines
+360
to
+370
| # Cross-invocation stitching: operation started in a prior | ||
| # invocation. Create + immediately end a linked span. | ||
| parent = self._resolve_parent(info.parent_id) | ||
| span = self._start_span( | ||
| operation_id=info.operation_id, | ||
| name=info.name or info.operation_id, | ||
| info=info, | ||
| parent=parent, | ||
| start_time=info.start_time, | ||
| existed=True, | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
I'm actually not sure what to do here since I'm not very familiar with how Python is resolving the parent operation span when we create an attempt span. Since on_operation_start is only called when we want to checkpoint the operation's start, in retry scenarios I'm guessing there can be attempt spans that are started when the operation span doesn't exist. I'm opting for leaving this here for now and solving it in a separate PR. I've also raised this in #578 (comment) which I think is trying to fix a related issue.
Contributor
Author
There was a problem hiding this comment.
I won't add any tests until this is fixed since I think it's a key problem that the Execution view has in Python
Contributor
There was a problem hiding this comment.
I'm actually not sure what to do here since I'm not very familiar with how Python is resolving the parent operation span when we create an attempt span.
Actually in Python we don't resolve a parent. Python just ensures parent span is always created before the child span.
ExecutionOtelPlugin created a separate continuation span (random ID plus a link back to the deterministic logical span) on replay/cross-invocation. Remove that concept: operation spans always use the deterministic span ID so a suspended-then-completed operation exports a single span on completion, matching the JS reference. Updates the cross-invocation test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes: Port aws-durable-execution-sdk-js#729 (ExecutionOtelPlugin) to Python.
Adds a self-contained execution-view plugin that produces the deterministic span hierarchy Workflow -> Invocation -> Operation -> Attempt, stitching one trace across every Lambda invocation of a durable execution. The Workflow span is the root (created in an empty context, deterministic span ID derived from the execution ARN) and is exported only on a terminal invocation status; non-terminal statuses (PENDING/RETRY) drop it. Operations are parented under the Workflow span and linked to the current invocation span.
Shared infrastructure is extracted so both plugins reuse it:
The existing OtelPlugin is aliased as InvocationOtelPlugin to match the JS rename while preserving the published public API. pyproject gains an optional "instrumentation" extra for the auto-configured provider path.
Tests: 22 new (test_execution_plugin.py, test_provider.py); full otel suite 76 passed.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.