Repository navigation
fix(telemetry): align records with the JS exporter - #803
ParidelPooya wants to merge 11 commits into
Conversation
The JS exporter is the reference shape that the record readers use for
every SDK. Align four Python record fields with it:
- error.name is the simple type name of the user's error (for example
PaymentError), not the module-qualified type or the SDK wrapper. An
SDK wrapper resolves to the failed operation's user error; with no
user error the SDK error's simple class name is used.
- An opted-in operation result is the plain JSON value. Results that
the default serializer wrote as a {"t", "v"} envelope are decoded
with it and converted to JSON values. Other payloads are unchanged,
and a decode failure falls back to the previous value.
- Every record built at invocation end carries endTime and durationMs,
including RUNNING records for a suspend or retry, as in JS.
- LambdaLogExporter writes the Lambda JSON log envelope (timestamp,
level, message) itself, because Lambda does not wrap Python print
output. The size limit still applies to the record.
| if not _is_extended_type_envelope(parsed): | ||
| return parsed | ||
| try: | ||
| decoded = EXTENDED_TYPES_SERDES.deserialize(raw_result, SerDesContext()) | ||
| return _to_json_compatible(decoded) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
/ai verdict out-of-scope
The hooks do not say which serializer wrote a result, so telling a JsonSerDes payload from an ExtendedTypeSerDes envelope needs serializer provenance from the core; the shape-based decoding stays as documented in _decode_checkpoint_result.
There was a problem hiding this comment.
Recorded out-of-scope for AI finding arf_v1_xhn5b5nmzbwy2pb5r35uf3l3zq.
| for op in operations.values() | ||
| if (op_error := op.error) is not None | ||
| and op_error.type | ||
| and not op_error.type.startswith(_SDK_TYPE_PREFIX) | ||
| and (op_error.message, op_error.data, op_error.stack_trace) == key |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment has been minimized.
This comment has been minimized.
Implement the error.name rule shared with the JS and Java exporters,
and fix two problems in the previous cause lookup.
- A qualified type is cut to its last segment after any "." or "$".
An empty segment keeps the string ("trailing.", "Outer$"), and a
blank or missing type is "Error". Whitespace is not trimmed.
- An SDK wrapper is resolved only from the failing operation's own
subtree. For the execution error, that is the failed operation of
the wrapper's kind with the same error fields. Before, any failed
operation in the execution with the same message could be the
cause, including a failure that user code had already handled.
- Non-wrapper SDK errors (ValidationError, SerDesError, ...) are
candidates. More than one distinct candidate type means no guess.
- The chain walk is limited to 16 levels and stops on a cycle.
- CallbackExternalError is a wrapper: an external failure with a
type is named by that type, and one without a type is
CallbackExternalError. A timed-out callback operation is
CallbackTimeoutError, as on its context and the execution.
- The lookup index is built once per record, so a record build is
linear in the number of operations. Before, each failed operation
scanned all operations: 2000 failed map items took 800 ms, and now
take 15 ms.
Tests: the shared Table A vectors are copied unchanged to
tests/fixtures/error-name-vectors.json and drive a parametrized test.
Every Table B row runs through the local runner, live and replayed.
BREAKING CHANGE: error.name values change for records with errors, and
a blank or missing type is now reported as "Error".
Opted-in results decoded from the default serializer lost Decimal
precision and sign, and non-finite floats made the record invalid
JSON.
- An integral Decimal is a JSON integer. Any other finite Decimal is
a JSON number only when that float converts back to the same
Decimal, and otherwise its exact string, str(value).
- NaN and Infinity, as Decimal or float, become null in decoded and
in plain JSON payloads, so the emitted record is strict JSON.
- The README and docstring describe the envelope collision: the hooks
do not carry the serializer, so a {"t", "v"} object written by
another SerDes with a default type tag is decoded as an envelope.
The README no longer says that custom SerDes payloads always pass
through.
BREAKING CHANGE: a Decimal result that a float cannot hold exactly is
now a string, and NaN or Infinity results are now null.
The LambdaLogExporter size limit is measured on the record, and the envelope and JSON escaping of message add bytes on top of it. State how much: about 20% for a typical record and up to about twice the record in the worst case, since escaping doubles each quote and backslash. CloudWatch Logs accepts events of up to 1 MB (PutLogEvents), so a line for a record at the default 256 KB limit fits.
| decoded = EXTENDED_TYPES_SERDES.deserialize(raw_result, SerDesContext()) | ||
| return _to_json_compatible(decoded) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
/ai verdict out-of-scope
The hooks do not say which serializer wrote a result, so telling a JsonSerDes payload from an ExtendedTypeSerDes envelope needs serializer provenance from the core; the shape-based decoding stays as documented in _decode_checkpoint_result.
| if value == value.to_integral_value() and not ( | ||
| value.is_zero() and value.is_signed() | ||
| ): | ||
| return int(value) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| for op in self._by_key.get(_error_key(error), ()): | ||
| if allowed is None or op.operation_type in allowed: | ||
| causes |= self._subtree_causes(op) | ||
| if len(causes) == 1: | ||
| return causes.pop() |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| if op.operation_type == OperationType.CALLBACK and not message: | ||
| # The SDK fills in a default message when the callback error has none. | ||
| is_timeout = op.status == OperationStatus.TIMED_OUT | ||
| message = "Callback timed out" if is_timeout else "Callback failed" | ||
| return (message, data, stack) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment has been minimized.
This comment has been minimized.
An integral Decimal with a large exponent, such as Decimal("1E+1000000"),
was converted with int(). That took tens of seconds on the handler thread,
and json then refused to write the integer, so the whole record was lost.
An integral Decimal now becomes a JSON integer only when it has at most 300
digits (adjusted() < 300). A larger one takes the float-or-string path and is
reported as its exact string, for example "1E+5000". The README now says that
a Decimal keeps its numeric value but not its scale.
Rule 3: the depth now counts wrapper hand-offs, not tree levels. One unwrap is a failed direct child with the same error fields, or a failed item or branch under a direct map or parallel child. At most 16 unwraps happen; the error reached then is reported. A cycle reports the last error visited before the repeat. The execution takes the name of the outermost operation it propagated from, so rule 8 holds at the bound. Children that do not carry the wrapper's fields are no longer searched, so a failure that user code handled inside the failing subtree no longer makes the name vague. Rule 6: blank means only the six ASCII whitespace characters. U+00A0, U+FEFF, U+001C and U+2003 are kept (vectors A13-A17). The vector file is replaced with version 2. A wait_for_callback submitter that fails with an empty message is re-raised by the core as "Callback submitter failed"; the execution now matches it to the submitter's context. A callback failed externally with the type "Error" is named Error (B13b), now covered by tests.
Adds B13b, B20 and B21, the 16/17/18 nested child context bound, a map in a map in a child context, and a failure handled inside the failing subtree. A callback failed externally with an empty message and a submitter that fails with an empty message exercise the core's real default messages, which the plugin has to match. The harness now waits for a terminal-status record and asserts on the last one, instead of reading the last record after the first arrives.
A handled failure that is a direct child of the failing operation, with the same message and data, cannot be told apart from the cause. The README now says the name is then the wrapper's, and a test pins it.
| if not _is_extended_type_envelope(parsed): | ||
| # Plain JSON already; only non-finite numbers (json.loads accepts NaN | ||
| # and Infinity) need mapping to null. | ||
| return _to_json_compatible(parsed) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Fixed in 459216c: _to_json_compatible now converts containers with an explicit stack (no recursion), and a normalization failure falls back to that result's raw string, so the rest of the record is still emitted. Tests: test_result_deeply_nested_plain_json_is_converted, test_json_conversion_does_not_recurse, test_result_normalization_failure_is_isolated_to_that_result.
| try: | ||
| return transform(parsed) | ||
| except Exception: # noqa: BLE001 - untrusted transform must never break emission | ||
| decoded = EXTENDED_TYPES_SERDES.deserialize(raw_result, SerDesContext()) | ||
| return _to_json_compatible(decoded) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
/ai verdict out-of-scope
The hooks do not say which serializer wrote a result, so telling a JsonSerDes payload from an ExtendedTypeSerDes envelope needs serializer provenance from the core; the shape-based decoding stays as documented in _decode_checkpoint_result.
| if ( | ||
| error_type == _CALLBACK_SUBMITTER_TYPE | ||
| and message == _CALLBACK_SUBMITTER_MESSAGE | ||
| ): | ||
| # The submitter step failed with no message; the core re-raised it | ||
| # with its default message, and the context keeps the empty one. | ||
| keys += [("", data, stack), (None, data, stack)] |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| for child in self._children.get(op.operation_id, ()): | ||
| if self._keys[child.operation_id] == key: | ||
| links.append(child) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment has been minimized.
This comment has been minimized.
The core now checkpoints the error a wrapper wraps, so every failed operation and the execution carry the user's type. Name an operation by its own ErrorType, reduced (Table A), with the callback cases: TIMED_OUT is CallbackTimeoutError and an external failure with a missing or blank type is CallbackExternalError. Name the execution by its own ErrorType, reduced; a wrapper type from an older core reports the wrapper's name (rule 9). Remove the cause matching: the error-name index, the subtree walk, the per-kind link filters, the default-message equivalences and the ambiguity unions. B20 and B21 now name the execution after the error that throw_if_error raised. Convert decoded results to plain JSON with an explicit stack, so a deeply nested result does not hit the recursion limit, and fall back to the raw string when one result fails to normalize, so the rest of the record is still emitted.
| value.adjusted() < _MAX_INT_DIGITS | ||
| and value == value.to_integral_value() | ||
| and not (value.is_zero() and value.is_signed()) | ||
| ): | ||
| return int(value) |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_hovewbithq67gm6ebbmi5dp6ix
[P2] Preserve DynamoDB compatibility when converting Decimals
With an identity result override, a step returning Decimal("9" * 39) now produces a 39-digit integer. DynamoDBExporter._marshal() passes that integer to boto3's TypeSerializer, which rejects numbers exceeding its 38-digit precision; the scheduler consequently drops the entire record. Previously, the decimal was a string inside its serializer envelope and exported successfully. Convert numbers outside DynamoDB's supported precision/range to exact strings during marshalling, or preserve them as strings here, and add a plugin-to-DynamoDB regression test.
To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.
Codex AI reviewTwo P2 findings in Suppressed repeats
Reviewed commit |
|
Closing in favour of a simpler approach. We no longer try to make |
Summary
The JS Durable Execution Telemetry exporter is the reference record shape: the JS telemetry readers (
-telemetry-coreand-telemetry-mcp) read every SDK's records. This PR aligns four Python fields with it, and implements theerror.namerule that the JS (aws/aws-durable-execution-sdk-js#1052) and Java (aws/aws-durable-execution-sdk-java#814) exporters implement too. Everything else is unchanged: timestamp precision, sampling, ordering, the custom exporter interface, and names.Stacked on #807 (
fix: checkpoint the wrapped error for failed contexts). This PR's base isfix/checkpoint-wrapped-error, and that branch is merged in, so the diff shows only the exporter changes. Merge #807 first; this PR then retargets tomain.All changes are in
packages/aws-durable-execution-sdk-python-telemetry-exporter:plugin.py,exporters/lambda_log_exporter.py, the README and the tests.The package is at
0.0.1. These go under "Breaking Changes" in the next telemetry-exporter release notes.lintcommit.pyrejects a!after the scope, so the commits carry aBREAKING CHANGE:footer instead.error.namevalues change for every Python record that has an error (executionerror,operations[].error,operationsByName.<name>.error). Before: the module-qualified type, or the SDK wrapper (aws_durable_execution_sdk_python.exceptions.StepError). After: the shared rule below, for examplePaymentError. A blank or missing type is nowError, not""/null.error.messageis unchanged.{"t", "v"}envelopes are decoded, and overrideresulttransforms receive the plain value. ADecimalthat a float cannot hold exactly is now a string, and NaN/Infinity are nownull.endTime/durationMs(on-changemode). Do not use the presence ofendTimeas a completion signal; usestatus.LambdaLogExporterstdout line is the Lambda JSON log envelope (timestamp,level,message). Consumers thatjson.loadsthe line and read record fields directly must parsemessagefirst.1.
error.nameThis implements version 3 of the shared rule (vectors file still
version: 2, unchanged). In version 3 the core SDK checkpoints the wrapped error, not the wrapper, which is what #807 does for Python. Every failed operation and the execution therefore carry the user's type, and the exporter only reduces a type string.The rule, in the same words in all three READMEs:
.or$. If that segment is empty, the string is kept as it is. A missing type, or a type made only of blank characters, isError. The blank characters are the six ASCII whitespace characters: space,\t,\n,\r,\fand\v. No other character counts as blank, and names are never trimmed.error.nameequals that operation'serror.name.A callback that is failed externally with an explicit
ErrorTypeis named after that type, after the reduction of rule 6. This includes the type"Error", which reportsError. Only a missing or blank type reports the SDK callback-failure class.How Python applies it:
ErrorObject.from_exceptionunwrapsStepError,ChildContextError,InvokeError,WaitForConditionError,CallbackSubmitterError,CallbackExternalErrorandDurableOperationError(exact class, at most 16 steps, cycle-safe).CallbackTimeoutError,SerDesError,ValidationError,BatchCompletionErrorand user subclasses are not unwrapped. A callback failed externally keeps the sender's type, so a missing type and"Error"stay different._simple_type_name(operation.error.type), with two callback cases: aTIMED_OUTcallback operation isCallbackTimeoutError(the service typeCallback.Timeoutwould reduce toTimeout), and a callback failed externally with a missing or blank type isCallbackExternalError._simple_type_name(execution error type). With fix: checkpoint the wrapped error for failed contexts #807 that is the user's type. A wrapper type from an older core reduces to the wrapper's name (StepError,ChildContextError, ...), per rule 9.Smaller design
Removed from
plugin.py:_ErrorNameIndex(children-by-parent and error-key indexes, subtree links, BFS with the 16-unwrap budget, cycle handling and caching, outermost-candidate search), the wrapper-to-operation-kind filter table, the default-message equivalences (Callback failed,Callback timed out,Callback submitter failed) and the ambiguity unions. What is left is about 60 lines: the reducer, the callback special cases and two one-line name functions. Tests that only covered the removed heuristics (subtree scoping, kind filters, ambiguity, nested-map unwrap counting, tree cycles, default-message matching, the linear-cost and 2000-item timing checks) are gone.git diff --stat fix/checkpoint-wrapped-error...fix/telemetry-align-with-js:45582afb)plugin.pygoes from +452 to +262 lines against main;tests/test_error_name.pyfrom 728 to 400.Table A (shared vectors)
tests/fixtures/error-name-vectors.json(version 2, unchanged, byte-identical in the three repositories).tests/test_error_name.pyruns every vector through the reducer and through a full record (execution,operations[],operationsByName). The vectors changed by this PR:trailing.""trailing.Outer$""Outer$./$""" PaymentError """," ",None""/""/NoneErrorErrorErrorError(str.strip())Error(str.strip())Error(str.strip())Table B (local runner, live and replayed, on this stacked branch)
tests/e2e/error_name_int_test.pyruns every row throughDurableFunctionTestRunnertwice. The first run is live. In the second, the handler catches the error, waits so that a new invocation starts, and rethrows it. Each row asserts the names below, thaterror.messageis the original message, that the live and replayed names are equal, and thatoperationsByNamematchesoperations[]. All rows pass in both modes (expected = actual).error.namePaymentErrorcharge:PaymentErrorPaymentErrorTypeError, then on retryPaymentErrorcharge:PaymentErrorPaymentErrorcheckout:PaymentErrorPaymentErrorcheckout,charge:PaymentErrorPaymentErrormap-item-0:PaymentErrorPaymentErrormap-item-0,charge:PaymentErrorPaymentErrorparallel-branch-1:PaymentErrorPaymentErrorwait_for_conditioncheck throwspoll:PaymentErrorPaymentErrorpay:PaymentErrorPaymentErrorwait_for_callbacksubmitter throwsapproval,approval submitter:PaymentErrorPaymentErrorapproval,approval create callback id:CallbackTimeoutErrorCallbackTimeoutErrorcom.acme.PaymentErrorapproval,approval create callback id:PaymentErrorPaymentErrorapproval,approval create callback id:CallbackExternalErrorCallbackExternalErrorErrorapproval,approval create callback id:ErrorErrorPaymentErrorwith causeTypeErrorPaymentErrorStepErrorWrappedPaymentErrorValueError("x")in a step, then a child fails withValidationError("x")lookup:ValueError;work:ValidationErrorValidationErrorcharge:SerDesErrorSerDesErrorPayment$Errorthrown from a stepcharge:ErrorErrorpromise.all)PaymentError, item 1FraudErrormap-item-0:PaymentError;map-item-1:FraudErrorPaymentError(raised bythrow_if_error)PaymentError,FraudErrorparallel-branch-0:PaymentError;parallel-branch-1:FraudErrorPaymentError(raised bythrow_if_error)B20 and B21 run with
max_concurrency=1andCompletionConfig.all_completed(), so both items fail and item 0's error is the onethrow_if_error()raises.Extra rows in the same test:
error.namePaymentError()approval,approval submitter:PaymentError(message"")PaymentError(messageCallback submitter failed)com.acme.PaymentError, message""approval:PaymentError(messageCallback failed);approval create callback id:PaymentError(message"")PaymentErrorouter, childinnercatches a step'sValueError; then stepchargethrowsPaymentErrorouter,charge:PaymentError;lookup:ValueErrorPaymentErrorPaymentErrorinside 18 nested child contextsPaymentErrorPaymentErrorPaymentErrorPaymentErrorThe tree depth no longer limits the name: each context records the user's type, so the 17- and 18-level rows that reported
StepError/ChildContextErrorin version 2 now givePaymentErroreverywhere. The 16-unwrap bound and cycle handling live in the core (#807 unit tests). The slim exporter depends on #807: run against themaincore, 21 of the 26 rows fail (the contexts and the execution report the wrapper, as rule 9 allows for old checkpoints). Only B11, B13, B14, B15 and B17 pass there.2. Opted-in operation
resultoperations[charge].result{"t":"m","v":{"amount":{"t":"f","v":99.99}}}{"amount":99.99}This applies to
operations[].result,operationsByName.<n>.result, and the value passed to an override'sresulttransform. AnExtendedTypeSerDesenvelope (root is exactly{"t", "v"}, withta string) is decoded withEXTENDED_TYPES_SERDESand converted to JSON values:Decimalkeeps its numeric value, not its scale or trailing zeros (1.10becomes1.1,1E+2becomes100). An integral value of at most 300 digits (adjusted() < 300) is an integer; a larger one takes the path below, so1E+5000and1E+1000000become the strings"1E+5000"and"1E+1000000"in milliseconds. Converting those withint()took tens of seconds, and the record was then dropped becausejsonrefuses integers over 4300 digits. Any other value is a float only whenDecimal(repr(float(v))) == v, and otherwisestr(v). So12345678901234567.89becomes"12345678901234567.89", and1E-400stays"1E-400"instead of0.0.Decimalorfloatvalues becomenull, in decoded and plain payloads alike, so the line is strict JSON.datetime/datebecomeisoformat()strings,UUIDa string,bytesbase64,tuplea list, andBatchResultitsto_dict()form.A payload that is not an envelope stays parsed JSON, or the raw string when it is not JSON. An envelope that fails to decode stays parsed JSON. Decoding never raises. The conversion uses an explicit stack, not recursion, so a deeply nested result (for example 600 nested arrays on Python 3.11) is converted. If converting one result fails anyway, that result falls back to its raw string and the rest of the record is still emitted.
Known collision (documented in the README). The hooks do not say which serializer wrote a result, so an envelope is recognised by shape alone. A
JsonSerDes, custom-SerDesor chained-invoke result that is exactly{"t": <default tag>, "v": ...}is decoded as if the default serializer wrote it. For example{"t":"s","v":"x"}becomes"x". An unknown tag passes through unchanged. Fixing this needs the serializer onOperationInfoin the core SDK.3.
endTime/durationMson non-terminal recordsJS
onInvocationEndpassesendTime: new Date()for every status (index.ts). Python now does the same: every record built at invocation end carriesendTimeanddurationMs, including the RUNNING record of a suspend or retry. Records built at invocation start or on an operation change have neither. Output and error are only set on terminal records.4.
LambdaLogExporterlineIn the JSON log format, the Node.js runtime wraps
console.logoutput in this envelope. The Python runtime capturesprintas plain text (Python logging) and does not double-encode JSON (log format), so the exporter writes the envelope itself:timestampis UTC with milliseconds andZ.levelisINFO.messageis the compact record JSON.requestIdis omitted, because exporters do not receive it.The size limit is still measured on the record (
message). The printed line is larger: about 20% for a typical record, and up to about 2x in the worst case, because escaping doubles each"and\. CloudWatch Logs accepts events of up to 1 MB (PutLogEvents), so a line for a record at the default 256 KB limit fits. The README and the docstring say so.Testing
Mirrors
.github/workflows/ci.ymllocally (Python 3.14, macOS; CI runs 3.11 to 3.14), on the stacked branch:hatch fmt --check(repository root, 750 files)hatch run types:checkhatch run test:cov(full)plugin.pyare covered; the uncoveredplugin.pylines predate this PRhatch buildin the core and telemetry-exporter packagespython .github/scripts/lintcommit.py --range origin/main..HEADNew or changed tests:
tests/fixtures/error-name-vectors.json: the shared Table A vectors, version 2.tests/test_error_name.py: Table A against the reducer and full records; the core records the wrapped type for every wrapper class; each operation named by its own type (B16); B20/B21 snapshots; a user subclass of a wrapper (B15); old checkpoints holding a wrapper type report the wrapper and never a child's type (rule 9); callback timeout, external failure with a type, no type,"Error", blank types and a non-ASCII space;include_errors=False.tests/e2e/error_name_int_test.py: Table B (B19 not applicable) and the extra rows, live and replayed, with the local runner.tests/test_plugin.py: exactDecimalvalues, NaN/Infinity tonull, 600 nested arrays, a 20000-level conversion without recursion, dict order and duplicate keys, and a normalization failure isolated to one result.