Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 122 additions & 2 deletions telemetry-exporter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,30 @@ one event per record to `{logStreamPrefix}YYYY/MM/DD` (IAM: `logs:CreateLogStrea
it is deliberately *not* a type-preserving clone. Mutating the argument is safe (it cannot corrupt
the cached input snapshot or any later emission), and a transform that throws omits the field and
logs the failure rather than silently dropping it or failing the execution.
- **`includeErrors`** gates **both** the execution-level error and each operation-level error; with
`includeErrors(false)` neither is emitted, so a sensitive failure message never reaches a record.
- **`includeErrors`** gates only the per-operation errors (`operations[].error` and
`operationsByName.<name>.error`), as in the JS exporter. **`includeErrors(false)` does not hide
the execution-level `error`**: its `name` and `message` are emitted on every FAILED record. In
2.3.0 and earlier, `includeErrors(false)` also suppressed the execution error.
In Java the execution error is usually the failed operation's own exception rethrown, so its
message is usually the operation's message too. No setting suppresses the execution error
message, and the input/output transforms do not apply to errors. Do not put secrets or personal
data in exception messages that can reach the handler's return path: catch such an exception and
rethrow one with a safe message, or, in a custom `TelemetryExporter`, remove or redact `error` in
`render`.
- **`error.name`** follows the cross-SDK rule in [`error.name`](#errorname) below.
- **`LambdaLogExporter` line shape.** Each record is printed to stdout as one Lambda JSON log line,
`{"timestamp":"<UTC, 3 fraction digits>Z","level":"INFO","message":"<compact record JSON>"}`,
because the JS readers of this destination filter on `level` and parse `message`. The Node
runtime adds this envelope around `console.log`; Java `System.out` is not wrapped, and Lambda
does not re-wrap a line that is already JSON. `maxRecordSizeBytes` caps the printed line,
envelope included. The record is embedded in `message` as a JSON string, so every `"` and `\`
in it is escaped again and the line can be up to about twice the record. When the line is over
the limit, the exporter truncates the record again, measuring the escaped envelope, until the
line fits. With the default 256,000, or any limit up to 1 MB, every line is within the
CloudWatch Logs limit of 1 MB per log event.
`LambdaLogExporter` lines are INFO; an application log level of WARN or above drops them.
This line shape is a breaking change: 2.3.0 printed the bare record JSON, so a reader that
parsed each stdout line as a record must now parse `message`.
- **Non-fatal plugin failures are isolated.** Record construction, snapshotting, transforms,
truncation and exporter render/export/flush contain and log ordinary failures, including
`AssertionError` and optional-dependency `NoClassDefFoundError`. `VirtualMachineError` and
Expand All @@ -216,6 +238,104 @@ one event per record to `{logStreamPrefix}YYYY/MM/DD` (IAM: `logs:CreateLogStrea
Concurrent flush requests may share one flush, and the final record is delivered before the
invocation returns.

## `error.name`

This rule is shared by the JS, Python and Java exporters. `error.message` is never changed by
this rule. The rule applies to the execution-level `error`, every `operations[].error`, and every
`operationsByName.<name>.error`.

1. A record names the error that caused the failure. A wrapper error that the SDK added is never
the name, except in the bounded cases of rules 3 and 9.
2. The SDK unwraps an error only when the error's type is one of the SDK's own wrapper types. A
user error is never unwrapped, even if it has a cause.
3. Unwrapping follows the chain of SDK wrappers. The depth counts unwrap steps, one per wrapper
passed, not levels in the operation tree. At most 16 unwraps happen. If the error reached after
16 unwraps is still a wrapper, its name is reported. If the chain reaches an error it has
already visited, the walk stops and the last error visited before the repeat is reported.
4. If the chain ends at a user error, the name is that error's type name.
5. If the chain ends at an SDK error that is not a wrapper, the name is that SDK error's simple
class name. A callback timeout is an example.
6. A type name that is a qualified string is reduced to its last segment after any `.` or `$`. If
that segment is empty, the string is kept as it is. A missing type, or a type made only of blank
characters, is `Error`. The blank characters are the six ASCII whitespace characters: space,
`\t`, `\n`, `\r`, `\f` and `\v`. No other character counts as blank, and names are never
trimmed.
7. A failure gets the same name whether the record is built live or from a replayed checkpoint.
8. When the execution fails because an operation's error propagated, the execution `error.name`
equals that operation's `error.name`.
9. If a checkpoint holds only a wrapper type, which an older SDK can write, the name is that
wrapper's name. The exporter never guesses a cause from other operations.

The core SDK checkpoints the wrapped error, not the wrapper. When a child context, map item or
parallel branch fails because an SDK wrapper error escaped it, the Java SDK writes the original
error's `ErrorType`, `ErrorMessage`, `ErrorData` and `StackTrace` into that context's checkpoint.
An SDK error that names the failure itself (a timed-out callback, a timed-out or stopped invoke) is
written under its own class name. A `waitForCallback` context keeps the callback's own error. So
the exporter names an operation by reducing its own `ErrorType` (rule 6) and names the execution
by unwrapping its error's cause chain (rules 2 to 5). It keeps no matching by message, timestamp or
tree position.

A callback that is failed externally with an explicit `ErrorType` is named after that type, after
the reduction of rule 6. This includes the type `"Error"`, which reports `Error`. Only a missing or
blank type reports the SDK callback-failure class.

User error types have the same name in every SDK, because the user names them. SDK error types
keep each language's own class name. For example, a callback timeout is `CallbackTimeoutError` in
JS and Python and `CallbackTimeoutException` in Java. Every SDK name for a callback timeout
contains `Timeout`. An invoke timeout does not always: Java names it `InvokeTimedOutException`, so
a `/Timeout/` query that should also match invoke timeouts needs `/Timeout|TimedOut/`.

How the Java exporter applies it:

- Rule 6 is applied to `Class.getName()` for live errors and to the checkpointed `errorType` for
replayed ones, so both give the same name. `com.acme.Outer$PaymentError` gives `PaymentError`;
a top-level class named `Payment$Error` gives `Error`; a local class `LocalError` declared in
`pkg.Foo` (binary name `pkg.Foo$1LocalError`) gives `1LocalError`; an anonymous class gives its
last segment, for example `1`. A string that ends with a separator (`trailing.`, `Outer$`) is
kept as it is. U+00A0, U+FEFF, U+001C and U+2003 are not blank, so a type made of them is kept.
- A hidden class's name ends in `/0x…`, a suffix that changes on every JVM start. Only a trailing
suffix matching `/0x[0-9a-fA-F]+$` is removed before rule 6, live and from the checkpoint, so
`pkg.Gen$HiddenErr/0x…` gives `HiddenErr`. Callback and invoke error types come from outside the
JVM and are never cut: `payments/v1.PaymentError` gives `PaymentError`.
- The SDK wrappers are an explicit list, matched on the exact runtime class:
`DurableOperationException`, `UnrecoverableDurableExecutionException`, `StepException`,
`CallbackException`, `InvokeException`, `StepFailedException`, `CallbackFailedException`,
`CallbackSubmitterException`, `ChildContextFailedException`, `InvokeFailedException`,
`MapIterationFailedException`, `ParallelBranchFailedException` and
`WaitForConditionFailedException`. A subclass of one of them, such as a user exception, is not a
wrapper and reports its own name. Every other class in `software.amazon.lambda.durable.exception`
is an SDK error that reports its simple class name: `CallbackTimeoutException`,
`InvokeTimedOutException`, `InvokeStoppedException`, `StepInterruptedException`,
`IllegalDurableOperationException`, `NonDeterministicExecutionException`, `SerDesException` and
`DurableExecutionException`.
- A timed-out callback, and a timed-out or stopped invoke, are SDK errors like the callback
timeout. The operation and the execution report `CallbackTimeoutException`,
`InvokeTimedOutException` or `InvokeStoppedException`, whatever error type the checkpoint
carries. A stopped invoke is not named after its stop error: a stop with `errorType`
`States.Stopped` or `com.acme.PaymentError` reports `InvokeStoppedException`. Inside a child
context the core checkpoints that SDK error as the context's error, so the invoke, every
enclosing context and the execution report the same name, live and replayed.
- After 16 unwraps, or at a cycle, the reported wrapper's name is its checkpointed `errorType`,
reduced; with a blank type it is the SDK failure class below.
- When a callback or invoke failed with no error type (for example a callback failed externally
without an `errorType`), the operation and the execution report the SDK exception the core throws
for it, such as `CallbackFailedException`. A failed step or context always has a type in its
checkpoint. A context checkpoint written by an older SDK without one reports the wrapper's name
for the execution and no operation error (rule 9).
- When the checkpoint has no error at all, the operation error has the SDK exception's name and a
`null` message, because there is no original message to keep. This is the case for a timed-out
callback and a timed-out invoke in the local test runner, and for a stopped invoke stopped
without an error. Before this change such operations had no `error` object; a reader that tests
`error != null` now sees one with `"message": null`. The execution error of a timed-out callback
has the message `Callback timed out: <callbackId>`, built by `CallbackTimeoutException`; it is not
checkpointed, and the JS and Python exporters report `Callback timed out` without the id.
- Java `map` and `parallel` return item and branch failures as data and do not throw them. Each
item and branch is named after its own type. For `parallel` the execution error is the error of
the branch whose future the handler calls `get()` on (Table B, B21); `map` has no `throwIfError`,
so B5, B6 and B20 have no propagated execution error in Java.
- Non-class type strings are reduced too: an external callback failure with `errorType`
`States.Timeout` reports `Timeout`.

## Conformance

Validated against the Durable Execution Telemetry conformance suite behaviors `telemetry-1 … telemetry-18`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,18 @@
*
* <p>Mirrors the JS {@code ContentConfig}. {@code input}/{@code output} may be included as-is, excluded, or
* transformed; all three are honored — the plugin reads execution input from {@code InvocationInfo.executionInput()}
* and output from {@code InvocationEndInfo.executionResult()}. {@code includeErrors} is honored for <em>both</em> the
* execution-level error and the per-operation error: with {@code includeErrors(false)} neither is emitted.
* Per-operation result opt-in is honored via {@link OperationOverride#withResult} (see
* {@code OperationChangeItemInfo.result()}).
* and output from {@code InvocationEndInfo.executionResult()}. Per-operation result opt-in is honored via
* {@link OperationOverride#withResult} (see {@code OperationChangeItemInfo.result()}).
*
* <p><b>{@code includeErrors} gates only the per-operation errors</b> ({@code operations[].error} and
* {@code operationsByName.<name>.error}), as in the JS exporter. With {@code includeErrors(false)} the
* <em>execution-level</em> {@code error}, including its message, is still emitted on every FAILED record. In Java the
* execution error is usually the failed operation's own exception rethrown, so its message is usually the operation's
* message too. Earlier releases of this exporter also suppressed the execution error; that is no longer the case. No
* setting suppresses the execution error message, and the input/output transforms do not apply to errors. Do not put
* secrets or personal data in exception messages that can reach the handler's return path: catch such an exception and
* rethrow one with a safe message, or, for a custom {@link TelemetryExporter}, remove or redact {@code error} in its
* {@link TelemetryExporter#render render}.
*
* <p><b>Transform value shape.</b> The {@code inputTransform} and {@code outputTransform} functions receive a
* <em>detached, JSON-compatible</em> copy of the value, never the SDK's original Java object: a POJO is presented as a
Expand Down Expand Up @@ -101,6 +109,14 @@ public Builder outputTransform(Function<Object, Object> transform) {
return this;
}

/**
* Whether per-operation errors ({@code operations[].error}, {@code operationsByName.<name>.error}) are emitted.
* Default {@code true}. {@code false} does <em>not</em> hide the execution-level {@code error} or its message;
* see the class documentation.
*
* @param include {@code false} to omit per-operation errors
* @return this builder
*/
public Builder includeErrors(boolean include) {
this.includeErrors = include;
return this;
Expand Down
Loading
Loading