feat(gax): implement active scope management and attribute validation for T3 attempt spans - #13977
feat(gax): implement active scope management and attribute validation for T3 attempt spans#13977jinseopkim0 wants to merge 3 commits into
Conversation
… for T3 attempt spans
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces OpenTelemetry context scope management to OpenTelemetryTracingTracer by storing the active scope during an attempt and closing it upon completion, as well as setting the span status to error when an attempt fails. However, the review highlights a critical thread-safety issue: storing the thread-local Scope in an instance variable and closing it asynchronously across different threads (e.g., from the application thread to the executor thread) can lead to context leaks and corruption. Additionally, the new unit tests execute synchronously on a single thread, which masks this multi-threading issue.
…racer to resolve thread safety issue
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces error status tracking on OpenTelemetry spans during failed attempts and adds a corresponding unit test. However, the newly declared attemptScope field is never initialized or used. Feedback recommends implementing the scope activation, ensuring that attemptScope is closed in an exception-safe manner to prevent context leaks, and adding unit tests to verify proper scope management.
| private final String attemptSpanName; | ||
| private final ApiTracerContext apiTracerContext; | ||
| private @Nullable Span attemptSpan; | ||
| private io.opentelemetry.context.@Nullable Scope attemptScope; |
There was a problem hiding this comment.
The field attemptScope is declared here but is never initialized, used, or closed anywhere in the class. The PR description states that the OpenTelemetry context Scope is activated during attemptStarted(...), but that implementation is missing. Please implement the scope activation in attemptStarted(...) using otelContext.makeCurrent() as described.
| private void endAttempt() { | ||
| if (attemptSpan == null) { | ||
| return; | ||
| if (attemptSpan != null) { | ||
| attemptSpan.end(); | ||
| attemptSpan = null; | ||
| } | ||
|
|
||
| attemptSpan.end(); | ||
| attemptSpan = null; | ||
| } |
There was a problem hiding this comment.
To prevent context leaks, the attemptScope must be closed when ending the attempt. Ensure that closeable resources are closed in the reverse order of their creation (LIFO) and that the implementation is exception-safe so that all resources are closed even if an exception occurs during closing.
private void endAttempt() {
try {
if (attemptScope != null) {
attemptScope.close();
}
} finally {
attemptScope = null;
try {
if (attemptSpan != null) {
attemptSpan.end();
}
} finally {
attemptSpan = null;
}
}
}References
- When managing a collection of closeable resources (e.g., scopes), ensure they are closed in the reverse order of their creation (LIFO). The implementation must be exception-safe to prevent resource leaks, meaning all opened resources should be closed even if exceptions occur during their creation or closing.
| void testAttemptFailed_setsErrorStatus() { | ||
| openTelemetryTracingTracer.attemptStarted(new Object(), 1); | ||
| openTelemetryTracingTracer.attemptFailedDuration( | ||
| new RuntimeException("Test error"), java.time.Duration.ofSeconds(1)); | ||
|
|
||
| verify(span).setStatus(io.opentelemetry.api.trace.StatusCode.ERROR); | ||
| verify(span).end(); | ||
| } |
There was a problem hiding this comment.
This test only verifies that the span status is set to ERROR and that the span is ended. It does not verify that the active scope is managed (i.e., activated and closed/restored). Once the scope management is implemented, please add assertions/verifications to ensure the scope is correctly activated and closed.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request updates OpenTelemetryTracingTracer to set the span status to StatusCode.ERROR when an attempt fails, and adds a corresponding unit test. It also introduces an unused attemptScope field. The review feedback highlights that attemptScope is declared but never initialized or closed, which could lead to context leaks. Additionally, a null check is recommended for attemptSpan before calling methods on it in recordErrorAndEndAttempt to prevent potential NullPointerExceptions.
| private final String attemptSpanName; | ||
| private final ApiTracerContext apiTracerContext; | ||
| private @Nullable Span attemptSpan; | ||
| private io.opentelemetry.context.@Nullable Scope attemptScope; |
There was a problem hiding this comment.
The field attemptScope is declared here but is never initialized, used, or closed. The PR description states that active scope context propagation is implemented via otelContext.makeCurrent() and cleaned up in endAttempt(), but these changes are missing from the implementation. Please ensure that the scope is properly activated when the attempt starts and closed when the attempt ends to prevent context leaks.
| private void endAttempt() { | ||
| if (attemptSpan == null) { | ||
| return; | ||
| if (attemptSpan != null) { | ||
| attemptSpan.end(); | ||
| attemptSpan = null; | ||
| } | ||
|
|
||
| attemptSpan.end(); | ||
| attemptSpan = null; | ||
| } |
There was a problem hiding this comment.
The attemptScope is not closed in endAttempt(). To prevent context leaks, ensure that the scope is closed in endAttempt(). According to general rules, closeable resources should be closed in the reverse order of their creation (LIFO) and in an exception-safe manner.
private void endAttempt() {
try {
if (attemptScope != null) {
attemptScope.close();
}
} finally {
attemptScope = null;
try {
if (attemptSpan != null) {
attemptSpan.end();
}
} finally {
attemptSpan = null;
}
}
}References
- When managing a collection of closeable resources (e.g., scopes), ensure they are closed in the reverse order of their creation (LIFO). The implementation must be exception-safe to prevent resource leaks, meaning all opened resources should be closed even if exceptions occur during their creation or closing.
| if (error != null) { | ||
| attemptSpan.setStatus(StatusCode.ERROR); | ||
| if (!Strings.isNullOrEmpty(error.getMessage())) { | ||
| attemptSpan.setAttribute( | ||
| ObservabilityAttributes.STATUS_MESSAGE_ATTRIBUTE, error.getMessage()); | ||
| } | ||
| } |
There was a problem hiding this comment.
Since attemptSpan is annotated with @Nullable, we should perform a null check before calling methods on it to prevent potential NullPointerExceptions.
| if (error != null) { | |
| attemptSpan.setStatus(StatusCode.ERROR); | |
| if (!Strings.isNullOrEmpty(error.getMessage())) { | |
| attemptSpan.setAttribute( | |
| ObservabilityAttributes.STATUS_MESSAGE_ATTRIBUTE, error.getMessage()); | |
| } | |
| } | |
| if (attemptSpan != null && error != null) { | |
| attemptSpan.setStatus(StatusCode.ERROR); | |
| if (!Strings.isNullOrEmpty(error.getMessage())) { | |
| attemptSpan.setAttribute( | |
| ObservabilityAttributes.STATUS_MESSAGE_ATTRIBUTE, error.getMessage()); | |
| } | |
| } |
|
|
1 participant
Description
Implements active scope context propagation and attribute validation for individual RPC attempt spans (T3 spans) in
OpenTelemetryTracingTracer.Changes
ScopeduringattemptStarted(...)viaotelContext.makeCurrent().attemptScopeinendAttempt().StatusCode.ERRORonattemptSpanwhen an attempt fails inrecordErrorAndEndAttempt.OpenTelemetryTracingTracerTestverifying scope management and error status propagation.b/541322523