[ISSUE #10766] Fail fast on telemetry dispatch failure - #10767
Conversation
There was a problem hiding this comment.
Pull request overview
This PR addresses RocketMQ Proxy gRPC telemetry relay behavior by failing fast when telemetry command dispatch cannot be performed, preventing relay response futures from lingering in GrpcChannelManager until timeout cleanup.
Changes:
- Register relay response futures with a pre-generated nonce and immediately complete/remove them with
SYSTEM_BUSYwhen telemetry dispatch fails forGetConsumerRunningInfoandConsumeMessageDirectly. - Change telemetry write path to return a success/failure boolean and add a helper to complete pending relay futures on write failure.
- Add unit tests covering observer-missing and observer-write-failure fail-fast scenarios.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannel.java | Adds fail-fast completion/removal of relay futures when telemetry dispatch fails; telemetry write now reports success/failure. |
| proxy/src/test/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannelTest.java | Adds tests ensuring relay futures complete immediately with SYSTEM_BUSY on dispatch failure. |
Suppressed comments (1)
proxy/src/test/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannelTest.java:127
- Avoid the unchecked raw cast when stubbing
getAndRemoveResponseFuture; use an explicit generic type parameter so the stub remains type-safe and warning-free.
CompletableFuture<ProxyRelayResult<ConsumeMessageDirectlyResult>> responseFuture = new CompletableFuture<>();
when(grpcChannelManager.addResponseFuture(eq(responseFuture))).thenReturn("nonce-2");
when(grpcChannelManager.getAndRemoveResponseFuture(eq("nonce-2"))).thenReturn((CompletableFuture) responseFuture);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| public boolean writeTelemetryCommand(TelemetryCommand command) { | ||
| StreamObserver<TelemetryCommand> observer = this.telemetryCommandRef.get(); |
| public void testGetConsumerRunningInfoShouldFailFastWhenObserverIsMissing() throws Exception { | ||
| CompletableFuture<ProxyRelayResult<ConsumerRunningInfo>> responseFuture = new CompletableFuture<>(); | ||
| when(grpcChannelManager.addResponseFuture(eq(responseFuture))).thenReturn("nonce-1"); | ||
| when(grpcChannelManager.getAndRemoveResponseFuture(eq("nonce-1"))).thenReturn((CompletableFuture) responseFuture); |
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Review by github-manager-bot
Summary
Makes writeTelemetryCommand return a boolean and uses it to fail fast when telemetry dispatch fails for GetConsumerRunningInfo and ConsumeMessageDirectly. Previously, the registered response future would remain in GrpcChannelManager until timeout cleanup. Now it is completed/removed immediately with a SYSTEM_BUSY relay result.
Findings
- [Correctness]
GrpcClientChannel.java:239-254— The fix correctly captures the nonce fromaddResponseFutureand uses it to remove the pending future on write failure. Both the null-observer and write-failure paths now return structuredSYSTEM_BUSYresults instead of leaving callers pending indefinitely. - [Correctness]
GrpcClientChannel.java:280-295— Same pattern applied toprocessConsumeMessageDirectly. Consistent error handling across both relay-style telemetry paths. - [Info] The
addResponseFutureis called beforewriteTelemetryCommand. IfwriteTelemetryCommandwere to throw an unchecked exception, the nonce would leak. Consider wrapping in try-catch for robustness, though this is low-risk given the current implementation. - [Tests]
GrpcClientChannelTest.java— 72 lines of new tests covering both the null-observer and write-failure scenarios for both methods. Good coverage.
Verdict
Solid reliability fix with clean implementation. LGTM.
Automated review by github-manager-bot
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #10767 +/- ##
=============================================
- Coverage 48.29% 48.25% -0.05%
+ Complexity 13499 13490 -9
=============================================
Files 1380 1380
Lines 101093 101114 +21
Branches 13102 13110 +8
=============================================
- Hits 48821 48789 -32
- Misses 46304 46334 +30
- Partials 5968 5991 +23 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Which Issue(s) This PR Fixes
Fixes #10766
Brief Description
When a gRPC telemetry command cannot be dispatched for relay-style requests, the registered response future used to stay in
GrpcChannelManageruntil the timeout cleanup path handled it. This PR makes the write path report dispatch failure and completes/removes the pending future immediately for:GetConsumerRunningInfoConsumeMessageDirectlyThe observer-null and observer-write-failure paths now return a structured
SYSTEM_BUSYrelay result instead of leaving callers pending.How Did You Test This Change?
mvn -q -pl proxy -Dtest=GrpcClientChannelTest testThe test run passes with 3 tests. The local output still contains existing JaCoCo Java 17 instrumentation warnings, but Maven exits successfully.