Skip to content

[ISSUE #10766] Fail fast on telemetry dispatch failure - #10767

Open
Aias00 wants to merge 1 commit into
apache:developfrom
Aias00:fix/proxy-telemetry-dispatch-fail-fast
Open

[ISSUE #10766] Fail fast on telemetry dispatch failure#10767
Aias00 wants to merge 1 commit into
apache:developfrom
Aias00:fix/proxy-telemetry-dispatch-fail-fast

Conversation

@Aias00

@Aias00 Aias00 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Which Issue(s) This PR Fixes

Fixes #10766

Brief Description

When a gRPC telemetry command cannot be dispatched for relay-style requests, the registered response future used to stay in GrpcChannelManager until the timeout cleanup path handled it. This PR makes the write path report dispatch failure and completes/removes the pending future immediately for:

  • GetConsumerRunningInfo
  • ConsumeMessageDirectly

The observer-null and observer-write-failure paths now return a structured SYSTEM_BUSY relay result instead of leaving callers pending.

How Did You Test This Change?

  • mvn -q -pl proxy -Dtest=GrpcClientChannelTest test

The test run passes with 3 tests. The local output still contains existing JaCoCo Java 17 instrumentation warnings, but Maven exits successfully.

Copilot AI review requested due to automatic review settings August 3, 2026 06:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses RocketMQ Proxy gRPC telemetry relay behavior by failing fast when telemetry command dispatch cannot be performed, preventing relay response futures from lingering in GrpcChannelManager until timeout cleanup.

Changes:

  • Register relay response futures with a pre-generated nonce and immediately complete/remove them with SYSTEM_BUSY when telemetry dispatch fails for GetConsumerRunningInfo and ConsumeMessageDirectly.
  • Change telemetry write path to return a success/failure boolean and add a helper to complete pending relay futures on write failure.
  • Add unit tests covering observer-missing and observer-write-failure fail-fast scenarios.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannel.java Adds fail-fast completion/removal of relay futures when telemetry dispatch fails; telemetry write now reports success/failure.
proxy/src/test/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannelTest.java Adds tests ensuring relay futures complete immediately with SYSTEM_BUSY on dispatch failure.
Suppressed comments (1)

proxy/src/test/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannelTest.java:127

  • Avoid the unchecked raw cast when stubbing getAndRemoveResponseFuture; use an explicit generic type parameter so the stub remains type-safe and warning-free.
        CompletableFuture<ProxyRelayResult<ConsumeMessageDirectlyResult>> responseFuture = new CompletableFuture<>();
        when(grpcChannelManager.addResponseFuture(eq(responseFuture))).thenReturn("nonce-2");
        when(grpcChannelManager.getAndRemoveResponseFuture(eq("nonce-2"))).thenReturn((CompletableFuture) responseFuture);


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +272 to 273
public boolean writeTelemetryCommand(TelemetryCommand command) {
StreamObserver<TelemetryCommand> observer = this.telemetryCommandRef.get();
public void testGetConsumerRunningInfoShouldFailFastWhenObserverIsMissing() throws Exception {
CompletableFuture<ProxyRelayResult<ConsumerRunningInfo>> responseFuture = new CompletableFuture<>();
when(grpcChannelManager.addResponseFuture(eq(responseFuture))).thenReturn("nonce-1");
when(grpcChannelManager.getAndRemoveResponseFuture(eq("nonce-1"))).thenReturn((CompletableFuture) responseFuture);

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by github-manager-bot

Summary

Makes writeTelemetryCommand return a boolean and uses it to fail fast when telemetry dispatch fails for GetConsumerRunningInfo and ConsumeMessageDirectly. Previously, the registered response future would remain in GrpcChannelManager until timeout cleanup. Now it is completed/removed immediately with a SYSTEM_BUSY relay result.

Findings

  • [Correctness] GrpcClientChannel.java:239-254 — The fix correctly captures the nonce from addResponseFuture and uses it to remove the pending future on write failure. Both the null-observer and write-failure paths now return structured SYSTEM_BUSY results instead of leaving callers pending indefinitely.
  • [Correctness] GrpcClientChannel.java:280-295 — Same pattern applied to processConsumeMessageDirectly. Consistent error handling across both relay-style telemetry paths.
  • [Info] The addResponseFuture is called before writeTelemetryCommand. If writeTelemetryCommand were to throw an unchecked exception, the nonce would leak. Consider wrapping in try-catch for robustness, though this is low-risk given the current implementation.
  • [Tests] GrpcClientChannelTest.java — 72 lines of new tests covering both the null-observer and write-failure scenarios for both methods. Good coverage.

Verdict

Solid reliability fix with clean implementation. LGTM.


Automated review by github-manager-bot

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.21053% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.25%. Comparing base (89166af) to head (52603a9).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
...ketmq/proxy/grpc/v2/channel/GrpcClientChannel.java 84.21% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             develop   #10767      +/-   ##
=============================================
- Coverage      48.29%   48.25%   -0.05%     
+ Complexity     13499    13490       -9     
=============================================
  Files           1380     1380              
  Lines         101093   101114      +21     
  Branches       13102    13110       +8     
=============================================
- Hits           48821    48789      -32     
- Misses         46304    46334      +30     
- Partials        5968     5991      +23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Proxy telemetry dispatch failure leaves relay futures pending until timeout

4 participants