Skip to content

Add URMA runtime recovery and TCP fallback - #3708

Open
Spicy-cream wants to merge 23 commits into
apache:masterfrom
LinQuickDev:feature/rdma-runtime-recovery
Open

Spicy-cream wants to merge 23 commits into
apache:masterfrom
LinQuickDev:feature/rdma-runtime-recovery

Conversation

@Spicy-cream

@Spicy-cream Spicy-cream commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

URMA can fall back to TCP during connection setup, but established failures do not guide subsequent connection attempts. A failed server SEND can close the control socket while the client immediately reconnects over URMA to the same peer.

What is changed and the side effects?

  • Add process-level LEGACY, AUTO and TCP_ONLY modes. AUTO applies a configurable peer cooldown, defaulting to 5 seconds; LEGACY remains the default. New connections use TCP during cooldown and can select URMA afterward.
  • Report completion, JFC and WR-posting failures once per connection. Ignore reports from earlier policy epochs, count access errors separately and preserve EAGAIN for backpressure.
  • Report established client control-socket EOF/read failures so server-side faults also trigger peer cooldown. Normal server-side EOF from short clients closes the connection without cooling down the peer.
  • Select CTP/RTP paths from device capabilities, release short-connection resources, serialize TCP reads and initialize/validate send SGEs. Optional aligned copied sends add one payload copy.
  • Add a recovery client, process-local SEND fault injection, independent regression tests, documentation and a URMA mock CI workflow.
  • Publish mock SEND completions after payload delivery so polling consumers can release source buffers safely. Run the completion-order regression in URMA mock CI.

Mode changes affect new connections. Short connections can resume URMA after cooldown; established single/pooled TCP connections can continue serving RPCs. Existing v2/v3 wire formats remain compatible.

Validation

  • The local URMA mock suite passed all 16 regression groups with RDMA disabled, including the SEND completion-order test.
  • The server-SEND regression reproduces the missing client cooldown before the fix. All 8 CTP/RTP, v2/v3 and event/polling combinations pass after it, covering normal short closes, TCP cooldown, URMA recovery and 64 KiB attachment validation.
  • Library/example builds, C++14 compilation, injection-runner self-tests and fetched-SDK-header configuration checks passed locally.
  • Prior URMA RM/RTP hardware runs passed TCP_ONLY, v2/v3 AUTO, runtime mode switching and single/pooled 64 KiB/1 MiB attachment checks. Client SEND-post injection passed event short/single/pooled and polling short runs with 1 MiB attachments: one expected RPC failure per run, subsequent RPC recovery and zero content-validation failures. Short connections resumed URMA after TCP cooldown.

Related discussions and PRs

Check List

  • Build the library/examples and run related regressions.
  • Add independent tests, preserve existing community UT sources and update documentation.
  • Follow four-space indentation and C++ nullptr conventions.

@Spicy-cream

Copy link
Copy Markdown
Contributor Author

This is the necessary work for us to complete a plan,could you have a look? @chenBright @wwbmmm

@Spicy-cream

Copy link
Copy Markdown
Contributor Author

This PR is only for URMA, should we have also done the RDMA part?

@wwbmmm

wwbmmm commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

On the RDMA question: URMA-only is fine for this PR. RDMA has no equivalent runtime recovery today -- it only falls back to TCP while the handshake/queue-pair bring-up is failing (rdma_endpoint.cpp), and an established QP error is treated as unrecoverable. So there is no parity gap this PR has to close. If the mechanism proves useful, the generalization worth doing afterwards is a transport-agnostic "a transport failure cools this peer for the next N seconds / new connections degrade to TCP" hook shared by RDMA and URMA, rather than a second process-global copy of the policy inside RDMA. That is also an argument for keeping the policy interface small and transport-neutral where it is cheap to do so.

@zchuango

zchuango commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

On the RDMA question: URMA-only is fine for this PR. RDMA has no equivalent runtime recovery today -- it only falls back to TCP while the handshake/queue-pair bring-up is failing (rdma_endpoint.cpp), and an established QP error is treated as unrecoverable. So there is no parity gap this PR has to close. If the mechanism proves useful, the generalization worth doing afterwards is a transport-agnostic "a transport failure cools this peer for the next N seconds / new connections degrade to TCP" hook shared by RDMA and URMA, rather than a second process-global copy of the policy inside RDMA. That is also an argument for keeping the policy interface small and transport-neutral where it is cheap to do so.

fully agree with this viewpoint. First, we plan to first establish and improve the fallback capability through the URMA memory semantics, and then use this as a foundation to extend it to the upper layers of the entire architecture. We will attempt to expand the functionality of RDMA, UBShm, and other transport layers, such as the design of a common fallback hook function. We will first internally align and provide the relevant plan. @wwbmmm

@wwbmmm
wwbmmm requested a lite review from Copilot November 16, 2026 13:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical teardown and EAGAIN paths can report successful sends without peer delivery or lose retryable payloads, so the active issues must be fixed before approval.

3 open issues
What changed in this PR

This PR adds process-level URMA runtime recovery, peer cooldowns, TCP fallback, transport-path selection, safer SEND handling, and extensive mock-based regression coverage while preserving existing URMA handshake compatibility.

Changes:

  • Adds LEGACY, AUTO, and TCP_ONLY recovery modes with per-peer cooldowns, failure metrics, and established-connection failure reporting.
  • Adds CTP/RTP capability selection, aligned copied-send support, SGE validation, TCP-read serialization, and mock SEND completion ordering.
  • Adds recovery, transport, copy-path, fault-injection tests, CI coverage, examples, and English/Chinese documentation.
File Description
test/​CMakeLists.txt Builds SEND-failure injection support and registers URMA transport and recovery test variants.
test/​brpc_urma_transport_unittest.cpp Tests transport-path selection, handshake compatibility, payload exchange, and runtime mode switching.
test/​brpc_urma_runtime_unittest.cpp Covers path selection, SGE validation, copied sends, recovery reporting, and TCP fallback behavior.
test/​brpc_urma_remote_recovery_unittest.cpp Exercises server SEND failures, client cooldown, TCP admission, and URMA recovery.
test/​brpc_urma_recovery_unittest.cpp Unit-tests recovery-policy modes, epochs, cooldowns, cache behavior, metrics, and concurrency.
test/​brpc_urma_mock_completion_unittest.cpp Verifies that mock SEND completions follow successful payload delivery.
test/​brpc_urma_copy_unittest.cpp Tests aligned copied sends and multi-WR payload handling across URMA configurations.
src/​brpc/​urma/​urma_recovery_policy.h Defines recovery modes, failure reasons, status information, and policy interfaces.
src/​brpc/​urma/​urma_recovery_policy.cpp Implements peer cooldown admission, failure tracking, epochs, cache management, and metrics.
src/​brpc/​urma/​urma_helper.h Declares path-selection, SGE-initialization, and copied-send helper interfaces.
src/​brpc/​urma/​urma_helper.cpp Implements capability-based path selection, SGE validation, and aligned send-buffer copying.
src/​brpc/​urma/​urma_handshake.cpp Encodes TCP fallback state in URMA server hello messages.
src/​brpc/​urma/​urma_endpoint.h Adds recovery state, synchronized TCP reads, and runtime failure-reporting interfaces.
src/​brpc/​urma/​urma_endpoint.cpp Integrates recovery admission and reporting, TCP fallback, SGE preparation, copied sends, and completion handling.
src/​brpc/​urma/​mock_urma.cpp Adds mock path capabilities, segment metadata, and SEND completion publication after delivery.
example/​urma_performance/​test_send_failure_injector.c Provides self-test coverage for SEND-failure injector behavior.
example/​urma_performance/​test_run_recovery_injection.py Tests recovery-runner parsing, readiness handling, and phase validation.
example/​urma_performance/​send_failure_injector.c Implements process-local LD_PRELOAD SEND-post failure injection.
example/​urma_performance/​run_recovery_injection.py Runs end-to-end SEND-failure, cooldown, fallback, and recovery validation.
example/​urma_performance/​recovery_client.cpp Adds a configurable URMA/TCP recovery workload and attachment-validation client.
example/​urma_performance/​CMakeLists.txt Builds the recovery client and optional fault-injection tools.
docs/​en/​urma.md Documents transport-path selection, runtime recovery, fault injection, and validation procedures.
docs/​cn/​urma.md Adds the corresponding Chinese URMA configuration, recovery, and validation documentation.
.github/​workflows/​ci-urma.yml Adds Linux URMA mock build and regression-test CI coverage.

Review effort: Lite

Comment thread src/brpc/urma/mock_urma.cpp Outdated
Comment on lines 585 to 587
if (remote_it == jetty_id_map.end()) {
complete_send();
continue;
Comment thread src/brpc/urma/mock_urma.cpp Outdated
}
// A SEND completion lets the consumer release its source buffers.
// Finish copying the payload before publishing that completion.
complete_send();
<< ", configured_max_sge=" << GetUrmaMaxSge()
<< ", payload_size=" << this_len
<< " on " << _socket->description();
ReportFailure(UrmaFailureReason::SEND_POST);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants