Repository navigation
Add URMA runtime recovery and TCP fallback - #3708
Spicy-cream wants to merge 23 commits into
Conversation
|
This is the necessary work for us to complete a plan,could you have a look? @chenBright @wwbmmm |
|
This PR is only for URMA, should we have also done the RDMA part? |
|
On the RDMA question: URMA-only is fine for this PR. RDMA has no equivalent runtime recovery today -- it only falls back to TCP while the handshake/queue-pair bring-up is failing ( |
fully agree with this viewpoint. First, we plan to first establish and improve the fallback capability through the URMA memory semantics, and then use this as a foundation to extend it to the upper layers of the entire architecture. We will attempt to expand the functionality of RDMA, UBShm, and other transport layers, such as the design of a common fallback hook function. We will first internally align and provide the relevant plan. @wwbmmm |
(cherry picked from commit 351935a)
There was a problem hiding this comment.
🟡 Changes recommended
Critical teardown and EAGAIN paths can report successful sends without peer delivery or lose retryable payloads, so the active issues must be fixed before approval.
3 open issues
What changed in this PR
This PR adds process-level URMA runtime recovery, peer cooldowns, TCP fallback, transport-path selection, safer SEND handling, and extensive mock-based regression coverage while preserving existing URMA handshake compatibility.
Changes:
- Adds LEGACY, AUTO, and TCP_ONLY recovery modes with per-peer cooldowns, failure metrics, and established-connection failure reporting.
- Adds CTP/RTP capability selection, aligned copied-send support, SGE validation, TCP-read serialization, and mock SEND completion ordering.
- Adds recovery, transport, copy-path, fault-injection tests, CI coverage, examples, and English/Chinese documentation.
| File | Description |
|---|---|
| test/CMakeLists.txt | Builds SEND-failure injection support and registers URMA transport and recovery test variants. |
| test/brpc_urma_transport_unittest.cpp | Tests transport-path selection, handshake compatibility, payload exchange, and runtime mode switching. |
| test/brpc_urma_runtime_unittest.cpp | Covers path selection, SGE validation, copied sends, recovery reporting, and TCP fallback behavior. |
| test/brpc_urma_remote_recovery_unittest.cpp | Exercises server SEND failures, client cooldown, TCP admission, and URMA recovery. |
| test/brpc_urma_recovery_unittest.cpp | Unit-tests recovery-policy modes, epochs, cooldowns, cache behavior, metrics, and concurrency. |
| test/brpc_urma_mock_completion_unittest.cpp | Verifies that mock SEND completions follow successful payload delivery. |
| test/brpc_urma_copy_unittest.cpp | Tests aligned copied sends and multi-WR payload handling across URMA configurations. |
| src/brpc/urma/urma_recovery_policy.h | Defines recovery modes, failure reasons, status information, and policy interfaces. |
| src/brpc/urma/urma_recovery_policy.cpp | Implements peer cooldown admission, failure tracking, epochs, cache management, and metrics. |
| src/brpc/urma/urma_helper.h | Declares path-selection, SGE-initialization, and copied-send helper interfaces. |
| src/brpc/urma/urma_helper.cpp | Implements capability-based path selection, SGE validation, and aligned send-buffer copying. |
| src/brpc/urma/urma_handshake.cpp | Encodes TCP fallback state in URMA server hello messages. |
| src/brpc/urma/urma_endpoint.h | Adds recovery state, synchronized TCP reads, and runtime failure-reporting interfaces. |
| src/brpc/urma/urma_endpoint.cpp | Integrates recovery admission and reporting, TCP fallback, SGE preparation, copied sends, and completion handling. |
| src/brpc/urma/mock_urma.cpp | Adds mock path capabilities, segment metadata, and SEND completion publication after delivery. |
| example/urma_performance/test_send_failure_injector.c | Provides self-test coverage for SEND-failure injector behavior. |
| example/urma_performance/test_run_recovery_injection.py | Tests recovery-runner parsing, readiness handling, and phase validation. |
| example/urma_performance/send_failure_injector.c | Implements process-local LD_PRELOAD SEND-post failure injection. |
| example/urma_performance/run_recovery_injection.py | Runs end-to-end SEND-failure, cooldown, fallback, and recovery validation. |
| example/urma_performance/recovery_client.cpp | Adds a configurable URMA/TCP recovery workload and attachment-validation client. |
| example/urma_performance/CMakeLists.txt | Builds the recovery client and optional fault-injection tools. |
| docs/en/urma.md | Documents transport-path selection, runtime recovery, fault injection, and validation procedures. |
| docs/cn/urma.md | Adds the corresponding Chinese URMA configuration, recovery, and validation documentation. |
| .github/workflows/ci-urma.yml | Adds Linux URMA mock build and regression-test CI coverage. |
Review effort: Lite
| if (remote_it == jetty_id_map.end()) { | ||
| complete_send(); | ||
| continue; |
| } | ||
| // A SEND completion lets the consumer release its source buffers. | ||
| // Finish copying the payload before publishing that completion. | ||
| complete_send(); |
| << ", configured_max_sge=" << GetUrmaMaxSge() | ||
| << ", payload_size=" << this_len | ||
| << " on " << _socket->description(); | ||
| ReportFailure(UrmaFailureReason::SEND_POST); |

What problem does this PR solve?
URMA can fall back to TCP during connection setup, but established failures do not guide subsequent connection attempts. A failed server SEND can close the control socket while the client immediately reconnects over URMA to the same peer.
What is changed and the side effects?
Mode changes affect new connections. Short connections can resume URMA after cooldown; established single/pooled TCP connections can continue serving RPCs. Existing v2/v3 wire formats remain compatible.
Validation
Related discussions and PRs
Check List