Skip to content

Fix RDMA polling initialization failure cleanup - #3750

Open
yanglimingcn wants to merge 1 commit into
apache:masterfrom
yanglimingcn:bugfix/rdma_polling_initialize
Open

yanglimingcn wants to merge 1 commit into
apache:masterfrom
yanglimingcn:bugfix/rdma_polling_initialize

Conversation

@yanglimingcn

Copy link
Copy Markdown
Contributor

Reset the running state and join already-started pollers when creation fails. Release failed thread arguments, preserve the error code, and clear joined thread IDs.

Add regression coverage for creation failures, retries, and repeated release in both bthread and pthread-backed modes.

Validation: git diff --check passed. Build and unit tests were blocked by missing Protobuf development dependencies and protoc.

What problem does this PR solve?

Issue Number: resolve

Problem Summary:

What is changed and the side effects?

Changed:

Side effects:

  • Performance effects:

  • Breaking backward compatibility:


Check List:

Reset the running state and join already-started pollers when creation fails. Release failed thread arguments, preserve the error code, and clear joined thread IDs.

Add regression coverage for creation failures, retries, and repeated release in both bthread and pthread-backed modes.

Validation: git diff --check passed. Build and unit tests were blocked by missing Protobuf development dependencies and protoc.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The Bazel RDMA unit test still references a UNIT_TEST-only hook without receiving that definition, causing an undefined-symbol link failure and leaving the injection path unavailable.

1 open issue
What changed in this PR

Fixes RDMA polling initialization failure cleanup by rolling back partially created poller groups, preserving startup errors, releasing failed thread arguments, and supporting retries and repeated release.

Changes:

  • Reset polling state, join already-started pollers, and clear thread IDs after creation failures.
  • Add RAII-based argument cleanup and error preservation during poller startup.
  • Add parameterized regression tests for failure points and bthread- and pthread-backed modes.
File Description
test/​brpc_rdma_unittest.cpp Adds lifecycle regression tests for poller creation failures, retries, repeated release, and both threading modes.
src/​brpc/​rdma/​rdma_endpoint.cpp Implements poller startup rollback, argument ownership cleanup, error preservation, test injection, and thread ID reset.

Review effort: Lite

Comment on lines +1957 to +1961
#if defined(UNIT_TEST)
// Inject poller creation failures without requiring an RDMA device.
decltype(&bthread_start_background) g_start_rdma_poller_for_test =
bthread_start_background;
#endif
running.store(false, std::memory_order_relaxed);
for (size_t j = 0; j < i; ++j) {
bthread_join(pollers[j].tid, nullptr);
pollers[j].tid = INVALID_BTHREAD;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting running to false before the old poller exits allows another initialization with the same tag to pass the entry CAS. Specifically, after A starts the first poller, creation of the second fails and A enters join. If B then initializes another Channel and sets running back to true, the old poller, which has not yet observed the stop signal, will continue running, and A's initialization will remain blocked. Initializations of different Channels share this PollerGroup, and the call chain has no serialization protection. Consider separating the stop signal from the initialization state to prevent the next batch of tasks from starting before rollback completes, and add a barrier-controlled concurrency test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants