Skip to content

Add business QoS configuration for URMA - #3712

Open
Spicy-cream wants to merge 37 commits into
apache:masterfrom
LinQuickDev:feature/rdma-business-qos
Open

Spicy-cream wants to merge 37 commits into
apache:masterfrom
LinQuickDev:feature/rdma-business-qos

Conversation

@Spicy-cream

@Spicy-cream Spicy-cream commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

A process can serve latency-sensitive RPCs alongside bulk traffic. Channel and server options let each workload set its outgoing priority, with the process default available for other traffic.

This PR depends on PR3708. The source branch includes its recovery changes; merge PR3708 first.

What is changed and the side effects?

  • Add ChannelOptions and ServerOptions settings for outgoing SL 0-255 and priority 0-15; -1 selects automatic behavior. Channels configure request traffic, and servers configure response traffic.
  • Resolve SL through the selected CTP/RTP device mapping. Reject unsupported, conflicting or out-of-range settings during Channel initialization or Server startup, with supported priority:SL pairs in diagnostics.
  • Include QoS in connection signatures and propagate it through naming services, accepted sockets and rebuilt Jettys. Use prepared Jettys only when their priority matches.
  • Propagate QoS through GetSubSocketOptions() so pooled/short children retain QoS alongside keepalive, TCP user timeout and bthread tag settings.
  • Expose requested/effective values in diagnostics and examples, and add independent QoS and connection-lifecycle regressions.

The selected path exposes up to 16 priority/SL mappings per device. Live URMA connections each use a Jetty at each end. Connections with the default priority and compatible queues can use a prepared Jetty. Other connections allocate Jettys on demand. The channel signature includes the configured priority and SL, so equivalent settings can use separate connections. Existing v2/v3 wire formats remain compatible.

Validation

  • The local URMA mock suite passed all 22 regression groups with RDMA disabled, including the inherited recovery regressions.
  • The RTP/polling QoS churn suite passed 100 complete runs after the inherited mock SEND fix.
  • A regression verifies QoS, keepalive snapshots, TCP user timeout and bthread tag propagation to pooled and short sockets. All 3 existing community Socket option tests passed; their sources remain unchanged.
  • Library/example builds and C++14 compilation checks passed, including public headers with transports disabled.
  • Mock lifecycle regressions cover small-pool v2/v3 short connections and request SL 4/response SL 2 across TCP cooldown and fresh URMA connections.
  • Prior URMA RM/RTP hardware runs passed all 12 event/polling, v2/v3 and auto/SL/SL-plus-priority RPC cases, plus all 6 client/server configuration-rejection cases. A live server connection reported ESTABLISHED with effective priority 2 and SL 2.

Design discussion: URMA runtime recovery and business QoS configuration.

Check List

  • Build the library/examples and run related regressions.
  • Add independent tests, preserve existing community UT sources and update documentation.
  • Follow four-space indentation and C++ nullptr conventions.

@wwbmmm

wwbmmm commented Nov 7, 2026

Copy link
Copy Markdown
Contributor
  1. This PR is not reviewable as a QoS change. Its ~4.5k added lines also carry the runtime-recovery work from Add URMA runtime recovery and TCP fallback #3708 plus changes unrelated to QoS: --urma_tp_type path selection, --urma_copy_send, the recovery policy, the send-failure injection harness, and a new .github/workflows/ci-urma.yml. Please merge Add URMA runtime recovery and TCP fallback #3708 first and rebase so this PR contains only the QoS delta, and keep the CI workflow in the PR that introduces the recovery support. Otherwise reviewers cannot separate what QoS needs from what the dependency already provides.

  2. Adding UrmaQosOptions to ChannelOptions, ServerOptions, SocketOptions and Acceptor makes this a public API change on the core headers (channel.h/server.h/socket.h now include brpc/urma/urma_qos.h). The requirement is reasonable, but the PR should justify why a process-level default (the current behavior, which can stay a flag) is insufficient and how many distinct settings a process can carry: each distinct (priority, SL) combination needs its own Jetty per device, and prepared-Jetty reuse is already made conditional on a priority match, so every business channel with a custom priority adds Jetty creation/teardown cost. Please document the expected bound and the behavior when a device exposes few priorities.

@Spicy-cream

Copy link
Copy Markdown
Contributor Author
  1. This PR is not reviewable as a QoS change. Its ~4.5k added lines also carry the runtime-recovery work from Add URMA runtime recovery and TCP fallback #3708 plus changes unrelated to QoS: --urma_tp_type path selection, --urma_copy_send, the recovery policy, the send-failure injection harness, and a new .github/workflows/ci-urma.yml. Please merge Add URMA runtime recovery and TCP fallback #3708 first and rebase so this PR contains only the QoS delta, and keep the CI workflow in the PR that introduces the recovery support. Otherwise reviewers cannot separate what QoS needs from what the dependency already provides.
  2. Adding UrmaQosOptions to ChannelOptions, ServerOptions, SocketOptions and Acceptor makes this a public API change on the core headers (channel.h/server.h/socket.h now include brpc/urma/urma_qos.h). The requirement is reasonable, but the PR should justify why a process-level default (the current behavior, which can stay a flag) is insufficient and how many distinct settings a process can carry: each distinct (priority, SL) combination needs its own Jetty per device, and prepared-Jetty reuse is already made conditional on a priority match, so every business channel with a custom priority adds Jetty creation/teardown cost. Please document the expected bound and the behavior when a device exposes few priorities.

Thanks for the review. I’ll move the mock SEND completion fix and its CI wiring to #3708, then rebase this PR after #3708 lands so the diff contains only the QoS changes.

The use case is a single process with both latency-sensitive RPCs and bulk traffic. Those channels need different outgoing priorities, while channels with no explicit setting can inherit the process default.

I’ll document the device limits and resource cost. The selected device and path expose up to 16 effective priority/SL mappings, and unsupported settings fail initialization with the available mappings in the diagnostic. Jetty usage scales with live URMA connections and the prepared pool. Connections with non-default priorities allocate Jettys on demand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants