Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docs/cn/urma.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,3 +256,51 @@ URMA 恢复及 64 KiB 附件回显。正常短连接关闭也在同一测试中
`--urma_copy_send=true` 将每个数据 WR 合并到 64 字节对齐的注册缓冲区,并持有至完成,
增加一次 CPU 拷贝。两端保持设备、队列和消息大小一致,对比原发送路径与复制路径。
初始化日志记录 `copy_send`,失败日志记录完成方向、槽位及注册段范围。

## 业务 SL 与优先级

同一进程可以同时运行对延迟敏感的 RPC 和批量传输。两类业务可以分别设置发送优先级,
未显式配置的业务继承进程默认值。`ChannelOptions.urma_qos` 控制请求发送方向,
`ServerOptions.urma_qos` 控制响应发送方向。

```cpp
brpc::ChannelOptions client_options;
client_options.socket_mode = brpc::SOCKET_MODE_URMA;
client_options.urma_qos.service_level = 2;

brpc::ServerOptions server_options;
server_options.socket_mode = brpc::SOCKET_MODE_URMA;
server_options.urma_qos.service_level = 2;
```

`--urma_tp_type=auto` 按设备能力优先选择 CTP,其次 RTP;`ctp`、`rtp` 显式选择路径。
两端使用相同路径,进程初始化后保持固定。设备未提供能力表时,auto/ctp 使用 priority 6,
rtp 使用 priority 0。

| 配置 | 默认 | 显式范围 | 含义 |
| --- | --- | --- | --- |
| `service_level` | -1 | 0–255 | 选择当前路径下首个映射到该 SL 的 priority |
| `priority` | -1 | 0–15 | 指定当前路径支持的 priority |

两项均为 -1 时使用进程默认 priority。同时指定时,两项须匹配同一设备映射。
每个设备在所选路径下最多提供 16 个 priority/SL 映射,实际可用项取决于设备能力。
显式配置不受支持或发生冲突时,Channel 初始化或 Server 启动失败,日志会列出当前路径支持的映射。
平台配置设备映射与调度策略,brpc 使用进程初始化时的映射快照。

客户端连接签名包含原始 QoS 设置。即使 RTP 的 priority 0 是 SL 2 的首个匹配项,
只填 `service_level=2` 和同时填 `priority=0` 的两个 Channel 也会分别建连。
配置随服务发现、连接池、短连接和服务端 accept 传递,Jetty 重建时重新应用。

活跃的 URMA 连接在每端各占用一个 Jetty。匹配默认 priority 和队列大小的连接可以使用
预建池中的 Jetty,其他连接按需创建 Jetty。v2/v3 握手使用所选路径。

### 功能验证

用 `urma_admin show --dev "$URMA_DEVICE" --whole` 核对当前路径的 priority:SL。
`example/urma_performance` 的两端均支持 `--urma_tp_type=rtp --urma_sl=2`,
也可通过 `--urma_priority` 指定匹配项。先以默认配置完成 RPC,再分别改变请求和响应 SL,
验证合法映射、不支持的 SL 和 SL/priority 冲突。对 v2/v3、single/pooled 及 polling 模式重复验证。

连接调试信息提供 `state`、`urma_effective_priority` 和 `urma_effective_sl`。
核对 URMA 连接为 `ESTABLISHED`、映射符合设备配置、RPC 无错误。
调度效果测试结合竞争流量、设备队列计数器、吞吐与 P99。
59 changes: 59 additions & 0 deletions docs/en/urma.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,3 +279,62 @@ including normal short-connection closes before the injection.
buffer held until completion, adding one CPU copy. Keep device, queue and payload
settings consistent when comparing send paths. Startup logs report `copy_send`;
failure logs report direction, send slot and registered segment range.

## Business SL and priority

A process can serve latency-sensitive RPCs alongside bulk transfers. Configure
their outgoing priorities independently and leave other traffic on the process
default. Set `ChannelOptions.urma_qos` for outgoing requests and
`ServerOptions.urma_qos` for outgoing responses.

```cpp
brpc::ChannelOptions client_options;
client_options.socket_mode = brpc::SOCKET_MODE_URMA;
client_options.urma_qos.service_level = 2;

brpc::ServerOptions server_options;
server_options.socket_mode = brpc::SOCKET_MODE_URMA;
server_options.urma_qos.service_level = 2;
```

`--urma_tp_type=auto` prefers CTP, then RTP, using device capabilities.
Use `ctp` or `rtp` to select a path explicitly. Both peers use the same path,
fixed at process initialization. Providers without a capability table use
priority 6 for auto/CTP and priority 0 for RTP.

| Option | Default | Explicit range | Meaning |
| --- | --- | --- | --- |
| `service_level` | -1 | 0–255 | Select the first priority mapping to this SL on the selected path |
| `priority` | -1 | 0–15 | Select a priority supported by the selected path |

Both fields at -1 select the process default. When both are explicit, they must
match the same device mapping. Each device exposes up to 16 priority/SL mappings
on the selected path, depending on its capabilities. Channel initialization and
Server startup reject unsupported or conflicting settings and log the available
mappings. The platform configures mappings and scheduling. brpc uses the
initialization snapshot.

The client connection signature includes the configured QoS values. If RTP
priority 0 is the first mapping for SL 2, a channel that sets only
`service_level=2` uses a separate connection from one that also sets
`priority=0`. Naming services, pooled and short connections, accepted sockets
and rebuilt Jettys inherit the configuration.

Each active URMA connection uses one Jetty at each end. Connections with the
default priority and a compatible queue size can use a Jetty from the prepared
pool. Other connections allocate Jettys on demand. Both v2 and v3 use the
selected path.

### Functional validation

Inspect the selected path's priority:SL map with
`urma_admin show --dev "$URMA_DEVICE" --whole`.
Both examples in `example/urma_performance` accept `--urma_tp_type=rtp --urma_sl=2`
and an optional matching `--urma_priority`. Establish RPCs with defaults, then
change request and response SLs separately. Check supported mappings, unsupported
SLs and conflicting pairs across v2/v3, single/pooled and polling modes.

Connection diagnostics expose `state`, `urma_effective_priority` and
`urma_effective_sl`. Check that URMA is `ESTABLISHED`, the mapping matches the
device and RPCs succeed. Evaluate scheduling with competing traffic, device queue
counters, throughput and P99.
12 changes: 10 additions & 2 deletions example/urma_performance/client.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ DEFINE_int32(attachment_size, -1, "Attachment size is used (in Bytes)");
DEFINE_int32(rpc_timeout_ms, 5000, "Timeout for each RPC in milliseconds");
DEFINE_bool(echo_attachment, false, "Select whether attachment should be echo");
DEFINE_bool(use_urma, true, "Use URMA transport (true) or TCP (false)");
DEFINE_int32(urma_priority, -1,
"Outbound URMA priority: -1 automatic, or [0,15]");
DEFINE_int32(urma_sl, -1, "Outbound URMA SL: -1 unspecified, or [0,255]");

bvar::LatencyRecorder g_latency("client");
bvar::Adder<int64_t> g_error_count("client_error_count");
Expand Down Expand Up @@ -88,6 +91,8 @@ static void* worker(void* arg) {
int main(int argc, char* argv[]) {
gflags::ParseCommandLineFlags(&argc, &argv, true);
brpc::ChannelOptions options;
options.urma_qos.priority = FLAGS_urma_priority;
options.urma_qos.service_level = FLAGS_urma_sl;
options.socket_mode = FLAGS_use_urma ? brpc::SOCKET_MODE_URMA
: brpc::SOCKET_MODE_TCP;
options.connect_timeout_ms = FLAGS_rpc_timeout_ms;
Expand All @@ -110,7 +115,9 @@ int main(int argc, char* argv[]) {
stub.Test(&warmup_cntl, &warmup_req, &warmup_resp, nullptr);
if (warmup_cntl.Failed()) {
LOG(ERROR) << "Warm-up RPC failed after timeout_ms="
<< FLAGS_rpc_timeout_ms << ": "
<< FLAGS_rpc_timeout_ms << " use_urma=" << FLAGS_use_urma
<< " urma_priority=" << FLAGS_urma_priority
<< " urma_sl=" << FLAGS_urma_sl << ": "
<< warmup_cntl.ErrorText();
return -1;
}
Expand All @@ -131,7 +138,8 @@ int main(int argc, char* argv[]) {
}
LOG(INFO) << "URMA performance client started (server=" << FLAGS_server
<< ", use_urma=" << FLAGS_use_urma
<< ", threads=" << thread_num
<< ", urma_priority=" << FLAGS_urma_priority
<< ", urma_sl=" << FLAGS_urma_sl << ", threads=" << thread_num
<< ", rpc_timeout_ms=" << FLAGS_rpc_timeout_ms << ")";
while (!brpc::IsAskedToQuit()) {
sleep(1);
Expand Down
9 changes: 8 additions & 1 deletion example/urma_performance/server.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@

DEFINE_int32(port, 8003, "TCP Port of this server");
DEFINE_bool(use_urma, true, "Use URMA transport (true) or TCP (false)");
DEFINE_int32(urma_priority, -1,
"Outbound URMA priority: -1 automatic, or [0,15]");
DEFINE_int32(urma_sl, -1, "Outbound URMA SL: -1 unspecified, or [0,255]");

butil::atomic<uint64_t> g_last_time(0);

Expand Down Expand Up @@ -73,14 +76,18 @@ int main(int argc, char* argv[]) {
}

brpc::ServerOptions options;
options.urma_qos.priority = FLAGS_urma_priority;
options.urma_qos.service_level = FLAGS_urma_sl;
options.socket_mode = FLAGS_use_urma ? brpc::SOCKET_MODE_URMA
: brpc::SOCKET_MODE_TCP;
if (server.Start(FLAGS_port, &options) != 0) {
LOG(ERROR) << "Fail to start server";
return -1;
}
LOG(INFO) << "URMA performance server started on port " << FLAGS_port
<< " (use_urma=" << FLAGS_use_urma << ")";
<< " (use_urma=" << FLAGS_use_urma
<< ", urma_priority=" << FLAGS_urma_priority
<< ", urma_sl=" << FLAGS_urma_sl << ")";
server.RunUntilAskedToQuit();
return 0;
}
Expand Down
1 change: 1 addition & 0 deletions src/brpc/acceptor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,7 @@ void Acceptor::OnNewConnectionsUntilEAGAIN(Socket* acception) {
options.force_ssl = am->_force_ssl;
options.initial_ssl_ctx = am->_ssl_ctx;
options.socket_mode = am->_socket_mode;
options.urma_qos = am->_urma_qos;
options.bthread_tag = am->_bthread_tag;
if (Socket::Create(options, &socket_id) != 0) {
am->ReleaseConnectionSlot();
Expand Down
3 changes: 2 additions & 1 deletion src/brpc/acceptor.h
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
#include "butil/containers/flat_map.h"
#include "brpc/input_messenger.h"
#include "brpc/socket_mode.h"

#include "brpc/urma/urma_qos.h"

namespace brpc {

Expand Down Expand Up @@ -137,6 +137,7 @@ friend class Server;

// Choose to use a certain socket: 0 TCP, 1 RDMA
SocketMode _socket_mode;
UrmaQosOptions _urma_qos;

// Acceptor belongs to this tag
bthread_tag_t _bthread_tag;
Expand Down
20 changes: 15 additions & 5 deletions src/brpc/channel.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,9 @@ ChannelSSLOptions* ChannelOptions::mutable_ssl_options() {
}

static ChannelSignature ComputeChannelSignature(const ChannelOptions& opt) {
if (opt.auth == nullptr &&
!opt.has_ssl_options() &&
opt.client_host.empty() &&
opt.device_name.empty() &&
opt.connection_group.empty() &&
if (opt.socket_mode != SOCKET_MODE_URMA && opt.auth == nullptr &&
!opt.has_ssl_options() && opt.client_host.empty() &&
opt.device_name.empty() && opt.connection_group.empty() &&
opt.hc_option.health_check_path.empty()) {
// Returning zeroized result by default is more intuitive for users.
return ChannelSignature();
Expand Down Expand Up @@ -141,6 +139,10 @@ static ChannelSignature ComputeChannelSignature(const ChannelOptions& opt) {
buf.append("|rdma");
} else if (opt.socket_mode == SOCKET_MODE_URMA) {
buf.append("|urma");
buf.append("|priority=")
.append(std::to_string(opt.urma_qos.priority));
buf.append("|sl=").append(
std::to_string(opt.urma_qos.service_level));
}
butil::MurmurHash3_x64_128_Update(&mm_ctx, buf.data(), buf.size());
buf.clear();
Expand Down Expand Up @@ -187,6 +189,12 @@ int Channel::InitChannelOptions(const ChannelOptions* options) {
if (options) { // Override default options if user provided one.
_options = *options;
}
if (_options.socket_mode == SOCKET_MODE_URMA &&
!_options.urma_qos.IsValid()) {
LOG(ERROR) << "URMA priority must be -1 or in [0,15], "
"and SL must be -1 or in [0,255]";
return -1;
}
const Protocol* protocol = FindProtocol(_options.protocol);
if (nullptr == protocol || !protocol->support_client()) {
LOG(ERROR) << "Channel does not support the protocol";
Expand Down Expand Up @@ -407,6 +415,7 @@ int Channel::InitSingle(const butil::EndPoint& server_addr_and_port,
opt.local_side = client_endpoint;
opt.initial_ssl_ctx = ssl_ctx;
opt.socket_mode = _options.socket_mode;
opt.urma_qos = _options.urma_qos;
opt.hc_option = _options.hc_option;
opt.device_name = _options.device_name;
if (SocketMapInsert(SocketMapKey(server_addr_and_port, sig),
Expand Down Expand Up @@ -453,6 +462,7 @@ int Channel::Init(const char* ns_url,
ns_opt.succeed_without_server = _options.succeed_without_server;
ns_opt.log_succeed_without_server = _options.log_succeed_without_server;
ns_opt.socket_option.socket_mode = _options.socket_mode;
ns_opt.socket_option.urma_qos = _options.urma_qos;
ns_opt.channel_signature = ComputeChannelSignature(_options);
ns_opt.socket_option.hc_option = _options.hc_option;
ns_opt.socket_option.local_side = client_endpoint;
Expand Down
4 changes: 4 additions & 0 deletions src/brpc/channel.h
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
#include "brpc/naming_service_filter.h"
#include "brpc/health_check_option.h"
#include "brpc/socket_mode.h"
#include "brpc/urma/urma_qos.h"

namespace brpc {

Expand Down Expand Up @@ -111,6 +112,9 @@ struct ChannelOptions {
// Default: SOCKET_MODE_TCP
SocketMode socket_mode;

// Outbound URMA QoS for this Channel. Used when socket_mode is URMA.
UrmaQosOptions urma_qos;

// Turn on authentication for this channel if `auth' is not nullptr.
// Note `auth' will not be deleted by channel and must remain valid when
// the channel is being used.
Expand Down
8 changes: 8 additions & 0 deletions src/brpc/server.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -876,6 +876,13 @@ int Server::StartInternal(const butil::EndPoint& endpoint,
const ServerOptions default_opt;
const ServerOptions& real_opt = opt ? *opt : default_opt;

if (real_opt.socket_mode == SOCKET_MODE_URMA &&
!real_opt.urma_qos.IsValid()) {
LOG(ERROR) << "URMA priority must be -1 or in [0,15], "
"and SL must be -1 or in [0,255]";
return -1;
}

if (!real_opt.h2_settings.IsValid(true/*log_error*/)) {
LOG(ERROR) << "Invalid h2_settings";
return -1;
Expand Down Expand Up @@ -1163,6 +1170,7 @@ int Server::StartInternal(const butil::EndPoint& endpoint,
_am->_socket_mode = _options.socket_mode;
_am->_bthread_tag = _options.bthread_tag;
}
_am->_urma_qos = _options.urma_qos;
// Set `_status' to RUNNING before accepting connections
// to prevent requests being rejected as ELOGOFF
_status = RUNNING;
Expand Down
4 changes: 4 additions & 0 deletions src/brpc/server.h
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
#include "brpc/baidu_master_service.h"
#include "brpc/rpc_pb_message_factory.h"
#include "brpc/socket_mode.h"
#include "brpc/urma/urma_qos.h"

namespace brpc {

Expand Down Expand Up @@ -242,6 +243,9 @@ struct ServerOptions {
// Default: SOCKET_MODE_TCP
SocketMode socket_mode;

// Outbound URMA QoS for responses on accepted connections.
UrmaQosOptions urma_qos;

// [CAUTION] This option is for implementing specialized baidu-std proxies,
// most users don't need it. Don't change this option unless you fully
// understand the description below.
Expand Down
6 changes: 6 additions & 0 deletions src/brpc/socket.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -711,6 +711,10 @@ void SetSocketBufferOptions(int fd) {
// version: from version part of _versioned_nref, must be an EVEN number.
// slot: designated by ResourcePool.
int Socket::Create(const SocketOptions& options, SocketId* id) {
if (options.socket_mode == SOCKET_MODE_URMA && !options.urma_qos.IsValid()) {
errno = EINVAL;
return -1;
}
return VersionedRefWithId<Socket>::Create(id, options);
}

Expand All @@ -726,6 +730,7 @@ int Socket::OnCreated(const SocketOptions& options) {
});
// start build the transport
_socket_mode = options.socket_mode;
_urma_qos = options.urma_qos;
_transport = TransportFactory::CreateTransport(options.socket_mode);
CHECK(nullptr != _transport);
_transport->Init(this, options);
Expand Down Expand Up @@ -2814,6 +2819,7 @@ SocketOptions Socket::GetSubSocketOptions() const {
opt.keytable_pool = _keytable_pool;
opt.app_connect = _app_connect;
opt.socket_mode = _socket_mode;
opt.urma_qos = _urma_qos;
opt.keepalive_options = _keepalive_options;
opt.tcp_user_timeout_ms = _tcp_user_timeout_ms;
opt.bthread_tag = _io_event.bthread_tag();
Expand Down
4 changes: 4 additions & 0 deletions src/brpc/socket.h
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
#include "brpc/versioned_ref_with_id.h"
#include "brpc/health_check_option.h"
#include "brpc/socket_mode.h"
#include "brpc/urma/urma_qos.h"
#include "brpc/input_messenger_processor.h" // InputMessengerProcessor

namespace brpc {
Expand Down Expand Up @@ -310,6 +311,7 @@ struct SocketOptions {
bool force_ssl{false};
std::shared_ptr<SocketSSLContext> initial_ssl_ctx;
SocketMode socket_mode{SOCKET_MODE_TCP};
UrmaQosOptions urma_qos;
bthread_keytable_pool_t* keytable_pool{nullptr};
SocketConnection* conn{nullptr};
std::shared_ptr<AppConnect> app_connect;
Expand Down Expand Up @@ -640,6 +642,7 @@ friend class TransportFactory;

// The socket mode decided at creation.
SocketMode socket_mode() const { return _socket_mode; }
const UrmaQosOptions& urma_qos() const { return _urma_qos; }

// Get an UNUSED socket connecting to the same place as this socket
// from the SocketPool of this socket.
Expand Down Expand Up @@ -1017,6 +1020,7 @@ friend class TransportFactory;

// Should use SOCKET_MODE_RDMA or SOCKET_MODE_TCP or Other, default is SOCKET_MODE_TCP Transport
SocketMode _socket_mode;
UrmaQosOptions _urma_qos;
std::unique_ptr<Transport> _transport;

// Pass from controller, for progressive reading.
Expand Down
Loading
Loading