Conversation
|
为了便于理解，我想对flatbuffers构造消息的基本流程进行简要补充说明，该示例程序的Scheme文件（类似于.proto文件）如下。 table EchoRequest {
opcode:int;
attachment:int;
}Message需要通过MessageBuilder创建（定义在flatbuffers_impl.h中），该类继承了::flatbuffers::FlatBufferBuilder，它才是实际起作用的类，MessageBuilder的主要作用是向它传递自定义的内存分配器（SlabAllocator）。应用在创建message时，需要先定义一个MessageBuilder，然后基于它来添加元素，最终调用ReleaseMessage方法来获取完整消息，以下是一段示例代码。 brpc::flatbuffers::MessageBuilder mb;
auto message = mb.CreateString(g_request);
auto req = test::CreateBenchmarkRequest(mb, 123, 333, 1111, 2222, 0, message);
mb.Finish(req);
brpc::flatbuffers::Message request = mb.ReleaseMessage();在构造message过程中，flatbuffers会使用一段连续buffer存储数据和metadata，在代码里，该buffer使用vector_downward表示，由FlatBufferBuilder管理。该Buffer维护了scratch_和cur_两个指针。在初次向buffer中添加数据时，会申请max(需要长度，1024)字节的内存，之后会使用cur_将数据添加在buffer末尾，将偏移量信息(FieldLoc)添加在buffer起始位置，如下图所示。如果在添加过程中buffer不够了，则会重新分配一个足够大的buffer，并将数据按序拷贝过去，所以或许可以配置一个GFlag来调整初次分配buffer的大小（TODO）。 brpc只需要定义好内存分配器SlabAllocator，在添加元素的过程中，当内存不足时flatbuffers会调用allocate或reallocate_downward，这实际上会调用SingleIOBuf中的相关方法。我们发现将rpc请求的header部分与message连续存储可获得明显性能提升。因此，在申请buffer时我们在Buffer前面预留出了一部分空间，其大小由宏DEFAULT_RESERVE_SIZE指定，默认为64B（也可通过GFlag配置，TODO）。这部分内存的消费者是brpc中的协议处理逻辑，在创建message过程中不会被使用到。 flatbuffers读取消息的本质是进行间接寻址。参考上图，当读取EchoRequest中的opcode时，首先从buffer起始地址出读取Root Offset，获取Table Data相对于buffer起始地址的偏移量；然后从Table Data起始处获取Vtable的相对偏移量，再次跳转；然后从预先生成的FlatBuffersVTableOffset中获取opcode在vtable中的位置，访问它获取实际偏移量；当字段合法时，通过(Table Data + 偏移量)获取到实际值。该过程需要3次间接寻址。 |
There was a problem hiding this comment.
Pull request overview
This pull request adds FlatBuffers message construction logic to brpc as part 2 of a 3-part implementation to support the FlatBuffers protocol. The PR introduces custom allocators, message builders, and service descriptors that integrate FlatBuffers with brpc's zero-copy IOBuf system.
Changes:
- Adds FlatBuffers message construction infrastructure with custom SlabAllocator for zero-copy operations
- Implements Message and MessageBuilder classes for FlatBuffers message lifecycle management
- Provides ServiceDescriptor and MethodDescriptor classes for RPC method resolution
- Fixes a bug in SingleIOBuf::assign() by moving reset() call to ensure proper cleanup in all paths
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 23 comments.
Show a summary per file
| File | Description |
|---|---|
| src/butil/single_iobuf.cpp | Bug fix: moves reset() call before conditional to ensure proper cleanup in both code paths |
| src/brpc/details/flatbuffers_common.h | Defines base interfaces (Service, RpcChannel) for FlatBuffers RPC integration |
| src/brpc/details/flatbuffers_impl.h | Core implementation header with SlabAllocator, Message, MessageBuilder, and descriptor classes |
| src/brpc/details/flatbuffers_impl.cpp | Implementation of allocators, message handling, and service descriptor parsing |
| example/benchmark_fb/test_generated.h | Auto-generated FlatBuffers message definitions (temporary example) |
| example/benchmark_fb/test.fbs | FlatBuffers schema for benchmark service (temporary example) |
| example/benchmark_fb/test.brpc.fb.h | Auto-generated brpc service stub header (temporary example) |
| example/benchmark_fb/test.brpc.fb.cpp | Auto-generated brpc service stub implementation (temporary example) |
| example/benchmark_fb/server.cpp | Example benchmark server implementation (temporary example) |
| example/benchmark_fb/client.cpp | Example benchmark client implementation (temporary example) |
| example/benchmark_fb/CMakeLists.txt | Build configuration for benchmark example (temporary) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Key components: - flatbuffers_common.h: Defines abstract interfaces (RpcChannel and Service) for FlatBuffers-based RPC communication, similar to protobuf's RPC interfaces. - flatbuffers_impl.h: Implements FlatBuffers-specific message handling: * SlabAllocator: Custom allocator using SingleIOBuf for zero-copy operations. * Message: Wrapper for FlatBuffers messages with SingleIOBuf storage. * MessageBuilder: BRPC-specific FlatBufferBuilder with SlabAllocator. * ServiceDescriptor/MethodDescriptor: Service introspection support. - flatbuffers_impl.cpp: Implementation of allocation, serialization, and service descriptor initialization logic
|
@Q1ngbo 我们最近基于 #3196 和 #3197 的 FlatBuffers 支持代码进行了进一步验证和完善。
我们想先征求一下你的意见：你是否希望我们一起更新现有 PR，还是可以接受基于现有实现重新整理并提交一组便于评审的新 PR？ |
@Spicy-cream Hi，due to some previous issues, we have been delayed. Over the next couple of days, we will reorganize and submit a new set of PRs that are more suitable for review. |
|
Message/codegen follow-up: commits. |
|
Message/codegen follow-up: commits @wwbmmm @ivanallen @Spicy-cream: Could you please review it together? |
Please submit a PR |
What problem does this PR solve?
Issue Number: resolve #2354 #978
Problem Summary:
Hi, 我基于先前的commit #3062 完成了flatbuffers协议对brpc的适配，准备将完整实现提交至社区。在brpc中增加flatbuffers(fb)支持可分为fb message构造与fb协议处理两部分。前者解决的问题是如何使用flatbuffers库提供的接口去创建一个message，以及应用接收到message后如何从中读取出数据；后者解决的是brpc如何处理"fb"协议。因此，我将相关实现拆分为了两个提交。本提交重点在于message构造。
我之所以将message构造提交至brpc仓库中而不是像grpc一样在google/flatbuffers中实现，主要出于两点考虑：首先，为了与IOBuf兼容，创建message使用的底层数据结构为之前已经提交的SingleIOBuf，在flatbuffers中使用该数据结构会导致循环依赖；其次，flatbuffers提供了完善的接口，只需要在brpc里定义好内存分配器，即可调用相关接口来构造消息，实现上很方便。
特别说明：
Check List: