Skip to content

optimize HBase request and response CPU performance - #442

Merged
WeiXinChan merged 5 commits into
oceanbase:masterfrom
hhlgt:optimize/client-cpu-performance
Sep 1, 2026
Merged

optimize HBase request and response CPU performance#442
WeiXinChan merged 5 commits into
oceanbase:masterfrom
hhlgt:optimize/client-cpu-performance

Conversation

@hhlgt

@hhlgt hhlgt commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR optimizes CPU usage and object allocation on the OBKV HBase request and response hot paths.

The changes focus on HBase query/scan response decoding, Batch Get response decoding, Put V2 request encoding, object type lookup, CRC32C calculation, and result container allocation.

Solution Description

The optimization consists of four related changes:

  1. Optimize HBase query and scan response decoding

    • Add a compact ObHBaseCellBatch representation for K/Q/T/V results.
    • Decode HBase cells directly into compact arrays, avoiding per-cell ObObj and row-list allocations.
    • Preallocate query and streaming result containers based on decoded row counts.
    • Retain generic ObObj decoding as a compatibility fallback when the schema or object metadata does not match the expected HBase K/Q/T/V layout.
  2. Optimize HBase Put V2 request encoding

    • Add a dedicated compact codec for HBase Put cells.
    • Encode qualifiers, timestamps, values, and TTLs directly from byte-array slices.
    • Avoid temporary cell objects, intermediate lists, and unnecessary byte-array copies.
    • Reserve container capacity in advance for batch Put requests.
  3. Optimize HBase Batch Get response decoding

    • Propagate the HBase decoding context through LS operation responses.
    • Automatically enable compact K/Q/T/V decoding for eligible HKV GET_LIST and BATCH requests.
    • Preallocate Batch Get result containers and decode values directly into compact HBase cell batches.
    • Fall back to the generic decoder when the response type layout is incompatible.
  4. Optimize common request-path utilities

    • Replace hot-path enum map lookups with array-based lookup while retaining map fallback for extended type IDs.
    • Use slicing-by-8 CRC32C for request checksum calculation.
    • Remove response checksum verification from the response hot path, avoiding a full payload copy and an additional CRC traversal. Request checksum generation remains unchanged.
    • Correctly honor ObBytesString offset and length during serialization, conversion, and partition hashing.
    • Remove internal tuning switches and apply the optimized paths automatically only when their eligibility conditions are satisfied.

The wire format remains compatible. Existing generic decoding and legacy Put representations are preserved as fallback paths.

Targeted unit tests were added or updated for compact query decoding, stream decoding, Batch Get decoding, Put encoding, type lookup, byte-slice handling, hashing, and CRC32C calculation. The newly added optimization-path tests pass locally. Full integration tests require a configured OceanBase/ZDAL test environment and were not run locally.

hhlgt and others added 5 commits August 5, 2026 15:48
Make response CRC optional, accelerate object type lookup, preallocate query results, add KQTV fast decoding, and expose compact HBase result batches.
Reduce CPU usage and object allocation when constructing and encoding HBase Put V2 requests.
Preallocate Batch Get response containers and replace object type map lookups with a hybrid array/map lookup. Propagate Batch Get decode context through LS responses and decode K/Q/T/V values directly into compact HBase cell batches. Keep the generic ObObj decoder as a compatibility fallback.
@hhlgt hhlgt changed the title Optimize/client cpu performance perf: optimize HBase request and response CPU performance Aug 31, 2026
@hhlgt hhlgt changed the title perf: optimize HBase request and response CPU performance optimize HBase request and response CPU performance Aug 31, 2026
@hhlgt

hhlgt commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

AI Coding Review

审查范围:
Base: origin/master
Head: optimize/client-cpu-performance
Commits: 4

[P1] 不应在性能优化中无条件移除响应 CRC32C 校验

位置:[ObTableRemoting.java (line 107)]
基线代码在解析响应前,会根据 RPC Header 中的 checksum 校验整个响应内容。本 PR 最终完全删除了响应 CRC32C 校验,收到数据后直接开始解析 ResultCode 和 payload。
影响:

  • 网络传输、代理或内存损坏不再产生 BOLT_CHECKSUM_ERR。
  • 被破坏的响应可能继续进入协议解码。
  • 如果损坏内容仍能成功解码,调用方可能收到错误数据、错误路由信息或异常的结果数量。
  • 这是协议完整性行为变化,不只是 CPU/对象分配优化。

建议:

  • 保留响应 checksum 校验,优化计算过程本身。
  • 可以直接针对 ByteBuf 或其 NIO buffers 增量计算 CRC,避免复制整个响应到临时 byte[]。
  • 如果特定服务端版本不提供有效 checksum,应基于明确的协议字段或版本能力判断,而不是全局删除。
  • 增加损坏响应的回归测试,确认仍返回 TransportCodes.BOLT_CHECKSUM_ERR。
    ##其他审查结论
    紧凑 K/Q/T/V 解码、类型数组查找、Put V2 紧凑编码和 ObBytesString slice 处理暂未发现阻塞级正确性问题。通用 ObObj fallback 仍然保留。

Review 结论

合入前至少需要恢复或替代响应完整性校验。

@WeiXinChan

Copy link
Copy Markdown
Contributor

LGTM

@WeiXinChan
WeiXinChan merged commit 5a85940 into oceanbase:master Sep 1, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants