Skip to content

branch-4.1: [improvement](hive) Batch Hive metastore partition access - #68197

Open
CalvinKirs wants to merge 1 commit into
apache:branch-4.1from
CalvinKirs:backport-67186-branch-4.1
Open

branch-4.1: [improvement](hive) Batch Hive metastore partition access#68197
CalvinKirs wants to merge 1 commit into
apache:branch-4.1from
CalvinKirs:backport-67186-branch-4.1

Conversation

@CalvinKirs

@CalvinKirs CalvinKirs commented Sep 18, 2026

Copy link
Copy Markdown
Member

@CalvinKirs
branch-4.1: [improvement](hive) Batch Hive metastore partition access (…
cd984d2 
…apache#67186)

Adapted backport of apache#67186 to the pre-connector architecture on branch-4.1:

- The HMS partition batch executor (bounded chunking, adaptive size
  fallback on thrift message-size failures, strict response validation)
  is ported into org.apache.doris.datasource.hive with the thrift
  Partition payload, wired inside ThriftHMSCachedClient.getPartitions;
  a lenient getExistingPartitions omits names the metastore no longer
  has, preserving the historical getPartitionsByNames behavior for
  cache bulk loads. hive.hms_partitions_batch_size_per_rpc rides the
  existing hive.* catalog-property-to-HiveConf pipeline (default 5000).
- HiveExternalMetaCache.getAllPartitions with cache now serves hits
  from the cache and aggregates all misses into one bulk load instead
  of one single-partition RPC per missed key.
- The MTMV bulk snapshot path (MTMVRefreshContext prepare/load with
  per-partition failure and missing caches, the MTMVRelatedTableIf
  bulk default, call sites in MTMVPartitionUtil / MTMVRewriteUtil /
  MTMVTask) applies with one deviation: the master-only pinned-MVCC-
  snapshot plumbing is dropped because its prerequisites
  (MTMVPartitionExpander, query-used partition filtering) do not exist
  on branch-4.1; snapshots keep resolving from the statement context.
- HMSExternalTable routes bulk partition snapshots to a HiveDlaTable
  implementation that resolves names against one partition-value
  listing and loads all cache misses through one batched HMS request,
  normalizing connector runtime failures to AnalysisException.
- The scan batch mode and connector-cache changes of the original PR
  target the plugin scan path only and have no branch-4.1 counterpart;
  the equivalent benefit comes from the batched cache bulk load.
@CalvinKirs

CalvinKirs commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

run buildall

@CalvinKirs

CalvinKirs commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

Performance verification of this backport against a real PostgreSQL-backed Hive 2.3.2 metastore with 120,000 real partitions, driving the branch-4.1 production ThriftHMSCachedClient directly (pool size 1, warm connection, medians of repeated runs; identical result checksums for every before/after pair):

Partitions Before RPCs Before latency After RPCs After latency Speedup
1,000 1,000 5.702 s 1 0.113 s 50.5x
10,000 10,000 48.201 s 2 0.536 s 89.9x
120,000 120,000 625.284 s 24 6.509 s 96.1x

Before is the per-partition access shape of the base branch (one getPartitionsByNames RPC per name — cold per-key cache loads and per-partition MTMV freshness reads). After is one logical call chunked at the default batch size of 5,000.

Batch-size sensitivity at 120,000 partitions (single runs, exercising the hive.hms_partitions_batch_size_per_rpc catalog property end to end): 100 → 12.9 s, 500 → 7.5 s, 1,000 → 8.1 s, 2,500 → 7.2 s, 5,000 → 6.6 s.

Notably, the base branch's unbounded bulk shape (all 120,000 names in one getPartitionsByNames, the physical shape of HiveExternalMetaCache.loadPartitions before this change) failed outright in the same session: the metastore worker thread died with java.lang.StackOverflowError planning the 120,000-element name list and the client saw a TTransportException, while the chunked path read the same partitions in 6.2–6.7 s. At this scale bounded chunking is the difference between completing and crashing the metastore request.

@hello-stephen

hello-stephen commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 71.96% (349/485) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@CalvinKirs @hello-stephen