Skip to content

feat(store): implement pluggable cloud storage for HStore#3081

Open
vaijosh wants to merge 5 commits into
apache:masterfrom
vaijosh:Hstore+CloudStorage
Open

feat(store): implement pluggable cloud storage for HStore#3081
vaijosh wants to merge 5 commits into
apache:masterfrom
vaijosh:Hstore+CloudStorage

Conversation

@vaijosh

@vaijosh vaijosh commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Purpose

This PR introduces a cloud storage architecture for HugeGraph Store (HStore). In cloud-native environments, storage nodes are ephemeral, making reliance on local disk storage a single point of failure. This implementation decouples the storage layer from local disk dependencies, allowing SST files to be offloaded to durable, scalable cloud storage providers.

Key Changes

Pluggable Architecture

  • Created the CloudStorageProvider SPI interface to enable extensible storage backends.
  • Added CloudStorageProviderFactory for seamless provider discovery and lifecycle management.

S3 Provider Implementation

  • Introduced the hg-store-cloud-s3 module, leveraging AWS SDK v2 for production-ready S3/S3 compatible storage interaction.

Lifecycle & Event Integration

  • Integrated CloudStorageEventListener with RocksDBFactory. This hooks into critical SST lifecycle events (onTableFileCreated, onTableFileDeleted) to ensure synchronized state between local RocksDB and cloud storage.
  • Implemented startup hydration (onDBCreated) to backfill pre-existing files and read-miss on-demand hydration to ensure data availability.

Infrastructure & Testing

  • Added a comprehensive local development environment via docker/cloud-storage/ using MinIO.
  • Included an integration test script (test-graph-queries-and-sst.sh) to verify end-to-end data durability and query consistency.

Implementation Highlights

  • SPI Integration: Standardized service discovery via META-INF/services.
  • RocksDB Hooking: Registered a singleton RocksdbEventListener within RocksDBFactory to intercept file operations without modifying core RocksDB logic.
  • Read-Miss Protection: Added a guard window mechanism in onReadMiss to prevent redundant hydration requests during high-concurrency read scenarios.

Verifying These Changes

  • Unit Tests: Coverage for config parsing (CloudStorageConfigTest), factory registration, and event listener logic.
  • Integration Tests: Verified using the new docker-compose setup with MinIO.
  • Configuration: Verified application.yml bindings for credentials, bucket management, and sync intervals.

Impact

  • Dependencies: Added AWS SDK v2; updated LICENSE and NOTICE accordingly.
  • Configurations: Added cloud.storage namespace to application.yml (disabled by default).
  • Public API: No breaking changes to existing public-facing client APIs.

Documentation

  • Architecture and configuration details are documented in hugegraph-store/docs/pluggable-cloud-storage-architecture.md.

@vaijosh vaijosh marked this pull request as ready for review July 6, 2026 14:11
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. feature New feature store Store module labels Jul 6, 2026
vaijosh added a commit to vaijosh/hugegraph that referenced this pull request Jul 8, 2026
apache#3081
-added multipart upload for the S3 storage provider
@vaijosh vaijosh force-pushed the Hstore+CloudStorage branch from 6a3f496 to 8b6d597 Compare July 8, 2026 10:29
apache#3081
-added multipart upload for the S3 storage provider
@vaijosh vaijosh force-pushed the Hstore+CloudStorage branch from 8b6d597 to 4275426 Compare July 8, 2026 10:59
@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.19355% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.14%. Comparing base (03e6b8e) to head (4275426).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...graph/store/cloud/CloudStorageProviderFactory.java 71.42% 11 Missing and 3 partials ⚠️
...ache/hugegraph/store/cloud/CloudStorageConfig.java 90.00% 1 Missing ⚠️
...he/hugegraph/store/cloud/CloudStorageProvider.java 0.00% 1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (03e6b8e) and HEAD (4275426). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (03e6b8e) HEAD (4275426)
3 1
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #3081       +/-   ##
============================================
- Coverage     36.80%   0.14%   -36.66%     
+ Complexity      338      42      -296     
============================================
  Files           805     787       -18     
  Lines         68587   65951     -2636     
  Branches       9029    8550      -479     
============================================
- Hits          25241      97    -25144     
- Misses        40653   65849    +25196     
+ Partials       2693       5     -2688     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature size:XXL This PR changes 1000+ lines, ignoring generated files. store Store module

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

1 participant