HBASE-29081: Add HBase Read Replica Cluster feature

[anmolnar]

Hi all,

We would like to propose merging the feature “Read Replica Cluster” into
the main branch.

Background

We’d like to implement the open source version of Amazon’s Read Replica
Cluster on S3 feature for Apache HBase. It adds the ability of running
another HBase cluster on the same cloud storage location in read-only mode,
allowing users to share the read workload between multiple clusters. Due
to the characteristics of the implementation and the lack of automated
synchronization between the active and read-replica clusters, read replicas
are eventually consistent, hence they’re not suitable for reading most
recent data. However we still believe that users of open source Apache HBase
could take advantage of this feature and there are use cases out there which
read replicas could help with. Please find more information about the
feature in the linked blog post.

Pros

• Running multiple clusters in different Availability Zones adds HA to the
  entire workload,
• No need for data movement or duplication (active-active replication setup)
  which is cost and time efficient,
• No limit for the number of read replica clusters

Cons

• Read Replica clusters are eventually consistent: in memory data is not
  visible from read replicas,
• Read Replica clusters must be manually refreshed: flush on active cluster,
  refresh hfiles/meta on read replicas

A detailed description of the design and implementation can be found in the
following document.

Apache HBase Read Replica Cluster Feature

Please review and share your feedback or comments.

Best regards,
Andor

[anmolnar]

I have no idea what's wrong with CI builds. Code builds completely fine on my machine with the same JDK version and tests are also passing.

[charlesconnell]

That will be fixed in the next release of Yetus by apache/yetus#365

[taklwu]

[DISCUSS] I'm wondered if we should always use junit 5 when merging to master branch, if so, would it be possible to have it in this PR? or at least I want a follow up task on it.

[anmolnar]

[DISCUSS] I'm wondered if we should always use junit 5 when merging to master branch, if so, would it be possible to have it in this PR? or at least I want a follow up task on it.

I completely agree. I haven't actually noticed that we're still using JUnit 4 somewhere. I tried to migrate every test during the rebasing.

I created a sub-task for it: https://issues.apache.org/jira/browse/HBASE-30085

[taklwu]

another comment but otherwise LGTM,

I also assumed #7730 will be additions to this PR after we merge this change?

[anmolnar]

I also assumed #7730 will be additions to this PR after we merge this change?

Unfortunately we're not getting anywhere with that PR. Given that it's not a strict requirement to complete the feature, I'd like to go forward with this patch at the first place.

[anmolnar]

@charlesconnell Any news about the Yetus release?

[charlesconnell]

No news about the Yetus release

[kgeisz]

We have come across a bug where tables on the replica cluster are not getting new data post-refresh after the read-only flag has been flipped twice. This is being tracked in HBASE-30090, and we are working on a solution.

[anmolnar]

hey @Apache9 - You might want to take a look at this patch sooner rather than later. Please share your thoughts when you get a chance.

[kgeisz]

Hi @charlesconnell, PR #8115 addresses some of your comments and has been merged into this feature branch. Can you please mark the relevant comments as resolved? Thanks

[charlesconnell]

#8182 will help with Yetus failures once it's merged and you rebase

[anmolnar]

#8182 will help with Yetus failures once it's merged and you rebase

Rebased.

[anmolnar]

@charlesconnell Looks like Yetus is making progress with the builds this time.
We've finished all the outstanding review items on the patch. Could you please do another quick round of review and approve patch if everything is resolved?

[charlesconnell]

The yetus checks are failing for legit reasons, for example:

[ERROR] /home/runner/work/hbase/hbase/src/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java:[889,56] cannot find symbol
  symbol:   method getSuffixFileDataToWrite()
  location: variable mfs of type org.apache.hadoop.hbase.master.MasterFileSystem

and

/home/runner/work/hbase/hbase/src/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/master/BackupLogCleaner.java:[207,70] error: cannot find symbol
  symbol:   variable MASTER_REGION_DIR_NAME
  location: class MasterRegionFactory

I am happy to review again when the build is fixed

[ndimiduk]

I'm pretty sure what you're seeing is an artifact of how Yetus applies each commit in your PR branch (git apply --binary -v --stat --apply -p1 input.patch) against a clean checkout of master. Grepping logs, this looks important:

For ReadOnlyController.java, the apply log shows 9 stanzas:

• Line 544 of stat: 393 +++++++++ (HBASE-29083 adds the file)
• Lines 550, 552, 571, 649, 653, 655, 663, 669: subsequent modifications/re-adds across the feature branch
• Line 677: 913 -------------------- (HBASE-29841 deletion stanza, all -, no +)

Somehow it's not tracking changes to this file correctly across the entire history. Looks like this is a known issue in Yetus (YETUS-983) and has been around since 2020.

My best recommendation for now is to squish your history a bit and see what comes of it.

[ndimiduk]

Trying a fix. apache/yetus#384

[ndimiduk]

https://issues.apache.org/jira/browse/YETUS-983 is merged. Let's try it again.

[anmolnar]

Thanks @ndimiduk for the quick help. I triggered another round of CI.

[ndimiduk]

Glad to help.

FYI, last I looked, this commit history needs some cleanup before you land it on master. We still aim for one commit per JIRA issue. Not sure if you want to just squish it all down so that only the umbrella issue lands in the tree? In that case, be sure to update the fix version on all the subtasks appropriately (i.e., set to the feature branch release version, not the release version).

[anmolnar]

Glad to help.

FYI, last I looked, this commit history needs some cleanup before you land it on master. We still aim for one commit per JIRA issue. Not sure if you want to just squish it all down so that only the umbrella issue lands in the tree? In that case, be sure to update the fix version on all the subtasks appropriately (i.e., set to the feature branch release version, not the release version).

Thanks @ndimiduk for the heads-up, I didn't think about it yet. I can certainly squash the patch is a single commit that's quite straightforward. However, squashing multiple commits of a single sub-task would be a bit more time-consuming, but I think it's still feasible.

What do you recommend?

[ndimiduk]

I think that a single squished commit is easiest.

[charlesconnell]

I have one unresolved comment still, but other than that no new comments

[anmolnar]

I have one unresolved comment still, but other than that no new comments

I think we have resolved all the outstanding comments. Please carry on with your review when you have a chance.

[charlesconnell]

Yetus failures seem bogus. LGTM.

[anmolnar]

Thank you @charlesconnell ! Let me squash the commits.

[anmolnar]

@ndimiduk Looks like squashing the commits doesn't make any difference.

[ndimiduk]

It's not yetus, the failures are legit -- your merge target is shifting under you. You need to rebase onto master.

[ndimiduk]

Click through into a failed check and look at the Summary document, https://github.com/apache/hbase/actions/runs/26121214670 . Click on the House and Summary at the top and you'll see an overview.

[anmolnar]

Thanks @ndimiduk for looking. Let me do the rebase and fix the problems.