From 20bc15de3ae43212ed46aa2bbf1289bdd960b896 Mon Sep 17 00:00:00 2001 From: Nick Dimiduk Date: Sat, 23 May 2026 14:14:44 +0200 Subject: [PATCH 1/2] HBASE-30173: Add AGENTS.md for LLM-assisted codebase navigation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LLM-based tools (code assistants, security scanners, review agents) increasingly operate against large codebases. This adds an AGENTS.md convention file that teaches these tools how to navigate HBase's 37+ Maven modules, understand the client/server divide, use existing @InterfaceAudience annotations for orientation, and locate key entry points. The security model section points to the existing documentation rather than duplicating it. Also adds a dev-support/README.md index and fixes stale content across 12 existing READMEs — incorrect paths, removed modules, outdated versions, broken URLs, and a complete rewrite of hbase-endpoint and hbase-rest READMEs that described code no longer present. Three Java comment references to the renamed hbase-hadoop2-compat module were corrected as well. Co-Authored-By: Claude Opus 4.6 --- AGENTS.md | 187 ++++++++++++++++++ dev-support/README.md | 84 ++++++++ dev-support/code-coverage/README.md | 4 +- dev-support/hbase_docker/README.md | 2 +- dev-support/release-vm/README.md | 2 +- hbase-archetypes/README.md | 10 +- hbase-endpoint/README.txt | 25 +-- hbase-examples/README.txt | 4 +- .../hadoop/hbase/metrics/BaseSourceImpl.java | 8 +- .../hbase/mapreduce/TableMapReduceUtil.java | 2 +- hbase-metrics-api/README.txt | 8 +- hbase-protocol-shaded/README.txt | 3 +- hbase-rest/README.txt | 21 +- .../hbase/regionserver/MetricsRegion.java | 2 +- hbase-website/README.md | 10 +- 15 files changed, 323 insertions(+), 49 deletions(-) create mode 100644 AGENTS.md create mode 100644 dev-support/README.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000000..2d4125fb1f34 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,187 @@ + + +# AGENTS.md + +Apache HBase is a distributed, scalable big data store built on HDFS and cloud +object storage. + +## Repo Structure + +This is a multi-module Maven project. Modules live in arbitrarily nested +folders; enumerate them by searching for `pom.xml` files (excluding `target/` +directories). The root `pom.xml` defines the full reactor and build order. + +### Client and Server + +The fundamental divide in this codebase is client-side vs. server-side, with +several modules shared between them. + +- `hbase-client` -- The client library. Builds RPC requests, handles retries, + manages connections. This is the public API that external consumers depend on. +- `hbase-server` -- RegionServer and Master implementations. Processes RPCs, + manages regions, stores data. The largest module by far. +- Shared modules like `hbase-common`, `hbase-protocol-shaded`, and + `hbase-metrics-api` are dependencies of both sides. + +When orienting on unfamiliar code, first determine which side of this divide +you are on. + +### Module Roles + +**Core data path:** +`hbase-client` -> `hbase-server` (via protobuf RPCs defined in +`hbase-protocol-shaded`) + +**Gateways** (alternative client entry points): +`hbase-rest` (HTTP/JSON), `hbase-thrift` (Thrift RPC) + +**Coprocessors** are HBase's server-side extension framework. They allow custom +code to run inside RegionServer and Master processes, with the same privileges +as the host process. Coprocessor interfaces (observers, endpoints) are defined +in `hbase-client` and `hbase-server` under the `coprocessor` package; endpoint +implementations live in `hbase-endpoint`. The built-in `AccessController` +coprocessor enforces ACLs; `VisibilityController` enforces cell-level +visibility labels. Third-party coprocessors are loaded via configuration or +table schema. + +**Server subsystems** (separated from hbase-server for modularity): +`hbase-balancer`, `hbase-procedure`, `hbase-replication`, `hbase-asyncfs`, +`hbase-zookeeper`, `hbase-http` + +**Shared libraries:** +`hbase-common`, `hbase-metrics` + `hbase-metrics-api`, `hbase-logging`, +`hbase-hadoop-compat` + +**Storage codecs:** +`hbase-compression/*` (pluggable algorithms), `hbase-external-blockcache` + +**Packaging and shading:** +`hbase-shaded/*`, `hbase-assembly*`, `hbase-resource-bundle` + +**Tooling:** +`hbase-shell` (JRuby REPL), `hbase-hbtop`, `hbase-mapreduce`, `hbase-backup`, +`hbase-diagnostics` + +**Build infrastructure** (ignore for code tasks): +`hbase-build-configuration`, `hbase-checkstyle`, `hbase-annotations`, +`hbase-archetypes/*`, `hbase-dev-generate-classpath` + +**Testing:** +`hbase-testing-util`, `hbase-it`, `hbase-examples` + +### Navigating with @InterfaceAudience + +Classes are annotated with `@InterfaceAudience` to indicate their intended +consumer: + +- `Public` -- Stable client API. External consumers depend on these. +- `LimitedPrivate` -- Internal API shared across modules, scoped to a named + audience (e.g., `COPROC`, `CONFIG`, `REPLICATION`, `AUTHENTICATION`). The + audience name tells you who is expected to call this code. +- `Private` -- Module-internal. Not API. + +These annotations are the fastest way to determine whether a class is part of +the external surface or internal plumbing. + +### Key Entry Points + +When investigating a behavior, start from where it enters the system: + +- **Client RPCs**: `RSRpcServices` (RegionServer) and `MasterRpcServices` + (Master) handle all client-initiated RPCs. Trace from the method matching + the RPC name. +- **REST gateway**: resource classes in `hbase-rest` map HTTP verbs to + operations. +- **Thrift gateway**: handler classes in `hbase-thrift` map Thrift methods. +- **Coprocessor hooks**: observer interfaces (`RegionObserver`, + `MasterObserver`, etc.) define extension points. Implementations are loaded + via configuration or table schema. +- **Procedures**: `hbase-procedure` defines the framework; concrete procedures + (table create, region split, etc.) live in `hbase-server`. +- **Configuration**: properties are defined in `hbase-default.xml` (in + `hbase-common`) and overridden by operators in `hbase-site.xml`. +- **Wire format**: `.proto` files in `hbase-protocol-shaded` define every RPC + request/response and all persisted data structures. (Older branches had a + separate `hbase-protocol` module; it has been removed on master.) + +### Split Packages + +The same Java package often appears in multiple modules (e.g., the +`coprocessor` package exists in `hbase-client`, `hbase-server`, +`hbase-endpoint`, and `hbase-examples`). Each module contributes different +classes to the package. When searching for a class, check which module it +lives in -- the module determines the execution context. + +### Related Repositories + +[hbase-thirdparty](https://github.com/apache/hbase-thirdparty) is a companion +project that patches and shades key dependencies (protobuf, netty, gson, etc.) +so that HBase's internal use of these libraries does not conflict with +versions on the application classpath. The `hbase-shaded-*` artifacts from +that repo appear as dependencies throughout this project's `pom.xml`. Changes +to shaded dependency versions or patches happen in that repo, not here. + +### Developer Tooling + +`dev-support/` contains CI configuration, release automation, code analysis +scripts, and other maintainer tools. PR-level CI has migrated to GitHub +Actions (`.github/workflows/`), but nightly and branch-level CI still runs +via configurations in `dev-support/`. That directory also holds release +scripts, docker-based test environments, and various developer utilities. +See `dev-support/README.md` for a full index. + +`conf/` holds default configuration templates (`hbase-site.xml`, +`hbase-env.sh`, `log4j2.properties`). `bin/` holds shell scripts for cluster +lifecycle and operations. + +`dev-support/design-docs/` collects design documents and proposals for major +features. These capture the rationale behind complex subsystems and are useful +for understanding why the code is structured the way it is. + +### Conventions + +- Tests mirror source paths: `src/test/java` parallels `src/main/java` +- Generated code (protobuf, etc.) lives in `target/` and is not checked in +- Configuration properties use `hbase.` prefix +- The shell is JRuby wrapping the Java client API + +## Documentation + +The project website (https://hbase.apache.org) is maintained in this repo under +`hbase-website/`. User-facing and administrator-facing documentation covering +configuration, security, architecture, schema design, operations, APIs, and +more lives in `hbase-website/app/pages/_docs/docs/_mdx/`. The table of +contents and page ordering is defined in the `meta.json` files within that +tree. + +The site also serves https://hbase.apache.org/llms-full.txt, which +concatenates all documentation pages into a single text file suitable for +LLM context ingestion. + +## Security Model + +The project's security model is documented at +`hbase-website/app/pages/_landing/security-model/content.md` +(published at https://hbase.apache.org/security-model). +Read that document for the full security model including trust boundaries, +what constitutes a valid vulnerability, and what does not. + +When performing security analysis of this codebase, use the navigation +structure above to determine the role of the code under review, then apply +the security model to interpret findings in context. diff --git a/dev-support/README.md b/dev-support/README.md new file mode 100644 index 000000000000..9dc3f3143616 --- /dev/null +++ b/dev-support/README.md @@ -0,0 +1,84 @@ + + +# dev-support + +Developer and maintainer tooling for the HBase project. This directory +contains CI configuration, release automation, code analysis, and various +utility scripts. + +## CI + +PR-level CI runs via GitHub Actions (see `../.github/workflows/`). Nightly +builds, branch validation, and precommit checks still use Jenkins +configurations in this directory: + +- `Jenkinsfile`, `Jenkinsfile_GitHub` -- Pipeline definitions +- `hbase_nightly_yetus.sh`, `jenkins_precommit_github_yetus.sh` -- Yetus-based + precommit and nightly check scripts +- `hbase-personality.sh` -- Yetus personality plugin that customizes checks for + HBase +- `jenkinsEnv.sh`, `jenkins-scripts/` -- Shared Jenkins environment setup +- `HOW_TO_YETUS_LOCAL.md` -- Guide for running Yetus checks locally + +## Release Automation + +- `create-release/` -- Docker-based release candidate builder (tags, builds, + signs, publishes). Entry point is `do-release-docker.sh`. +- `make_rc.sh` -- Older release candidate script (superseded by + `create-release/`) +- `hbase-vote.sh` -- Generates release vote email content +- `git-jira-release-audit/` -- Audits git history against JIRA fixVersion + fields to find discrepancies between what was committed and what JIRA says + shipped + +## Code Quality and Analysis + +- `checkcompatibility.py` -- Checks API/ABI compatibility between versions +- `checkstyle_report.py` -- Generates checkstyle reports +- `spotbugs-exclude.xml` -- SpotBugs exclusion rules +- `code-coverage/` -- Scripts for generating code coverage reports +- `flaky-tests/` -- Flaky test detection, reporting, and dashboards +- `license-header` -- Apache License header template + +## Docker and Test Environments + +- `docker/` -- Dockerfile for CI build environment +- `hbase_docker/`, `hbase_docker.sh` -- Docker-based local test cluster +- `adhoc_run_tests/` -- Scripts for running test suites outside CI +- `integration-test/` -- Integration test support + +## Utility Scripts + +- `smart-apply-patch.sh`, `make_patch.sh` -- Patch creation and application +- `rebase_all_git_branches.sh` -- Rebases all local tracking branches +- `zombie-detector.sh` -- Detects leaked processes from test runs +- `gather_machine_environment.sh` -- Captures build machine info for debugging +- `gh_hide_old_comments.sh` -- Hides outdated bot comments on GitHub PRs + +## IDE Configuration + +- `hbase_eclipse_formatter.xml` -- Eclipse code formatter settings +- `eclipse.importorder` -- Eclipse import ordering +- `HBase Code Template.xml` -- IntelliJ code template + +## Design Documents + +`design-docs/` collects design documents and proposals for major features. +These capture the rationale behind complex subsystems and are useful for +understanding why the code is structured the way it is. diff --git a/dev-support/code-coverage/README.md b/dev-support/code-coverage/README.md index 0b3eaf044acb..8e56a127229f 100644 --- a/dev-support/code-coverage/README.md +++ b/dev-support/code-coverage/README.md @@ -31,7 +31,7 @@ the related modules. Here is how you can generate the code coverage report: -```sh dev/code-coverage/run-coverage.sh``` +```sh dev-support/code-coverage/run-coverage.sh``` ## Publishing coverage results to SonarQube @@ -45,5 +45,5 @@ The project name is an optional parameter. Here is an example command for running and publishing the coverage data: -`./dev/code-coverage/run-coverage.sh -l ProjectCredentials +`./dev-support/code-coverage/run-coverage.sh -l ProjectCredentials -u https://exampleserver.com -k Project_Key -n Project_Name` diff --git a/dev-support/hbase_docker/README.md b/dev-support/hbase_docker/README.md index 3d0641afaee9..d37e329cca19 100644 --- a/dev-support/hbase_docker/README.md +++ b/dev-support/hbase_docker/README.md @@ -42,5 +42,5 @@ this image will start the HMaster and launch the HBase shell when run. bash` to start a container without a running HMaster. Within this environment, HBase is built in `/root/hbase-bin`. -> NOTE: When running on mac m1 platforms, the docker file requires setting platfrom flag explicitly. +> NOTE: When running on mac m1 platforms, the docker file requires setting platform flag explicitly. > You may use same instructions above running from to the "./m1" sub-dir. diff --git a/dev-support/release-vm/README.md b/dev-support/release-vm/README.md index 74bb4392d2eb..a9509ed6bad7 100644 --- a/dev-support/release-vm/README.md +++ b/dev-support/release-vm/README.md @@ -24,7 +24,7 @@ for running an Apache HBase release. Requires: * [VirtualBox](http://virtualbox.org) -* [Vagrant](http://virtualbox.org) +* [Vagrant](https://www.vagrantup.com/) * The private portion of your signing key avilable in the local GPG agent * The private portion of your Github authentication key available in either the local GPG agent or local SSH agent diff --git a/hbase-archetypes/README.md b/hbase-archetypes/README.md index 6ab3b3d88e46..c40b454dc92d 100644 --- a/hbase-archetypes/README.md +++ b/hbase-archetypes/README.md @@ -45,7 +45,7 @@ exemplar Maven project containing: For example, the components of the hbase-client-project consist of (a) sample code `./src/main/.../HelloHBase.java` and `./src/test/.../TestHelloHBase.java`, (b) a `pom.xml` file establishing dependency upon hbase-client and test-scope -dependency upon hbase-testing-util, and (c) a `log4j.properties` resource file. +dependency upon hbase-testing-util, and (c) a `log4j2.properties` resource file. #### How archetypes are created during the hbase install process During the `mvn install` process, all standalone exemplar projects in the @@ -108,19 +108,19 @@ to choose from for generation of a new Maven project. ## Footnotes: 1 -- [Maven Archetype -](http://maven.apache.org/archetype/index.html) ("About" page). +](https://maven.apache.org/archetype/index.html) ("About" page). -- [↩](#a1) 2 -- [Maven Archetype Catalog -](http://repo1.maven.org/maven2/archetype-catalog.xml) (4MB+ xml file). +](https://repo1.maven.org/maven2/archetype-catalog.xml) (4MB+ xml file). -- [↩](#a2) -3 -- [Maven Central Repository](http://search.maven.org/) +3 -- [Maven Central Repository](https://search.maven.org/) (search engine). -- [↩](#a3) 4 -- [Maven Archetype Plugin - archetype:generate -](http://maven.apache.org/archetype/maven-archetype-plugin/generate-mojo.html). +](https://maven.apache.org/archetype/maven-archetype-plugin/generate-mojo.html). -- [↩](#a4) 5 -- Prior to archetype creation, each exemplar project's diff --git a/hbase-endpoint/README.txt b/hbase-endpoint/README.txt index 2d2cb4b36049..3b2c15a2d54f 100644 --- a/hbase-endpoint/README.txt +++ b/hbase-endpoint/README.txt @@ -1,13 +1,14 @@ -ON PROTOBUFS -This maven module has protobuf definition files ('.protos') used by hbase -Coprocessor Endpoints that ship with hbase core (including tests). Coprocessor -Endpoints are meant to be standalone, independent code not reliant on hbase -internals. They define their Service using protobuf. The protobuf version -they use can be distinct from that used by HBase internally since HBase started -shading its protobuf references. Endpoints have no access to the shaded protobuf -hbase uses. They do have access to the content of hbase-protocol -- the -.protos found in this module -- but avoid using as much of this as you can as it is -liable to change. +This module contains coprocessor endpoint implementations that ship with +HBase core. Coprocessor endpoints are standalone RPC services deployed on +region servers (or master) via the coprocessor framework. -Generation of java files from protobuf .proto files included here is done as -part of the build. +Included endpoints: + - AggregateImplementation -- server-side aggregation (sum, min, max, avg, etc.) + - Export -- server-side table export to HDFS + +Client helpers for invoking these endpoints are in the +org.apache.hadoop.hbase.client.coprocessor package (AggregationClient, +AsyncAggregationClient). + +The protobuf service definitions used by these endpoints live in +hbase-protocol-shaded, not in this module. diff --git a/hbase-examples/README.txt b/hbase-examples/README.txt index 5425af092339..7b7fefcbe99f 100644 --- a/hbase-examples/README.txt +++ b/hbase-examples/README.txt @@ -38,7 +38,7 @@ Example code. should only be specified when the client connects to a secure cluster. It's default value is "hbase". 4. Here is a lazy example that just pulls in all hbase dependency jars and that goes against default location on localhost. It should work with a standalone hbase instance started by doing ./bin/start-hbase.sh: - {java -cp ./hbase-examples/target/hbase-examples-2.0.0-SNAPSHOT.jar:`./bin/hbase classpath` org.apache.hadoop.hbase.thrift.DemoClient localhost 9090} + {java -cp ./hbase-examples/target/hbase-examples-4.0.0-alpha-1-SNAPSHOT.jar:`./bin/hbase classpath` org.apache.hadoop.hbase.thrift.DemoClient localhost 9090} * Ruby: hbase-examples/src/main/ruby/DemoClient.rb 1. Modify the import path in the file to point to {$THRIFT_HOME}/lib/rb/lib. @@ -64,7 +64,7 @@ Example code. * CPP: hbase-examples/src/main/cpp/DemoClient.cpp 1. Make sure you have Thrift C++ libraries; modify Makefile if necessary. - The recent (0.14.1 as of this writing) version of Thrift can be downloaded from http://thrift.apache.org/download/. + The recent (0.14.1 as of this writing) version of Thrift can be downloaded from https://thrift.apache.org/download/. 2. Execute {make}. 3. Execute {./DemoClient }. diff --git a/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseSourceImpl.java b/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseSourceImpl.java index 2db502deb152..0b84fcb4d488 100644 --- a/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseSourceImpl.java +++ b/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseSourceImpl.java @@ -32,9 +32,9 @@ import org.apache.yetus.audience.InterfaceAudience; /** - * Hadoop 2 implementation of BaseSource (using metrics2 framework). It handles registration to + * Hadoop implementation of BaseSource (using metrics2 framework). It handles registration to * DefaultMetricsSystem and creation of the metrics registry. All MetricsSource's in - * hbase-hadoop2-compat should derive from this class. + * hbase-hadoop-compat should derive from this class. */ @InterfaceAudience.Private public class BaseSourceImpl implements BaseSource, MetricsSource { @@ -53,9 +53,9 @@ synchronized void init(String name) { inited = true; DefaultMetricsSystem.initialize(HBASE_METRICS_SYSTEM_NAME); JvmMetrics.initSingleton(name, ""); - // initialize hbase-metrics module based metric system as well. GlobalMetricRegistriesSource + // initialize hbase-metrics module based metric system as well. GlobalMetricRegistriesAdapter // initialization depends on the metric system being already initialized, that is why we are - // doing it here. Once BaseSourceSourceImpl is removed, we should do the initialization of + // doing it here. Once BaseSourceImpl is removed, we should do the initialization of // these elsewhere. GlobalMetricRegistriesAdapter.init(); } diff --git a/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java b/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java index 555e9220728b..cd5b5a71cea0 100644 --- a/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java +++ b/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java @@ -839,7 +839,7 @@ public static void addHBaseDependencyJars(Configuration conf) throws IOException org.apache.hadoop.hbase.client.Put.class, // hbase-client org.apache.hadoop.hbase.ipc.RpcServer.class, // hbase-server org.apache.hadoop.hbase.CompatibilityFactory.class, // hbase-hadoop-compat - org.apache.hadoop.hbase.mapreduce.JobUtil.class, // hbase-hadoop2-compat + org.apache.hadoop.hbase.mapreduce.JobUtil.class, // hbase-hadoop-compat org.apache.hadoop.hbase.mapreduce.TableMapper.class, // hbase-mapreduce org.apache.hadoop.hbase.metrics.impl.FastLongHistogram.class, // hbase-metrics org.apache.hadoop.hbase.metrics.Snapshot.class, // hbase-metrics-api diff --git a/hbase-metrics-api/README.txt b/hbase-metrics-api/README.txt index 5642fb33772a..3ffc239bd7c2 100644 --- a/hbase-metrics-api/README.txt +++ b/hbase-metrics-api/README.txt @@ -1,7 +1,7 @@ Overview ======== hbase-metrics and hbase-metrics-api are two modules that define and implement the "new" metric -system used internally within HBase. These two modules (and some other code in hbase-hadoop2-compat) +system used internally within HBase. These two modules (and some other code in hbase-hadoop-compat) module are referred as "HBase metrics framework". HBase-metrics-api Module @@ -64,15 +64,15 @@ classes. See HBASE-9774 [4] for more context. hbase-metrics module right now only deals with metrics tracking and collection. It does not do JMX reporting or reporting to console, ganglia, opentsdb, etc. We use Hadoop's Metrics2 for reporting metrics to different sinks or exporting via JMX. However, this is contained within the -hbase-hadoop2-compat module completely, so that rest of the code does not know anything about the +hbase-hadoop-compat module completely, so that rest of the code does not know anything about the Metrics2 dependency. HBaseMetrics2HadoopMetricsAdapter is the adapter that can collect metrics in a MetricRegistry using the metric2 MetricsCollector / MetricRecordBuilder interfaces. -GlobalMetricRegistriesSource is the global Metrics2 Source that collects all of the metrics in all +GlobalMetricRegistriesAdapter is the global Metrics2 Source that collects all of the metrics in all of the metric registries in the MetricRegistries.global() instance. References 1. https://hbase.apache.org/docs/upgrading/version-number 2. http://metrics.dropwizard.io/ -3. https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/metrics2/package-summary.html +3. https://hadoop.apache.org/docs/r3.4.3/api/org/apache/hadoop/metrics2/package-summary.html 4. https://issues.apache.org/jira/browse/HBASE-9774 diff --git a/hbase-protocol-shaded/README.txt b/hbase-protocol-shaded/README.txt index 96e82fabf5d2..31329da2adf2 100644 --- a/hbase-protocol-shaded/README.txt +++ b/hbase-protocol-shaded/README.txt @@ -7,12 +7,11 @@ Hbase 3 and later versions. proto files layout: protobuf/client - client to server messages, client rpc service and protos, used in hbase-client exclusively; -protobuf/rest - hbase-rest messages; protobuf/rpc - rpc and post-rpc tracing messages; protobuf/server/coprocessor - coprocessor rpc services; protobuf/server/coprocessor/example - coprocessors rpc services examples from hbase-examples; protobuf/server/io - filesystem and hbase-server/io protos; -protobuf/server/maser - master rpc services and messages; +protobuf/server/master - master rpc services and messages; protobuf/server/region - region rpc services and messages (except client rpc service, which is in Client.proto); protobuf/server/rsgroup - rsgroup protos; protobuf/server/zookeeper - protos for zookeeper and ones used exclusively in hbase-zookeeper module; diff --git a/hbase-rest/README.txt b/hbase-rest/README.txt index f08ec29bfd83..94346244c8d1 100644 --- a/hbase-rest/README.txt +++ b/hbase-rest/README.txt @@ -1,11 +1,14 @@ ON PROTOBUFS -This maven module has core protobuf definition files ('.protos') used by hbase -REST that ship with hbase core including tests. The protobuf version -they use can be distinct from that used by HBase internally since HBase started -shading its protobuf references. REST Endpoints have no access to the shaded protobuf -hbase uses. They do have access to the content of hbase-protocol -- the -.protos found in here -- but avoid using as much of this as you can as it is -liable to change. +This module defines protobuf message types used by the HBase REST gateway +(see src/main/protobuf/). These are the REST-specific wire formats for +resources like cells, tables, scanners, and cluster status -- they are +separate from the internal HBase RPC protobufs that live in +hbase-protocol-shaded. -Generation of java files from protobuf .proto files included here is done as -part of the build. +The REST protos are compiled and shaded using the same pipeline as +hbase-protocol-shaded (protobuf-maven-plugin + replacer plugin rewriting +com.google.protobuf to the hbase-thirdparty shaded package). The build +config comment in pom.xml notes it is "copied directly from +hbase-shaded-protocol, and should be kept in sync." + +Java sources are generated from the .proto files as part of the build. diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java index e8e64b634c49..3955e3de0f3a 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java @@ -23,7 +23,7 @@ /** * This is the glue between the HRegion and whatever hadoop shim layer is loaded - * (hbase-hadoop1-compat or hbase-hadoop2-compat). + * (hbase-hadoop-compat). */ @InterfaceAudience.Private public class MetricsRegion { diff --git a/hbase-website/README.md b/hbase-website/README.md index a67a85699757..0685e87f6372 100644 --- a/hbase-website/README.md +++ b/hbase-website/README.md @@ -46,7 +46,7 @@ Legacy documentation is preserved for those users who have old bookmarked links **Examples:** -- `app/pages/_landing/team/content.md` - Markdown content for team page +- `app/pages/_landing/downloads/content.md` - Markdown content for downloads page - `app/pages/_landing/powered-by-hbase/companies.json` - JSON data for companies - `app/pages/_landing/news/events.json` - JSON data for news/events - `app/pages/_docs/docs/_mdx/(multi-page)/...` - MDX content for documentation @@ -61,7 +61,7 @@ Before you begin, ensure you have the following installed: - **Node.js version 22** - JavaScript runtime (like the JVM for Java) - Download from [nodejs.org](https://nodejs.org/) - - Verify installation: `node --version` (should show v20.19+ or v22.12+) + - Verify installation: `node --version` (should show v22.12+) - **NPM** - Node Package Manager (like Maven for Java) - Comes bundled with Node.js @@ -270,7 +270,7 @@ This downloads all required packages from npm (similar to Maven Central). npm run extract-developers ``` -This extracts the developer information from the parent `pom.xml` file and creates `app/pages/team/developers.json`, which is required for the Team page to work properly. Re-run this command whenever the developers section in `pom.xml` changes. The output json is ignored by git, and this command also runs at a build time, so there is no need to `git commit` the generated file. +This extracts the developer information from the parent `pom.xml` file and creates `app/pages/_landing/team/developers.json`, which is required for the Team page to work properly. Re-run this command whenever the developers section in `pom.xml` changes. The output json is ignored by git, and this command also runs at a build time, so there is no need to `git commit` the generated file. **Important:** Generate the HBase configuration markdown before starting the development server: @@ -388,7 +388,7 @@ npm run test:e2e:ui **Writing new tests:** -Use the `renderWithProviders` utility in `test/utils.tsx` to ensure components have access to routing and theme context: +Use the `renderWithProviders` utility in `unit-tests/utils.tsx` to ensure components have access to routing and theme context: ```typescript import { renderWithProviders, screen } from './utils' @@ -462,7 +462,7 @@ When you run `mvn site`, the website module automatically: - Installs to `node_modules/` 4. **Extracts developers data** from the parent `pom.xml` - - Creates `app/pages/team/developers.json` + - Creates `app/pages/_landing/team/developers.json` - Required for the Team page 5. **Runs a website CI command**: From c0144dbdc1ab95bf844f79e0747f284175fbc860 Mon Sep 17 00:00:00 2001 From: Nick Dimiduk Date: Sat, 23 May 2026 15:00:32 +0200 Subject: [PATCH 2/2] Review feedback on AGENTS.md and README fixes - Clarify coprocessor interface locations (observers in hbase-server, not hbase-client) - Add hbase-extensions module to AGENTS.md - Note ghost directories from removed modules - Remove stale protobuf/server/coprocessor/example reference - Fix VirtualBox link and typo in release-vm README Co-Authored-By: mimo-v2.5-pro --- AGENTS.md | 19 +++++++++++++------ dev-support/release-vm/README.md | 4 ++-- hbase-protocol-shaded/README.txt | 1 - 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2d4125fb1f34..a1ef12e9658c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,6 +26,10 @@ object storage. This is a multi-module Maven project. Modules live in arbitrarily nested folders; enumerate them by searching for `pom.xml` files (excluding `target/` directories). The root `pom.xml` defines the full reactor and build order. +Note that some directories from removed or merged modules (e.g., +`hbase-hadoop2-compat/`, `hbase-protocol/`, `hbase-rsgroup/`) may still exist +as empty shells with only `target/` remnants. If a directory has no `pom.xml`, +it is not part of the active build. ### Client and Server @@ -53,12 +57,12 @@ you are on. **Coprocessors** are HBase's server-side extension framework. They allow custom code to run inside RegionServer and Master processes, with the same privileges -as the host process. Coprocessor interfaces (observers, endpoints) are defined -in `hbase-client` and `hbase-server` under the `coprocessor` package; endpoint -implementations live in `hbase-endpoint`. The built-in `AccessController` -coprocessor enforces ACLs; `VisibilityController` enforces cell-level -visibility labels. Third-party coprocessors are loaded via configuration or -table schema. +as the host process. The base `Coprocessor` interface lives in `hbase-client`; +observer and endpoint interfaces (`RegionObserver`, `MasterObserver`, etc.) live +in `hbase-server`. Endpoint implementations live in `hbase-endpoint`. The +built-in `AccessController` coprocessor enforces ACLs; `VisibilityController` +enforces cell-level visibility labels. Third-party coprocessors are loaded via +configuration or table schema. **Server subsystems** (separated from hbase-server for modularity): `hbase-balancer`, `hbase-procedure`, `hbase-replication`, `hbase-asyncfs`, @@ -68,6 +72,9 @@ table schema. `hbase-common`, `hbase-metrics` + `hbase-metrics-api`, `hbase-logging`, `hbase-hadoop-compat` +**Extensions:** +`hbase-extensions` (currently `hbase-openssl` for native TLS support) + **Storage codecs:** `hbase-compression/*` (pluggable algorithms), `hbase-external-blockcache` diff --git a/dev-support/release-vm/README.md b/dev-support/release-vm/README.md index a9509ed6bad7..37890f4b1f4d 100644 --- a/dev-support/release-vm/README.md +++ b/dev-support/release-vm/README.md @@ -23,9 +23,9 @@ This is a vagrant project that provides a virtual machine environment suitable for running an Apache HBase release. Requires: -* [VirtualBox](http://virtualbox.org) +* [VirtualBox](https://www.virtualbox.org/) * [Vagrant](https://www.vagrantup.com/) -* The private portion of your signing key avilable in the local GPG agent +* The private portion of your signing key available in the local GPG agent * The private portion of your Github authentication key available in either the local GPG agent or local SSH agent diff --git a/hbase-protocol-shaded/README.txt b/hbase-protocol-shaded/README.txt index 31329da2adf2..b2c065a56bb9 100644 --- a/hbase-protocol-shaded/README.txt +++ b/hbase-protocol-shaded/README.txt @@ -9,7 +9,6 @@ proto files layout: protobuf/client - client to server messages, client rpc service and protos, used in hbase-client exclusively; protobuf/rpc - rpc and post-rpc tracing messages; protobuf/server/coprocessor - coprocessor rpc services; -protobuf/server/coprocessor/example - coprocessors rpc services examples from hbase-examples; protobuf/server/io - filesystem and hbase-server/io protos; protobuf/server/master - master rpc services and messages; protobuf/server/region - region rpc services and messages (except client rpc service, which is in Client.proto);