diff --git a/.github/workflows/pd-store-ci.yml b/.github/workflows/pd-store-ci.yml index 1a6825e7e4..f0c2231a4f 100644 --- a/.github/workflows/pd-store-ci.yml +++ b/.github/workflows/pd-store-ci.yml @@ -134,6 +134,10 @@ jobs: done echo "can_run=true" >> "$GITHUB_OUTPUT" + - name: Run PD docker entrypoint secret override tests + run: | + $TRAVIS_DIR/test-pd-docker-entrypoint.sh + - name: Run start-hugegraph-pd.sh foreground mode tests if: steps.pd-preflight.outputs.can_run == 'true' run: | diff --git a/docker/README.md b/docker/README.md index 0bb74cf81f..b26d0b8060 100644 --- a/docker/README.md +++ b/docker/README.md @@ -39,8 +39,9 @@ contains a single quote or newline. echo ".env already exists; edit it instead of overwriting it" >&2 exit 1 } - printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\nHUGEGRAPH_AUTH_TOKEN_SECRET='%s'\n" \ - 'replace-with-your-password' "${jwt_secret}" > .env + pd_secret="$(openssl rand -hex 24)" + printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\nHUGEGRAPH_AUTH_TOKEN_SECRET='%s'\nHG_PD_AUTH_SECRET_KEY='%s'\n" \ + 'replace-with-your-password' "${jwt_secret}" "${pd_secret}" > .env ) ``` @@ -60,12 +61,56 @@ behind an HTTPS reverse proxy and trusted network controls. first authenticated startup. Changing `.env` does not rotate an existing administrator password; use the HugeGraph user API for credential changes. -For the verification commands below, set the password in your current shell: +For the verification commands below, load `.env` into your current shell and +set the password: ```bash +set -a; . ./.env; set +a ADMIN_PASSWORD='the-same-password-used-in-.env' ``` +Compose reads `.env` on its own; the line above is so that the `curl` and +`sed` commands on this page can use `${HG_PD_AUTH_SECRET_KEY}` too. + +The PD REST API (port 8620, HStore topologies only) has its own credential: +requests other than health probes need HTTP Basic auth with an internal +service name (for example `hg`) and the PD secret as the password. PD ships +no default secret, so `HG_PD_AUTH_SECRET_KEY` is required and the HStore +Compose files refuse to start without it. The `.env` command above generates +one. To list registered stores: + +```bash +curl -u "hg:${HG_PD_AUTH_SECRET_KEY}" http://localhost:8620/v1/stores +``` + +Three consumers read this credential, and all three have to agree or startup +fails: + +- PD itself, through `HG_PD_AUTH_SECRET_KEY`. +- The Server, whose `bin/wait-storage.sh` polls `/v1/stores` before the + Server starts. Both Compose files pass `PD_AUTH_PASSWORD` to it from the + same variable, so setting `HG_PD_AUTH_SECRET_KEY` in `.env` covers it. If + the Server sends the wrong secret it retries until + `WAIT_STORAGE_TIMEOUT_S` (300s) expires and the container exits with + `ERROR: Timeout waiting for storage backend`. +- Hubble, through `operations.pd.password` in the file under `conf/hubble/`. + That file is mounted read-only and is not templated, so write the same value + into it by hand. Until you do, Hubble's PD-backed views get 401 from PD; + everything else in Hubble works. + +Write it in, after loading `.env` as above. Use `hstore.properties` for the +Minimal HStore topology and `hstore-ha.properties` for HA: + +```bash +if [ -n "${HG_PD_AUTH_SECRET_KEY:-}" ]; then + sed -i.bak \ + "s#^operations.pd.password=.*#operations.pd.password=${HG_PD_AUTH_SECRET_KEY}#" \ + conf/hubble/hstore.properties +else + echo 'HG_PD_AUTH_SECRET_KEY is empty; load .env first' >&2 +fi +``` + ### Standalone This is the recommended quickstart. @@ -312,7 +357,9 @@ docker compose -f docker-compose-hstore.yml up -d --wait ### Hubble configuration The three small files under `conf/hubble/` contain only topology-specific -discovery settings and container paths: +discovery settings, the PD REST credential (`operations.pd.username` and +`operations.pd.password`, which must match PD's `auth.secret-key`), and +container paths: - `conf/hubble/standalone.properties` uses direct Server mode. - `conf/hubble/hstore.properties` uses one PD and one Store REST target. diff --git a/docker/conf/hubble/hstore-ha.properties b/docker/conf/hubble/hstore-ha.properties index a50c52c96f..2a818d6fb2 100644 --- a/docker/conf/hubble/hstore-ha.properties +++ b/docker/conf/hubble/hstore-ha.properties @@ -20,6 +20,11 @@ pd.enabled=true server.direct_url=http://server0:8080 pd.peers=pd0:8686,pd1:8686,pd2:8686 pd.server=pd0:8620 +# PD REST credential. The password must equal PD's auth.secret-key, which has +# no default: set it to the same value as HG_PD_AUTH_SECRET_KEY in .env. While +# it is empty, Hubble's PD-backed views get HTTP 401 from PD. +operations.pd.username=hubble +operations.pd.password= operations.store.allowed_targets=[http://store0:8520,http://store1:8520,http://store2:8520] upload_file.location=/hubble/data/upload-files dashboard.address= diff --git a/docker/conf/hubble/hstore.properties b/docker/conf/hubble/hstore.properties index 5de43c212f..55796d6e29 100644 --- a/docker/conf/hubble/hstore.properties +++ b/docker/conf/hubble/hstore.properties @@ -20,6 +20,11 @@ pd.enabled=true server.direct_url=http://server:8080 pd.peers=pd:8686 pd.server=pd:8620 +# PD REST credential. The password must equal PD's auth.secret-key, which has +# no default: set it to the same value as HG_PD_AUTH_SECRET_KEY in .env. While +# it is empty, Hubble's PD-backed views get HTTP 401 from PD. +operations.pd.username=hubble +operations.pd.password= operations.store.allowed_targets=[http://store:8520] upload_file.location=/hubble/data/upload-files dashboard.address= diff --git a/docker/docker-compose-3pd-3store-3server.yml b/docker/docker-compose-3pd-3store-3server.yml index 6f599c6870..ba02f27eb1 100644 --- a/docker/docker-compose-3pd-3store-3server.yml +++ b/docker/docker-compose-3pd-3store-3server.yml @@ -70,6 +70,8 @@ x-server-environment: &server-environment HG_SERVER_REQUIRE_AUTH_TOKEN_SECRET: "true" HG_SERVER_AUTH_TOKEN_SECRET: ${HUGEGRAPH_AUTH_TOKEN_SECRET:-} PASSWORD: ${HUGEGRAPH_ADMIN_PASSWORD:-} + # bin/wait-storage.sh polls the PD REST API, so it needs the same secret + PD_AUTH_PASSWORD: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} x-server-common: &server-common image: hugegraph/server:${HUGEGRAPH_VERSION:-latest} @@ -107,6 +109,7 @@ services: HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 HG_PD_DATA_PATH: /hugegraph-pd/pd_data HG_PD_INITIAL_STORE_COUNT: 3 + HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: ["8620:8620", "8686:8686"] volumes: - hg-pd0-data:/hugegraph-pd/pd_data @@ -125,6 +128,7 @@ services: HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 HG_PD_DATA_PATH: /hugegraph-pd/pd_data HG_PD_INITIAL_STORE_COUNT: 3 + HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: ["8621:8620", "8687:8686"] volumes: - hg-pd1-data:/hugegraph-pd/pd_data @@ -143,6 +147,7 @@ services: HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 HG_PD_DATA_PATH: /hugegraph-pd/pd_data HG_PD_INITIAL_STORE_COUNT: 3 + HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: ["8622:8620", "8688:8686"] volumes: - hg-pd2-data:/hugegraph-pd/pd_data diff --git a/docker/docker-compose-hstore.yml b/docker/docker-compose-hstore.yml index d201430692..ba5a421c2c 100644 --- a/docker/docker-compose-hstore.yml +++ b/docker/docker-compose-hstore.yml @@ -39,6 +39,7 @@ services: HG_PD_RAFT_PEERS_LIST: pd:8610 HG_PD_INITIAL_STORE_LIST: store:8500 HG_PD_DATA_PATH: /hugegraph-pd/pd_data + HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: - "8620:8620" volumes: @@ -94,6 +95,8 @@ services: HG_SERVER_INIT_STORE_ENABLED: "false" HG_SERVER_AUTH_TOKEN_SECRET: ${HUGEGRAPH_AUTH_TOKEN_SECRET:-} PASSWORD: ${HUGEGRAPH_ADMIN_PASSWORD:-} + # bin/wait-storage.sh polls the PD REST API, so it needs the same secret + PD_AUTH_PASSWORD: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: - "8080:8080" healthcheck: diff --git a/docker/test-compose.sh b/docker/test-compose.sh index ecd5ab5b2d..aa74a88a30 100644 --- a/docker/test-compose.sh +++ b/docker/test-compose.sh @@ -21,6 +21,7 @@ set -Eeuo pipefail DOCKER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PASSWORD="ci-compose-password" SECRET="0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" +PD_SECRET="ci-compose-pd-secret" VERSION="ci-version" RENDER_HUBBLE_IMAGE="example.invalid/hugegraph/hubble:ci" DATASOURCE="jdbc:h2:file:/hubble/data/hubble;DB_CLOSE_ON_EXIT=FALSE" @@ -34,6 +35,7 @@ compose_auth() { HUBBLE_IMAGE="${RENDER_HUBBLE_IMAGE}" \ HUGEGRAPH_ADMIN_PASSWORD="${PASSWORD}" \ HUGEGRAPH_AUTH_TOKEN_SECRET="${SECRET}" \ + HG_PD_AUTH_SECRET_KEY="${PD_SECRET}" \ docker compose "$@" } @@ -271,6 +273,7 @@ compose_active() { HUBBLE_IMAGE="${HUBBLE_IMAGE:-hugegraph/hubble:latest}" \ HUGEGRAPH_ADMIN_PASSWORD="${PASSWORD}" \ HUGEGRAPH_AUTH_TOKEN_SECRET="${SECRET}" \ + HG_PD_AUTH_SECRET_KEY="${PD_SECRET}" \ COMPOSE_PROGRESS=plain \ docker compose -p "${ACTIVE_PROJECT}" "${ACTIVE_FILES[@]}" "$@" } diff --git a/hugegraph-pd/README.md b/hugegraph-pd/README.md index 794dba9b98..fa5d8de86c 100644 --- a/hugegraph-pd/README.md +++ b/hugegraph-pd/README.md @@ -100,6 +100,7 @@ Key configuration file: `conf/application.yml` | `raft.address` | `127.0.0.1:8610` | Raft service address for this PD node | | `raft.peers-list` | `127.0.0.1:8610` | Comma-separated list of all PD nodes in the Raft cluster | | `pd.data-path` | `./pd_data` | Directory for storing PD metadata and Raft logs | +| `auth.secret-key` | none (required) | Password required by the REST API with an internal service name (`hg`, `store`, `hubble`, `vermeer`) via HTTP Basic auth. No default is shipped; generate one per deployment and configure every REST client (e.g. Hubble's `operations.pd.password`) with the same value | #### Single-Node Example @@ -236,11 +237,17 @@ Build PD Docker image: # From project root docker build -f hugegraph-pd/Dockerfile -t hugegraph/pd:latest . +# Generate the REST secret once and keep it: every PD REST client needs this +# same value, and a new one silently breaks the clients already using the old +# one. Store it somewhere durable rather than only in this shell. +export HG_PD_AUTH_SECRET_KEY="$(openssl rand -hex 24)" + # Run container docker run -d \ -p 8620:8620 \ -p 8686:8686 \ -p 8610:8610 \ + -e HG_PD_AUTH_SECRET_KEY="${HG_PD_AUTH_SECRET_KEY}" \ -e HG_PD_GRPC_HOST= \ -e HG_PD_RAFT_ADDRESS=:8610 \ -e HG_PD_RAFT_PEERS_LIST=:8610 \ @@ -280,6 +287,30 @@ docker/docker-compose-3pd-3store-3server.yml - Ensure low latency (<5ms) between PD nodes for Raft consensus - Open required ports: `8620` (REST), `8686` (gRPC), `8610` (Raft) +### Security + +- Keep all three ports on a trusted network. The REST API on `8620` includes + management endpoints that mutate the cluster (peer changes, store removal, + data movement), and the gRPC and Raft ports carry no authentication. +- REST requests need HTTP Basic auth: one of the internal service names + (`hg`, `store`, `hubble`, `vermeer`) with the `auth.secret-key` value as + the password. Health probes (`/v1/health`, `/actuator/*`, + `/v1/prom/targets/*`) stay unauthenticated. +- `auth.secret-key` has no shipped default, because a secret in the source + tree is published to everyone. Generate one per deployment (`openssl rand + -hex 24`) and set it in the config file, or through + `HG_PD_AUTH_SECRET_KEY`, which the Docker image requires. Give every REST + client the same value: the Server's `bin/wait-storage.sh` reads + `PD_AUTH_PASSWORD` (and `PD_AUTH_USER`, default `store`), and Hubble reads + `operations.pd.password`. A client left on a stale secret gets 401, and for + `wait-storage.sh` that means Server startup aborts after + `WAIT_STORAGE_TIMEOUT_S`. +- An existing `conf/application.yml` carried over from an earlier release has + no `auth` block. PD then starts with an empty secret and refuses every + authenticated REST request, logging an error that names `auth.secret-key`. + Add the key before upgrading. PD refuses to start if the key is set to the + placeholder value that earlier revisions of this repository carried. + ### Monitoring PD exposes metrics via REST API at: diff --git a/hugegraph-pd/docs/api-reference.md b/hugegraph-pd/docs/api-reference.md index aa8cce8473..7535bc8f62 100644 --- a/hugegraph-pd/docs/api-reference.md +++ b/hugegraph-pd/docs/api-reference.md @@ -760,6 +760,22 @@ for (Map.Entry entry : results.entrySet()) { PD exposes a REST API for management and monitoring (default port: 8620). +### Authentication + +Every endpoint below except the probes needs HTTP Basic auth: one of the +internal service names (`hg`, `store`, `hubble`, `vermeer`) as the user, and +the `auth.secret-key` value from PD's `conf/application.yml` as the password. +A missing or wrong credential gets HTTP 401. The `curl` examples that follow +omit `-u` for readability; add it to every call except `/v1/health`, +`/actuator/*` and `/v1/prom/targets/*`, which stay unauthenticated for probes. + +```bash +curl -u hg: http://localhost:8620/v1/stores +``` + +Endpoints under `/v1` mutate the cluster (peer list changes, store removal, +partition balancing), so keep port 8620 on a trusted network regardless. + ### Health Check ```bash diff --git a/hugegraph-pd/docs/configuration.md b/hugegraph-pd/docs/configuration.md index e3ae4f6f25..e3cbd1ed3a 100644 --- a/hugegraph-pd/docs/configuration.md +++ b/hugegraph-pd/docs/configuration.md @@ -79,6 +79,31 @@ server: - Metrics: `http://:8620/actuator/metrics` - Prometheus: `http://:8620/actuator/prometheus` +### REST Authentication Settings + +Every REST request except the probes below must carry HTTP Basic auth: one of +the internal service names (`hg`, `store`, `hubble`, `vermeer`) as the user, +and the shared secret as the password. A missing or wrong credential gets +HTTP 401. Unauthenticated paths: `/v1/health`, `/actuator/*` and +`/v1/prom/targets/*`. + +```yaml +auth: + secret-key: +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `auth.secret-key` | String | none (required) | Password checked against the Basic credential. There is no default: a secret shipped in the source tree would be published to everyone. While it is empty PD refuses every authenticated REST request and logs an error naming this parameter, and PD refuses to start at all if it is set to the value that earlier revisions carried as a placeholder. | + +Every REST client needs the same value: the Server's `bin/wait-storage.sh` +reads it from `PD_AUTH_PASSWORD`, Hubble from `operations.pd.password`, and +the Docker image takes `HG_PD_AUTH_SECRET_KEY`. + +```bash +curl -u hg: http://:8620/v1/stores +``` + ### Raft Consensus Settings Controls Raft consensus for PD cluster coordination. @@ -253,13 +278,13 @@ management: endpoints: web: exposure: - include: "*" # Expose all actuator endpoints + include: "health,metrics,prometheus" # Allowlist; see note below ``` | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `management.metrics.export.prometheus.enabled` | Boolean | `true` | Enable Prometheus-compatible metrics at `/actuator/prometheus`. | -| `management.endpoints.web.exposure.include` | String | `"*"` | Actuator endpoints to expose. `"*"` = all, or specify comma-separated list (e.g., `"health,metrics"`). | +| `management.endpoints.web.exposure.include` | String | `"health,metrics,prometheus"` | Actuator endpoints to expose. `/actuator/*` is excluded from the REST authentication interceptor, so every endpoint listed here is reachable without a credential on port 8620. Prefer an allowlist over `"*"`. | ## Deployment Scenarios diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java index acfc2ec290..1eece31655 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java @@ -25,6 +25,7 @@ import org.apache.hugegraph.pd.ConfigService; import org.apache.hugegraph.pd.IdService; +import org.springframework.beans.factory.InitializingBean; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Configuration; @@ -38,7 +39,15 @@ */ @Data @Component -public class PDConfig { +public class PDConfig implements InitializingBean { + + /** + * The secret that earlier revisions carried as the placeholder default for + * `auth.secret-key`. It is published in this repository, so a deployment + * still using it authenticates anyone who can read the source. Refuse to + * start rather than let a well-known string look like authentication. + */ + private static final String PUBLISHED_SECRET_KEY = "FXQXbJtbCLxODc6tGci732pkH1cyf8Qg"; // cluster ID @Value("${pd.cluster_id:1}") @@ -69,7 +78,11 @@ public class PDConfig { @Autowired private ThreadPoolGrpc threadPoolGrpc; - @Value("${auth.secret-key: 'FXQXbJtbCLxODc6tGci732pkH1cyf8Qg'}") + // No default: Spring takes the text after the first ':' literally, so a + // quoted default would resolve to a value including the quotes and the + // leading space, and no client would ever match it. An absent key must + // yield "" so the REST interceptor can refuse every request and say why. + @Value("${auth.secret-key:}") @ToString.Exclude private String secretKey; @@ -85,6 +98,17 @@ public class PDConfig { private ConfigService configService; private IdService idService; + @Override + public void afterPropertiesSet() { + if (PUBLISHED_SECRET_KEY.equals(this.secretKey)) { + throw new IllegalStateException( + "auth.secret-key is set to the value published in the HugeGraph source " + + "tree, which authenticates anyone who can read it. Set a " + + "deployment-specific secret in conf/application.yml, or through the " + + "HG_PD_AUTH_SECRET_KEY environment variable for the Docker image."); + } + } + public Map getInitialStoreMap() { if (initialStoreMap == null) { initialStoreMap = new HashMap<>(); diff --git a/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh b/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh index 529936d06a..d38230b37d 100755 --- a/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh +++ b/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh @@ -26,10 +26,31 @@ require_env() { fi } +# Escape a value for use inside a JSON string: backslash and quote, then every +# remaining C0 control character as \uXXXX. Dropping only LF, as an earlier +# version did, left CR and TAB to produce invalid JSON and a container that +# failed before startup. json_escape() { - local s="$1" - s=${s//\\/\\\\}; s=${s//\"/\\\"}; s=${s//$'\n'/} - printf "%s" "$s" + local s="$1" out="" i c + s=${s//\\/\\\\} + s=${s//\"/\\\"} + for (( i = 0; i < ${#s}; i++ )); do + c=${s:i:1} + case "$c" in + $'\n') out+='\n' ;; + $'\r') out+='\r' ;; + $'\t') out+='\t' ;; + $'\b') out+='\b' ;; + $'\f') out+='\f' ;; + *) + if [[ "$c" < $'\x20' || "$c" == $'\x7f' ]]; then + printf -v c '\\u%04x' "'$c" + fi + out+="$c" + ;; + esac + done + printf "%s" "$out" } migrate_env() { @@ -51,14 +72,22 @@ require_env "HG_PD_GRPC_HOST" require_env "HG_PD_RAFT_ADDRESS" require_env "HG_PD_RAFT_PEERS_LIST" require_env "HG_PD_INITIAL_STORE_LIST" +# The REST API refuses every authenticated request without this, and the image +# ships no default because a published secret is not a secret. +require_env "HG_PD_AUTH_SECRET_KEY" : "${HG_PD_GRPC_PORT:=8686}" : "${HG_PD_REST_PORT:=8620}" : "${HG_PD_DATA_PATH:=/hugegraph-pd/pd_data}" : "${HG_PD_INITIAL_STORE_COUNT:=1}" +# Secret for REST Basic authentication (auth.secret-key). Required above and +# never logged. +AUTH_JSON="\"auth\": { \"secret-key\": \"$(json_escape "${HG_PD_AUTH_SECRET_KEY}")\" }," + SPRING_APPLICATION_JSON="$(cat < - * WARNING: This class currently implements only basic internal authentication - * validation for internal modules (hg, store, hubble, vermeer). The authentication mechanism - * is designed for internal service-to-service communication only. + * WARNING: This class validates a Basic credential for internal modules + * (hg, store, hubble, vermeer): the service name must be one of the four and the + * password must match the shared secret configured via `auth.secret-key`. The + * mechanism is designed for internal service-to-service communication only. *

* *

Important SEC Considerations:

@@ -57,10 +64,16 @@ * and regular security audits. *

*/ +@Slf4j @Component public class Authentication { private static final Set innerModules = Set.of("hg", "store", "hubble", "vermeer"); + private static final AtomicBoolean missingSecretLogged = new AtomicBoolean(); + + @Autowired + private PDConfig pdConfig; + protected T authenticate(String authority, String token, Function tokenCall, Supplier call) { try { @@ -70,26 +83,49 @@ protected T authenticate(String authority, String token, Function } byte[] bytes = authority.getBytes(StandardCharsets.UTF_8); byte[] decode = Base64.getDecoder().decode(bytes); - String info = new String(decode); + // RFC 7617: Basic credentials are UTF-8. Decoding with the platform + // default would compare against a UTF-8 secret only when the host + // locale happens to agree. + String info = new String(decode, StandardCharsets.UTF_8); int delim = info.indexOf(':'); if (delim == -1) { throw new BadCredentialsException(invalidBasicInfo); } String name = info.substring(0, delim); - // TODO: password validation is skipped — only service name is checked against - // innerModules. Full credential validation should be added as part of the auth refactor. - //String pwd = info.substring(delim + 1); - if (innerModules.contains(name)) { - return call.get(); - } else { + String pwd = info.substring(delim + 1); + if (!innerModules.contains(name)) { throw new AccessDeniedException("invalid service name"); } + if (!verifySecret(pwd)) { + throw new BadCredentialsException("invalid credential"); + } + return call.get(); } catch (Exception e) { throw new RuntimeException(e); } } + /** + * Compare the password of the Basic credential with the shared secret + * configured via `auth.secret-key`. A missing or empty secret refuses every + * request instead of falling back to name-only authentication. + */ + private boolean verifySecret(String pwd) { + String secret = this.pdConfig == null ? null : this.pdConfig.getSecretKey(); + if (StringUtils.isEmpty(secret)) { + // Logged once: this path is reachable by unauthenticated callers + if (missingSecretLogged.compareAndSet(false, true)) { + log.error("auth.secret-key is not configured, so every authenticated REST " + + "request is refused. Add it to conf/application.yml (or set " + + "HG_PD_AUTH_SECRET_KEY) and give every REST client the same value."); + } + return false; + } + return MessageDigest.isEqual(pwd.getBytes(StandardCharsets.UTF_8), + secret.getBytes(StandardCharsets.UTF_8)); + } + public static String getTokenKey(String name) { return "PD/TOKEN/" + name; } diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java index 2b1103739b..224d417cc4 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java @@ -41,7 +41,12 @@ public void configure(ServerBuilder serverBuilder) { poolGrpc.getQueue())); serverBuilder.maxInboundMessageSize(MAX_INBOUND_MESSAGE_SIZE); // TODO: GrpcAuthentication is instantiated as a Spring bean but never registered - // here — add serverBuilder.intercept(grpcAuthentication) once auth is refactored. + // here - add serverBuilder.intercept(grpcAuthentication) once auth is refactored. + // It extends Authentication, which now also checks the Basic password against + // auth.secret-key. Registering it therefore requires giving that value to every + // gRPC client first: ServiceConstant.AUTHORITY (Server) and DefaultPdProvider + // .authority (Store) are "" and "default" today, and hg-pd-cli sends "". + // Otherwise no store can register once the interceptor is enabled. } } diff --git a/hugegraph-pd/hg-pd-service/src/main/resources/application.yml b/hugegraph-pd/hg-pd-service/src/main/resources/application.yml index 5a03595f7b..44adc2ed1f 100644 --- a/hugegraph-pd/hg-pd-service/src/main/resources/application.yml +++ b/hugegraph-pd/hg-pd-service/src/main/resources/application.yml @@ -27,7 +27,9 @@ management: endpoints: web: exposure: - include: "*" + # Allowlist, not "*": /actuator/* is excluded from the REST auth + # interceptor, so anything exposed here is anonymous on this port. + include: "health,metrics,prometheus" grpc: port: 8686 @@ -43,6 +45,14 @@ license: server: port: 8620 +auth: + # Shared secret checked against the password of the Basic credential on every + # authenticated REST request. Required and deliberately empty: a secret in + # the source tree is published to everyone. Set a deployment-specific value + # here (or via HG_PD_AUTH_SECRET_KEY) and give every REST client the same + # value. While empty, PD refuses every authenticated REST request. + secret-key: + pd: # Periodically check whether the cluster is healthy at intervals, in seconds patrol-interval: 300 diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java index 0836120c73..d7df26d59e 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java @@ -17,6 +17,9 @@ package org.apache.hugegraph.pd; +import java.nio.charset.StandardCharsets; +import java.util.Base64; + import org.apache.hugegraph.pd.client.PDConfig; public class BaseTest { @@ -24,9 +27,12 @@ public class BaseTest { protected static String pdGrpcAddr = "127.0.0.1:8686"; protected static String pdRestAddr = "http://127.0.0.1:8620"; protected static String user = "store"; - protected static String pwd = ""; + // Must match the auth.secret-key that travis/start-pd.sh gives the PD + // under test; the shipped config carries no secret by design + protected static String pwd = "pd-ci-test-secret-not-for-production"; protected static String key = "Authorization"; - protected static String value = "Basic c3RvcmU6YWRtaW4="; + protected static String value = "Basic " + Base64.getEncoder().encodeToString( + (user + ":" + pwd).getBytes(StandardCharsets.UTF_8)); protected PDConfig getPdConfig() { return PDConfig.of(pdGrpcAddr).setAuthority(user, pwd); diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java index 95b044c76b..a65b53e886 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java @@ -17,6 +17,7 @@ package org.apache.hugegraph.pd.core; +import org.apache.hugegraph.pd.service.interceptor.AuthenticationTest; import org.apache.hugegraph.pd.core.meta.MetadataKeyHelperTest; import org.apache.hugegraph.pd.core.store.HgKVStoreImplTest; import org.apache.hugegraph.pd.raft.IpAuthHandlerTest; @@ -32,6 +33,7 @@ MetadataKeyHelperTest.class, HgKVStoreImplTest.class, PDConfigTest.class, + AuthenticationTest.class, ConfigServiceTest.class, IdServiceTest.class, KvServiceTest.class, diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java index 4aff85d1e9..5c808b6527 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java @@ -18,15 +18,29 @@ package org.apache.hugegraph.pd.rest; import java.net.http.HttpClient; +import java.nio.charset.StandardCharsets; +import java.util.Base64; import org.junit.After; import org.junit.BeforeClass; public class BaseServerTest { + // Must match the auth.secret-key that travis/start-pd.sh gives the PD + // under test; the shipped config carries no secret by design + protected static final String SECRET = "pd-ci-test-secret-not-for-production"; + protected static final String AUTH_HEADER = "Authorization"; + protected static final String VALID_AUTH = basicAuth("store", SECRET); + protected static HttpClient client; protected static String pdRestAddr; + protected static String basicAuth(String name, String pwd) { + String credential = name + ":" + pwd; + return "Basic " + Base64.getEncoder() + .encodeToString(credential.getBytes(StandardCharsets.UTF_8)); + } + @BeforeClass public static void init() { client = HttpClient.newHttpClient(); diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java index fb2b71d480..3c0b6e475b 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java @@ -35,7 +35,7 @@ public void testQueryIndexInfo() throws URISyntaxException, IOException, Interru String url = pdRestAddr + "/"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -54,7 +54,7 @@ public void testQueryClusterInfo() throws URISyntaxException, IOException, Inter String url = pdRestAddr + "/v1/cluster"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -68,7 +68,7 @@ public void testQueryClusterMembers() throws URISyntaxException, IOException, String url = pdRestAddr + "/v1/members"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -82,7 +82,7 @@ public void testQueryStoresInfo() throws URISyntaxException, IOException, Interr String url = pdRestAddr + "/v1/stores"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -96,7 +96,7 @@ public void testQueryGraphsInfo() throws IOException, InterruptedException, JSON String url = pdRestAddr + "/v1/graphs"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -110,7 +110,7 @@ public void testQueryPartitionsInfo() throws IOException, InterruptedException, String url = pdRestAddr + "/v1/highLevelPartitions"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -124,7 +124,7 @@ public void testQueryDebugPartitionsInfo() throws URISyntaxException, IOExceptio String url = pdRestAddr + "/v1/partitions"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -137,11 +137,62 @@ public void testQueryShards() throws URISyntaxException, IOException, Interrupte String url = pdRestAddr + "/v1/shards"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); JSONObject obj = new JSONObject(response.body()); assert obj.getInt("status") == 0; } + + @Test + public void testMissingCredentialGets401() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/v1/members"; + HttpRequest request = HttpRequest.newBuilder() + .uri(new URI(url)) + .GET() + .build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + assert response.statusCode() == 401; + } + + @Test + public void testWrongPasswordGets401() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/v1/members"; + HttpRequest request = HttpRequest.newBuilder() + .uri(new URI(url)) + .header(AUTH_HEADER, basicAuth("store", "wrong-password")) + .GET() + .build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + assert response.statusCode() == 401; + } + + @Test + public void testEmptyPasswordGets401() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/v1/members"; + HttpRequest request = HttpRequest.newBuilder() + .uri(new URI(url)) + .header(AUTH_HEADER, basicAuth("hg", "")) + .GET() + .build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + assert response.statusCode() == 401; + } + + @Test + public void testUnknownServiceNameGets401() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/v1/members"; + HttpRequest request = HttpRequest.newBuilder() + .uri(new URI(url)) + .header(AUTH_HEADER, basicAuth("nobody", SECRET)) + .GET() + .build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + assert response.statusCode() == 401; + } } diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/service/interceptor/AuthenticationTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/service/interceptor/AuthenticationTest.java new file mode 100644 index 0000000000..5bb801f6d1 --- /dev/null +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/service/interceptor/AuthenticationTest.java @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.pd.service.interceptor; + +import java.lang.reflect.Field; +import java.nio.charset.StandardCharsets; +import java.util.Base64; + +import org.apache.hugegraph.pd.config.PDConfig; +import org.junit.Assert; +import org.junit.Test; + +/** + * In-process cover for the REST credential check. The suites that exercise it + * over HTTP talk to a PD in another JVM, so nothing here is covered by them. + */ +public class AuthenticationTest { + + private static final String SECRET = "unit-test-secret"; + + private static Authentication authWithSecret(String secret) throws Exception { + Authentication auth = new Authentication(); + PDConfig config = new PDConfig(); + config.setSecretKey(secret); + Field field = Authentication.class.getDeclaredField("pdConfig"); + field.setAccessible(true); + field.set(auth, config); + return auth; + } + + private static String credential(String name, String pwd) { + return Base64.getEncoder().encodeToString( + (name + ":" + pwd).getBytes(StandardCharsets.UTF_8)); + } + + private static boolean accepts(Authentication auth, String authority) { + try { + return auth.authenticate(authority, null, t -> Boolean.TRUE, () -> Boolean.TRUE); + } catch (RuntimeException e) { + return false; + } + } + + @Test + public void testEveryInnerModuleIsAcceptedWithTheSecret() throws Exception { + Authentication auth = authWithSecret(SECRET); + for (String name : new String[]{"hg", "store", "hubble", "vermeer"}) { + Assert.assertTrue(name + " should be accepted with the right secret", + accepts(auth, credential(name, SECRET))); + } + } + + @Test + public void testPasswordIsActuallyChecked() throws Exception { + Authentication auth = authWithSecret(SECRET); + Assert.assertFalse("wrong password must be refused", + accepts(auth, credential("hg", "wrong-password"))); + Assert.assertFalse("empty password must be refused", + accepts(auth, credential("hg", ""))); + Assert.assertFalse("secret as the name must not help", + accepts(auth, credential(SECRET, SECRET))); + } + + @Test + public void testUnknownServiceNameIsRefused() throws Exception { + Authentication auth = authWithSecret(SECRET); + Assert.assertFalse(accepts(auth, credential("nobody", SECRET))); + Assert.assertFalse(accepts(auth, credential("admin", SECRET))); + } + + @Test + public void testMissingOrMalformedCredentialIsRefused() throws Exception { + Authentication auth = authWithSecret(SECRET); + Assert.assertFalse(accepts(auth, null)); + Assert.assertFalse(accepts(auth, "")); + // no colon + Assert.assertFalse(accepts(auth, Base64.getEncoder().encodeToString( + "hg".getBytes(StandardCharsets.UTF_8)))); + } + + @Test + public void testUnconfiguredSecretRefusesEveryone() throws Exception { + for (String secret : new String[]{null, ""}) { + Authentication auth = authWithSecret(secret); + Assert.assertFalse("an unset secret must not fall back to a name check", + accepts(auth, credential("hg", ""))); + Assert.assertFalse(accepts(auth, credential("hg", SECRET))); + } + } + + @Test + public void testNonAsciiSecretDoesNotDependOnTheDefaultCharset() throws Exception { + String secret = "sécrèt-2026"; + Authentication auth = authWithSecret(secret); + Assert.assertTrue(accepts(auth, credential("hg", secret))); + Assert.assertFalse(accepts(auth, credential("hg", "secret-2026"))); + } + + @Test + public void testPublishedSecretRefusesStartup() { + PDConfig config = new PDConfig(); + config.setSecretKey("FXQXbJtbCLxODc6tGci732pkH1cyf8Qg"); + Assert.assertThrows(IllegalStateException.class, config::afterPropertiesSet); + } + + @Test + public void testOwnSecretStarts() { + PDConfig config = new PDConfig(); + config.setSecretKey(SECRET); + config.afterPropertiesSet(); + } +} diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh index 93c3a19dd2..6842cb5563 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh @@ -39,7 +39,25 @@ log() { echo "[wait-storage] $1" } -PD_AUTH_ARGS="-u ${PD_AUTH_USER:-store}:${PD_AUTH_PASSWORD:-admin}" +# PD REST credential. PD checks the password against its auth.secret-key and +# ships no default, so this has to be provided by the deployment. +# The value is deliberately kept out of the inner script's source text and out +# of curl's argv: the inner shell reads it from the environment and hands it to +# curl on stdin as a config file. +PD_AUTH_USER="${PD_AUTH_USER:-store}" +PD_AUTH_PASSWORD="${PD_AUTH_PASSWORD:-}" +if [ -z "${PD_AUTH_PASSWORD}" ]; then + log "WARN: PD_AUTH_PASSWORD is empty; PD will answer 401 unless it runs without auth" +fi +# curl -K takes a quoted string, so escape backslash first and then quote +escape_curlrc() { + local v=$1 + v=${v//\\/\\\\} + printf '%s' "${v//\"/\\\"}" +} +PD_AUTH_CURL_USER=$(escape_curlrc "${PD_AUTH_USER}") +PD_AUTH_CURL_PASSWORD=$(escape_curlrc "${PD_AUTH_PASSWORD}") +export PD_AUTH_CURL_USER PD_AUTH_CURL_PASSWORD function key_exists { local key=$1 @@ -101,10 +119,12 @@ if env | grep '^hugegraph\.' > /dev/null; then check_any_pd_stores() { for peer in \$(echo \"\$PD_REST_LIST\" | tr ',' ' '); do - if curl ${PD_AUTH_ARGS} -f -s \ + if printf 'user = \"%s:%s\"\n' \ + \"\$PD_AUTH_CURL_USER\" \"\$PD_AUTH_CURL_PASSWORD\" | \ + curl -K - -f -s \ --connect-timeout ${WAIT_STORAGE_PD_CONNECT_TIMEOUT_S} \ --max-time ${WAIT_STORAGE_PD_MAX_TIMEOUT_S} \ - http://\${peer}/v1/stores 2>/dev/null | \ + \"http://\${peer}/v1/stores\" 2>/dev/null | \ grep -qi '\"state\"[[:space:]]*:[[:space:]]*\"Up\"'; then echo \"\$peer\" return 0 diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh index 0c137489e1..7e3198bedb 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh @@ -31,6 +31,11 @@ fi PD_DIR=$HOME_DIR/hugegraph-pd/apache-hugegraph-pd-$VersionInBash +# conf/application.yml ships auth.secret-key empty on purpose, so PD would +# refuse every authenticated REST request. Supply a test-only secret; it must +# match the value the PD test suites send. +export SPRING_APPLICATION_JSON='{"auth":{"secret-key":"pd-ci-test-secret-not-for-production"}}' + pushd $PD_DIR . bin/start-hugegraph-pd.sh sleep 10 diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-docker-entrypoint.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-docker-entrypoint.sh new file mode 100755 index 0000000000..ea8874dc37 --- /dev/null +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-docker-entrypoint.sh @@ -0,0 +1,114 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Checks that the PD Docker entrypoint turns HG_PD_AUTH_SECRET_KEY into valid +# SPRING_APPLICATION_JSON, whatever the secret contains, and that the value +# Spring would read back is the secret that went in. + +set -euo pipefail + +ENTRYPOINT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../.." && pwd)/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh" +TMP_DIR=$(mktemp -d "${TMPDIR:-/tmp}/pd-entrypoint-test.XXXXXX") +trap 'rm -rf "${TMP_DIR}"' EXIT + +PASS=0 +FAIL=0 + +[[ -f "${ENTRYPOINT}" ]] || { echo "entrypoint not found at ${ENTRYPOINT}" >&2; exit 1; } +command -v python3 >/dev/null || { echo "python3 required" >&2; exit 1; } + +mkdir -p "${TMP_DIR}/bin" +cp "${ENTRYPOINT}" "${TMP_DIR}/docker-entrypoint.sh" +# Stand in for the launcher: record the generated config instead of starting PD +cat > "${TMP_DIR}/bin/start-hugegraph-pd.sh" <<'STUB' +#!/usr/bin/env bash +printf '%s' "${SPRING_APPLICATION_JSON}" > ./spring.json +STUB +chmod +x "${TMP_DIR}/bin/start-hugegraph-pd.sh" "${TMP_DIR}/docker-entrypoint.sh" + +run_case() { + local name="$1" secret="$2" + local out + if ! out=$(cd "${TMP_DIR}" && env \ + HG_PD_GRPC_HOST=pd0 \ + HG_PD_RAFT_ADDRESS=pd0:8610 \ + HG_PD_RAFT_PEERS_LIST=pd0:8610 \ + HG_PD_INITIAL_STORE_LIST=store0:8500 \ + HG_PD_AUTH_SECRET_KEY="${secret}" \ + ./docker-entrypoint.sh 2>&1); then + echo " FAIL ${name}: entrypoint exited non-zero" + printf '%s\n' "${out}" | tail -3 + FAIL=$((FAIL + 1)) + return + fi + + if ! SECRET="${secret}" python3 - "${TMP_DIR}/spring.json" <<'PY' +import json, os, sys +with open(sys.argv[1], encoding="utf-8") as fh: + doc = json.load(fh) +got = doc["auth"]["secret-key"] +want = os.environ["SECRET"] +if got != want: + print(" round-trip mismatch: %r != %r" % (got, want)) + sys.exit(1) +PY + then + echo " FAIL ${name}: invalid JSON or secret did not round-trip" + FAIL=$((FAIL + 1)) + return + fi + echo " PASS ${name}" + PASS=$((PASS + 1)) +} + +echo "PD docker-entrypoint secret override" +run_case "plain secret" 'aVerySecretValue123' +run_case "carriage return" "$(printf 'a\rb')" +run_case "tab" "$(printf 'a\tb')" +run_case "double quote" 'a"b' +run_case "backslash" 'a\b' +run_case "backslash and quote" 'a\"b' +run_case "non-ascii" 'sécrèt-2026' +run_case "spaces" 'two words' + +# The secret is required, and must never be echoed to the log +if (cd "${TMP_DIR}" && env \ + HG_PD_GRPC_HOST=pd0 HG_PD_RAFT_ADDRESS=pd0:8610 \ + HG_PD_RAFT_PEERS_LIST=pd0:8610 HG_PD_INITIAL_STORE_LIST=store0:8500 \ + ./docker-entrypoint.sh >/dev/null 2>&1); then + echo " FAIL missing secret: entrypoint started without HG_PD_AUTH_SECRET_KEY" + FAIL=$((FAIL + 1)) +else + echo " PASS missing secret is refused" + PASS=$((PASS + 1)) +fi + +log_out=$(cd "${TMP_DIR}" && env \ + HG_PD_GRPC_HOST=pd0 HG_PD_RAFT_ADDRESS=pd0:8610 \ + HG_PD_RAFT_PEERS_LIST=pd0:8610 HG_PD_INITIAL_STORE_LIST=store0:8500 \ + HG_PD_AUTH_SECRET_KEY='do-not-log-this-value' \ + ./docker-entrypoint.sh 2>&1) +if printf '%s' "${log_out}" | grep -q 'do-not-log-this-value'; then + echo " FAIL secret was written to the log" + FAIL=$((FAIL + 1)) +else + echo " PASS secret is not logged" + PASS=$((PASS + 1)) +fi + +echo "${PASS} passed, ${FAIL} failed" +[[ "${FAIL}" -eq 0 ]] diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh index da6a008f05..8764806519 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh @@ -25,6 +25,7 @@ DIST_ROOT="${TMP_DIR}/dist" MOCK_BIN="${TMP_DIR}/mock-bin" CALL_LOG="${TMP_DIR}/curl-calls" ARGS_LOG="${TMP_DIR}/curl-args" +CONFIG_LOG="${TMP_DIR}/curl-config" COUNT_FILE="${TMP_DIR}/store-call-count" TIMEOUT_LOG="${TMP_DIR}/timeout-arg" CASE_OUTPUT="" @@ -54,8 +55,12 @@ assert_contract() { ! grep -q '/v1/health' "${CALL_LOG}" || \ fail "/v1/health must not gate readiness" [[ -s "${ARGS_LOG}" ]] || fail "curl was not called" - if grep -Fv -- '-u test-user:test-password' "${ARGS_LOG}" | grep -q .; then - fail "authentication arguments were not preserved" + if grep -Fq -- 'test-password' "${ARGS_LOG}"; then + fail "credential leaked into curl argv" + fi + [[ -s "${CONFIG_LOG}" ]] || fail "curl was not given a credential config" + if grep -Fv -- 'user = "test-user:test-password"' "${CONFIG_LOG}" | grep -q .; then + fail "authentication credential was not preserved" fi if grep -Fv -- '--connect-timeout 2' "${ARGS_LOG}" | grep -q .; then fail "per-peer connect timeout was not preserved" @@ -70,6 +75,7 @@ run_case() { local scenario="$1" peers="$2" abort_after="$3" : > "${CALL_LOG}" : > "${ARGS_LOG}" + : > "${CONFIG_LOG}" : > "${COUNT_FILE}" : > "${TIMEOUT_LOG}" : > "${DIST_ROOT}/conf/graphs/hugegraph.properties" @@ -80,6 +86,7 @@ run_case() { MOCK_ABORT_AFTER="${abort_after}" \ MOCK_CALL_LOG="${CALL_LOG}" \ MOCK_ARGS_LOG="${ARGS_LOG}" \ + MOCK_CONFIG_LOG="${CONFIG_LOG}" \ MOCK_COUNT_FILE="${COUNT_FILE}" \ MOCK_TIMEOUT_LOG="${TIMEOUT_LOG}" \ HG_SERVER_PD_REST_ENDPOINT="${peers}" \ @@ -132,6 +139,14 @@ url="${!#}" printf '%s\n' "$*" >> "${MOCK_ARGS_LOG}" printf '%s\n' "${url}" >> "${MOCK_CALL_LOG}" +# The credential must arrive as a config file on stdin, never in argv +for arg in "$@"; do + if [[ "${arg}" == "-K" ]]; then + cat >> "${MOCK_CONFIG_LOG}" + break + fi +done + if [[ "${url}" == */v1/health ]]; then printf '{}\n' exit 0 diff --git a/hugegraph-store/docs/operations-guide.md b/hugegraph-store/docs/operations-guide.md index f46b5559d7..9b942683f0 100644 --- a/hugegraph-store/docs/operations-guide.md +++ b/hugegraph-store/docs/operations-guide.md @@ -2,6 +2,21 @@ This guide covers monitoring, troubleshooting, backup & recovery, and operational procedures for HugeGraph Store in production. +> **PD REST credential.** Calls to a PD REST endpoint on port 8620, other than +> `/v1/health`, `/actuator/*` and `/v1/prom/targets/*`, need HTTP Basic auth: +> one of the internal service names (`hg`, `store`, `hubble`, `vermeer`) and +> PD's `auth.secret-key` value as the password. A call without it gets HTTP +> 401, which for a mutating step such as `balanceLeaders` means the step did +> nothing. Export the secret before following a procedure that uses +> `${PD_SECRET}`: +> +> ```bash +> read -rs PD_SECRET && export PD_SECRET +> ``` +> +> Store endpoints on port 8520 are unaffected. Some PD examples in this guide +> still omit the credential; add `-u hg:"${PD_SECRET}"` when a call returns 401. + ## Table of Contents - [Monitoring and Metrics](#monitoring-and-metrics) @@ -604,19 +619,20 @@ curl http://192.168.1.10:8620/v1/partitionsAndStatus 2. **Verify Registration**: ```bash - curl http://192.168.1.10:8620/v1/stores + curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/stores # New Store should appear ``` 3. **Trigger Rebalancing** (optional): ```bash - curl -X POST http://192.168.1.10:8620/v1/balanceLeaders + curl -u hg:"${PD_SECRET}" -X POST http://192.168.1.10:8620/v1/balanceLeaders ``` 4. **Monitor Rebalancing**: ```bash # Watch partition distribution - watch -n 10 'curl http://192.168.1.10:8620/v1/partitionsAndStatus' + watch -n 10 'curl -u hg:"${PD_SECRET}" \ + http://192.168.1.10:8620/v1/partitionsAndStatus' ``` 5. **Verify**: Wait for even distribution (may take hours)