fix: reach READY in headless kubernetes deployments

[joaopaulosr95]

Summary

On a Kubernetes deployment with no head node, every node stays in WAITING for the
lifetime of the process and returns 503 to every request. The cause is a readiness gate
added in ed6b5a01 that is only ever satisfied by a head node's /register response,
so the k8s discovery path can never satisfy it. This is deterministic, not a race, and
independent of replica count.

This fixes it by deriving cluster_state from the dn roster in the k8s path, and by
re-checking that roster while a node is not yet READY.

Why this affects the shipped manifests

All three deployment manifests under admin/kubernetes/ run headless, each saying so
explicitly:

manifest	setting
k8s_deployment_aws.yml	HEAD_PORT: null # no head container
k8s_deployment_azure.yml	HEAD_PORT: "0" # no head container
k8s_deployment_posix.yml	HEAD_PORT: "0" # no head container

None defines a head container, only NODE_TYPE: sn and NODE_TYPE: dn. Headless is
therefore not an unusual configuration but the only Kubernetes topology this project
documents, which is why the fix belongs in the k8s path rather than in guidance to
deploy a head node.

Root cause

ed6b5a01 ("avoid race condition in ready state logic") added a gate in
updateReadyState():

elif app.get("cluster_state") != "READY":
    is_ready = False

cluster_state is initialised to "WAITING" in baseInit() and is only ever
reassigned inside docker_update_dn_info(), from the head node's /register response.
But update_dn_info() branches on whether a head url exists:

if "is_k8s" in app and not getHeadUrl(app):
    await k8s_update_dn_info(app)     # never touches cluster_state
else:
    await docker_update_dn_info(app)  # the only writer of cluster_state

With no head node the first branch is taken on every health check, so cluster_state
never leaves "WAITING" and is_ready is permanently False.

Reproduced at a single replica with a fully converged roster, so no race is involved:

scaling - updating dn_ids to: ['dn-d21ee']
scaling - node numbers complete          <- roster complete on the first pass
healthCheck - node_state: WAITING        <- and still never goes READY

Why configuration cannot work around it

getHeadUrl() hardcodes dns_name = "127.0.0.1" when KUBERNETES_SERVICE_HOST is
set, and there is no head_endpoint/head_host config key, so a head node has to be a
container in the same pod. With N replicas that yields N independent heads, each seeing
only its own pod's sn/dn and each concluding the cluster is complete. Pods would then
disagree on getObjPartition() for the same obj_id, which is the hazard ed6b5a01
set out to prevent.

The fix

Two changes to k8s_update_dn_info(), both using values the function already computes.

1. Derive cluster_state from the dn roster. The existing if/elif/else chain
already separates "roster complete and self-consistent" from every partial state, so
the else branch sets READY and the partial branches set WAITING.

This covers only the dn dimension of isClusterReady(). That function also
compares the sn count against target_sn_count, which is deliberately not checked
here: getObjPartition() partitions by dn count, so dn completeness is what decides
whether nodes agree on partitioning, whereas an sn that is not up is simply not
serving. Happy to add an sn-count condition if you would rather the two paths match
exactly, though it would need a target_sn_count that headless deployments do not
currently set (it defaults to 0).

It also enforces the function's own documented condition 1, "node_count ==
len(dn_urls) for all dn's" — min_node_count/max_node_count were being computed and
then never read.

2. Re-check the roster while not READY. scale_update was only set when dn_urls
changed, which made the roster check one-shot. During a rescale a dn can be observed
before it has assigned its own node_number (reporting -1), or two dn's can
transiently report the same number. The check logged a warning and, with the pod set
then stable, never ran again, leaving the cluster wedged.

Observed on a 3-replica deployment; every rollout hit one of these transient states:

observed dn_node_numbers	without fix 2	with fix 2
[-1]	wedged in WAITING	re-fetched, converged, READY
[-1, 0, 1]	wedged in WAITING	re-fetched, converged, READY
[0, 1, 1]	wedged in WAITING	re-fetched, converged, READY

Probes in the example manifests

Related, and the reason this outage went unnoticed for as long as it did: all three
manifests pointed their liveness probes at /info, which is in INFO_METHODS and so
bypasses the node_state gate. It returns 200 even while the node is wedged in
WAITING and returning 503 to every real request, so pods reported Running, never
restarted, and passed every probe throughout a total outage. There were also no
readiness probes, so wedged pods stayed in their Service's endpoints.

Adds a hsds-node-state console script that inspects node_state and exits non-zero
unless it is READY. It takes its port from NODE_TYPE, so the sn and dn containers
share one identical command rather than each hardcoding a port:

livenessProbe:
  exec:
    command: ["hsds-node-state"]
  initialDelaySeconds: 30
  periodSeconds: 60
  failureThreshold: 5
readinessProbe:
  exec:
    command: ["hsds-node-state"]
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

All three manifests now use it for both probes. Readiness keeps a wedged pod out of the
Service, so a bad rollout stalls with the old pods still serving instead of replacing
them with 503-ing ones. Liveness restarts a genuinely stuck node. Without it, readiness
alone would pull every wedged pod from the Service and nothing would recover them,
leaving zero endpoints indefinitely. The 5 x 60s liveness budget is deliberately far
longer than the ~10s WAITING dips a rescale causes, so only a real wedge trips it.

It prints the state on stdout as well, which makes the same command useful for
diagnostics (kubectl exec ... -- hsds-node-state).

Trade-offs worth weighing

Two consequences that reviewers should see up front rather than discover.

Rescales now briefly return 503. Any partial view of the roster sets WAITING, so
a scale event 503s until the roster reconverges, roughly one health check interval.
Before ed6b5a01 a node holding a complete dn_ids stayed READY through roster churn.
The trade here is inconsistent partitioning versus short unavailability.
Measured on a 3-replica rollout: 2 requests received 503 across the entire rollout.

Fix 2 polls while not READY. It costs one /info request per dn per health check,
so n² across the cluster, though only while not READY. A cluster that never converges
therefore polls indefinitely where it previously checked once. Negligible at small node
counts; there is a comment marking where backoff would go if it matters at larger
scale.

Testing

Adds tests/k8s/: a k3d integration test that deploys HSDS headless on hermetic POSIX storage,
asserts every node reaches READY, then scales to 3 replicas and re-asserts.

Two properties this failure mode required designing around:

• /info bypasses the node_state gate, so probing it proves nothing about whether
  a node is serving (see the probe section above). The test asserts node_state
  explicitly and sends its requests to a gated route, where the pass condition is
  "not 503".
• The race only surfaces on rescale, so a single-replica test can pass against
  broken code. The test scales up and re-asserts.

Readiness is asserted per container. sn and dn run separate health checks and
converge independently, so an sn can report READY while its own dn is still WAITING,
and a request that shards to that dn gets a 503 from the dn's own gate. Asserting on sn
alone produced a false pass during development.

Verified in both directions: fails on the parent commit (0/1 READY, sn=WAITING dn=WAITING)
and passes with this change, holding at 2/3 READY while a dn converged
rather than falsely reporting 3/3.

Also adds .github/workflows/k8s-integration.yml to run it on push and PR.

Tested on k3s v1.35.5 (via k3d 5.9.0) and on a 3-replica EKS deployment at this commit,
on both linux/amd64 and linux/arm64.

[joaopaulosr95]

@jreadey I tried upgrading our production HSDS server after yesterday's release and found this bug.