Expose heartbeat health and failure telemetry

[jeregrine]

Problem

Heartbeat failures currently expose only the final write outcome, which makes it difficult to distinguish a transient network interruption from a permanent configuration failure, understand how close a node is to fencing, or quantify the impact when the watchdog terminates a supervisor tree.

The object-store retry policy also enumerates specific transport reasons. New DNS, socket, TLS, or network reason variants can therefore bypass retries even though a transport failure does not provide an authoritative HTTP response.

Fix

Record every heartbeat backend or HTTP attempt with bounded classifications for result, HTTP status, transport class, and error class. The lifecycle manager now exposes a node-local snapshot of attempt totals, consecutive failures, last-success age, remaining watchdog budget, and cache-degraded duration, while telemetry events provide externally aggregatable attempt, cache-transition, and watchdog-termination measurements.

Failure logs are sampled across a consecutive failure run and emit a single recovery message. Telemetry metadata never includes raw errors, request URLs, or object keys.

Treat every Req.TransportError as retryable within the existing absolute heartbeat deadline. Authoritative authentication and configuration responses still fail immediately. When the watchdog deadline expires, emit the latest heartbeat health and a non-blocking count of children being fenced before terminating the tree.

Restart claims also recheck the discovery-degraded gate inside the storage mutation path, including claims whose lock was previously verified as expired. A retained stale heartbeat snapshot therefore cannot authorize an orphan claim once reconciliation is known to be incomplete.

Supporting information

This is stacked on #24 because that change separates local heartbeat liveness from peer-cache reconciliation and defines the discovery_degraded lifecycle that these metrics observe. The PR should be retargeted to main after #24 merges.

Node-local snapshots reset with the lifecycle manager. Long-term termination and fenced-child totals should be aggregated from telemetry outside the DurableServer supervision tree.