Keep live servers running during degraded discovery

[jeregrine]

Problem

A heartbeat cycle treated two different health signals as one operation: publishing this node's heartbeat and refreshing the cache of every peer heartbeat. If this node successfully proved its own liveness but one peer read failed, the lifecycle manager raised and stopped the supervisor tree, unnecessarily terminating already-owned DurableServers.

At the same time, proceeding with discovery from a partial peer view would be unsafe because it could make a live owner appear orphaned.

Fix

Separate local liveness from cluster discovery:

• The heartbeat loop only publishes this node's heartbeat, renews the watchdog immediately, and schedules the next PUT.
• An independent, single-in-flight reconciliation loop refreshes peer heartbeats for both polling and subscription modes.
• A complete reconciliation updates the heartbeat cache and enables discovery.
• A partial reconciliation retains the last complete cache snapshot and enters discovery_degraded.
• While degraded, orphan detection, lock stealing, and new remote placements fail closed, while already-owned DurableServers and heartbeat PUTs continue.
• Reconciliation keeps retrying until a complete snapshot clears the degraded state.
• Unexpected reconciliation task failures remain visible as errors and degrade discovery without taking down the heartbeat owner.

flowchart TD
  A["Heartbeat timer"] --> B["PUT this node's heartbeat"]
  B -->|Success| C["Renew watchdog"]
  B -->|Failure| D["Stop supervisor tree"]
  C --> E["Continue serving owned processes"]

  F["Reconciliation timer"] --> G["Read all peer heartbeats"]
  G -->|Complete| H["Apply complete cache snapshot"]
  H --> I["Enable discovery and remote placement"]
  G -->|Partial| J["Retain last complete snapshot"]
  G -->|Task failure| J
  J --> K["Set discovery_degraded"]
  K --> L["Block orphan claims, lock stealing, and remote placement"]
  L --> F

Loading

The resulting behavior is deliberately asymmetric:

Condition	Existing owners	Heartbeat PUTs	Takeovers	Remote placement
Own heartbeat PUT succeeds, peer view complete	Continue	Continue	Enabled	Enabled
Own heartbeat PUT succeeds, peer view partial	Continue	Continue	Disabled	Disabled
Own heartbeat PUT fails	Supervisor stops	Stopped	Not applicable	Not applicable

Supporting information

The degraded marker is published through the supervisor's shared ETS coordination table so lock and placement paths can fail closed immediately without synchronously calling the lifecycle manager. The lifecycle manager also rechecks the marker immediately before restart claims and remote placement attempts, closing races with work that began before reconciliation degraded.

No public configuration or storage schema changes are required.