Wait for the lease a CAS mount holder declares before fencing or reclaiming its slot

docs/en/antalya/cas/architecture/garbage-collection.md

@@ -193,8 +193,8 @@ token of each `mount` object.
 - **Writes / deletes:** a token-guarded `PUT` per fenced mount (`gc_fenced = true`, `seq + 1`); no
   deletes
 - **Safety:** fences only after this leader's *own* monotonic clock has watched the mount's
-  write-token hold unchanged for `cas_mount_lease_ttl_ms + 5% + cas_mount_renew_period_ms`
-  (defaults 30 s and 10 s; every server sharing the pool must run the same values); the write is
+  write-token hold unchanged for `L + 5% + cas_mount_renew_period_ms`, where `L` is the longer of this server's
+  `cas_mount_lease_ttl_ms` and the lease the slot's body declares (defaults 30 s and 10 s); the write is
   guarded by that exact token. A completed fence-out stays valid even if this GC later loses
   leadership. `expires_at_ms` (another host's wall clock) is never trusted.
 - **Fails the round if:** nothing — a per-mount `PUT` conflict is re-observed and re-classified

docs/en/antalya/cas/architecture/mounts-and-leases.md

@@ -119,22 +119,23 @@ inside authority already proved by the last confirmed lease.
 
 GC's own view of a dead server is symmetric and clock-skew-immune: a slot becomes fence-eligible
 only after the leader observes the *same* renewal token hold stable, on its own monotonic clock,
-for `TTL + floor(TTL/20) + period` — close to, but not identical to, the threshold a re-mounting
-server uses to wait out a predecessor, which observes `TTL + floor(TTL/20) + max(1,
-floor(period/2))`. Both thresholds are evaluated purely on the observer's own clock and its own
-configured `TTL`/`period`; nothing about the writer's timing travels on the wire. The stamped
-`expires_at_ms` never participates in either decision — it is a writer-stamped diagnostic used by
-`system.cas_mounts` and by the non-authoritative decommission epoch-recovery precheck, never an
-authorization; local fencing is derived instead from the confirmed request's pre-I/O `BOOTTIME`
+for `L + floor(L/20) + period`. `L` is the longer of the leader's own `cas_mount_lease_ttl_ms` and the
+lease the slot's body declares (`expires_at_ms − started_at_ms`). A body without stamps, or with a
+non-positive interval (a farewell), declares nothing. A re-mounting server waits out a predecessor the
+same way with the step `max(1, floor(period/2))`. `period` is always the observer's own. The two stamps
+are compared only with each other, never with the observer's clock. `expires_at_ms` alone is a
+writer-stamped diagnostic used by `system.cas_mounts` and the non-authoritative decommission
+epoch-recovery precheck. Local fencing is derived from the confirmed request's pre-I/O `BOOTTIME`
 anchor plus the TTL, and wall-clock `now` stays audit-only.
 
-Every server sharing a pool must therefore run the identical `cas_mount_lease_ttl_ms` and
-`cas_mount_renew_period_ms`: a member or GC leader configured with a shorter threshold than its
-peers can fence out a healthy peer whose token-update gap merely exceeds that shorter threshold —
-a peer renewing frequently stays live, one that missed a renewal does not. Change these values only
-with every member of the pool stopped; a graceful restart removes only that member's own startup
-observation and does not make mixed thresholds safe. With the defaults (TTL 30 s, period 10 s,
-margin 2 s), `TTL − margin − period − 2 × envelope = 4 s` is the scheduling-lateness budget before
+`cas_mount_lease_ttl_ms` is local to each server. It is how long this server may write after its last
+confirmed renewal, and every other server waits at least this long before fencing or reclaiming its
+slot. A configured value above 10 minutes (600000 ms) is clamped to 10 minutes with a warning at startup, so no holder declares a longer lease. Mixed values are safe, and so is changing the value one server at a time in any order. A longer
+lease on one server delays recovery after that server's crash, and the GC leader logs one `WARNING` per
+slot whose declared lease differs from its own. A server built without this rule waits only its own
+lease: keep the values equal until every server of the pool runs a build with the rule.
+
+With the defaults (TTL 30 s, period 10 s, margin 2 s), `TTL − margin − period − 2 × envelope = 4 s` is the scheduling-lateness budget before
 the first renewal attempt of a period can begin, where `envelope = attempt_timeout + 2 × cap` and
 `cap` is `attempt_timeout` when the disk's `connect_timeout_ms` is `0`, else
 `min(connect_timeout_ms, attempt_timeout)` (7 s with defaults).
@@ -175,7 +176,7 @@ a `MountClaimResult::Kind` together with a `MountPriorState` describing which ce
 | `None` | no reclaim needed (fresh claim or same-epoch refresh) |
 | `Clean` | the predecessor's own graceful farewell (`min_active_build_sequence == UINT64_MAX`) |
 | `Fenced` | GC's own threshold-gated fence-out (`gc_fenced`) |
-| `UncleanObserved` | this claimant's own token-stability observation held for the full `TTL + drift` window |
+| `UncleanObserved` | this claimant's own token-stability observation held for the full observation threshold (the longer of the two leases, plus drift and step) |
 | `UncleanUnsafe` | the operator's explicit `cas_unsafe_remount_no_delay` authorization carried the slot's exact token — not a certificate of death |
 
 ## Behavioral mount-slot model {#mount-state-machines}

docs/en/antalya/cas/configuration.md

@@ -96,7 +96,7 @@ entirely before release. Treat this table as a snapshot of the current build, no
 | `cas_blob_hash` | `cityhash128` | Pool blob content-hash function (`cityhash128` \| `xxh3-128` \| `sha256`). Recorded in the pool at creation; a mismatching config is refused at mount |
 | `cas_blob_hash_allow_new` | `false` | Explicit opt-in to admit a new hash algorithm into an existing pool. One-way: once admitted, the pool carries both algorithms permanently |
 | `skip_access_check` | `false` | Skip the boot-time capability probe (start now, fix later). Only the preflight probe is skipped — the conditional-write correctness check still runs on every writable mount. **Not available on a writable generation-token (GCS) disk**, which refuses to mount with it: there, the probe battery is the only proof that a token-exact delete carries its generation precondition. Mount such a disk read-only if you need to defer the check |
-| `cas_mount_lease_ttl_ms` | `30000` | Milliseconds for which a mount lease remains valid after a successful claim or renewal (≥ 1). Lower values shorten stale-mount recovery but reduce tolerance for object-storage and scheduling delays |
+| `cas_mount_lease_ttl_ms` | `30000` | How long this server may write after its last confirmed renewal; every other server waits at least this long before fencing or reclaiming its slot (≥ 1, at most 600000 — a larger value is clamped to 10 minutes with a warning at startup, so no holder declares a longer lease). Lower values shorten recovery after this server's crash but reduce tolerance for object-storage and scheduling delays |
 | `cas_mount_renew_period_ms` | `10000` | Milliseconds between background mount-lease renewals (≥ 1). It must leave enough time for two attempt envelopes (a renewal write and its settlement read) and the lease safety margin before the TTL expires: `period + 2 × envelope + margin < TTL` |
 | `cas_gc_snapshot_generations_to_keep` | `3` | GC snapshot generations retained |
 | `cas_gc_shards` | `1` | Blob-hash-prefix reducer shards (≥ 1). Recorded in the pool at creation; a mismatching config is refused at mount |
@@ -112,17 +112,15 @@ entirely before release. Treat this table as a snapshot of the current build, no
 | `cas_unsafe_remount_no_delay` | `0` | Reclaim a mount slot that carries this server's own uuid at once after a hard restart, without observing the slot's token for the lease TTL. Unsafe whenever two processes can hold the same `server_uuid` (a copied uuid file, a stalled predecessor). After such a reclaim the predecessor can still start conditional writes until its own cutoff (`confirmed deadline − cas_lease_safety_margin_ms − 2 × envelope`) or until its next renewal meets the token guard, and a request it already sent may still materialize later. That is not a data hazard: ref-log keys carry `(writer_epoch, sequence)` and creates are conditional, so two writers can never commit different bodies to one key, and recovery's epoch seal settles any straggler (recovery fails closed after 64 successive seal-create attempts displaced by newly materializing old-epoch transactions). The exposure is availability, not data. Intended for test stands and deployments that guarantee one process per uuid |
 | `cas_staging_backend` | `local` | Blob staging backend (`local` \| `s3`); `s3` is opt-in and requires native same-store copy on writable mount |
 
-All servers sharing a pool must run the same `cas_mount_lease_ttl_ms` and `cas_mount_renew_period_ms`.
-Startup reclaim and GC's fence-out both judge liveness by the mount slot's write token holding stable
-on the observer's own `CLOCK_BOOTTIME`, using the observer's own threshold — nothing about a writer's
-timing travels on the wire. Startup observes `cas_mount_lease_ttl_ms + floor(cas_mount_lease_ttl_ms /
-20) + max(1, floor(cas_mount_renew_period_ms / 2))`; GC observes `cas_mount_lease_ttl_ms +
-floor(cas_mount_lease_ttl_ms / 20) + cas_mount_renew_period_ms`. A pool member or GC leader
-configured with a shorter threshold than its peers can therefore fence out a healthy peer whose
-token-update gap exceeds that shorter threshold — a peer renewing frequently stays live, one that
-missed a renewal does not. Change these values only with every member of the pool stopped: a
-graceful restart removes only that member's own startup observation and does not make mixed
-thresholds safe.
+Both settings are local. Startup reclaim and GC's fence-out judge liveness by the mount slot's write
+token holding stable on the observer's own `CLOCK_BOOTTIME` for `L + floor(L / 20) + step`. `L` is the
+longer of the observer's `cas_mount_lease_ttl_ms` and the lease the holder's body declares
+(`expires_at_ms − started_at_ms`; a body without stamps or with a non-positive interval declares
+nothing). Startup's step is `max(1, floor(cas_mount_renew_period_ms / 2))` and GC's is
+`cas_mount_renew_period_ms`, both the observer's own. Mixed values are therefore safe in any order;
+a longer lease on one server delays recovery after that server's crash. A server built without this
+rule waits only its own lease, so keep the values equal until every server of the pool runs a build
+with it.
 
 A shorter TTL reduces the tolerance for object-storage delays; a shorter renewal period increases it
 (renewal starts earlier) at the cost of more background traffic. With the defaults,

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp

@@ -28,6 +28,10 @@ namespace ErrorCodes
 
 namespace
 {
+/// Observers wait for the lease a holder declares, so a long lease would block reclaim of a silently
+/// dead member for that long. The configured TTL is clamped here, before anything derives from it.
+constexpr UInt64 max_mount_lease_ttl_ms = 600'000;
+
 /// Poco renders repeated XML elements as `name`, `name[1]`, `name[2]` -- the convention
 /// `StorageURL` and `HTTPDictionarySource` both handle when reading repeated `<header>` elements.
 /// A key of ours that appears twice must be recognised by its base name rather than passed over as
@@ -252,6 +256,14 @@ void ContentAddressedSettings::validate()
             "content_addressed disk: cas_mount_lease_ttl_ms must be >= 1 (got {})",
             settings[ContentAddressedSetting::mount_lease_ttl_ms].value);
 
+    if (settings[ContentAddressedSetting::mount_lease_ttl_ms] > max_mount_lease_ttl_ms)
+    {
+        LOG_WARNING(getLogger("ContentAddressedSettings"),
+            "content_addressed disk: cas_mount_lease_ttl_ms={} exceeds the maximum of {} ms; using {} ms",
+            settings[ContentAddressedSetting::mount_lease_ttl_ms].value, max_mount_lease_ttl_ms, max_mount_lease_ttl_ms);
+        settings[ContentAddressedSetting::mount_lease_ttl_ms] = max_mount_lease_ttl_ms;
+    }
+
     if (settings[ContentAddressedSetting::mount_renew_period_ms] == 0)
         throw Exception(ErrorCodes::BAD_ARGUMENTS,
             "content_addressed disk: cas_mount_renew_period_ms must be >= 1 (got {})",

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp

@@ -696,24 +696,19 @@ RoundReport Gc::runRegularRound(std::function<void()> on_lease_acquired, bool al
     }
 
     /// Etag-guarded fence-out of dead mounts (liveness only — graduation itself paces on GC
-    /// rounds via `new_round`, not on heartbeat acks). Fencing no longer trusts a predecessor's stamped
-    /// `expires_at_ms` against our wall clock — it
-    /// fences ONLY once `mount_obs` has watched the mount's write-token hold unchanged for the full
-    /// threshold on THIS leader's own monotonic clock (shares `claimMountAwaitingExpiry`'s
-    /// `TTL + Drift` formula for a mount's own reopen wait, but with the full renewal period as the
-    /// cadence term below instead of half of it, so the two thresholds are close, not identical).
+    /// rounds via `new_round`, not on heartbeat acks). A slot is fenced only after `mount_obs` has
+    /// watched its token hold unchanged on this leader's own monotonic clock for
+    /// `mountObservationThresholdMs` over the longer of this server's lease and the one the slot
+    /// declares, with the full renewal period as the step.
     const uint64_t ttl_ms = static_cast<uint64_t>(store->poolConfig().mount_lease_ttl_ms.count());
-    /// The formula is shared with `claimMountAwaitingExpiry` via
-    /// `mountObservationThresholdMs` -- see its doc comment (CasServerRoot.h).
-    const uint64_t stable_threshold_ms = mountObservationThresholdMs(
-        ttl_ms, static_cast<uint64_t>(store->poolConfig().mount_renew_period.count()));
+    const uint64_t renew_period_ms = static_cast<uint64_t>(store->poolConfig().mount_renew_period.count());
 
     /// PHASE 3/18 `heartbeat_floor`: one LIST of `gc/server-roots/`, one GET per mount slot, and a fence
     /// PUT per newly-fenced mount.
     {
         GcPhaseTimer t(phase_sink, "heartbeat_floor");
-        const HeartbeatFloor floor = computeHeartbeatFloor(op, layout, now_ms_fn(), mono_ms_fn(),
-                                                           stable_threshold_ms, mount_obs);
+        const HeartbeatFloor floor = computeHeartbeatFloor(op, layout, now_ms_fn(), mono_ms_fn,
+                                                           ttl_ms, renew_period_ms, mount_obs, lease_mismatch_warned);
         report.fence_outs = floor.fenced_now;
         if (floor.fenced_now > 0)
             ProfileEvents::increment(ProfileEvents::CASGCHeartbeatFenceOuts, floor.fenced_now);
@@ -4759,11 +4754,8 @@ RebuildReport Gc::rebuildBaseline(bool force)
     /// returned classification counts are not needed for the round mint — graduation paces on rounds).
     /// Use the same threshold/`mount_obs` as the regular round.
     const uint64_t ttl_ms = static_cast<uint64_t>(store->poolConfig().mount_lease_ttl_ms.count());
-    /// Share the identical formula with `claimMountAwaitingExpiry` via
-    /// `mountObservationThresholdMs` -- see its doc comment (CasServerRoot.h).
-    const uint64_t stable_threshold_ms = mountObservationThresholdMs(
-        ttl_ms, static_cast<uint64_t>(store->poolConfig().mount_renew_period.count()));
-    computeHeartbeatFloor(op, layout, now_ms_fn(), mono_ms_fn(), stable_threshold_ms, mount_obs);
+    const uint64_t renew_period_ms = static_cast<uint64_t>(store->poolConfig().mount_renew_period.count());
+    computeHeartbeatFloor(op, layout, now_ms_fn(), mono_ms_fn, ttl_ms, renew_period_ms, mount_obs, lease_mismatch_warned);
 
     /// Retired-in-snapshot: the rebuilt seal's `condemned_summary` must be TOTAL over gc_shards so a
     /// subsequent regular round reads graduation/carry decisions zero-I/O off it (and its `carryParentRefs`

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h

@@ -427,7 +427,7 @@ class Gc
     /// OBSERVATION clock: monotonic on this
     /// process, injected for tests, defaults to `Pool::bootMs()`, and the ONLY clock the heartbeat
     /// gate's own fence-out threshold is measured against (mirrors `claimMountAwaitingExpiry`'s
-    /// `mono_ms_fn`, but at heartbeat-gate granularity — one GC round is one observation tick).
+    /// `mono_ms_fn`).
     /// Everything else in the round stays deterministic/clock-free.
     ///
     /// `log_` is the logger every round-engine log line is emitted through; pass a disk/srid-scoped
@@ -1009,6 +1009,9 @@ class Gc
     /// leader (after a steal, or a process restart) starts empty, which only delays fencing an
     /// already-dead mount by one extra round (safe: never fences early).
     MountObservationMap mount_obs;
+    /// Srids `computeHeartbeatFloor` already warned about for a lease length different from ours.
+    /// In-memory: a new leader names each mismatched holder once more.
+    std::set<String> lease_mismatch_warned;
 
     /// Throttle for the orphan sweep's retention rollup. The premise retains on any pass whose epochs are
     /// not closed-and-folded, so an unconditional `LOG_INFO` would repeat the same sentence for as long as