Renew the CAS mount lease until a definitive answer; an expired lease is a pause

[filimonov]

The background CAS mount lease renewal used to give up when it got no definitive answer from the object store inside its budget. It then tripped the fence and started a remount under a fresh epoch. A short object-store outage (tens of seconds) cost about 40 s without writes, restarted merges and counted a CASMountLeaseLost, although nothing had touched the slot and the holder still had its lease on its own clock. Now the renewal retries until the store gives a definitive answer. While the lease cannot cover a write, writes pause and resume under the same epoch; the mount is no longer lost.

The branch is rebased on antalya-26.6-20261008, which already contains #2509 and #2513. It is not stacked any more. Read it commit by commit.

Rollout

• Date a GC mount-token sighting after the read that returned it #2509 (GC dates a sighting after the read) must run on the GC servers before this change reaches the holders, because a live holder now keeps one slot token for the whole outage. It is merged and in the base.
• Wait for the lease a CAS mount holder declares before fencing or reclaiming its slot #2516 (observers wait for the lease the holder declares) is still open. Until it is merged and rolled out, every pool member must use the same cas_mount_lease_ttl_ms.

How it works

• Every cas_mount_renew_period_ms the renewal thread builds one tuple and keeps it for the whole series of attempts: key, body bytes, the expected ETag of the last confirmed body, write_attempt_id, and start, taken before the first request.
• The conditional PUT runs on the lease plane under Retry::untilDefinitive, which has no deadline. After an unclear attempt, the next one waits a uniform draw in [0.8, 1.2] of 1000 ms minus what the attempt took (Retry::pauseAfterAttempt). There is no growing backoff, so the first good second after an outage ends the series.
• Any store refusal (credentials, AccessDenied, MalformedXML, …) waits one renewal period, then a new engine call sends the same tuple. A refused read waits the same period. A clean 412 whose read failed repeats only the read. A refusal after an unclear attempt is settled inside the call.
• Four things end the series:
  • A commit. DriverLease::finish publishes the deadline start + lease under driver_mutex, before the driver leaves the call, so a remount cannot arm a new epoch in between. The store is unconditional; it is monotonic because a renewal starts only after the previous one finished, and the renewer cannot be replaced during a call.
  • A slot that moved: another owner, our own body gc_fenced, a deleted slot, or our token over other bytes. This goes to the existing loss handling.
  • A local failure.
  • A stop or a remount request. Every pause on the lease plane wakes on both.
• Stop semantics.
  • A stop between engine calls, after a refusal or after a read that found the slot unchanged, is NotAttempted: nothing of ours is in flight, and the farewell still hands the slot back.
  • A stop during a call that sent an attempt, or while the reads after a refused precondition repeat, is terminal with classification cancelled: the request may still land, or the slot may have moved. No farewell runs.
  • A remount request (park) always follows a fence trip and stays a loss.
• The pause.
  • Writes stop being admitted once the lease has less than 2 × envelope + cas_lease_safety_margin_ms left. This rule is unchanged.
  • From the same moment, computed by the same predicate, system.cas_mounts shows lifecycle = 'not_live' and lifecycle_reason = 'lease_expired', with lifecycle_since = when writes paused.
  • At the next renewal attempt the server logs one WARNING per pause. It names the writer_epoch and the last renewal error: CAS mount lease of '<srid>' under writer_epoch <E> has too little time left for a write … Last renewal error: ….
  • A confirmed renewal re-admits writes under the same epoch only if it leaves more than that reservation. A renewal confirmed later than that leaves writes paused, and the next one starts at once.
  • A refused write's error says writes are paused and resume once the renewal is confirmed.
  • A clean shutdown during a pause skips the farewell, which has no lease room left, logs one INFO, and leaves the slot for GC to observe.
• The bounded renewals stay bounded by the last confirmed lease minus the safety margin:
  • the startup re-anchor;
  • the remount re-anchor;
  • the background renewal of a pool opened for decommission (bounded_background_renewal).
• Late commit. A renewal confirmed after an outage longer than the lease stores an expires_at_ms already in the past. GC does not fence the slot for it, because GC fences only a token it watched unchanged for its whole threshold.

Settings (no new ones)

• cas_mount_lease_ttl_ms: the length of the right to write from each confirmed renewal.
• cas_mount_renew_period_ms: the interval between renewal starts, and the wait after a refusal.
• cas_attempt_timeout_ms: the budget of one attempt, hence the admission envelope.
• cas_lease_safety_margin_ms: unchanged.
• The 1000 ms spacing is a constant.

Risks / notes

• Nothing changes on the store or in the observer protocol. The remount machinery stays as the reaction to a definitive loss.
• CASMountRenewalDeadlineExceeded now counts only bounded renewals: the startup and remount re-anchors, and a decommission's background renewal. An outage of a normal mount no longer moves it or CASMountLeaseLost; alert on system.cas_mounts.lifecycle_reason = 'lease_expired' instead.
• During a pause, background merges restart about once a second and fail at commit (the pool stays Live while admission is closed). Tracked as backlog task CAS-256.
• The S39 soak card still expects a remount after the long fault and needs CAS-344.
• Docs: mounts-and-leases.md, debugging.md, monitoring.md, troubleshooting.md, metrics_and_events.md, cas_mounts.md.

Testing

• Unit, 30 new or rewritten tests on two shared fixtures:
  • CASRetrySpacing.* and CASRequestsSpacing.*: no deadline, spaced attempts, the standard backoff unchanged.
  • CASHeartbeatUntilDefinitive.*, one tuple past the lease and the refusal pause: RetriesOneTuplePastTheLease, ARefusedRenewalIsSentAgainAfterTheRefusalPause, AfterACleanPreconditionRefusalOnlyTheReadRepeats.
  • CASHeartbeatUntilDefinitive.*, every way a series ends after a refused precondition: AReadThatFindsAForeignHolderEndsTheSeries, AReadThatFindsOurSlotFencedEndsTheSeries, AReadThatFindsTheSlotDeletedEndsTheSeries, OurTokenOverOtherBytesEndsTheSeries.
  • CASHeartbeatUntilDefinitive.*, stops and liveness: AStopBetweenCallsEndsTheSeriesAsCancelled, AStopDuringTheReadsEndsTheSeriesWithoutAFarewell, ALostLivenessDuringTheReadsEndsTheSeriesAsALoss.
  • CASHeartbeatUntilDefinitive.ALateCommitRestartsTheGcObservation: a late commit against GC.
  • CASMountRuntime.*: a long outage is a pause under the same epoch; an old epoch's renewal does not move the new deadline; a late commit reports a true reason; a remount request ends a series.
  • CASMountRuntime.*, stop and park: they end the refusal wait at once, and the farewell survives a stop.
  • CASMountRuntime.*, the rest: one refusal waits one period; the decommission renewal ends inside its lease; the pause is logged once with the last error; a clean shutdown during a pause leaves the slot.
  • CASLifecycleSnapshot.*: the row is not_live from the admission edge.
  • CASHeartbeat.CancellationAfterSendIsTerminalAndForbidsRelease now also asserts classification cancelled.
  • Each new behaviour was shown red by reverting it on the final tree.
• Integration: test_long_mount_put_outage_pauses_under_the_same_epoch; test_cas_mount_renewal_retry 6 passed.
• CAS* gate: 2670 passed (release), 2674 passed (ASan, no error).
• TLA+ model run models/mount-lease-series-v2 @1e4f3222015, all batteries green. Pause safety ResumedEpochNeverWritesAfterGcActed holds in CaRenewalDriver.

Closes: #2431
Related: #2509
Related: #2513
Related: #2516

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

A CAS mount lease renewal retries until the object store gives a definitive answer instead of giving up at a deadline. While the lease is too short to admit a write, writes pause and resume under the same epoch once a renewal is confirmed, instead of losing the mount and remounting.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [e3588e3]