CAS: SYSTEM CAS GC RUN on a follower silently does nothing (acquired_lease=0)

[alsugiliazova]

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Usability / operator contract — the command the operator uses to “clean the pool now” can succeed with no reclamation and no error.

Describe the situation

SYSTEM CAS GC RUN is node-local. One GC leader holds gc/state. A manual run uses allow_steal = false (CasGcScheduler::runOneRoundNow). If this process is not the leader, acquireOrRenewLease fails, runRegularRound returns immediately with acquired_lease = false, and the interpreter still returns a success row of zeros.

The operator intent of GC RUN is “reclaim unreferenced blobs now.” A follower run does not do that. Nothing in the SQL error stream says so. You have to notice acquired_lease = 0 in the result block.

This is not the same as “always steal and run here.” Stealing from a live leader while that leader may be in a destructive phase is how two GC actors overlap (see tracking #2031 CAS-003). Forwarding the query to the leader is also not free: durable GcLease.owner is a random gc_id (UInt128), not a hostname, so a follower cannot print “leader is replica-2” from gc/state today.

This issue:

• Silent no-op on a follower is the bug
• Automatic steal of a live leader is the wrong fix
• “Run as the leader” as a product goal is right; the cheap honest contract is fail loud (and, later, name the holder once gc/state carries a host identity)

---

How to reproduce the behavior

Environment

• CAS disk, two (or more) replicas, gc_enabled = 1 (background scheduler running)
• Version: PR [experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 #2159 / antalya-26.6 CAS

Steps

1. Let background GC elect a leader (default: first writable mount that wins gc/state).
2. On a replica that is not the GC leader:

SYSTEM CAS GC RUN cas_disk;

3. Optionally compare:

SELECT hostName(), disk, server_root_id, is_leader
FROM clusterAllReplicas('{cluster}', system.cas_mounts)
WHERE disk = 'cas_disk';

is_leader is only populated on the local server_root_id row (StorageSystemContentAddressedMounts.cpp — NULL on other members’ mounts). Finding the leader requires querying every replica.

---

Expected behavior

A command whose purpose is pool reclamation should not look like success when it did not reclaim.

Reasonable contract (in order of preference):

1. Fail closed if this node did not acquire the lease: exception (or a non-zero exit / NOT_A_LEADER code) stating that this RUN had no effect, that another GC holds gc/state, and how to proceed (RUN ON CLUSTER, or system.cas_mounts / clusterAllReplicas where is_leader = 1).
2. Do not steal from a live leader on manual RUN. Overlapping destructive rounds is worse than a loud no-op.
3. If we want “always actually clean, from any node” without steal: either ON CLUSTER must fail if zero nodes acquired the lease, or gc/state must publish hostname / server_root_id so the error can name the holder, or the query must be forwarded to that holder. None of that exists today.

SYSTEM CAS GC RUN ON CLUSTER … already fans out and the leader’s row gets acquired_lease = 1. That is a workaround, not a contract: the operator still has to read N rows and notice the zeros.

---

Actual behavior

On a follower (release)

The query returns a row, acquired_lease = 0, all reclaim counters 0. Finish in system.cas_gc_log is NotALeader. No exception.

Anchors:

• CasGcScheduler::runOneRoundNow — allow_steal = false
• Gc::runRegularRound — if !acquired_lease, return empty RoundReport
• InterpreterSystemQuery::appendContentAddressedGcRoundRow — still emits a success row
• GcLease (CasGcStateFormat.h) — {owner: UInt128, seq} only; no host

ON CLUSTER

Same query on every replica. One row may show work; the others are silent no-ops. Easy to treat the statement as “the cluster GCd” after looking at any single follower row.

---

Root cause analysis

Leadership is a pool-wide compare-and-swap on gc/state. Manual RUN correctly refuses to steal (so it will not interrupt a live leader). It then fails open in the SQL contract: “we didn’t run” is encoded as a flag in a successful result, not as a failure. system.cas_mounts.is_leader cannot name a remote leader from one node. Together, the operator cannot achieve “clean the pool” from a random replica without already knowing the GC-leadership model.

---

Additional context

• Docs already say a manual run executes regardless of GC STOP (background pause only). They do not say a follower run is a no-op.
• Related: CAS — consolidated static analysis audit findings (tracking) #2031 CAS-003 (steal vs live destructive GC). This issue is the operator-facing half: don’t steal, but don’t pretend the pool was cleaned.