CAS: UNFREEZE on one replica deletes another replica's FREEZE (shadow namespace is missing server_root_id)

[alsugiliazova]

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report — data loss on a shared CAS pool

Describe the situation

ALTER TABLE … FREEZE WITH NAME on a CAS disk publishes frozen parts into a pool-global shadow namespace. Live parts on the same pool are namespaced by server_root_id; frozen parts are not. ALTER TABLE … UNFREEZE WITH NAME on any other replica of the same table therefore acts on that one shared namespace and can release a backup it never took. After DROP TABLE + GC, the frozen blobs are gone.

This is not how FREEZE/UNFREEZE work on ordinary MergeTree (local disk), and it is not how live CAS refs work on the same pool. Confirmed by a six-property isolation suite: P1/P2/P4/P6 pass, P3 and P5 fail.

This issue:

• Is CAS-001 from the 2026-08-12 audit, split out of tracking issue CAS — consolidated static analysis audit findings (tracking) #2031
• Is a real runtime bug on PR [experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 #2159, not a static-only finding
• Hits the intended production shape: several ClickHouse servers, one CAS pool, one ReplicatedMergeTree (CREATE … ON CLUSTER / shared UUID, distinct server_root_id)
• FREEZE/UNFREEZE are local statements. A client that freezes on replica 1 and later unfreezes through a load balancer, ON CLUSTER, or a cleanup job on replica 2 will destroy the backup

---

How to reproduce the behavior

Environment

• Version: 26.6.2.20000.altinityantalya (arm64 release)
• PR: [experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 #2159 (feature/antalya-26.6/CAS)
• Object store: RustFS (rustfs/rustfs:1.0.0-beta.12), advertised as minio:9000, bucket warehouse

Two ClickHouse nodes, one ZooKeeper, macros <replica>clickhouse1</replica> / <replica>clickhouse2</replica>.

Config (each node)

Same pool, different server_root_id. Either a named disk:

<clickhouse>
<storage_configuration>
    <disks>
        <cas_disk>
            <type>object_storage</type>
            <object_storage_type>s3</object_storage_type>
            <metadata_type>cas</metadata_type>
            <server_root_id>cas-{replica}</server_root_id>
            <endpoint>http://minio:9000/warehouse/data/cas/</endpoint>
            <access_key_id>admin</access_key_id>
            <secret_access_key>password</secret_access_key>
        </cas_disk>
    </disks>
    <policies>
        <cas_policy>
            <volumes>
                <main>
                    <disk>cas_disk</disk>
                </main>
            </volumes>
        </cas_policy>
    </policies>
</storage_configuration>
</clickhouse>

or the equivalent inline disk(...) used below (private pool prefix, named disk so SYSTEM CAS GC RUN can address it).

Steps

Use the same table UUID on both replicas (what CREATE … ON CLUSTER already does).

On clickhouse1:

CREATE TABLE t UUID '08761b7c-4921-41d5-82cb-970d5c9088a6'
(
    p UInt8, i UInt64
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/t', 'replica1')
PARTITION BY p
ORDER BY i
SETTINGS disk = disk(
    name = 'cas_t',
    type = object_storage,
    object_storage_type = s3,
    metadata_type = cas,
    server_root_id = 'cas-t-clickhouse1',
    endpoint = 'http://minio:9000/warehouse/data/t/',
    access_key_id = 'admin',
    secret_access_key = 'password');

INSERT INTO t SELECT 1, number FROM numbers(10);
INSERT INTO t SELECT 2, number FROM numbers(10);
INSERT INTO t SELECT 3, number FROM numbers(10);

SYSTEM SYNC REPLICA t;

ALTER TABLE t FREEZE WITH NAME 'pre_upgrade';

On clickhouse2:

CREATE TABLE t UUID '08761b7c-4921-41d5-82cb-970d5c9088a6'
(
    p UInt8, i UInt64
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/t', 'replica2')
PARTITION BY p
ORDER BY i
SETTINGS disk = disk(
    name = 'cas_t',
    type = object_storage,
    object_storage_type = s3,
    metadata_type = cas,
    server_root_id = 'cas-t-clickhouse2',
    endpoint = 'http://minio:9000/warehouse/data/t/',
    access_key_id = 'admin',
    secret_access_key = 'password');

List the pool (RustFS via mc). Live manifests are correctly per-server. Frozen manifests are not:

mc find minio/warehouse/data/t --print '{}'

Still on clickhouse2 — this replica never froze anything:

ALTER TABLE t UNFREEZE WITH NAME 'pre_upgrade';

That statement succeeds. It appends into the freeze's ref stream (P3). Then:

-- either replica
DROP TABLE t SYNC;

-- both replicas, a few times (GC is staged and only one node holds the lease)
SYSTEM CAS GC RUN 'cas_t';

The freeze manifests and the blobs are gone (P5). The same sequence without clickhouse2's UNFREEZE leaves the freeze in the pool (P1). The same sequence on a local-disk ReplicatedMergeTree leaves /var/lib/clickhouse/shadow/pre_upgrade/ on clickhouse1 (P6).

Automated repro: cas/tests/freeze_isolation.py in Altinity/clickhouse-regression (--only '/cas/content addressed storage/freeze isolation/*').

---

Exact wrong paths (recorded run, 2026-08-13)

Pool prefix data/freeze_d5e407a4_971e_11f1_b853_de7b9eea3491/, backup name backup_d5e407a4_971e_11f1_b853_de7b9eea3491, table UUID 08761b7c-4921-41d5-82cb-970d5c9088a6.

Live parts — correct, per server_root_id:

cas/manifests/cas-freeze_d5e407a4_971e_11f1_b853_de7b9eea3491-clickhouse1/store/087/08761b7c-4921-41d5-82cb-970d5c9088a6@cas@/0000000000000001-0000000000000001/000001.zst
cas/manifests/cas-freeze_d5e407a4_971e_11f1_b853_de7b9eea3491-clickhouse1/store/087/08761b7c-4921-41d5-82cb-970d5c9088a6@cas@/0000000000000001-0000000000000002/000001.zst
cas/manifests/cas-freeze_d5e407a4_971e_11f1_b853_de7b9eea3491-clickhouse1/store/087/08761b7c-4921-41d5-82cb-970d5c9088a6@cas@/0000000000000001-0000000000000003/000001.zst
cas/manifests/cas-freeze_d5e407a4_971e_11f1_b853_de7b9eea3491-clickhouse2/store/087/08761b7c-4921-41d5-82cb-970d5c9088a6@cas@/…

Frozen parts published by clickhouse1 — wrong: no server_root_id, so both replicas share one namespace:

cas/manifests/shadow/backup_d5e407a4_971e_11f1_b853_de7b9eea3491/store/087/08761b7c-4921-41d5-82cb-970d5c9088a6/0000000000000001-0000000000000004/000001.zst
cas/manifests/shadow/backup_d5e407a4_971e_11f1_b853_de7b9eea3491/store/087/08761b7c-4921-41d5-82cb-970d5c9088a6/0000000000000001-0000000000000005/000001.zst
cas/manifests/shadow/backup_d5e407a4_971e_11f1_b853_de7b9eea3491/store/087/08761b7c-4921-41d5-82cb-970d5c9088a6/0000000000000001-0000000000000006/000001.zst

Full RustFS keys are those paths under warehouse/data/freeze_d5e407a4_971e_11f1_b853_de7b9eea3491/.

The freeze's ref table (hashed; not named after the backup):

cas/ns/state/8b00fc76ed4a241ca9c58df36cb1836f/
cas/ns/stream/8b00fc76ed4a241ca9c58df36cb1836f/

clickhouse2's UNFREEZE (it took no backup) appended into that same stream:

cas/ns/stream/8b00fc76ed4a241ca9c58df36cb1836f/_log/0000000000000001-0000000000000007.zst
cas/ns/stream/8b00fc76ed4a241ca9c58df36cb1836f/_log/0000000000000001-0000000000000008.zst
cas/ns/stream/8b00fc76ed4a241ca9c58df36cb1836f/_log/0000000000000001-0000000000000009.zst

Nothing was removed at UNFREEZE time — the drop of the shadow refs is a log append. After DROP TABLE + SYSTEM CAS GC RUN, P5 observed 0 freeze manifests and 0 blobs.

---

Expected behavior

A freeze belongs to the server that took it, the same way live CAS refs and local-disk shadow/ do.

• clickhouse2's UNFREEZE WITH NAME 'pre_upgrade' must not write into clickhouse1's freeze namespace (no-op or "nothing of that name here").
• After drop + GC, clickhouse1's freeze must still be in the pool until clickhouse1 unfreezes it.
• Two replicas freezing the same name must not be concurrent writers on one ref table.

RQ.SRS-048.CAS.SharedPool.IndependentRefs: each server's refs stay independent under its server_root_id, even when they resolve to the same blobs.

---

Actual behavior

shadowNamespace() is a static method and cannot see server_root_id. liveNamespace() (12 lines above it) prepends serverPrefix().

// ContentAddressedMetadataStorage.cpp
Cas::RootNamespace liveNamespace(const std::string & table_uuid) const
{
    return Cas::RootNamespace{serverPrefix() + "/" + Cas::mirroredArchiveNamespace(table_uuid)};
}

Cas::RootNamespace shadowNamespace(const std::string & shadow_table_dir)  // static
{
    return Cas::RootNamespace{canonicalDiskPath(shadow_table_dir)};
}

ownsNamespace() only matches namespaces that start with server_root_id + "/", so a shadow namespace is owned by nobody. UNFREEZE is removeRecursive("shadow/<name>/…") → dropNamespace(shadowNamespace(path)) with no owner check (ContentAddressedTransaction.cpp, MergeTreeData.cpp).

---

Why this is a bug, not expected CAS behaviour

Sharing blobs across replicas is the point of CAS. Sharing the freeze ref table is not.

Check	Result
P1 freeze → drop → GC, no extra UNFREEZE	backup survives
P2 owner UNFREEZE → drop → GC	pool emptied (GC works)
P4 clickhouse2 UNFREEZE of an unrelated name	backup untouched
P6 same as P5 on local-disk ReplicatedMergeTree	/var/lib/clickhouse/shadow/ on clickhouse1 survives
P3 clickhouse2 UNFREEZE of the same name	writes into clickhouse1's freeze namespace
P5 P1 + that foreign UNFREEZE	0 manifests, 0 blobs

P1 shows GC does not eat a live freeze by itself. P4 shows UNFREEZE is not globally destructive — the shared backup name is the collision. P6 shows ordinary MergeTree does not behave this way. There is no freeze refcount, no "cluster-wide backup" protocol, and every other CAS exclusion primitive is per-server_root_id. A pool-global shadow namespace is the missing prefix, not a designed shared-backup feature.

---

How to fix

Make shadowNamespace() a non-static member like liveNamespace(), and prefix it with serverPrefix():

Cas::RootNamespace ContentAddressedMetadataStorage::shadowNamespace(const std::string & shadow_table_dir) const
{
    return Cas::RootNamespace{serverPrefix() + "/" + canonicalDiskPath(shadow_table_dir)};
}

Frozen keys then look like live ones:

cas/manifests/<server_root_id>/shadow/<backup>/store/<3hex>/<uuid>/…

UNFREEZE on replica 2 computes a different namespace and cannot drop replica 1's freeze. ownsNamespace() and the GC watermark floor start working for freeze namespaces.

Also prefix the listNamespaces(path) walk in removeRecursive for SYSTEM UNFREEZE (shadow/<name>/ on the disk → serverPrefix() + "/" + path), otherwise the whole-backup listing will miss the new keys.

---

Additional context

• Tracking: CAS — consolidated static analysis audit findings (tracking) #2031 (CAS-001)
• PR: [experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 #2159
• Code: ContentAddressedMetadataStorage.cpp (liveNamespace / shadowNamespace / ownsNamespace), ContentAddressedTransaction.cpp (removeRecursive on shadow paths), MergeTreeData.cpp (freezePartitionsByMatcher / unfreezePartitionsByMatcher)