CAS: concurrent ATTACH PARTITION / INSERT loses the 90s ref_catalog/_ckpt policy (Code 210); common-pool num_tries then 300s-backs off REPLACE_RANGE

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report - something's broken

Describe the situation

Under the concurrent /alter/attach partition combinatorics, CAS cannot finish ordinary writes. A 10-row INSERT and ALTER TABLE … ATTACH PARTITION … FROM fail with Code: 210 NETWORK_ERROR after the CAS write policy gives up on the pool-global cas/ref_catalog object or a per-namespace _ckpt. That is a user-visible failure on MergeTree as well as ReplicatedMergeTree — no replica handshake is required.

On replicated destinations the same load also ages REPLACE_RANGE queue entries that never ran: selectQueueEntry increments num_tries before trySchedule, a saturated background_common_pool (default 8, max_tasks == max_threads, no wait queue) refuses without rolling the counter back, and the first real CAS error then hits the 300 s max_postpone_time_for_failed_replicated_tasks_ms cap. Replicas miss rows; SYSTEM SYNC REPLICA loses to that budget.

This is not #2310. #2310 is the later relink-confirm Unknown → NO_REPLICA_HAS_PART path (part 1 replica divergence). This issue is the control-plane write that dies first on part 3, including on non-replicated tables. They compose: catalog timeouts leave the ledger recovering, which feeds #2310.

Discovered on PR #2300 cas_s3_cache_alter_attach_3 (job), SHA f377ba3a499beacf5d032f3d34a69185765b8699, package 26.6.4.20001.altinityantalya. Same class on every CAS attach part-3 shard of that run (x86/arm × --cas / --cas-s3-cache, attempts 1 and 2 = 8/8). Isolated /alter/attach partition/part 1/* and a 63-scenario subset pass; the full part-3 Pool() is what saturates the catalog.

This issue:

Fails user INSERT / ATTACH with NETWORK_ERROR after a 90 s CAS policy, not a network fault
Is CAS-only and load-sensitive. Local-disk attach part 3 is not in this PR’s job set; isolated CAS runs pass
Is not unique to CAS improvements #2300 — same attach-load class already failed on the CAS baseline (R7). CAS improvements #2300 still owns making antalya-26.6 CAS attach reliable
Must not be xfailed. The assertion is correct

How to reproduce the behavior

Environment

Version: 26.6.4.20001.altinityantalya (PR CAS improvements #2300 head f377ba3)
Build type: release (Altinity Antalya)
Arch: x86_64 and aarch64 (both fail)
Flags: --use-keeper --with-analyzer
Storage: --cas or --cas-s3-cache (storage_policy = 'cas_policy', shared pool)

CI combinatorics (reliably hits it)

From altinity/clickhouse-regression/alter. Part 3 runs seven features in one Pool(), and simple attach is @Repeat(100) with an inner Pool(6):

./regression.py --local --with-analyzer \
  --clickhouse-binary-path docker://<pr-2300-package> \
  --cas \
  --only '/alter/attach partition/part 3/*'

--cas-s3-cache fails the same way. --only of a single failed leaf does not reproduce — the catalog needs the concurrent CREATE/INSERT/ATTACH storm.

On this SHA, /alter/attach partition/part 1/* and part 2/* passed on all four CAS attach jobs. Part 3 is the reliable trigger.

Local measurements (CAS author, part 1 full set, ~4 min)

These are the amplifier, not the part-3 INSERT failure:

REPLACE_RANGE / DROP_RANGE run in the common background pool
On CAS, one REPLACE_RANGE is ~1.4 s p50 (mean 1.73 s)
Suite keeps ~120 replicated tables in flight; pool ~95 % busy
Queue rows wait minutes (20+ s typical, up to 214 rows/node)
116 / 1006 queue rows had num_tries ≥ 19, empty last_exception, num_postponed = 0 (max 58) — never executed, only refused admission
First real error then gets the 300 s backoff immediately

Expected behavior

A 10-row INSERT into a CAS MergeTree table succeeds. ATTACH PARTITION FROM on the same disk succeeds.
CAS control-plane contention (cas/ref_catalog token-CAS, _ckpt RMW) is queued or retried inside the write, or surfaced as a retryable error the client/test can wait out — not a 90 s hard fail mapped to NETWORK_ERROR.
A replication-queue entry that was never scheduled must not increment num_tries. The first real exception on that entry must start backoff from try 1, not from a counter already at 19–58.
After attach, every replica has the source partition’s rows (RQ.SRS-048.CAS.Relink.AttachPartitionFrom).

Actual behavior

On release builds (`26.6.4.20001.altinityantalya`, #2300 `f377ba3`)

This job (cas_s3_cache_alter_attach_3), first leaf:

/alter/attach partition/part 3/check simple attach partition 2/simple attach partition/run #9/partition key _intDiv_a_2__intDiv_b_2__ tables partitioned_MergeTree empty_partitioned_MergeTree

Code: 210. DB::Exception: Received from localhost:9000. DB::Exception: CAS write could not be committed (read of 'data/cas/ref_catalog': gave up at the policy deadline after 13 attempt(s)); retrying later. (NETWORK_ERROR)
(query: INSERT INTO source_41bbc7e0_acde_11f1_92d3_9200088b5af9 (p, i) SELECT 3, rand64() FROM numbers(10))

Same burst (~06:11–06:12 UTC) also failed rbac/pattern #159 and two partition key ReplicatedMergeTree combinations (TestFlows only prints the first traceback).

Sibling part-3 job (aarch64 cas_s3_cache_alter_attach_3):

Code: 210. DB::Exception: CAS write could not be committed (CAS _ckpt for namespace 'alter-cas-clickhouse1/store/adf/adf9421d-…@cas@': persistent CAS contention, the checkpoint contribution was not published); retrying later. (NETWORK_ERROR)
(query: ALTER TABLE destination_14b532f0_… ATTACH PARTITION 3 FROM source_14b5327a_…)

No crash, no sanitizer, no server death. Move/replace CAS jobs on the same SHA are green.

Consistency on this SHA

Job	part 1	part 2	part 3	move	replace
CAS x86 / arm	OK	OK	Fail ×2	OK	OK
CAS S3 cache x86 / arm	OK	OK	Fail ×2	OK	OK

8/8 part-3 CAS jobs failed. Different leaves each time (run #0 vs #9, different keys/engines).

Root cause analysis

Two defects compose. Line numbers are #2300 f377ba3 (altinity/feature/antalya-26.6/CAS-improvements), read from a local clone after fetching that branch. Do not use the older #2310 source map (684161d).

Defect 1 — CAS control-plane write loses a 90 s policy (this job)

cas/ref_catalog is one object for the whole pool (spec INV-3):

/// The whole-pool namespace catalog (spec INV-3): one object, key `cas/ref_catalog`
/// (`Layout::refCatalogKey`), read on every fold round and every recovery, mutated by one token-CAS
/// write per lifecycle transition.

Key: prefix + "/cas/ref_catalog" (CasLayout.h refCatalogKey()). Every namespace CREATE/recovery/fold re-GETs it. Concurrent attach combinatorics (many tables, many namespaces) turns that object into a hotspot.

Default write policy is 90 s (CasRetry.h):

    /// `within(90'000)` -- the default write policy.
    static Retry standard() { return within(90'000); }

When the read/RMW loop hits that bound it throws NETWORK_ERROR:

void throwCasWriteRetryLater(const String & why)
{
    logCasWriteRetryLater(why);
    throw Exception(ErrorCodes::NETWORK_ERROR, "CAS write could not be committed ({}); retrying later", why);
}

void CasOperation::giveUpReadDeadline(...)
{
    last_read_stop = ReadStop::PolicyExhausted;
    throwCasWriteRetryLater(fmt::format("{} of '{}': gave up at the {} deadline after {} attempt(s)",
                                        verb, subject, bound.lease_bound ? "lease" : "policy", attempts_made));
}

That is the exact job-log string: read of 'data/cas/ref_catalog': gave up at the policy deadline after 13 attempt(s).

The sibling ATTACH failure is the same class on the per-namespace checkpoint. After readModifyWrite returns GaveUp (not FenceLost):

        throwCasWriteRetryLater("CAS _ckpt for namespace '" + life.ns.string()
            + "': persistent CAS contention, the checkpoint contribution was not published");

The comment on throwCasWriteRetryLater says the condition is “expected to self-heal (the caller retries)”. The caller here is a user INSERT / ATTACH with no retry. The test (and a client) sees a hard error.

Defect 2 — `num_tries` increments on an entry that was never scheduled (amplifier)

ATTACH PARTITION FROM on ReplicatedMergeTree is a dummy-range REPLACE_RANGE (StorageReplicatedMergeTree.cpp ~457–460). That type is not fetch and not merge/mutate, so it goes to the common executor:

    assignee.scheduleCommonTask(
        std::make_shared<ExecutableLambdaAdapter>(
            [this, selected_entry]() mutable { return processQueueEntry(selected_entry); }, ...),
        /* need_trigger */ true);
    return true;   // return value of scheduleCommonTask is ignored

Common pool: max_threads == max_tasks == background_common_pool_size (default 8). Unlike merge/mutate, there is no extra task slot.

    shared->common_executor = std::make_shared<OrdinaryBackgroundExecutor>(
        ThreadName::MERGETREE_COMMON,
        background_common_pool_size,
        background_common_pool_size,   // max_tasks == max_threads
        ...
    );

trySchedule refuses immediately when value >= max_tasks_count (MergeTreeBackgroundExecutor.cpp ~161–171). No queue.

Order that ages a never-run entry:

selectQueueEntry() → selectEntryToProcess() constructs CurrentlyExecuting, which does ++entry->num_tries (ReplicatedMergeTreeQueue.cpp:2019) and sets currently_executing.
scheduleCommonTask → trySchedule. If the pool is full this returns false. The ExecutableLambdaAdapter is destroyed; ~CurrentlyExecuting clears currently_executing and future-part tags but does not decrement num_tries (ReplicatedMergeTreeQueue.cpp:2077–2111).
scheduleDataProcessingJob still return true.
Backoff (getPostponeTimeMsForEntry) is 1 << num_tries ms, capped at max_postpone_time_for_failed_replicated_tasks_ms = 300 s, but only after last_exception_time_ms is set. Schedule refusals leave last_exception empty, so the entry is immediately eligible again and the counter keeps climbing (author: 116/1006 rows, num_tries ≥ 19, empty exception, num_postponed = 0).
The first real failure (Defect 1, or CAS relink confirm returns Unknown after ATTACH PARTITION FROM; replicas never receive attached parts #2310 confirm Unknown) therefore starts at try 19–58 and is postponed for the full 300 s. SYNC REPLICA in the suite is also 300 s.

How this relates to #2310

#2310 is still correct and still open: confirm Unknown is collapsed with No, taxonomy row 3 throws NO_REPLICA_HAS_PART and forbids byte-fetch (DataPartsExchange.cpp:1632–1636, ContentAddressedExchange.h:19–22). That is the first real error once the queue finally runs.

This issue is why the ledger is not quiescent (catalog/_ckpt timeouts → recovery) and why that first error is immediately unrecoverable inside the 300 s test budget (Defect 2). Fixing only #2310 leaves part-3 INSERT/ATTACH red. Fixing only the pool size leaves confirm Unknown and the num_tries accounting bug.

Suggested fix

Do all three. Raising background_common_pool_size to 16–32, or cutting suite parallelism, is a CI workaround only.

A. Catalog / `_ckpt` must not fail a user write in 90 s (this issue)

Serialize writers of cas/ref_catalog (single in-process mutex / admission), or give the catalog its own longer/unbounded policy distinct from blob PUTs
On _ckpt GaveUp (contention, not fence): retry under the caller’s lease, or return a value the attach/insert path can wait on — do not throw NETWORK_ERROR to the client
Do not map “policy exhausted on a shared control object” to NETWORK_ERROR for a user query; that code is the replication-queue retry class and is the wrong signal here

B. Do not increment `num_tries` until the task is admitted (this issue, also worth upstream)

Increment num_tries in processQueueEntry / executeLogEntry after trySchedule succeeded, or decrement in ~CurrentlyExecuting when func never ran
Honour scheduleCommonTask’s false in scheduleDataProcessingJob (today it is ignored)
Optionally: give REPLACE_RANGE the fetch pool, or give the common executor max_tasks > max_threads like merge/mutate

C. Confirm `Unknown` ≠ “source in doubt” (#2310)

Already specified there: split No / Unknown on the wire, or byte-fetch when gate 0 still has the part.

Do not

Xfail /alter/attach partition/…
Only lengthen the test timeout
Call Defect 1 “the test is too aggressive”. Isolated runs pass; a 10-row INSERT must survive concurrent attaches on the same pool

Tests that must go green

cas_alter_attach_3 and cas_s3_cache_alter_attach_3 (x86 + aarch64) on this package
The first leaf of this job: MergeTree → MergeTree simple attach tuple-key INSERT
Existing cas/tests/replicated.py replicated_attach_partition_from
CAS relink confirm returns Unknown after ATTACH PARTITION FROM; replicas never receive attached parts #2310’s confirm-Unknown follower-convergence test, once filed there

Additional context

CI failure

PR: #2300 CAS improvements
Job: RegressionTestsRelease / CASS3CacheAlter (attach, 3) / cas_s3_cache_alter_attach_3
Run: 34388936376 attempt 2 (attempt 1 failed the same four part-3 jobs)
Job log: https://github.com/Altinity/ClickHouse/actions/runs/34388936376/job/102757852231?pr=2300
Siblings (same SHA, same class):
- cas_alter_attach_3 x86
- cas_s3_cache_alter_attach_3 arm (_ckpt contention on ATTACH)
- cas_alter_attach_3 arm
Commit: f377ba3a499beacf5d032f3d34a69185765b8699
Package: 26.6.4.20001.altinityantalya
CI report: https://s3.amazonaws.com/altinity-build-artifacts/PRs/2300/f377ba3a499beacf5d032f3d34a69185765b8699/34388936376/ci_run_report.html
Base: antalya-26.6

Related issues

#2310 — relink confirm Unknown after ATTACH (part 1 replica divergence). Keep that issue. This one is the write-deadline + num_tries pair that part 3 hits first.
#2173 — cross-disk tmp_replace_from_* unique-ref (different path)
#2233 — relink NETWORK_ERROR storm on soak (related load)
#2031 — static audit; CAS-112 is the uncached ref_catalog GET on every positive ref-log append

Requirements

RQ.SRS-048.CAS.MergeTree.InsertSelect / Transparency
  A 10-row INSERT on cas_policy must succeed.

RQ.SRS-048.CAS.Relink.AttachPartitionFrom
  ATTACH PARTITION FROM SHALL preserve correct logical partition contents
  on all replicas.

RQ.SRS-048.CAS.Relink.CrossPool.ByteFallback
  SHALL fall back to byte fetch when confirmation is unavailable.

Scope (do not file one issue per leaf)

All unexpected Fail leaves on the four part-3 CAS jobs of run 34388936376 are this class: simple attach (tuple keys, @Repeat(100)), partition key / partition key datetime, rbac. Ancestor Fail rows are not separate bugs.

Workarounds (CI only)

Raise background_common_pool_size 2–4× (8 → 16 or 32)
Cut intra-part TestFlows Pool() concurrency / Repeat(100)
Neither fixes Defect 1 on a 10-row INSERT once the catalog is hot

Part 1 is build on assumptions, and is (most probably) incorrect.
Part 2 is real defect of the upstream code.

Both are not release blockers, let's try background_common_pool_size=32 and/or reduce test concurrency

Part 2, restated: `num_tries` is charged before the entry is admitted to a pool

Where. ReplicatedMergeTreeQueue::CurrentlyExecuting::CurrentlyExecuting (ReplicatedMergeTreeQueue.cpp:2019) does ++entry->num_tries when selectQueueEntry picks the entry. StorageReplicatedMergeTree::scheduleDataProcessingJob (StorageReplicatedMergeTree.cpp:4430) then calls assignee.scheduleCommonTask, which is trySchedule on the common executor. When the pool is full (background_common_pool_size, default 8, max_tasks == max_threads, no wait queue) trySchedule returns false, the SelectedEntry is destroyed, the entry goes back to the queue, and the increment stays. scheduleFetchTask and scheduleMergeMutateTask have the same shape for the fetch and merge pools.

Effect. ReplicatedMergeTreeQueue::getPostponeTime (:1637) computes the backoff after a failure as 2^num_tries ms, capped by max_postpone_time_for_failed_replicated_tasks_ms (300 s). An entry that was refused admission 19 times and never ran has num_tries = 19; its first real exception postpones it for the full 300 s instead of 2 ms. Measured on cas_alter_attach_1 (run 3, x86): 116 of 1006 queue rows had num_tries >= 19, empty last_exception, num_postponed = 0. The queue is also slow without any exception: on aarch64 cas_alter_attach_3 (run 10) replicas diverged 2700 times with zero CAS exceptions in the logs, purely from waiting on the 8-slot pool where a CAS REPLACE_RANGE takes about 1.4 s.

Scope. Not CAS-specific. The code is identical in upstream 26.6 and master (ReplicatedMergeTreeQueue.cpp:2052 and :1649 on master). CAS only makes REPLACE_RANGE slow enough to saturate the default pool. The regression-suite change (background_common_pool_size=32, Altinity/clickhouse-regression 52e8775ae6) hides it on the CI stand; a server with the default pool and heavy partition operations still hits it.

Fix. Charge num_tries only when the task actually starts, or roll it back when trySchedule refuses. Either:

in scheduleDataProcessingJob, on scheduleCommonTask / scheduleFetchTask / scheduleMergeMutateTask returning false, decrement num_tries and clear last_attempt_time (three call sites, no behavior change otherwise); or
move ++num_tries and last_attempt_time from the CurrentlyExecuting constructor into processQueueEntry / the merge and mutate task entry points, so refusals never touch the counter.

The first is smaller and keeps CurrentlyExecuting as the single owner of the executing state. A unit test can drive it: select an entry with a full pool, observe num_tries == 0 after the refusal, then fail the entry once and check the postpone time is 2 ms, not 300 s.

ps. this can be reported as a separate issue (also in the upstream)

Part 1

Verification pending. No CI run has exercised attach part 3 with the 32-thread pool yet (the regression change landed after run 10 attempt 2 started), and MasterCI carries no regression jobs.

@filimonov

@filimonov Here is failed run of attach part 3 with the 32-thread pool https://github.com/Altinity/clickhouse-regression/actions/runs/35472736745/job/105976572282 and another one https://github.com/Altinity/clickhouse-regression/actions/runs/34722153395/job/104410417483

Upstream issue for part 2 ClickHouse#122106

What remains after the upstream `num_tries` fix

ClickHouse/ClickHouse#122106 fixes the replication-queue amplifier. It will stop rejected executor admissions from turning the first real failure into a 300-second backoff. It will not stop the CAS failures that start this sequence.

The two linked runs already use background_common_pool_size=32, yet attach part 3 still fails. The newer run gives two particularly useful signatures:

read of 'data/cas/ref_catalog': gave up at the policy deadline after 13 attempt(s)
CAS ref catalog 'data/cas/ref_catalog' update: gave up at the policy deadline after zero attempt(s)

The same failure window also contains policy exhaustion on manifest GETs and data/cas/ns/stream/ LIST operations. This means the initial event is broader object-store unavailability, not just a conditional-write conflict on ref_catalog. The CAS control plane then amplifies that event.

1. Remove the pool-global catalog bottleneck

ref_catalog is one object for the whole pool (CasLayout.h:460-465). Every namespace lifecycle mutation decodes, copies, re-encodes, and conditionally replaces the complete object (CasRefCatalog.cpp:154-203). Namespace birth performs a catalog read, a Creating catalog update, _ckpt creation, a Live catalog update, and another catalog read (CasRefLedger.cpp:1297-1347, CasRefCatalog.cpp:600-710).

CasHotKeys already serializes same-process catalog writers and caches the latest body. Therefore the problem is not an uncontrolled same-process 412 race. The remaining problems are:

each ClickHouse process has its own lane, so different servers still compete on the same object;
plain CasRefCatalog::read bypasses the hot-key cache;
every operation transfers and parses the full, growing catalog;
the lane starts the caller's 90-second deadline before queueing (CasHotKeys.cpp:62-142).

The zero attempt(s) failure proves head-of-line blocking: that update spent its complete budget waiting behind another holder and never sent an S3 request.

Fix direction: shard catalog ownership by namespace/hash, or move lifecycle state to bounded per-namespace objects. Avoid requiring full-catalog GETs on namespace recovery. A larger timeout or pool only delays the same saturation point.

2. Serialize `_ckpt` publishers and report the real failure

_ckpt is per namespace, but publishCkpt uses generic readModifyWrite directly (CasRefCkpt.cpp:195-284). Birth, recovery, append-frontier publication, and snapshot publication can touch the same key independently.

The current error is also misleading. CasRefCkpt.cpp:270-280 maps every non-fence GaveUp to persistent CAS contention. That includes transport/read deadline exhaustion and ambiguous writes, not only ETag conflicts. The log text cannot distinguish contention from an object-store outage.

Fix direction: route _ckpt updates through CasHotKeys or the namespace append lane. Preserve the GaveUp reason, source, attempt count, and last backend exception instead of replacing all outcomes with persistent CAS contention.

3. Preserve the underlying object-store error

CasRequests::readLoop retries classified backend exceptions, then replaces the final cause with a generic policy-deadline error (CasRequests.h:455-524, CasRequests.cpp:495-500). The artifacts contain no object-store logs, so they cannot tell us whether the trigger was throttling, connection pressure, or backend latency.

Add per-operation metrics and logs for:

verb and key class (ref_catalog, _ckpt, manifest, LIST);
attempt latency and final backend exception;
hot-key queue depth and wait time;
clean ETag conflicts versus transport-ambiguous conflicts;
policy exhaustion before the first request versus after requests were sent.

4. Fix relink confirmation separately

#2310 is a separate protocol problem. No and Unknown collapse to the same unproven wire response (DataPartsExchange.cpp:285-313), and the receiver throws NO_REPLICA_HAS_PART instead of byte-fetching (:1624-1635). Catalog recovery and object-store slowdown can increase Unknown, but Unknown can also occur without either condition. The attach suite will remain vulnerable until this path has its own fix.

Acceptance criteria

Run CAS and CAS-S3-cache attach part 3 on x86 and ARM with the default common pool, not only 32 threads.
Stress multiple Pool/server instances sharing one prefix while creating at least the current suite's namespace count.
Inject slow/failing GET, PUT, and LIST independently. Existing operations must recover without a user-visible CREATE, INSERT, or ATTACH failure.
Verify catalog updates don't expire with zero attempts behind one slow holder.
Run concurrent birth, recovery, append, and snapshot _ckpt publishers. Verify all contributions survive and diagnostics identify the actual failure class.
Verify every replica converges after an unproven relink confirmation.

Until these changes land, reducing attach-suite concurrency is a valid CI workaround. Raising background_common_pool_size is not sufficient, as both linked 32-thread runs demonstrate.

Re-read both linked "pool=32" runs with the artifacts.

Run 34722153395 (Sep 15) is package 26.6.2, before #2300. It fails with a raw Code: 499 S3_ERROR ... Poco Timeout ... key data/cas/ref_catalog, object size 172391 on CREATE TABLE ON CLUSTER. It does not say anything about 26.6.4.

Run 35472736745 (Sep 20) is 26.6.4 with background_common_pool_size=32. 1 of 4 part-3 CAS jobs failed (x86 --cas), the rerun was green. All 6 failed leaves are one event, 02:06–02:08 UTC, TestFlows printed only one traceback:

RustFS stopped answering within the 5 s request timeout for about 80 s. AWSClient: Failed to make request to: http://minio:9001/... Poco::TimeoutException every 5 s, 13 attempts = the 90 s policy. Bursts per minute on clickhouse2: 515, 264, 213 timeouts.
The timeouts hit ref_catalog, manifest GETs and background merges alike, so the store was slow as a whole, not one key.
gave up ... after zero attempt(s) is the catalog lane waiting behind a holder that was itself stuck in the same timeouts.
The _ckpt "persistent CAS contention" ATTACH failure is the same minute. The message is misleading: any non-fence give-up is reported as contention.

So on 26.6.4 the remaining failure is "S3 is down for longer than the retry budget". Retries exist everywhere, but they cannot help if the store is down for 90 s. Any test on any storage fails in that case.

The catalog hot-path story is true only in part. ref_catalog grew from 89 KB to 1.37 MB in 2.5 hours (Removing rows are pruned only by GC), each namespace birth is 3 GETs + 2 conditional PUTs, and clickhouse2 alone lost 6796 conditional catalog PUTs with 412 during the run. But this load comes from the test creating and dropping thousands of tables. Table churn at this rate is not a normal workload.

What we cannot answer today: why RustFS stalled. Its logs are not collected. RustFS writes to /logs inside the container, so docker logs (_service_logs/minio.log) has 3 lines. And the report prefix in the bucket is overwritten by the rerun, the failed attempt's service logs are gone.

Proposed next steps on the infra side:

Collect RustFS logs in regression: RUSTFS_OBS_USE_STDOUT and the other RustFS knobs from CI: cap the RustFS blocking-thread pool on the CAS-S3 lanes and collect its log #2384, same as the stateless CI lanes.
Bump RustFS from 1.0.0-beta.12 to 1.0.0 (https://github.com/rustfs/rustfs/releases/tag/1.0.0).
Upload reports per attempt, so a rerun does not overwrite the failed one.

The num_tries amplifier stays with ClickHouse#122106.

New data from CI runs on 2026-10-08 … 10-15. The _ckpt failures are the same RustFS stall as above. They are not contention.

What happens

RustFS stops answering GET/HEAD for 90–100 s on all three servers at once. PUTs keep working (10–100 ms) in the same seconds.
CAS reads retry until the 90 s policy runs out, then the query fails with Code 210.
The text CAS write could not be committed (_ckpt persistent CAS contention) is misleading: publishCkpt prints it for any give-up. In the failure windows there are zero 412 responses on _ckpt, and the failing call never sent its PUT; it was stuck on the first GET. The "policy deadline after N attempt(s)" variant (e.g. Never resend a CAS blob's object-creating S3 request #2555) is the same event, printed by another code path.

How often

Seen on unrelated PRs on both bases: Antalya 26.6 - Read clustering columns from delta lake and use it for data skipping #2395, Add CAS disk to existing backup tests #2437, Antalya 26.6: Backport of #116171 to write iceberg v3 next_seq_num and first_row_id #2456, Show CAS mount renewal progress in system.cas_mounts #2513, Wait for the lease a CAS mount holder declares before fencing or reclaiming its slot #2516, Never resend a CAS blob's object-creating S3 request #2555.
Attach part 3, both CAS jobs: 37/116 runs red (2026-09-24 … 10-07), 16/38 (10-08 … 10-15). No trend; the RustFS image (1.0.0-beta.12) has not changed since July.

Reproduced locally: a proxy that holds only GETs to _ckpt gives exactly this error on INSERT. Without the stall, 200 concurrent ATTACH PARTITION give 0 errors.

Next

Collect RustFS logs and host metrics in the CAS regression jobs. Without them we cannot tell why the read path stalls.
Try RustFS 1.0.0 instead of beta.12.
The _ckpt message will say the real reason (deadline vs. fence), PR to follow.

CAS: concurrent ATTACH PARTITION / INSERT loses the 90s ref_catalog/_ckpt policy (Code 210); common-pool num_tries then 300s-backs off REPLACE_RANGE #2343

Description

Type of problem

Describe the situation

How to reproduce the behavior

Environment

CI combinatorics (reliably hits it)

Local measurements (CAS author, part 1 full set, ~4 min)

Expected behavior

Actual behavior

On release builds (26.6.4.20001.altinityantalya, #2300 f377ba3)

Consistency on this SHA

Root cause analysis

Defect 1 — CAS control-plane write loses a 90 s policy (this job)

Defect 2 — num_tries increments on an entry that was never scheduled (amplifier)

How this relates to #2310

Suggested fix

A. Catalog / _ckpt must not fail a user write in 90 s (this issue)

B. Do not increment num_tries until the task is admitted (this issue, also worth upstream)

C. Confirm Unknown ≠ “source in doubt” (#2310)

Do not

Tests that must go green

Additional context

CI failure

Related issues

Requirements

Scope (do not file one issue per leaf)

Workarounds (CI only)

Activity

filimonov commented on Sep 10, 2026

filimonov commented on Sep 14, 2026

Part 2, restated: num_tries is charged before the entry is admitted to a pool

filimonov commented on Sep 14, 2026

alsugiliazova commented on Sep 24, 2026

alsugiliazova commented on Sep 25, 2026

alsugiliazova commented on Sep 25, 2026

What remains after the upstream num_tries fix

1. Remove the pool-global catalog bottleneck

2. Serialize _ckpt publishers and report the real failure

3. Preserve the underlying object-store error

4. Fix relink confirmation separately

Acceptance criteria

filimonov commented on Sep 25, 2026

filimonov commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions