CAS: object-store connections exhaust the container's ephemeral ports, fencing out the mount lease

🤖 Generated with AI assistance — Claude Opus 5 (claude-opus-5[1m]) — from the CI logs and artifacts linked below, reviewed before posting.

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report — availability failure under read-heavy concurrency on a CAS default policy.

Describe the situation

With cas_disk as the default storage policy, a read-heavy concurrent phase drives the ClickHouse container to exhaust its local ephemeral ports against the object store. Every CAS request then fails at connect() with errno 99:

Poco::Exception. Code: 1000, e.code() = 99,
Net Exception: Cannot assign requested address: 172.18.0.2:9001

Nothing on the object-store side is down — the port pool on the ClickHouse side is empty. Because the mount-lease renewal is a conditional write over the same path, the outage takes down the lease, and the whole disk stops answering reads and writes until a self-remount.

Found in RegressionTestsRelease / CAS (selects) / cas_selects, test /selects/final/force/concurrent/SELECT WHERE parallel inserts, deletes, updates. Both retries of the job failed, at the same phase, from the same cause — this is not flaky.

This issue:

Turns a client-side resource exhaustion into a full read+write outage of cas_disk (1330 queries refused over ~42 s in retry 2)
Fences out the mount lease and forces a self-remount, discarding an in-flight PartWriteTxn
Is not caused by the store: RustFS 1.0.0-beta.12 honours HTTP keep-alive (verified — 3 requests, 1 connect, no Connection: close), so the connection churn is client-side
Is not bounded by any existing limit: disk_connections_store_limit=10000 / disk_connections_hard_limit=200000 against 28 232 ephemeral ports in the container netns

How to reproduce the behavior

Environment

Version: 26.6.2.20000.altinityantalya
Build type: release, amd (build_amd_release)
Package: PR #2159 7b274f0a461d67fff65a56aafb0865ba0e1af4c8
- https://altinity-build-artifacts.s3.amazonaws.com/PRs/2159/7b274f0a461d67fff65a56aafb0865ba0e1af4c8/build_amd_release/clickhouse
Store: RustFS 1.0.0-beta.12, single container (selects/selects_env/rustfs-service.yml)
Disk: cas_disk overriding the default policy (selects/cas_mode.py), no filesystem cache in front of it
Container netns: ip_local_port_range = 32768 60999 (28 232 ports), tcp_fin_timeout = 60

Steps

python3 -u selects/regression.py --clickhouse https://altinity-build-artifacts.s3.amazonaws.com/PRs/2159/7b274f0a461d67fff65a56aafb0865ba0e1af4c8/build_amd_release/clickhouse --cas --local --clickhouse-version 26.6.2.20000.altinityantalya --log log.log  --only '/selects/final/*'

While it runs, watch the port pressure inside the server container:

docker exec <clickhouse1> grep -c ' 06 ' /proc/net/tcp   # TIME_WAIT

Scope matters. Running only the failing scenario (--only '/selects/final/force/concurrent/SELECT WHERE parallel inserts, deletes, updates/*') passes — 280 s, 3837 OK, 0 fail, peak TIME_WAIT 1051 of 28 232. Exhaustion is cumulative: it needs the whole final subtree, a store with accumulated manifests (GC sweep running), and enough parts for FINAL to re-read.

Expected behavior

Concurrent SELECT ... FINAL load should not be able to drain the port pool the mount lease depends on. If the data plane does saturate, the lease should survive and the failure should stay scoped to the affected queries — not take the disk offline for reads.

Actual behavior

Same root cause, two blast radii across the two retries of the same job:

Retry 1 — lease survived; the write budget did not:

05:33:32 <Warning> CasWriteRetryLater: CAS write could not be committed
  (stageManifest: part-manifest PUT at 'data/cas/manifests/selects-cas-clickhouse1/store/6c6/
   6c641193-...@cas@/0000000000000001-0000000000000a6e/000001.zst' is UNCERTAIN
   (retry budget exhausted) — nothing conclusive was named); retrying later

Surfaces to the client as the INSERT failing:

Code: 210. DB::Exception: CAS write could not be committed (...) ; retrying later:
while pushing to view default.MergeTree_table_..._core_mview:
While executing WaitForAsyncInsert. (NETWORK_ERROR)

16 attempts / 90 s operation deadline (CasRequestControl.h) against ~2.5 min of unavailable ports.

Retry 2 — the lease was lost:

12:18:01 <Error> CasMountLeaseKeeper: background renewal failed transiently, retrying while the lease is still valid: ... e.code() = 99 ...
12:19:41 <Error> CasMountLeaseKeeper: (same)
12:19:51 <Error> CasMountLeaseKeeper: (same)
12:20:01 <Error> CasMountLeaseKeeper: background renewal failed, the mount-lease stops advancing

The pool goes TransientNotLive and every query on the disk is refused for ~42 s:

Code: 210. DB::Exception: content-addressed disk 'cas_disk' -- mount lease not held;
backing may be temporarily unreachable; ... TRANSIENT unavailability, not damage

→ 1330 occurrences in the TestFlows log, including plain SELECTs and the async-insert queue.

Then the self-remount invalidates work already in flight:

12:23:16 <Warning> CasWriteRetryLater: PartWriteTxn (writer_epoch 1) belongs to a superseded
  mount incarnation (live epoch 2) — the mount was fenced out and self-remounted; restart the build

Root cause analysis

The trigger is measurable in the server log. MergeTreePrefetchedReadPool activity jumps ~25× in exactly the minute the ports run out — cas_disk has no cache, so each of the 4 parallel SELECT ... FINAL variants re-reads every part from the store:

	prefetch baseline	prefetch peak	first `errno 99`	total `errno 99`
retry 1	~1 000/min	22 973/min (05:31)	05:31:24	7 362 (until 05:33:46)
retry 2	~1 000/min	25 818/min (12:17)	12:17:32	8 722 (until 12:20:06)

All 8 722 target 172.18.0.2:9001 — the object store, nothing else.

~430 GETs/s with tcp_fin_timeout=60 puts on the order of 26 000 sockets in TIME_WAIT, against 28 232 available ports. The retry loops then make it worse rather than better: errno 99 is local resource exhaustion, but it is classified as an ordinary retryable network error, so 16 CAS attempts (and the base client's s3_retry_attempts=500) keep opening more sockets. The lease renewal shares that path and has only 3 chances (mount_lease_ttl_ms=30000 / mount_renew_period=10000).

Suggested fix

Bound port usage. disk_connections_store_limit=10000 / disk_connections_hard_limit=200000 are both far above the 28 232 ports a container has. Cap disk connections for CAS disks below the ephemeral range (and/or investigate why the pool churns connections when the store honours keep-alive).
Do not let the data plane fence out the lease. Give the mount-lease renewal reserved capacity, or widen the margin — 30 s TTL / 10 s renew is 3 attempts.
Classify EADDRNOTAVAIL as local exhaustion, not as a retryable remote error: back off hard instead of spending 16 attempts opening further sockets.
CI mitigation (separate PR): put a filesystem cache in front of cas_disk in selects/cas_mode.py, and/or raise ip_local_port_range / lower tcp_fin_timeout in the CAS test envs.

Additional context

Job: RegressionTestsRelease / CAS (selects) / cas_selects — failed on both retries
Server logs (cas_selects-artifacts-release_zookeeper, path inside the zip: selects/_instances/clickhouse1/logs/clickhouse-server.log) — every excerpt above comes from these:
- retry 1 → artifact 9392364351
- retry 2 → artifact 9402740372
- both expire 2026-11-18; the TestFlows raw.log is in the same zip
Possibly related: #2233 (replica HTTP dies under a CAS NETWORK_ERROR storm on soak) — worth checking whether port exhaustion is the shared mechanism there too

Confirmed — good report, and the diagnosis is close. Adjudicated against cas-gc-rebuild head; the mechanism is fully explainable from code, and an in-house precedent exists (2026-07-06: a 20,000-column merge exhausted ephemeral ports the same way; it was mis-scoped then as "cost only" — this issue shows the same condition takes the lease down, so it is re-rated to availability class).

The exact churn mechanism (refines the report's "investigate why the pool churns"): the S3 client's keep-alive default is 5 seconds (S3Defines.h:12), not the store's. A pooled DISK connection expires after 0.8×5 s of idleness (0.1×5 s once the group crosses disk_connections_soft_limit), mustReconnect() resets any connection whose request started >4.5 s ago (Poco stamps _lastRequest at request start, so long-streaming GETs are reset even on success), and http_keep_alive_max_requests=100 rotates the rest. With P pooled connections and R req/s, reuse survives only while P/R < 4 s; concurrent FINAL prefetch puts P in exactly the dead band, so nearly every request opens a fresh socket and the client closes it on return — TIME_WAIT at request rate, 430 × 60 ≈ 26k, matching the measurement. RustFS honoring keep-alive is beside the point: the client throws the sockets away.

On the suggested fixes:

Capping disk connections below the ephemeral range will not help and can make it worse. All disk_connections_* limits gate keeping a connection, not creating one (only hard_limit gates creation, by throwing); TIME_WAIT sockets are closed and invisible to every limit — that is why the failure is EADDRNOTAVAIL and never HTTP_CONNECTION_LIMIT_REACHED; and a lower soft limit flips the pool into the 0.1× (0.5 s) TTL regime sooner, i.e. more churn. Separately, s3_max_connections is dead code on the disk path — it sets an AWS-SDK field that PocoHTTPClient never reads.
Lease resilience: agreed, and it is worse than the issue states in one way, better in another. The renewal is a bare single-attempt conditional PUT once per 10 s through the same DISK pool — no reserved capacity, no retry inside the period (a failed renewal continues into a full wait_for(period)), so the lease has less resilience than a data write (16 attempts / 90 s). But the margin arithmetic (ttl 30 s, period 10 s, margin 2 s) allows at most two ride-out warnings per lease generation — the three warnings over ~2.5 min in retry 2 imply interleaved successful renewals, i.e. the exhaustion was intermittent rather than a continuous blackout. Queued: in-period fast retry + reserved capacity for the renewal.
EADDRNOTAVAIL classification: agreed. It is treated as a remote transient at every layer (Poco → CoreErrors::NETWORK_CONNECTION → retryable; the CAS controller's classifier → Unresolved, the most retry-hungry class). Each retry consumes another port, so the retry policy is the positive-feedback term. The DNS branch in PocoHTTPClient.cpp:840 is the template for a local-exhaustion sub-classification (EADDRNOTAVAIL/EMFILE/ENFILE → hard backoff). Queued.
CI mitigation: the effective one-liner is raising the keep-alive in the disk's S3 config — <http_keep_alive_timeout>60</http_keep_alive_timeout> (optionally <http_keep_alive_max_requests> too) — which moves the reuse threshold from P<~1700 to P<~25000 at 430 req/s. One caveat to verify on the wire: Poco mins the client value down to whatever Keep-Alive: timeout=N the server advertises, so check what RustFS sends. A filesystem cache in front also works (it removes R rather than fixing reuse).

Everything downstream of the trigger worked as designed: fence → TransientNotLive → refusal with the transient-not-damage message → self-remount → superseded-epoch PartWriteTxn refusal is the intended fail-close chain. The ~42 s outage length is constructive, not backoff: the self-remount's token-stability observation wait is ttl + ttl/20 + poll = 36.5 s with these settings, plus claim/quiesce work — i.e. the outage is ≈1.2× lease TTL by construction. Shortening that window (or holding queries in a bounded wait instead of refusing during it) is a pre-existing tracked design question that this issue gives field data for.

Diagnostic worth adding to the repro before any tuning: the pool is fully instrumented — system.events: DiskConnectionsCreated/Reused/Preserved/Expired/Reset; system.metrics: DiskConnectionsTotal/Stored. The TTL-churn mechanism predicts Created ≈ request rate, Expired ≫ Reset, Stored ≈ 0. If instead Reset ≫ Expired, the driver is the non-drained-response branch and the fix belongs in the read path — worth capturing either way.

Re #2233: different mechanism there (interserver relink refusals; adjudicated separately), but the port-exhaustion angle is worth checking in that repro too — grep -c ' 06 ' /proc/net/tcp during the soak would settle it.

TLDR version: add <http_keep_alive_timeout>60</http_keep_alive_timeout> to disk config as a workaround.

UPD: need to make those settings passed through.

🤖 AI-assisted (Claude Opus 5).

Ran the diagnostic you asked for, on --only '/selects/final/*' with --cas (26.6.2.20000, PR 2159 7b274f0a), baseline config. It reproduces outside CI — peak TIME_WAIT to the store hit 28 192 of 28 232 ports (99.9%), 3 679 Cannot assign requested address, same stageManifest ... UNCERTAIN signature as retry 1.

The counters go the other way from the TTL-churn prediction. Over 16.4 min / 617 849 DiskS3* requests:

counter	total	/s
`DiskConnectionsCreated`	208 047	212.1
`DiskConnectionsReused`	424 775	433.0
`DiskConnectionsExpired`	2 265	2.3
`DiskConnectionsReset`	206 958	211.0

Created ≈ Reset (208 047 vs 206 958) — every reset connection is replaced by a fresh socket; 33.7% of all requests open one. Reset exceeds Expired by 91×, so by your own discriminator this is the non-drained-response branch and the fix belongs in the read path.

Narrowing which of the five reset causes in atConnectionDestroy:

mustReconnect() — ruled out: avg read 0.6–2.2 ms, avg write 12–56 ms, against a 5 s keep-alive.
isStoreLimitReached() — ruled out: peak pooled 58, store_limit 10 000.
!isCompleted() / buffered() — the remaining explanation.

Note Expired also covers the idle-TTL wipe in wipeExpiredImpl, not just keep_alive_max_requests, so the TTL regime would have surfaced there — it didn't. Reset is exactly 0 for minutes 0–3 and jumps to 8 961 at minute 4, when read volume roughly doubles.

On the keep-alive workaround — it can't be applied on a CAS disk at all. ContentAddressedSettings::loadFromConfig iterates every key in the disk block and forwards anything outside the 15-entry non_cas_keys allow-list to the CAS settings setter, so <http_keep_alive_timeout> throws UNKNOWN_SETTING and the server dies at startup (CI run, reverted since). No alternative route: the S3AuthSettings fallback looks for the user setting s3_http_keep_alive_timeout, which doesn't exist in Settings.cpp. Looks like the same one-line addition already made for expect_continue_min_bytes / http_client. I checked the wire caveat you raised as well, in case it's useful later: rustfs 1.0.0-beta.12 advertises neither Keep-Alive nor Connection on /health or on a real S3 path, so nothing would have negotiated the client value down.

Withdrawing suggested fix #1 from the report — agreed the disk_connections_* limits gate keeping rather than creating, and a lower soft limit would make it worse.

Confirming your intermittency read on retry 2, with the timeline: the EADDRNOTAVAIL come in two bursts — 12:17:30→12:18:19, then ~80 s with zero errors, then 12:19:40→12:20:09. The renewal warnings sit inside the bursts (12:18:01; then 12:19:41 / 12:19:51 and the give-up at 12:20:01), so the lease rode out the first burst exactly as you described.

One correction to my own report: the 1 051 TIME_WAIT figure was an unattributed total. Counting by remote port, inbound clickhouse client connections on :9000 peaked at 3 073 on their own and are excluded from every number above.

Still open from your list: I haven't run the #2233 soak repro with grep -c ' 06 ' /proc/net/tcp yet.

Caveat: Reset lumps five causes into one counter, so "non-drained response" is reached by elimination, not direct attribution — splitting that counter (or a debug build) would confirm it, and I haven't identified the read-path call site.

ca: backlog — issue #2243 CONFIRMED: keep-alive TTL churn exhausts po…

test: pin that a CAS disk carries its backend's own settings

cas: give the CAS disk settings their own `cas_` config-key namespace

raise S3 keep-alive on the CAS disk

@filimonov

I closed this on the commits that referenced it, but the failure is still happening.

Our cas_selects suite still wasn't configured with http_keep_alive_timeout. As I reported here in August, setting it on a CAS disk killed the server at startup, so we reverted it and had no way to apply the mitigation. It's accepted now, and I configured it today — 60s, plus http_keep_alive_max_requests 1000 — in the helper that generates the CAS disk for every --cas suite.

cas_selects since 2026-08-26, on builds new enough to accept the setting: 15 of 47 job executions, every one of them inside /selects/final/force/concurrent.

date	build	arch	`mount lease not held`	`EADDRNOTAVAIL`	run
2026-08-26	`antalya-26.6` (MasterCI)	aarch64	10	0	32963968181
2026-08-26	`antalya-26.6` (MasterCI)	x86_64	11	0	32963968181
2026-08-27	PR #2272	x86_64	0	5	33094585276
2026-08-31	PR #2251	x86_64	12	0	33412880754
2026-09-03	PR #2308	x86_64	11	0	33772852646
2026-09-04	PR #2309	x86_64	12	0	33885981325
2026-09-05	PR #2300	x86_64	12	0	33931084631
2026-09-06	PR #2318	x86_64	0	5	34040193616
2026-09-07	PR #2309	x86_64	0	5	34103296626
2026-09-08	PR #2292	aarch64	11	0	34143503608
2026-09-08	PR #2321	x86_64	23	5	34214700789
2026-09-08	PR #2251	x86_64	5	0	34255981201
2026-09-09	PR #2321	x86_64	0	5	34297124531

Same-window controls: cas_s3_cache_selects 0 of 44, non-CAS selects 0 of 76. The cache variant passed in every run where the uncached one failed.

With the setting applied, the same job is green: 3020/3020. It had failed three times on this run before the change, on the same ClickHouse build.

@filimonov Your triage listed a few fix directions besides the keep-alive setting, and I'm not sure how many of them landed. Is there anything else planned on the product side for this issue? If so, we should reopen it. Otherwise I'll leave it closed now that the setting is applied.

@filimonov

@filimonov — I'm reopening this because upstream found and fixed what looks like the cause.

Upstream fixed the underlying bug last week

ClickHouse#119072, opened 2026-09-09 and closed 2026-09-10: the pool compared disk_connections_store_limit against the total connections in the group, active plus stored, instead of against the stored ones alone. Once concurrency passes the limit, the cache stops storing anything. Every request then opens its own TCP connection and closes it.

Their counters show it:


`DiskConnectionsTotal`	11,987
`DiskConnectionsStored`	0
`DiskConnectionsCreated`	1,935,132
`DiskConnectionsReset`	1,919,921 (99.2%)

That chain ends where this issue started. The cache goes off, the churn starts, the ephemeral ports run out, and the server reports Cannot assign requested address.

ClickHouse#119078 fixes it, merged 2026-09-10. It replaces isStoreLimitReached() with tryReserveStoredConnection() and counts stored connections separately.

Our CAS lanes cross the same limit

Here is every test that this warning failed in the last 45 days (query):

Day	PR	Lane	Test	Pool	Links
2026-09-02	#2301	`amd_asan_ubsan cas, 1/2`	`03100_lwu_23_apply_patches`	8,100	json · job
2026-09-08	branch	`amd_tsan cas, 2/2`	`00504_mergetree_arrays_rw`	9,600	json · job
2026-09-10	branch	`amd_tsan cas, 1/2`	`03827_row_policy_partition_column_group_by`	8,000	json · job
2026-09-10	branch	`amd_tsan cas, 1/2`	`04033_tpc_ds_q17`	9,900	json · job
2026-09-10	branch	`amd_tsan cas, 1/2`	`02950_parallel_replicas_used_count`	11,500	json · job
2026-09-11	#2290	`amd_asan_ubsan cas, 2/2`	`00636_partition_key_parts_pruning`	8,900	json · job

Only CAS lanes produce it. No other stateless lane has.

Treat these six as a floor rather than a count. The CI database stores the warning only when it lands in a running test's stderr. How often the pool actually crosses the limit, we can't see.

The two symptoms look like one chain

I read the port exhaustion in this issue and the pool warning above as two points on the same chain, not as two problems. That reading also explains why I reported this still failing after I closed it: the keep-alive settings we added cut the churn, but they never switch the cache back on.

I haven't measured that last part. It comes from reading the diff in ClickHouse#119078.

Questions for you

Does the upstream diagnosis match what you saw in the original reproduction?
Should we backport Compare the HTTP connection store limit against stored connections, not the group total ClickHouse/ClickHouse#119078 to antalya-26.6 and antalya-26.8?
Would anything in the CAS request pattern still bite us after that fix?
If we backport, do the keep-alive settings on the CAS disks stay or come out with it?

cas: count disk connection resets and expiries by reason

cas: recommend the keep-alive settings on every CAS S3 disk config, i…

docs(cas): groom cas-gc-immutable-key-round-cost, cas-gc-teardown-sto…

Navigation Menu

Repository navigation

CAS: object-store connections exhaust the container's ephemeral ports, fencing out the mount lease #2243

Type of problem

Describe the situation

How to reproduce the behavior

Environment

Steps

Expected behavior

Actual behavior

Root cause analysis

Suggested fix

Additional context

Upstream fixed the underlying bug last week

Our CAS lanes cross the same limit

The two symptoms look like one chain

Questions for you

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CAS: object-store connections exhaust the container's ephemeral ports, fencing out the mount lease #2243

Description

Type of problem

Describe the situation

How to reproduce the behavior

Environment

Steps

Expected behavior

Actual behavior

Root cause analysis

Suggested fix

Additional context

Activity

filimonov commented on Aug 20, 2026

filimonov commented on Aug 20, 2026

CarlosFelipeOR commented on Aug 20, 2026

CarlosFelipeOR commented on Sep 9, 2026

CarlosFelipeOR commented on Sep 14, 2026

Upstream fixed the underlying bug last week

Our CAS lanes cross the same limit

The two symptoms look like one chain

Questions for you

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions