Repository navigation
CAS-S3 ASan stateless hits MEMORY_LIMIT_EXCEEDED with cas_gc_interval_sec 60 #2299
Description
Type of problem
CI OOM — with GC at the product default of 60s, ASan CAS-over-S3 stateless shards finish under 6h but the shared server hits the ~16.5 GiB memory cap. OvercommitTracker then kills unrelated tests.
Summary
Follow-up to the 5s-interval timeout (#2298). A throwaway PR that only changed the test disk from 5s to 60s still failed:
PR #2296 (cas_gc_interval_sec 60, cas_gc_enabled 1)
| Shard | Wall time | FAIL | BROKEN |
|---|---|---|---|
amd_asan_ubsan, cas s3 storage, parallel, 1/2 |
3h14m | 14 | 8 |
amd_asan_ubsan, cas s3 storage, parallel, 2/2 |
2h59m | 10 | 1 |
Most FAILs are Code: 241 MEMORY_LIMIT_EXCEEDED with RSS ~15–16 GiB, clustered in time (shard 1 ~05:59–06:00 UTC+5, shard 2 ~05:14). Examples: 01073_bad_alter_partition, 01031_mutations_interpreter_and_context, 01455_opentelemetry_distributed, 05025_cas_attach_partition_cross_disk, 02841_parallel_replicas_summary.
This is the product default (ContentAddressedSettings: gc_interval_sec 60). The test policy is in tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml. The disk is uncached cas_s3; system logs sit on that policy.
Control
PR #2297 (cas_gc_enabled 0, same ASan CAS-S3 shards):
| Shard | Wall time | FAIL | MEMORY_LIMIT_EXCEEDED |
|---|---|---|---|
| 1/2 | 2h07m | 2 | 0 |
| 2/2 | 2h51m | 0 (pass) | 0 |
The two remaining FAILs on GC-off 1/2 are not GC: 00172_early_constant_folding (random-settings EXPLAIN PIPELINE width vs reference) and 01010_partial_merge_join (RustFS Connection refused on cas_s3/cas/ref_catalog).
Likely fix direction
GC at 60s is still too expensive under ASan + uncached CAS-S3 + this runner memory cap. Options: lengthen or disable background GC for the full sanitizer stateless lane (keep it in a dedicated GC test), put a cache in front of cas_s3, or pin system logs off CAS. Do not paper over it with sleep.
Links
Related: #2298
Related: #2296
Related: #2297
Related: https://github.com/Altinity/ClickHouse/actions/runs/32969478793
Project Antalya Build Version: 26.6.2.20001.altinityantalya
Activity
filimonov commentedon Sep 4, 2026
trying to tune GC - will not help to speed it up.
disabling it - will decrease the tests coverage (and asan is important to capture use after free and other types of issues in the code which is EXECUTED).
The real problem here is GC performance, which should be improved. The performance of GC was secondary initially (correctness firsts). I guess we are coming to the place when perfomance become important too. I will push the first set of optimizations to GC in #2300 later today.
Should improve the situation (probably will still be not enough, but let's see)
filimonov commentedon Sep 7, 2026
Under ASan the server's resident set is dominated by the sanitizer runtime, not by tracked heap:
detect_stack_use_after_return(on by default) gives every thread a fake stack whose touched pages stay mapped for the thread's lifetime (the setting is needed to detect stack-use-after-return)It seems we just create too many CAS disks during the tests run, and every disk keeps some thread pools (about 25 threads per disk).
Will try to address it.
filimonov commentedon Sep 7, 2026
Extra threads may also explain the slowness of msan.
DimensionWieldr commentedon Sep 10, 2026
Hit this again on PR #2300, run 34388936376 — 808 FAILs on
amd_asan_ubsan, cas s3 storage, parallel, 2/2. 744 of them areCode: 241.metric_lognarrows the mechanism a bit.It isn't gradual growth.
CurrentMetric_MemoryTrackinggoes 0.06 GiB → 18.32 GiB in one second and stays 18–20 GiB for the remaining 2h10 (95% of samples > 15 GiB). At that second no otherCurrentMetric_*moves, no query inquery_loguses more than 14 MiB, and there are 6 queries / 43 tables.CurrentMetric_MemoryTrackingUncorrectedgoes negative in the same second.That is the
#elsebranch ofMemoryWorker.cpp(no jemalloc under ASan → Cgroups source): when the global tracker goes transiently negative,updateAllocated(resident)re-baselines it to the process resident set — which under ASan is the sanitizer runtime, not the tracked heap.What decides the run is when that first fires, not how much memory is used:
The healthy run reached a higher resident set and survived, because its correction landed while resident was still ~2.5 GiB. Signature is exactly one >4 GiB step in 1 s in each bad run, and none in the good one — which also explains the rate: 3 mass events in 69 runs of this job over 60 days, on unrelated PRs (#2251, #2318, #2300).
The 64 non-241 FAILs are downstream too — none of them fail in the healthy run.
Code: 210matchesTCPHandler: Cannot initialize connection: Code: 241; the result-differs ones are pipeline width (MergingSortedTransform8 → 1 → 3 → 1) from per-query thread reduction.Upstream tracks the same re-baseline: ClickHouse/ClickHouse#117681, with #117738 and #118094. Worth noting neither covers the ASan lane: ClickHouse#117738 switches to jemalloc
stats.allocated, which doesn't exist here, and ClickHouse#118094 enables continuous correction from resident, which would pin the tracker to the sanitizer's ~21 GiB instead of leaving it at 3.3 GiB. For a build without jemalloc the useful behaviour is probably to not re-baseline to resident at all and just clamp the tracker at 0.AI-assisted analysis. The tables and the one-second step are read from
metric_logandclickhouse-server.logof the four runs; the reading ofMemoryWorker.cppand the suggestion at the end are interpretation, not measurement.filimonov commentedon Sep 16, 2026
This one should additionally help from the other side: #2384