Skip to content

CAS: refuse to republish a namespace life retired during a cold admission - #2570

Open
filimonov wants to merge 3 commits into
antalya-26.6-20261008from
fix/antalya-26.6/cas-378-retired-life-republish
Open

filimonov wants to merge 3 commits into
antalya-26.6-20261008from
fix/antalya-26.6/cas-378-retired-life-republish

Conversation

@filimonov

@filimonov filimonov commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

A cold admission of a namespace runtime could publish a life that was retired while it waited, and a re-created table then wrote acknowledged files into that dead life. The cold path reads the catalog row (Live, life L), then takes ref_queue_mutex to install the runtime; no lock is held in between. If DROP TABLE and the GC drain retire L in that window, invalidateRemovedCatalogLife erases the name's slot, and the delayed insertion installs a runtime for L into the empty slot. A table re-created with the same UUID then gets L from the warm namespaceLife, putNamespaceFile acknowledges its files, and the dead-life janitor deletes them.

Changes

  • RefNameSlot keeps a retirement count. invalidateRemovedCatalogLife bumps it under ref_queue_mutex and clears the slot's runtime instead of erasing the slot, whatever runtime is resident, including a successor's. A slot is evicted only while its count is zero.
  • The cold paths capture the count before their first catalog read: acquireReadableRefTableRuntime, namespaceLife, namespaceStillLogicallyPresent, dropNamespaceImpl, acquireMutableRefTableRuntime. acquireRefTableRuntime refuses with a retryable error if the count moved; the retry re-reads the catalog.
  • The warm namespaceLife refuses a runtime whose recovered state is terminal.

Risks / notes

  • A retirement record stays in memory until the pool remounts: one small entry per retired name. Clearing it earlier would reopen the window.
  • A concurrent cold admission of another life of the same name can take one extra retry.
  • No on-store format change.

Testing

  • New gtests (fail on the base): a cold reader and a cold writer whose catalog read is followed by DROP and two GC rounds (a test backend runs them during the second catalog GET); a retirement recorded while a successor is resident survives its eviction; namespaceLife refuses a recovered terminal life.
  • CAS* unit tests green in release.

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

CAS: a table dropped and re-created with the same UUID could have acknowledged table files deleted by GC, when a concurrent read admitted the dropped table's namespace while the drop completed.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

filimonov and others added 3 commits October 13, 2026 02:24
A cold reader could see the catalog row `Live`, then lose the CPU before
`acquireRefTableRuntime` took `ref_queue_mutex`. Removal plus the GC drain
retired the life and erased the slot, so the reader inserted a fresh runtime
for the dead life. A warm `namespaceLife` then returned it and
`putNamespaceFile` acknowledged bytes the dead-life janitor later deletes.

- `RefNameSlot` keeps a per-name `retirements` count;
  `invalidateRemovedCatalogLife` bumps it instead of erasing the slot, also
  when no runtime is resident. Eviction keeps slots with a non-zero count.
- Every cold path captures the count before its first catalog GET;
  `acquireRefTableRuntime` throws retry-later if it moved.
- `namespaceLife` refuses a recovered runtime whose terminal removal is
  applied.

Draft: not built or tested yet.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`invalidateRemovedCatalogLife` returned without bumping the name's
retirement count when a successor life was resident. A delayed cold
admission of the predecessor could then insert it once the successor was
evicted. The count is now bumped for every retirement, while invalidation
and detachment stay limited to the exact predecessor runtime.

New tests:
- a delayed predecessor retirement with a successor resident advances the
  count, leaves the successor attached, and the count survives eviction;
- `namespaceLife` refuses a recovered terminal runtime whose removal-admission
  flag is clear, on the cold call and on the warm one;
- the cold `namespaceLife` window between its catalog read and its insertion.

Draft: not built or tested yet.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Draft: not built or tested yet.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Workflow [PR], commit [19b5df8]

@filimonov

Copy link
Copy Markdown
Member Author

Stateless tests (amd_binary, cas s3 storage, parallel) and Stateless tests (amd_debug, parallel): Server died with Logical error: GroupingAggregatedTransform pushed bucket A twice. The stack is in aggregation (MergingAggregatedMemoryEfficientTransform), no CAS frames; the same failure hits other PRs on this base. Not related to this PR. Tracked upstream: ClickHouse#115986

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant