Create custom disks outside storage_policies_mutex

[filimonov]

Run the creator of a disk(...) custom disk outside the server-wide storage_policies_mutex.

Context::getOrCreateDisk ran the disk creator under that mutex. A CAS disk whose previous process exited without a farewell waits out the stale mount lease inside the creator (about 36.5 s with the default 30 s lease). While it waited, every other custom-disk creation and every getDisk, getDisksMap and getStoragePolicy call queued behind it. With three stale mounts, server start would take about 110 s; the stress test harness gave up at its limit with "Cannot start clickhouse-server":
#2516 (Stress test (amd_debug), https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2516&sha=b7e424247ab997ae2b0d3c06853e10a2c445f2a5&name_0=PR&name_1=Stress%20test%20%28amd_debug%29)

Changes

• getOrCreateDisk releases the mutex while the creator runs. The mutex still guards selector initialization, lookup, the disks-map snapshot passed to the creator, the in-flight creation map, the final insert into DiskSelector, and configuration reload.
• Concurrent requests for the same name share one creation. A second open of one CAS root would fail as a live twin.
• Concurrent callers of a failed creation all get its failure and do not run their own creator; previously each blocked caller ran its creator in turn. Each caller gets its own copy of a Poco::Exception; the dynamic type is kept for exception types that override rethrow (ErrnoException, NetException). Other exceptions are rethrown unchanged.
• A failed creation is not cached: the next request creates the disk again, as before.
• A configuration reload skips a configured disk whose name is being created as a custom disk and logs a warning: creating it could open the same storage twice.
• The stale-mount wait itself is unchanged.

Risks / notes

• Parallel stale-mount waits apply to custom disks created from independent requests (for example, table loads). Config-defined disks are still created one by one in DiskSelector::initialize.
• One stale mount still costs about 36.5 s of server start.
• A configured disk skipped by a reload is not created by that reload. If the custom creation succeeds, the name then refers to the custom disk. If it fails, the configured disk stays absent until the next reload, and storage policies of that reload that use it are not applied.

Testing

• src/Disks/tests/gtest_cas_startup_stale_mounts.cpp, all also with --gtest_repeat=3:
  • 3 stale disks open in about 3.6 s instead of 10.8 s.
  • A getDisksMap call during a stale-mount wait returns under the 1.15 s bound instead of after 3.6 s.
  • Two concurrent creations of one disk share one open.
  • A failed creation is retried by the next request.
  • Two concurrent requests share one failure: one creator call, each gets its own ErrnoException copy.
  • A configuration reload during a creation does not create the configured disk of the same name.

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Creating a disk(...) custom disk no longer blocks other disk creations and disk or storage policy lookups. Server start with several CAS disk(...) custom disks that wait out a stale mount lease now waits for them in parallel instead of one after another.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

No documentation change: no setting, syntax or documented behavior changes.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [c905ab3]