Skip to content

[Bugfix] SortedSet Race Condition - Mutating under Shared Lock#1642

Merged
hamdaankhalid merged 4 commits intomainfrom
hkhalid/race-condition-RCU-object-store
Mar 27, 2026
Merged

[Bugfix] SortedSet Race Condition - Mutating under Shared Lock#1642
hamdaankhalid merged 4 commits intomainfrom
hkhalid/race-condition-RCU-object-store

Conversation

@hamdaankhalid
Copy link
Copy Markdown
Contributor

@hamdaankhalid hamdaankhalid commented Mar 19, 2026

TLDR:
SortedSetTimeToLive/ZTTL uses the Read Tsavorite method which uses Shared Lock, and then in SortedSetObjectImpl layer it was mutating shared data structure.

Reasoning:
As I understand, we handle thread safety for SortedSet data structures by depending on the state mutating methods like DeleteExpiredItem to be called for commands under the ExclusiveLock commands using Tsavorite Upsert or RMW methods.
If a SortedSet command is using the Read methods and then it mutates state, that breaks the invariant.

image

Copilot AI review requested due to automatic review settings March 19, 2026 23:02
Comment thread test/Garnet.test/SortedSetShallowCloneRaceTests.cs Outdated
@hamdaankhalid hamdaankhalid force-pushed the hkhalid/race-condition-RCU-object-store branch from a9a1fc1 to b7c4bf8 Compare March 19, 2026 23:06
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a SortedSet thread-safety issue where ZTTL/ZPTTL (read-path operations) mutated SortedSet state by calling DeleteExpiredItems() under Tsavorite’s shared/read lock, breaking the object-store locking invariant. Adds a new test intended to reproduce a related race involving shallow cloning during flush serialization.

Changes:

  • Remove DeleteExpiredItems() from SortedSetTimeToLive to prevent mutations on the read path.
  • Add SortedSetShallowCloneRaceTests to exercise a flush + CopyUpdater shallow-clone concurrency scenario.

Reviewed changes

Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.

File Description
libs/server/Objects/SortedSet/SortedSetObjectImpl.cs Stops ZTTL from mutating SortedSet internals during read operations by removing DeleteExpiredItems().
test/Garnet.test/SortedSetShallowCloneRaceTests.cs Introduces a new race-repro test intended to demonstrate shallow-clone mutation racing with serialization during flush.
Comments suppressed due to low confidence (1)

libs/server/Objects/SortedSet/SortedSetObjectImpl.cs:868

  • Removing DeleteExpiredItems() avoids mutating under a shared/read lock, but SortedSetTimeToLive now treats an expired member like a non-expiring member: GetExpiration returns the stored ticks, and ConvertUtils.*FromDiffUtcNowTicks returns -1 once the timestamp is in the past. That conflates “expired” with “no expiration”, and can change ZTTL/ZPTTL semantics. Consider returning -2 for expired members without mutating state (e.g., check IsExpired(member) or compare the returned expiration ticks to UtcNow and treat as not found).
        private void SortedSetTimeToLive(ref ObjectInput input, ref GarnetObjectStoreOutput output, byte respProtocolVersion)
        {
            var isMilliseconds = input.arg1 == 1;
            var isTimestamp = input.arg2 == 1;
            var numFields = input.parseState.Count;

            using var writer = new RespMemoryWriter(respProtocolVersion, ref output.SpanByteAndMemory);

            writer.WriteArrayLength(numFields);

            foreach (var item in input.parseState.Parameters)
            {
                var result = GetExpiration(item.ToArray());

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@hamdaankhalid hamdaankhalid force-pushed the hkhalid/race-condition-RCU-object-store branch from b7c4bf8 to c4040cf Compare March 19, 2026 23:51
@hamdaankhalid hamdaankhalid merged commit 87083ec into main Mar 27, 2026
69 of 71 checks passed
@hamdaankhalid hamdaankhalid deleted the hkhalid/race-condition-RCU-object-store branch March 27, 2026 17:51
vazois added a commit that referenced this pull request Mar 27, 2026
* fixed

* SortedSet Race Condition - Mutation under Shared Lock

---------

Co-authored-by: Hamdaan Khalid <hkhalid@microsoft.com>
Co-authored-by: Vasileios Zois <96085550+vazois@users.noreply.github.com>
vazois added a commit that referenced this pull request Apr 2, 2026
* Add es-metadata.yml (#1641)

* Fix INFO all/default/everything returning empty response (#1645)

* Fix INFO all/default/everything returning empty response

Implement support for INFO all, default, and everything options:
- all: returns all DefaultInfo sections excluding module-generated ones
- default: returns the default set of sections (same as no-arg INFO)
- everything: returns all DefaultInfo sections including modules

Added pre-declared HashSet collections (AllInfoSet, EverythingInfoSet)
in GarnetInfoMetrics.cs derived from DefaultInfo to support these options.

Updated InfoCommand.cs to use UnionWith with the new HashSets instead of
silently skipping the ALL keyword.

Added tests for all three options, verifying correct section inclusion/exclusion.

Fixes #1643

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* make everything option use DefaultInfo

* Add null/empty guard in GetSectionHeaders test helper

Adds explicit asserts before splitting INFO output so test failures
surface a clear message instead of a NullReferenceException.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* wip; restructuring cluster tests to reduce CI duration

* separate dispose from close and configure socket to allow rapid connect

* ensure socket is disposed succesfully

* fix failing test

* update global.json

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix revivification CLI flag ordering dependency (#1644)

The validation in Options.GetServerOptions() checked the raw
enableRevivification flag (from --reviv) instead of the computed
useRevivBinsPowerOf2 state. This caused specifying --reviv alongside
explicit --reviv-bin-record-sizes and --reviv-bin-record-counts to
always fail, despite the --reviv help text documenting that it can be
overridden by the combination of these flags.

Changed the check from enableRevivification to useRevivBinsPowerOf2,
which correctly reflects that --reviv-bin-record-sizes already
overrides the power-of-2 default when present.

Co-authored-by: Hamdaan Khalid <hkhalid@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Vasileios Zois <96085550+vazois@users.noreply.github.com>

* [Bugfix] SortedSet Race Condition - Mutating under Shared Lock (#1642)

* fixed

* SortedSet Race Condition - Mutation under Shared Lock

---------

Co-authored-by: Hamdaan Khalid <hkhalid@microsoft.com>
Co-authored-by: Vasileios Zois <96085550+vazois@users.noreply.github.com>

* Bugfix: Accept Loop socket error handling (#1646)

* Bugfix Accept Loop

* race condition and tiny loggin fix

* Dispose potentially partial sockets.

* Fix accept loop zombie state: tiered error handling with backoff retry

The accept loop in GarnetServerTcp.HandleNewConnection previously treated
all SocketError != Success as fatal, disposing the SocketAsyncEventArgs and
permanently killing the accept loop. This left the server in a zombie state
where existing connections worked but no new connections were accepted.

Changes:
- Extract HandleAcceptError with three-tier error categorization:
  - Tier 1 (fatal): OperationAborted, NotSocket, etc. — stop loop
  - Tier 2 (resource pressure): TooManyOpenSockets, NoBufferSpaceAvailable,
    etc. — backoff via Timer and retry (100ms initial, 5s cap)
  - Tier 3 (client-caused transient): ConnectionReset, etc. — log and continue
- Add ScheduleAcceptRetry using Timer to avoid blocking IOCP threads
- Dispose retry timer on server shutdown
- Add AcceptLoopTests with RST flood and extended attack simulation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* PR comments

* Nit

* Nit

* simplify greatly

* easy win

* small updates

---------

Co-authored-by: Hamdaan Khalid <hkhalid@microsoft.com>
Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Vasileios Zois <96085550+vazois@users.noreply.github.com>

* Skip Value Expiration Check When Scanning a Tombstoned Record (#1612)

* fix migrate write test

* prevent expiration check on tombstoned key while scanning

* fix formatting

* ensure reviv pause signal is observed through epoch protection

* make revivPauseEvent readonly

* Update test/Garnet.test.cluster/ClusterMigrateTests.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* addressing first comments

* update comment in MigrateScanFunctions

* release epoch when acquiring exlucisve SuspendConfigMerge lock

* add ReaderWriterLock custom implementation

* fix formatting

* fixing simple tests

* make pause reviv thread safe

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Bump picomatch from 2.3.1 to 2.3.2 in /website (#1649)

Bumps [picomatch](https://github.com/micromatch/picomatch) from 2.3.1 to 2.3.2.
- [Release notes](https://github.com/micromatch/picomatch/releases)
- [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md)
- [Commits](micromatch/picomatch@2.3.1...2.3.2)

---
updated-dependencies:
- dependency-name: picomatch
  dependency-version: 2.3.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump brace-expansion from 1.1.12 to 1.1.13 in /website (#1651)

Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 1.1.12 to 1.1.13.
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](juliangruber/brace-expansion@v1.1.12...v1.1.13)

---
updated-dependencies:
- dependency-name: brace-expansion
  dependency-version: 1.1.13
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Handle SocketException on accepted socket configuration (#1648)

When a client RSTs between accept completing and socket setup (e.g. NoDelay),
the SocketException was unhandled and crashed the process. Wrap the post-accept
socket configuration in a try/catch, dispose the dead socket, and continue
the accept loop.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix CLUSTER NODES and CLUSTER SHARDS metadata output (#1657)

* Fix CLUSTER NODES and CLUSTER SHARDS metadata output (#1650)

- CLUSTER NODES: Only append ,hostname when hostname is non-empty
- CLUSTER SHARDS: Replace 'address' field with Redis-compatible 'ip',
  'endpoint', and optional 'hostname' fields
- CLUSTER SHARDS: Honor ClusterPreferredEndpointType for endpoint field
  (ip address when Ip, hostname when Hostname)
- CLUSTER SHARDS: Fix role output to use 'master'/'slave' instead of
  enum names 'PRIMARY'/'REPLICA'
- Update NodeInfo struct and ClusterShards parser in test utilities
  for dynamic key-value field parsing
- Add ClusterShardsTest and ClusterNodesHostnameTest unit tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refactor CLUSTER SLOTS/SHARDS to use StringBuilder

Replace string concatenation with StringBuilder in GetShardsInfo,
GetSlotsInfo, and their helper methods (AppendFormattedSlotInfo,
AppendNodeNetworkingInfo) to reduce intermediate string allocations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments and fix endpoint validation

- Fix endpoint validation (Options.cs): allow cluster-announce-ip when
  bind address is 0.0.0.0/:: (wildcard), since the server listens on
  all interfaces. Only require port match in that case.
- Handle ClusterPreferredEndpointType.Unknown in CLUSTER SHARDS: set
  endpoint to '?' consistent with CLUSTER SLOTS and redirects.
- Add ClusterShardsTest cases for Unknown endpoint type.
- Improve ClusterNodesHostnameTest to handle both hostname-present
  and hostname-absent branches.
- Add even-length guard and explicit role validation in ClusterShards
  test parser.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hamdaan Khalid <42720645+hamdaankhalid@users.noreply.github.com>
Co-authored-by: Hamdaan Khalid <hkhalid@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants