Skip to content

fix(batch): stop a stale stream sender from stealing requests after a fast reconnect - #2262

Open
shadowmodder wants to merge 2 commits into
weaviate:mainfrom
shadowmodder:shadowmodder/fix/batch-sync-stale-sender
Open

shadowmodder wants to merge 2 commits into
weaviate:mainfrom
shadowmodder:shadowmodder/fix/batch-sync-stale-sender

Conversation

@shadowmodder

Copy link
Copy Markdown
Contributor

Fixes #2261

What changes

After a hang-up, recv_wrapper reconnects and starts a new stream. If the reconnect finishes within the old sender's 1s queue wait, the old __send generator — still blocked in __reqs.get() — wakes with the next request, adds its uuids to __inflight_objs, and yields it into the dead call, where gRPC drops it. The batch is lost silently and, with __inflight_objs wedged at the batch size, every later add_object hangs. The same steal applies to the None stop sentinel, which the live stream then never receives.

  • __recv now bumps a stream generation counter on every new stream and passes it to __send.
  • After a successful get, __send checks that its generation is still current (and that the stream has not hung up). A stale sender returns the request to the queue for the live stream's sender and exits, without touching __inflight_objs.
  • The identical race in the async client (weaviate/collections/batch/async_.py) is fixed the same way.

Testing

  • Three new regression tests in mock_tests/test_batch.py drive the race deterministically without a server: a stale sync sender returning a data request, a stale sync sender returning the stop sentinel, and the async variant. Each asserts the request is handed back, __inflight_objs stays empty, and the live sender then delivers the request exactly once.
  • Mutation check: with the staleness check disabled, all three tests fail; with it, all three pass.
  • mock_tests/test_batch.py: the 3 new tests pass. The 5 pre-existing mock-server tests error at fixture setup in my sandbox both before and after this change (local gRPC mock port), so no regression is attributable to this change; CI will exercise them.
  • Ruff check and format are clean on all three changed files.

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@weaviate-git-bot

Copy link
Copy Markdown

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?

@shadowmodder

Copy link
Copy Markdown
Contributor Author

I agree with the CLA

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

collections/batch/sync.py: after a fast hang-up reconnect the old stream's sender steals a request, hanging add_object

2 participants