Fix ZMQ REQ identity collisions causing 60s salt-call hangs (backport #69920 to 3006.x) - #70196
Open
jzandbergen wants to merge 1 commit into
Open
Fix ZMQ REQ identity collisions causing 60s salt-call hangs (backport #69920 to 3006.x)#70196jzandbergen wants to merge 1 commit into
jzandbergen wants to merge 1 commit into
Conversation
…altstack#69920) Give each daemon AsyncReqMessageClient a per-instance uuid.uuid4().hex as its ZMQ IDENTITY, replacing the earlier process-wide _REQ_IDENTITY_SLOT counter and SALT_REQ_IDENTITY_SLOT_MAX cap. Each RequestClient is opened and closed by Salt itself, so a per-instance UUID matches the object's lifetime and gives the master ROUTER's routing-id table a 1:1 mapping to a client we control. Fork inheritance of the earlier counter -- root cause of saltstack#69753 -- is impossible by construction, since each child draws a fresh UUID. (cherry picked from commit 098e4be)
twangboy
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backports upstream commit 098e4be
("zeromq: per-instance UUID IDENTITY for daemon AsyncReqMessageClient
(#69920)") from
masterto3006.x.Problem
3006.27 introduced a stable ZMQ IDENTITY for the AsyncReqMessageClient
REQ socket used by long-lived minion/syndic daemons. A running
salt-miniondaemon and a one-offsalt-callinvocation both setopts["__role"] = "minion"and share the sameminion_id, so they endup requesting the same IDENTITY. This causes the master's ROUTER
socket to drop replies for one of the two clients, resulting in
requests stalling for the full ~60s REQUEST_TIMEOUT before retrying.
See #70127 for the full report and root-cause analysis.
Fix
Each
AsyncReqMessageClientnow generates a per-instanceuuid.uuid4().hexas its ZMQ IDENTITY instead of relying on aprocess-wide counter (
_REQ_IDENTITY_SLOT/SALT_REQ_IDENTITY_SLOT_MAX).Since every RequestClient is opened and closed by Salt itself, a
per-instance UUID matches the object's lifetime 1:1, so concurrent
salt-callandsalt-minionprocesses (or any other short-livedclients) can no longer collide on the same identity, regardless of
role or minion ID.
Testing
3006.xwith one trivial import-orderconflict in
salt/transport/zeromq.py(unrelatedzlibimport frommaster history was dropped; not present on 3006.x).
(
tests/pytests/unit/transport/test_zeromq_identity_uuid.py) andupdated existing tests in
test_zeromq.py.Fixes #70127