Skip to content

fix: Reject state changes from resolvers of an ended sync - #417

Merged
erezrokah merged 2 commits into
cloudquery:mainfrom
jpollard-cs:fix/reject-state-changes-of-ended-syncs
Oct 20, 2026
Merged

erezrokah merged 2 commits into
cloudquery:mainfrom
jpollard-cs:fix/reject-state-changes-of-ended-syncs

Conversation

@jpollard-cs

@jpollard-cs jpollard-cs commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #415. Its rejection of checkpoints from an ended sync's resolvers can leave the next sync unable to save any state.

When a plugin keeps the current sync's state client on an attribute its resolvers read, a resolver left over from an ended sync can call set_key on the next sync's client. That key is tied to the ended sync's scope, so the next flush() is rejected. A failed flush keeps its changes for a retry, so the key stays and every later flush of the new sync raises too, including the post-sync hook. The new sync can also read the leftover's value back through get_key. A state client reused across syncs hits the same problem when a change set during one sync is not flushed before that sync closes, which happens on main too.

Once another sync runs, state changes made by resolvers of an ended sync are now dropped, as _send already does with their checkpoints. set_key ignores them, and flush and get_key discard any that are already recorded, restoring the key's last flushed value.

New tests cover a leftover resolver writing into the next sync's client and a reused client whose sync closed before a flush.

A resolver still running after its sync ended could change the state client of
the next sync, e.g. one a plugin keeps on an attribute. Its key then failed every
flush of that sync, as a failed flush keeps its changes, and the next sync read
its value. set_key now raises before it records the change.
@jpollard-cs
jpollard-cs requested a review from a team as a code owner October 20, 2026 13:33
@jpollard-cs
jpollard-cs requested review from alonnoga and removed request for a team October 20, 2026 13:33

@erezrokah erezrokah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed 4a51ea4: state changes from ended syncs are now dropped instead of rejected. This also fixes a case the first commit misses. Details inline.

"""
with self._flush_lock:
with self._lock:
for key in list(self.changes):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A state client that is kept across syncs is still blocked. If a change is set during its own sync and that sync is closed before a flush, every later flush raises, and get_key returns the stale value. This happens on main too. Now flush and get_key drop these changes and restore the last flushed value. A new test covers it.

return self.mem.get(key)

def set_key(self, key: str, value: str) -> None:
scope = self.checkpoints.current_scope() if self.checkpoints else None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check and the write are not atomic, so the sync can switch between them. Because flush now drops stale changes, this is harmless.


def set_key(self, key: str, value: str) -> None:
scope = self.checkpoints.current_scope() if self.checkpoints else None
if self.checkpoints and self.checkpoints.superseded(scope):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A leftover resolver that calls set_key on its own old client now raises, and on main it doesn't. So the "aren't affected" claim in the description is not true. Now set_key ignores these changes, the same as _send does with checkpoints of ended syncs.

"""Returns the scope of the table resolver running in this context, or None outside table resolvers."""
return self._scope.get()

def superseded(self, scope: Optional[Scope]) -> bool:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same check was in check_scope and schedule. I replaced both with one superseded(scope).



def test_resolvers_of_an_ended_sync_cannot_change_the_state_of_the_next_one():
plugin_attributes: Dict[str, StateClient] = {}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the comment (shared → plugin_attributes), the waits are now asserted, and the test now expects no error.

@erezrokah
erezrokah merged commit bbd5b2e into cloudquery:main Oct 20, 2026
5 checks passed
kodiakhq Bot pushed a commit that referenced this pull request Oct 20, 2026
🤖 I have created a release *beep* *boop*
---


## [0.1.56](v0.1.55...v0.1.56) (2026-10-20)


### Bug Fixes

* Reject state changes from resolvers of an ended sync ([#417](#417)) ([bbd5b2e](bbd5b2e))
* Send checkpoints from resource hooks after their resources ([#419](#419)) ([fe8cc8f](fe8cc8f))
* Use the computed queue size for child resolver pools ([#416](#416)) ([0a4b408](0a4b408))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants