Skip to content

sources: preserve cursor when inbox persistence fails - #410

Open
fallintoplace wants to merge 3 commits into
ClickHouse:mainfrom
fallintoplace:fix/source-cursor-on-persistence-failure
Open

sources: preserve cursor when inbox persistence fails#410
fallintoplace wants to merge 3 commits into
ClickHouse:mainfrom
fallintoplace:fix/source-cursor-on-persistence-failure

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What changed

  • Fail closed when inbox persistence fails.
  • Let the database transaction roll back the full batch.
  • Return an ingestion error and keep the source cursor unchanged.
  • Retry the batch on the next healthy run. Existing dedup keeps retries safe.

Why

A bad row used to be logged and skipped. The rest of the batch still committed and the cursor moved forward, so the bad row could be lost permanently.

Tests

  • uv run pytest tests/test_source_runner_persistence.py tests/test_source_filters.py tests/test_source_resurface.py tests/test_db_atomicity.py
  • uv run python -m compileall -q nerve/db/sources.py nerve/sources/runner.py tests/test_source_runner_persistence.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant