Skip to content

fix(connectors): defer postgres source progress until ack - #3957

Open
rohankumardubey wants to merge 4 commits into
apache:masterfrom
rohankumardubey:fix/postgres-source-ack
Open

fix(connectors): defer postgres source progress until ack#3957
rohankumardubey wants to merge 4 commits into
apache:masterfrom
rohankumardubey:fix/postgres-source-ack

Conversation

@rohankumardubey

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Closes #3635

Rationale

The PostgreSQL source advanced tracking offsets, deleted or marked rows, and consumed CDC changes before Iggy confirmed delivery. A failed send could therefore permanently skip source records.

What changed?

PostgreSQL polling now stages cursor updates and row operations until the runtime reports a successful batch acknowledgment. NACK discards the staged work, while ACK commits the state and performs the pending delete or mark operations.

CDC now peeks logical-slot changes and advances the slot only after acknowledgment. A deterministic regression test stops Iggy during delivery and verifies that the PostgreSQL rows are redelivered after restart.

Local Execution

  • Passed cargo fmt --all -- --check
  • Passed cargo clippy -p iggy_connector_postgres_source -p integration --all-features --all-targets -- -D warnings
  • Passed all 70 PostgreSQL source unit tests
  • Passed all 7 PostgreSQL polling integration tests
  • Passed both PostgreSQL CDC integration tests
  • Passed the deterministic kill-server regression test
  • Passed git diff --check

@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 23, 2026
@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.88235% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.69%. Comparing base (a65f177) to head (9ed427d).

Files with missing lines Patch % Lines
core/connectors/sources/postgres_source/src/lib.rs 80.88% 19 Missing and 7 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3957       +/-   ##
=============================================
- Coverage     83.93%   57.69%   -26.24%     
  Complexity     1358     1358               
=============================================
  Files          1213     1212        -1     
  Lines        168441   139746    -28695     
  Branches     135999   107313    -28686     
=============================================
- Hits         141373    80622    -60751     
- Misses        23389    55625    +32236     
+ Partials       3679     3499      -180     
Components Coverage Δ
Rust Core 50.16% <80.88%> (-34.56%) ⬇️
Java SDK 66.67% <ø> (ø)
C# SDK 76.52% <ø> (ø)
Python SDK 90.13% <ø> (ø)
PHP SDK 84.48% <ø> (ø)
Node SDK 95.81% <ø> (-0.10%) ⬇️
Go SDK 68.37% <ø> (+0.07%) ⬆️
Files with missing lines Coverage Δ
core/connectors/sources/postgres_source/src/lib.rs 73.79% <80.88%> (+0.47%) ⬆️

... and 405 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rohankumardubey

Copy link
Copy Markdown
Contributor Author

/ready

@rohankumardubey

Copy link
Copy Markdown
Contributor Author

/request-review @hubcio

@github-actions
github-actions Bot requested a review from hubcio August 24, 2026 08:40

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few things without diff lines to hang them on:

  • .claude/skills/connector-source/SKILL.md still teaches the pre-ack pattern this pr removes (the "matches poll_tables" snippet writes cursors during poll, "always return state in every ProducedMessages" no longer holds for empty polls, and state-serialization failure is now a hard poll error, lib.rs:288-296) and doesn't mention on_batch_result at all - needs an update to the new contract.
  • mark_or_delete_processed_rows maps db errors to Error::InvalidRecord (lib.rs:732, :751) - misleading now that these errors surface through the ack path; Error::Connection like advance_replication_slot uses would fit better.
  • README.md:15 still promises offset tracking "avoid duplicates" - at-least-once redelivery means duplicates are possible; and README.md:57 documents the poll_interval default as 1s while the code fallback is 10s (lib.rs:180).

Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/README.md
Comment thread core/integration/tests/connectors/postgres/postgres_source.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs Outdated
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs Outdated
Comment thread core/integration/tests/connectors/postgres/postgres_source.rs Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-author PR is waiting on author response

Projects

None yet

Development

Successfully merging this pull request may close these issues.

connectors: no send acknowledgment from runtime to source plugins — state advances and rows are deleted before delivery is confirmed

2 participants