Skip to content

[flink] Honor snapshot limit in exactly-once monitor source - #9677

Open
jianguotian wants to merge 1 commit into
apache:masterfrom
jianguotian:fix/exactly-once-monitor-snapshot-limit-upstream
Open

[flink] Honor snapshot limit in exactly-once monitor source#9677
jianguotian wants to merge 1 commit into
apache:masterfrom
jianguotian:fix/exactly-once-monitor-snapshot-limit-upstream

Conversation

@jianguotian

Copy link
Copy Markdown
Contributor

Purpose

FlinkSourceBuilder selects MonitorSource for streaming reads with a consumer-id in EXACTLY_ONCE mode. This path did not honor scan.max-snapshot.count, so it could continue planning non-empty snapshots before checkpoint completion confirmed progress for the snapshots already in flight.

This PR makes the monitor source enforce the configured snapshot limit and ties snapshot-credit release to completed checkpoints.

Changes

  • read scan.max-snapshot.count from the table options in the MonitorSource build path
  • track next-snapshot cursors for non-empty streaming plans while the limit is enabled
  • pause planning when the configured number of snapshots is in flight
  • release only credits covered by a completed checkpoint and wake the source reader
  • require checkpointing when a streaming monitor source enables the snapshot limit
  • reopen the transient planning gate after restore
  • keep the existing public constructor and Flink checkpoint state format unchanged

The limit applies to concurrently planned snapshots. It does not cap the records or splits within one snapshot and does not change scan.max-splits-per-task semantics.

Tests

  • Flink 1.x module compile/install with Checkstyle, Spotless, Maven Enforcer, and RAT
  • OperatorSourceTest with Flink 1.x: 10 tests passed
  • OperatorSourceTest with Flink 2.x: 10 tests passed
  • git diff --check

@jianguotian jianguotian closed this Sep 7, 2026
@jianguotian jianguotian reopened this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant