[FLINK-40211][checkpoint] Add Regional Checkpoint metrics, REST API and Web UI support - #28965
Open
raoraoxiong wants to merge 6 commits into
Open
Conversation
raoraoxiong
force-pushed
the
raorao/FLINK-40211-regional-checkpoint-metrics-rest
branch
2 times, most recently
from
August 17, 2026 11:54
b7e7100 to
44f8346
Compare
…iguration and refCheckpointId infrastructure - Add CheckpointListener.notifyRegionalCheckpointComplete(long, RegionalCheckpointInfo) for healthy-region tasks - Add CheckpointListener.notifyRegionalCheckpointFallback(long, long) for failed-region tasks - Add RegionalCheckpointInfo class with fallback checkpoint subtask mapping - Add OperatorCoordinator.supportsRegionCheckpoint() and checkpointCoordinatorForRegionFallback() - Wire OperatorCoordinatorCheckpointContext and OperatorCoordinatorHolder for forwarding - Add 3 config options: region.enabled, region.max-failure-ratio, region.max-consecutive-failures - Add refCheckpointId field to OperatorSubtaskState for tracking historical state references - Add MetadataV7Serializer for backward-compatible serialization of refCheckpointId - Add refCheckpointId to SubtaskStateStats/TaskStateStats for REST API aggregation - Add CheckpointSubsumeHelper for reference-aware checkpoint cleanup - Add regional config fields to CheckpointCoordinatorConfiguration Generated-by: CodeBuddy Code (GLM-5.2)
…d SourceCoordinator fallback - Add RegionalCheckpointHandler: decline buffering, region failure judgment, state recombination, two-tier max-consecutive-failures (Tier 1: force next global; Tier 2: abort + reset) - Wire CheckpointCoordinator to delegate regional checkpoint logic to RegionalCheckpointHandler - Add PendingCheckpoint methods: recordDecline, areAllTasksResponded, markUnacknowledgedTasksAsDeclined, reportFallbackSubtaskStats, finalizeRegionalCheckpoint - Add DefaultCompletedCheckpointStore.computeReferencedCheckpointIds for cleaner reference protection - Implement SourceCoordinator.supportsRegionCheckpoint() and checkpointCoordinatorForRegionFallback() - Implement SplitAssignmentTracker per-checkpoint assignment history with backward-compatible serialization - Wire Regional Checkpoint config through StreamGraph - Add unit tests: success path, consecutive limit, state assembly, deferred abort, cleaner, config Generated-by: CodeBuddy Code (GLM-5.2)
…dispatch and local state cleanup - Extend confirmCheckpoint RPC with fallbackCheckpointId parameter (reuses task-side checkpoint-complete RPC path so notification survives task restarts) - Add Task.notifyRegionalCheckpointFallback + CheckpointableTask.notifyRegionalCheckpointFallbackAsync - Implement StreamTask.notifyRegionalCheckpointFallbackAsync with SubtaskCheckpointCoordinator - Add SubtaskCheckpointCoordinator.notifyRegionalCheckpointFallback + OperatorChain propagation - Add AbstractUdfStreamOperator forwarding to user functions - Add TaskStateManager.pruneStateForCheckpoint for local state cleanup (FLIP-600 Section 9) - TaskExecutor.confirmCheckpoint dispatches to notifyRegionalCheckpointFallback or notifyCheckpointComplete Generated-by: CodeBuddy Code (GLM-5.2)
…ST API fields - Rename metric to regional_checkpoint_count (snake_case) and change from Gauge to Counter - Add ref_checkpoint_id to CompletedSubtaskCheckpointStatistics - Add oldest_ref_checkpoint_id to TaskCheckpointStatistics and CheckpointStatistics (cross-subtask/task min aggregation) - Add regional_checkpoint_enabled, regional_max_failure_ratio, regional_max_consecutive_failures to CheckpointConfigInfo - Wire data channel: OperatorSubtaskState.refCheckpointId -> SubtaskStateStats -> TaskStateStats -> CheckpointStatistics - Regenerate REST API snapshot Generated-by: CodeBuddy Code (GLM-5.2)
- Display Regional Checkpoint configuration in checkpoint config page - Display ref_checkpoint_id per subtask in checkpoint detail page - Add oldest_ref_checkpoint_id column in checkpoint statistics Generated-by: CodeBuddy Code (GLM-5.2)
raoraoxiong
force-pushed
the
raorao/FLINK-40211-regional-checkpoint-metrics-rest
branch
from
August 19, 2026 02:50
44f8346 to
d7f9ea3
Compare
…ST API docs
Two CI failures on this branch:
- DefaultCheckpointStatsTrackerTest#testMetricsRegistration and
#testMetricsAreUpdated asserted 15 registered gauges, but
regional_checkpoint_count is registered via metricGroup.counter(), not
metricGroup.gauge(). Both tests capture registrations by overriding
gauge() only, so the counter never appears and the real count stays at
14. Reverted both assertions to 14. The counter itself is already
covered by RegionalCheckpointMetricsTest.
- RuntimeOpenRestAPIDocsCompletenessITCase failed because
rest_v1_dispatcher.html was not regenerated after adding the
regional_checkpoint_enabled, regional_max_failure_ratio,
regional_max_consecutive_failures and oldest_ref_checkpoint_id REST
fields. Only the .yml spec had been updated. Regenerated with:
mvn package -Dgenerate-rest-docs -pl flink-docs -am -nsu -DskipTests
Generated-by: CodeBuddy Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Adds observability for Regional Checkpoint (FLIP-600 Phase 7-9): metrics, REST API fields, and Web UI display.
Changes
Commit 1: Metrics and REST API
regional_checkpoint_count(snake_case) and change from Gauge to Counterref_checkpoint_idto CompletedSubtaskCheckpointStatisticsoldest_ref_checkpoint_idto TaskCheckpointStatistics and CheckpointStatistics (cross-subtask/task min aggregation)regional_checkpoint_enabled,regional_max_failure_ratio,regional_max_consecutive_failuresto CheckpointConfigInfoCommit 2: Web UI
ref_checkpoint_idper subtask in checkpoint detail pageoldest_ref_checkpoint_idcolumn in checkpoint statisticsTesting
Dependencies
Generated-by: CodeBuddy Code (GLM-5.2)