Skip to content

fix(table): validate snapshot timestamp drift on add snapshot#3062

Open
mrutunjay-kinagi wants to merge 2 commits intoapache:mainfrom
mrutunjay-kinagi:fix-2938-snapshot-timestamp
Open

fix(table): validate snapshot timestamp drift on add snapshot#3062
mrutunjay-kinagi wants to merge 2 commits intoapache:mainfrom
mrutunjay-kinagi:fix-2938-snapshot-timestamp

Conversation

@mrutunjay-kinagi
Copy link
Contributor

Rationale for this change

Align snapshot timestamp validation with other Iceberg implementations by rejecting snapshots that drift backwards by more than one minute.

The new guard in AddSnapshotUpdate checks:

  • snapshot timestamp vs latest snapshot_log entry
  • snapshot timestamp vs table last_updated_ms

Both checks allow up to 60 seconds of clock skew tolerance.

Are these changes tested?

Yes.

  • Added test_update_metadata_add_snapshot_rejects_old_timestamp_vs_snapshot_log.
  • Added test_update_metadata_add_snapshot_rejects_old_timestamp_vs_last_updated.
  • Verified with targeted pytest run.

Are there any user-facing changes?

No API changes.

  • Invalid commit metadata now fails fast with explicit ValueError when timestamp drift exceeds tolerance.

Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrutunjay-kinagi Can you explain more about the use-case? I'm hesitant with these kind of features since it isn't implemented in other OSS implementations AFAIK.

elif (
base_metadata.snapshot_log and update.snapshot.timestamp_ms - base_metadata.snapshot_log[-1].timestamp_ms < -ONE_MINUTE_MS
):
raise ValueError(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe CommitFailedException?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I kept ValueError for consistency with the existing validation branches in this same method (sequence/row-id checks all currently raise ValueError). If you prefer, I can switch this validation path to CommitFailedException in a follow-up commit.

@mrutunjay-kinagi
Copy link
Contributor Author

@Fokko Thanks for the review. The use case here is handling malformed/clock-skewed commit metadata safely and aligning behavior with Java/Rust, which already apply a 1-minute backward-drift tolerance for snapshot timestamps.\n\nWithout this guard, a client can add a snapshot timestamp that is significantly older than current table metadata ( / latest ), which can make timeline semantics inconsistent for timestamp-based operations.\n\nThis change keeps the same 60s tolerance and only rejects larger backward drift; small skew remains allowed.

@mrutunjay-kinagi
Copy link
Contributor Author

Clarification on my previous comment (formatting issue):

The guard is intended to prevent adding snapshots that are significantly older than the table timeline, specifically older than last_updated_ms or the latest snapshot_log entry by more than 60 seconds.

This follows the same tolerance approach used in Java/Rust and keeps small clock skew allowed while rejecting larger backward drift.

@kevinjqliu
Copy link
Contributor

This follows the same tolerance approach used in Java/Rust and keeps small clock skew allowed while rejecting larger backward drift.

could you point to the java/rust references?

@mrutunjay-kinagi
Copy link
Contributor Author

The existing validation branches inside are raising , so for symmetry I left this guard as as well. If you’d prefer switching to later that’s easy to follow up on but I’d keep the current behavior for consistency.

@mrutunjay-kinagi
Copy link
Contributor Author

Reposting cleanly: the guard maintains the same exception type as the surrounding validation branches, so I kept it as ValueError for consistency. If you’d prefer CommitFailedException later, happy to follow up.

@mrutunjay-kinagi
Copy link
Contributor Author

mrutunjay-kinagi commented Feb 18, 2026

@kevinjqliu Here are the references we aligned with:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments