Repository navigation
[python] Check the bucket count of existing files when writing - #10871
Closed
plusplusjiajia wants to merge 1 commit into
Closed
plusplusjiajia wants to merge 1 commit into
plusplusjiajia wants to merge 1 commit into
Conversation
PyPaimon restored the sequence numbers of a partition without checking the bucket count its existing files were written with. After the bucket option was changed, for example from 2 to 4, writes to an existing partition used the new bucket count, so primary keys already stored under the old layout were written again to other buckets and the table returned duplicate keys. As in Java AbstractFileStoreWrite, writing to a bucket whose existing files have a different total bucket number now fails with Java's message, asking to rescale with INSERT OVERWRITE first. The check uses the existing per-partition restore scan, and overwrite, which does not restore existing files, is not affected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
When writing to a partition, PyPaimon restored the sequence numbers of its existing files but did not check the bucket count those files were written with. After the
bucketoption was changed, for example from 2 to 4, writes to an existing partition used the new bucket count. Primary keys already stored under the old layout were written again to other buckets, and the table returned duplicate keys.As in Java
AbstractFileStoreWrite, writing to a bucket whose existing files have a different total bucket number now fails with Java's message, which asks to rescale withINSERT OVERWRITEfirst. The check reuses the existing per-partition restore scan. Overwrite does not restore existing files, like Java'signorePreviousFiles, so it is not affected.As in Java, the check applies only to buckets that already have files. A write to a bucket that has no files yet, for example bucket 3 after changing the bucket count from 2 to 4, is not rejected. This case is only caught by the commit conflict check, which is not enabled for a default batch commit.
Tests
TableWriteTest#test_write_bucket_count: writing to a non-partitioned and a partitioned table after changing the bucket count fails with Java's message, and overwrite still rescales the data.TableWriteTest#test_postpone_checks_restored_bucket_count: the postpone fixed-bucket writer checks the bucket count of restored files and skips the check for overwrite.trusted_scan_metadata_test.py: the restore scan test sets the new writer fields.table_write_test.pyandtrusted_scan_metadata_test.py, and 1276 in the write and bucket tests. flake8 is clean.