Repository navigation
[python] Include block size in the local disk cache key - #10866
Open
Stephen0421 wants to merge 1 commit into
Open
Stephen0421 wants to merge 1 commit into
Stephen0421 wants to merge 1 commit into
Conversation
The local disk cache keys blocks by path and index only, but block contents are sized by local-cache.block-size. Because the cache directory outlives the process, reopening it after retuning block-size matches blocks that were split at the previous size, so reads return bytes from the wrong file offset. Java's LocalDiskCacheManager already includes the block size in its key; this aligns PyPaimon with it. Stale blocks are no longer matched after the change, so a retuned cache starts cold and the existing LRU reclaims the old entries. Add regression coverage at the cache manager, input stream and end-to-end Parquet read levels. The existing disk cache tests all pin block_size=64, which is why this went unnoticed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
The local disk cache keys blocks by path and index only:
But a block's contents are sized by
local-cache.block-size. Since the cache directory outlives the process, reopening it after retuning block-size matches blocks that were split at the previous size, and reads return bytes from the wrong file offset.This is reachable through ordinary configuration tuning: the local cache docs show
local-cache.dirandlocal-cache.block-sizeset together in the same example, the cache directory is never self-cleaned, and the default whitelist (meta,global-index) covers manifest, snapshot and schema files that feed query planning.Reading a real Parquet table after changing only
local-cache.block-size, reusing the same cache directory:At the stream level the corruption is silent rather than raising: a
read_at(512, 512)can return bytes taken from offset 40, and in one direction it returns more bytes than requested, breaking theread_atcontract.Java already does this correctly —
LocalDiskCacheManager.cachePathbuildsblockSize + ":" + filePath + ":" + blockIndex. This change aligns PyPaimon with the Java implementation:No migration is needed. Stale blocks simply stop matching, so a retuned cache starts cold instead of returning wrong data, and the existing LRU reclaims the old entries.
Note this does not address cache isolation between catalogs sharing one cache directory. Java scopes keys with a per-catalog
cacheNamespace(CachingFileIO), which PyPaimon also lacks; that touches cache manager lifecycle and is better handled separately.Tests
Three regression tests, at the cache manager, input stream and end-to-end Parquet read levels:
LocalDiskCacheManagerTest.test_restart_with_changed_block_size_does_not_reuse_blocksCachingInputStreamTest.test_reads_stay_correct_after_block_size_changetest_block_size_change_reuses_disk_cache_dir_safely(parametrized both directions)All three fail without the fix and pass with it.
The existing disk cache tests all pin
block_size=64and never reuse a directory across sizes, which is why this went unnoticed — 76 cache tests passed on the unfixed code.