API, Core: Add CloseableIterable.filter with keep/skip callback for hetergeneous element#17118
Open
dramaticlly wants to merge 1 commit into
Open
API, Core: Add CloseableIterable.filter with keep/skip callback for hetergeneous element#17118dramaticlly wants to merge 1 commit into
dramaticlly wants to merge 1 commit into
Conversation
Add a new filter overload that invokes a Consumer on each accepted and rejected element, so callers can attribute both survivors and skipped entries to metrics in a single pass. The counter-based filter and count helpers delegate to this new primitive. Migrate the manifest-level scan pipelines in DeleteFileIndex, PositionDeletesTable, and ManifestGroup — each previously chained a skip-counting filter with a scan-counting count call — to a single filter call with both onKeep and onSkip observers. File-level filters in ManifestReader and ManifestGroup remain on the counter-based overload since ScanMetrics only exposes a skipped counter at that layer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
FYI @stevenzwu @anoopj @nastra if you want to take a look. |
wombatu-kun
reviewed
Jul 7, 2026
| iterable.close(); | ||
| } | ||
| }; | ||
| return filter(iterable, item -> true, item -> counter.increment(), item -> {}); |
Contributor
There was a problem hiding this comment.
count now delegates to the FilterIterator-based filter, which increments during hasNext()/advance() rather than on next() like CloseableIterator.count. The javadoc still says the counter is incremented on each Iterator#next() call, so peeking via hasNext() without a following next() over-counts. Update the javadoc, or keep count on CloseableIterator.count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add a new filter overload that invokes a Consumer on each accepted and rejected element, so callers can attribute both kept and skipped entries to metrics in a single pass.
Why
This help provide a path for hetergeneous entry of
TrackedFilein v4, where a given entry can carry contentType of DATA, EQUALITY_DELETES, DATA_MANIFEST, or DELETE_MANIFEST within a single root manifest. Callback is better suited than a dedicated counter to allow more flexibility. Plus, importorg.apache.iceberg.metrics.Counterinorg.apache.iceberg.io. CloseableIterableis not ideal for a general purpose iteration utility.Existing caller
Migrate the manifest-level scan pipelines in
DeleteFileIndex,PositionDeletesTable, andManifestGroupwhere each previously chained a skip-counting filter with a scan-counting count call. Now replaced with a single filter call with both onKeep and onSkip callback. File-level filters in ManifestReader and ManifestGroup remain on the counter-based overload since ScanMetrics only exposes a skipped counter at that layer.