fix: close the outer file stream of inline reads - #19936
Merged
Merged
Conversation
InLineFsDataInputStream wraps the outer file's FSDataInputStream in an offset-adjusting InputStream and passes that to its super constructor. Neither class overrides close(), so closing an inline stream ends at InputStream.close(), a no-op, and the outer file's stream stays open until garbage collection. Every inline HFile read (metadata table log blocks) leaked a file handle this way. Spark's DebugFilesystem, which tracks the streams it hands out, reports it in TestStreamingSource as "possibly leaked file streams" whenever it is the cached local filesystem of the JVM. close() now closes the wrapped stream and then the outer one. A unit test checks the outer stream is closed exactly once.
3 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19936 +/- ##
============================================
- Coverage 80.23% 80.22% -0.01%
- Complexity 34747 34748 +1
============================================
Files 2546 2546
Lines 142608 142611 +3
Branches 17362 17362
============================================
- Hits 114419 114411 -8
- Misses 20277 20289 +12
+ Partials 7912 7911 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
hudi-agent
reviewed
Sep 13, 2026
hudi-agent
left a comment
hudi-agent
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the contribution! This PR overrides close() in InLineFsDataInputStream so that closing an inline stream also closes the wrapped outer FSDataInputStream, fixing a file-handle leak on every inline HFile read from a log block. I verified that the only production caller (InLineFileSystem.open) creates a dedicated outer stream per inline stream, so the ownership assumption behind the override is sound, and the try/finally ensures the outer handle is released even if the wrapper's close throws. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.
cc @yihua
Collaborator
danny0405
approved these changes
Sep 14, 2026
voonhous
added a commit
to voonhous/hudi
that referenced
this pull request
Sep 14, 2026
InLineFsDataInputStream wraps the outer file's FSDataInputStream in an offset-adjusting InputStream and passes that to its super constructor. Neither class overrides close(), so closing an inline stream ends at InputStream.close(), a no-op, and the outer file's stream stays open until garbage collection. Every inline HFile read (metadata table log blocks) leaked a file handle this way. Spark's DebugFilesystem, which tracks the streams it hands out, reports it in TestStreamingSource as "possibly leaked file streams" whenever it is the cached local filesystem of the JVM. close() now closes the wrapped stream and then the outer one. A unit test checks the outer stream is closed exactly once. (cherry picked from commit 1d74ee5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
InLineFsDataInputStreamreads an HFile embedded in a log file. It wraps the outer file'sFSDataInputStreamin an offset-adjustingInputStreamand passes that to its super constructor. Neither class overridesclose(), so closing an inline stream ends atInputStream.close(), a no-op, and the outer file's stream stays open until garbage collection. Every inline HFile read of a metadata table log block leaks a file handle this way.Found by TestStreamingSource on #19932, where Spark's
DebugFilesystemreported the outer streams as leaked once it was the JVM's cached local filesystem. On master the check never sees Hudi's reads, because the SQL suites cache a plain LocalFileSystem first.Summary and Changelog
InLineFsDataInputStream.close()closes the wrapped stream and then the outer stream.FSDataInputStreamand asserts it is closed exactly once.Impact
Metadata table reads through the inline filesystem release their file handles when the reader closes instead of at GC.
Risk Level
low. One
closeoverride; the outer stream is owned by this object.Documentation Update
none
Contributor's checklist