Skip to content

Fix residual leak in downloadDirectory by skipping file downloads after cancellation.#6983

Open
RanVaknin wants to merge 2 commits into
masterfrom
rvaknin/fix-directory-download-recreate-after-cancel
Open

Fix residual leak in downloadDirectory by skipping file downloads after cancellation.#6983
RanVaknin wants to merge 2 commits into
masterfrom
rvaknin/fix-directory-download-recreate-after-cancel

Conversation

@RanVaknin
Copy link
Copy Markdown
Collaborator

In #6875, we addressed canceling in flight transfers when a directory download is cancelled. That fix reduced the leaked
files per orphaned directory from thousands to single digits in stress test. However there was a residual leak for downloads that were already dispatched to doDownloadSingleFile (the method that creates the destination directory and initiates the file download) before the cancellation signal propagated.

When cancellation occurs, thread A calls subscription.cancel() to stop new items from being delivered. Meanwhile, thread B has already picked up an item from onNext and is inside doDownloadSingleFile. Since it entered before thread A's cancellation took effect, it creates the destination directory and starts a download into it.

This PR adds a isDone() guard at the top of doDownloadSingleFile so that any thread that entered the method before
cancellation took effect will return early before touching the filesystem.

The two tests:

  1. Updated downloadDirectory_cancel_shouldCancelAllFutures - now waits for both downloads to start before cancelling. The original test called cancel() immediately after downloadDirectory(), implicitly assuming all downloads would always start regardless of cancellation state. With the new isDone() guard, that assumption is not always true since cancel can now prevent downloads from starting. The test's intent is unchanged (in flight futures get cancelled), we just ensure both
    futures are actually in flight before cancelling. (Sanity check tested this with a repeated test suite 1000 times to make sure its deterministic)

  2. Added downloadDirectory_cancelledFuture_shouldNotCreateDirectories - The test cancels the directory download future, then delivers an S3 object through the publisher. Asserts that the destination subdirectory was never created on the filesystem. Without the fix, doDownloadSingleFile would call createParentDirectoriesIfNeeded() and create the directory despite the cancellation.

RanVaknin added 2 commits May 19, 2026 21:04
@RanVaknin RanVaknin requested a review from a team as a code owner May 20, 2026 05:19
@RanVaknin RanVaknin changed the title Rvaknin/fix directory download recreate after cancel Fix residual leak in downloadDirectory by skipping file downloads after cancellation. May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant