s3 killer: paginate to find complete list of files to delete#19009
Open
abannon wants to merge 2 commits intoapache:masterfrom
Open
s3 killer: paginate to find complete list of files to delete#19009abannon wants to merge 2 commits intoapache:masterfrom
abannon wants to merge 2 commits intoapache:masterfrom
Conversation
kfaraz
reviewed
Feb 11, 2026
...nsions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java
Outdated
Show resolved
Hide resolved
kfaraz
reviewed
Feb 11, 2026
...nsions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java
Outdated
Show resolved
Hide resolved
abannon
force-pushed
the
abannon/kill-1001
branch
from
bb4d3df to
b38b929
Compare
abannon
force-pushed
the
abannon/kill-1001
branch
from
b38b929 to
16e4e66
Compare
kfaraz
reviewed
Mar 3, 2026
| } | ||
| catch (Exception e) { | ||
| log.error("Error occurred while deleting segment files from s3. Error: %s", e.getMessage()); | ||
| return true; |
Contributor
There was a problem hiding this comment.
Please invert these return values as it is more natural to return true for success scenarios and false for failures.
| ); | ||
| } | ||
| catch (Exception e) { | ||
| log.error("Error occurred while deleting segment files from s3. Error: %s", e.getMessage()); |
Contributor
There was a problem hiding this comment.
This would log the stack trace as well.
Suggested change
| log.error("Error occurred while deleting segment files from s3. Error: %s", e.getMessage()); | |
| log.error(e, "Error occurred while deleting segment files from s3."); |
| ); | ||
| } | ||
| catch (Exception e) { | ||
| log.error("Error occurred while deleting segment files from s3. Error: %s", e.getMessage()); |
Contributor
There was a problem hiding this comment.
Suggested change
| log.error("Error occurred while deleting segment files from s3. Error: %s", e.getMessage()); | |
| log.error(e, "Error occurred while deleting segment files from s3."); |
Comment on lines
+106
to
+107
| boolean hadException = deleteSegmentFilesFromS3(s3Bucket, path); | ||
| if (hadException) { |
Contributor
There was a problem hiding this comment.
Suggested change
| boolean hadException = deleteSegmentFilesFromS3(s3Bucket, path); | |
| if (hadException) { | |
| boolean success = deleteSegmentFilesFromS3(s3Bucket, path); | |
| if (!success) { |
| for (S3ObjectSummary objectSummary : list.getObjectSummaries()) { | ||
| log.info("Removing index file[s3://%s/%s] from s3!", s3Bucket, objectSummary.getKey()); | ||
| s3Client.deleteObject(s3Bucket, objectSummary.getKey()); | ||
| try { |
Contributor
There was a problem hiding this comment.
Nit: Maybe just call the new method deleteSegmentFilesFromS3 here, since the logic seems to be the same.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
For segments stored in s3, the segment killer looks for files to remove within the segment prefix by calling listObjectsV2, which returns at most 1000 items. In the case where there are more than 1000 files per segment prefix, the segment killer did not find and delete all of the files. The fix is to add pagination by reusing existing 'deleteSegmentFilesFromS3' function.
Release note
Fixed: Kill task for s3 will remove all segment files, instead of up to 1000 files.
Key changed/added classes in this PR
This PR has: