[Observability] Log interrupted processing tasks on unexpected worker…#9261
Draft
prince8273 wants to merge 2 commits into
Draft
[Observability] Log interrupted processing tasks on unexpected worker…#9261prince8273 wants to merge 2 commits into
prince8273 wants to merge 2 commits into
Conversation
… death When a worker drops off the cluster unexpectedly (e.g., due to an OOM kill), the scheduler tracks the processing_keys but previously did not log them to the console. This change surfaces exactly which tasks were interrupted, significantly improving debugging provenance for cluster hangs and memory crashes.
Contributor
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 31 files ±0 31 suites ±0 11h 17m 10s ⏱️ - 13m 52s For more details on these failures, see this check. Results for commit 8213721. ± Comparison against base commit cf508b9. This pull request removes 1 and adds 1 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Author
|
Hi @fjetter ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #9263
This PR introduces targeted observability for interrupted processing tasks during unexpected worker disconnections (e.g., OOM kills or hardware failures).
Previously, when
remove_worker()was triggered withexpected=False, the scheduler correctly tracked theprocessing_keysbut failed to emit them to the logs. This created an observability gap where cluster operators could see a worker disconnect, but could not identify which specific task caused the failure without waiting for the task's retry limit to be exhausted.This change adds a surgical
logger.warningto surface the interruptedprocessing_keysprecisely at the time of the worker's death, significantly improving failure provenance and debugging workflows for large clusters.Implementation Details
distributed/scheduler.py::remove_workerto printprocessing_keyswhen a worker drops unexpectedly (expected=False).recompute_keysandlost_keyslogging blocks.test_log_remove_workertodistributed/tests/test_scheduler.pyto correctly test scheduler-level logs and expect the new observability warning during ungraceful shutdowns.