Skip to content

Fix FileNotFoundError when clear_cache() races with separate_files=True - #400

Open
lode-braced wants to merge 1 commit into
python-cachier:masterfrom
lode-braced:bugfix/concurrent-cache-clear
Open

lode-braced wants to merge 1 commit into
python-cachier:masterfrom
lode-braced:bugfix/concurrent-cache-clear

Conversation

@lode-braced

Copy link
Copy Markdown

Closes #399

Problem

With the pickle backend and separate_files=True, _PickleCore._clear_all_cache_files() lists the cache directory and then removes each matching file. If two clear_cache() calls run concurrently (same wrapper from two threads, or separate wrappers/processes sharing a cache_dir), both can list the same file. One removes it; the other's os.remove() then raises FileNotFoundError, aborting that clear before it processes the remaining files.

The existing retry loop only handles PermissionError (Windows file locking), so the missing-file case propagates.

Fix

Treat FileNotFoundError from the per-file os.remove() as "already cleared" and continue with the next file. The handler wraps only the individual deletion. PermissionError retry behavior and propagation of other errors are unchanged. No new locking is introduced.

This does not address in-flight computations re-creating entries after a clear; that is a separate invalidation concern.

Tests

Adds test_concurrent_clear_cache_separate_files, parametrized for a shared wrapper and for separate wrappers on the same directory. It uses real cache files and real deletions, and coordinates scheduling deterministically: both callers list the directory before either deletes, and the second deletion of each file is attempted only after the first completes. It asserts that both clears succeed, that all matching files are removed, and that unrelated files are untouched. The test fails on the current master and passes with this change.

… files

Two concurrent clear_cache() calls with separate_files=True could both
list the same cache file; the slower caller then failed with
FileNotFoundError. Treat a file that is already gone as cleared and
continue with the remaining files.

Closes python-cachier#399
@lode-braced
lode-braced requested a review from shaypal5 as a code owner October 7, 2026 11:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent clear_cache() with separate_files=True raises FileNotFoundError

1 participant