Repository navigation
Fix a 5.8.2 regression and the SafeDisk gaps found by testing downstream libraries - #42
Merged
Merged
Conversation
…t override _con to open another database work again.
…e) keys pass the write check, and refuse defaultdict(bool), which the reader refuses.
…ich raised ValueError from pickle.PickleBuffer.
…with_types disks do, and make refusal messages point to with_numpy() or say when no allowlist can help.
…erable of strings, as the code accepts.
…with_types, as numpy is, because crafted pickles of them crash the reader or run code.
…t protect reads, and explain why Dash needs a cache folder of its own.
Documentation build overview
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes found by running diskcache-community 5.8.2 against 42 downstream libraries, for a 5.8.3 patch release. There is one commit per fix, and the Unreleased section of
CHANGELOG.mdhas the user-facing wording.Regression
Cache.gettook its cursor from_connect(), which always openscache.db. Subclasses written for 5.6.3 that override_conto keep their rows in another file, as dwave-cloud-client's credentials store does, gotOperationalError: no such table: Cacheon every read._curnow takes its cursor fromself._con. The new test fails on master with that error.SafeDisk
memoize(typed=True)keys:SafeDiskstores the classes on its allowlist, such asintandstr, which everySafeDiskalready read back.NoneTypeandboolstay refused, because the reader refuses them.defaultdict(bool), which was written but could not be read back, is now refused when written.with_numpy()storesdatetime64andtimedelta64arrays. The reader already accepted them and onlypickle.PickleBufferrefused them, so the writer now passes a flat byte view of the same memory. Rows for every other dtype are byte-for-byte unchanged.SafeDiskchecks a value outside the standard-library types against the names its pickle looks up, aswith_typesdisks already did. A dict subclass that pickles as adict, such as DRF'sReturnDict, is stored and reads back as adict.with_types(), numpy objects point towith_numpy(), and an array of objects mentions pandas DataFrames.with_typesrefuses scipy, torch, bitstruct and pyahocorasick classes, as it refuses numpy: crafted pickles of them crash the reader or run code.with_typesaccepts a string or any iterable of strings formodules, as the code does.Docs
JSONDiskdoes not protect reads, and point toSafeJSONDisk.DiskcacheManagertakes only an upstream cache and keeps its signing secret in it, so its folder should be its own.memoize(typed=True)needs the defaultDisk.Behaviour changes to call out in the release
with_typeswith scipy, torch, bitstruct or pyahocorasick, also throughDISKCACHE_SAFE_MODULES, now raisesValueError.with_typesdisks, an object that cannot be pickled raisesTypeErrorinstead ofPicklingError..pylintrcraisesmax-module-linesfrom 4200 to 4300, sincecore.pyis now 4234 lines.Not in this PR
The changelog lists these as known issues: a cache owned by another account opens read-only, the write check is slow on very large values, and signed entries are planned for 5.9.
Testing
tests/typecheck_stubs.py), pylint, doc8 and the Sphinx build are clean.test_pickle_globals_reads_past_large_strings_and_bufferstests code this PR does not change. It failed once in five local full runs and passed 10 out of 10 runs on its own. It measures peak memory and looks flaky under load.