Fix a 5.8.2 regression and the SafeDisk gaps found by testing downstream libraries

.pylintrc

@@ -341,7 +341,7 @@ indent-string='    '
 max-line-length=100
 
 # Maximum number of lines in a module.
-max-module-lines=4200
+max-module-lines=4300
 
 # Allow the body of a class to be on the same line as the declaration if body
 # contains single statement.

CHANGELOG.md

@@ -4,9 +4,30 @@ diskcache-community forked from DiskCache at upstream commit ebfa37c (3 March 20
 
 ## Unreleased
 
+### Security
+
+- `SafeDisk.with_types` refuses the classes of scipy, torch, bitstruct and pyahocorasick, by class, by module name and through `DISKCACHE_SAFE_MODULES`, as it refuses numpy: a crafted scipy sparse matrix, pyahocorasick automaton or bitstruct compiled format crashes the reader, and torch builds some objects by compiling code. Code that allowed them gets `ValueError` from `with_types` (PR #42).
+
 ### Fixes
 
+- `Cache.get` works again in subclasses that override `_con` to keep their rows in another database file, as subclasses written for 5.6.3 do. Since 5.8.2 its fast path took its cursor from a connection to `cache.db`, so every read raised `OperationalError: no such table: Cache` (PR #42).
+- `SafeDisk` stores the classes in its allowlist, such as `int` and `str`, so `memoize(typed=True)` works with it: those keys hold the classes of the arguments, which every `SafeDisk` reads but plain `SafeDisk` and `with_types()` with no arguments refused to write. `NoneType` and `bool` stay refused, because the reader refuses them, and a `defaultdict(bool)`, which was written but could not be read back, is now refused too (PR #42).
+- `SafeDisk.with_numpy()` stores `datetime64` and `timedelta64` arrays, which raised `ValueError` from `pickle.PickleBuffer`. Their scalars still raise `TypeError` (PR #42).
+- Every `SafeDisk` checks a key or value outside the standard-library types against the names its pickle looks up, as the disks from `with_types` already did. A plain `SafeDisk` therefore stores a dict subclass that pickles as a `dict`, such as DRF's `ReturnDict`, and reads back a `dict`, and an object that cannot be pickled raises `TypeError` on every `SafeDisk` (PR #42).
+- Refusal messages say what to do: a refused function no longer points to `with_types()`, which allows only classes; numpy objects point to `with_numpy()`; an array of objects mentions pandas DataFrames and what to store instead; and a class from a package `with_types` refuses says so (PR #42).
 - `incr` and `decr` on `Cache` and `FanoutCache` type-check with `default=None`, which makes a missing key raise `KeyError`: the stubs typed `default` as `int`, so type checkers rejected the call. `tox -e mypy` now also checks `tests/typecheck_stubs.py`, a file of calls that must type-check against the stubs (PR #41).
+- The `modules` argument of `SafeDisk.with_types` type-checks with a single string or any iterable of strings, as the code accepts (PR #42).
+
+### Docs
+
+- The CVE page and the migration guide say up front that `JSONDisk` does not protect reads, because it reads each row as its `mode` column says, and point to `SafeJSONDisk`. The migration guide explains that Dash's `DiskcacheManager` takes only an upstream cache and keeps its signing secret there, so its folder should be its own (PR #42).
+- The tutorial no longer says that `memoize(typed=True)` needs the default `Disk` (PR #42).
+
+### Known issues
+
+- A cache whose directory or `cache.db` the process cannot write opens read-only, as since 5.7.0, where 5.6.3 failed to open it. A cache owned by another account is then read, and a library's fallback for an unwritable cache is skipped. A later release will warn, refuse or offer an option that requires a writable cache.
+- The write check of `SafeDisk` scans the pickle in Python, which takes about ten times as long as pickling a very large value. Reads are not affected.
+- Objects that no allowlist can hold, such as those whose pickle calls a function, need the default `Disk`. Signed entries, planned for 5.9, are meant to cover them.
 
 ## 5.8.2 (2026-11-02)
 

SECURITY.md

@@ -20,7 +20,7 @@ Where this fork stands:
 - 5.8.0 is imported as `diskcache_community`, next to the upstream `diskcache` package. `SafeDisk.with_types` allows your own classes and the classes defined in named modules while everything else stays refused; numpy classes cannot be allowed, because numpy unpickling is not memory-safe. `configure()` and `DISKCACHE_DISK=safe` select `SafeDisk` for caches created without a disk, including those that libraries create through `diskcache_community`. `Disk.with_options(on_error='miss')` deletes entries that cannot be read and treats them as missing. The steps are in [Fix CVE-2025-69872 in diskcache](https://diskcache-community.readthedocs.io/en/latest/fix-cve-2025-69872.html).
 - `JSONDisk` writes no pickles but still reads one from a crafted row, as it does in 5.6.3. `SafeJSONDisk` reads and writes the same files and never reads a pickle; `SafeDisk` does not decode what `JSONDisk` wrote.
 - `SafeDisk.with_numpy()` returns a `SafeDisk` that also stores numpy arrays, in the `.npy` format rather than as pickles. numpy pickles stay refused, because unpickling an array is not memory-safe.
-- The default stays as it is. Existing caches hold objects outside any allowlist, such as instances of your own classes, Django responses and the keys of `memoize(typed=True)`, so `SafeDisk` remains an explicit choice.
+- The default stays as it is. Existing caches hold objects outside any allowlist, such as instances of your own classes and Django responses, so `SafeDisk` remains an explicit choice.
 
 ## Dependencies
 

diskcache_community/arrays.py

@@ -93,7 +93,9 @@ def _pack_array(numpy, array):
     The buffer is the array's own memory, so pickle copies it once, into
     the row, as a bytearray. An array that is not contiguous or not
     writable is copied first, since pickle stores a read-only buffer as
-    bytes, which the reader refuses.
+    bytes, which the reader refuses. The buffer is a flat view of bytes,
+    since numpy exports no buffer for ``datetime64`` and ``timedelta64``
+    data; the dtype is in the header.
 
     """
     # pylint: disable=no-member
@@ -114,11 +116,16 @@ def _pack_array(numpy, array):
         header = buffer.getvalue()
         _read_header(numpy, header, array.nbytes)
     except (ValueError, pickle.UnpicklingError) as error:
-        raise TypeError(
-            'SafeDisk cannot store an array of dtype %s' % (array.dtype,)
-        ) from error
-
-    return header, pickle.PickleBuffer(array)
+        message = 'SafeDisk cannot store an array of dtype %s' % (array.dtype,)
+        if array.dtype.hasobject:
+            message += (
+                '; a pandas DataFrame holds its labels and text columns in'
+                ' such arrays, so store its records or columns instead'
+            )
+        raise TypeError(message) from error
+
+    data = array.reshape(-1, order='A').view(numpy.uint8)
+    return header, pickle.PickleBuffer(data)
 
 
 def _unpack_array(header, data):

diskcache_community/core.py

@@ -557,9 +557,11 @@ def fetch(self, mode, filename, value, read):
     'uuid': frozenset({'UUID'}),
 }
 
-# Python types that SafeDisk accepts for writing: exactly the types that the
-# names above read back. Subclasses are rejected because pickle records the
-# subclass, which the reader would refuse.
+# Python types that SafeDisk accepts for writing without pickling them first:
+# exactly the types that the names above read back. Anything else is pickled
+# and accepted only if the reader resolves every name the pickle looks up, so
+# a subclass is refused, because pickle records the subclass, unless it
+# pickles as one of these types.
 
 SAFE_TYPES = frozenset(
     {
@@ -593,15 +595,35 @@ def fetch(self, mode, filename, value, read):
     }
 )
 
+# Classes that SafeDisk accepts as objects, such as the classes that
+# memoize(typed=True) puts into keys and the factory of a defaultdict: the
+# classes in SAFE_TYPES that pickle as a name the reader resolves. NoneType
+# pickles as a call to builtins.type, and bool as builtins.bool, which the
+# reader refuses.
+
+_SAFE_CLASSES = SAFE_TYPES - {type(None), bool}
+
 _SAFE_MAPPINGS = (dict, co.Counter, co.OrderedDict, co.defaultdict)
 _SAFE_SEQUENCES = (list, tuple, set, frozenset, co.deque)
 _SAFE_TZ_TYPES = (type(None), dt.timezone)
 
-# Packages whose classes SafeDisk.with_types refuses to allow. numpy restores
-# the internal flags of a dtype from its pickle without checking them, so a
-# crafted pickle can make an array read raw bytes as object pointers.
+# Packages whose classes SafeDisk.with_types refuses to allow, because a
+# crafted pickle of one can corrupt memory or run code. numpy restores the
+# internal flags of a dtype from its pickle without checking them, so a
+# crafted pickle can make an array read raw bytes as object pointers. scipy's
+# sparse matrices and pyahocorasick's automatons keep index data that their
+# compiled code trusts, bitstruct's compiled formats crash on state they did
+# not build, and torch builds some objects by compiling or running code.
+
+_UNSAFE_MODULES = ('numpy', 'ahocorasick', 'bitstruct', 'scipy', 'torch')
+
+
+def _unsafe_reason(package):
+    """Return why with_types refuses classes of the unsafe `package`."""
+    if package == 'numpy':
+        return '; use with_numpy() to store arrays'
+    return ': a crafted pickle of a %s object is not safe to read' % package
 
-_UNSAFE_MODULES = ('numpy',)
 
 _PICKLE_STRING_OPS = frozenset(
     {'BINUNICODE', 'BINUNICODE8', 'SHORT_BINUNICODE', 'UNICODE'}
@@ -838,10 +860,12 @@ class SafeDisk(Disk):
     ``FanoutCache(directory, disk=SafeDisk)``, ``Deque(disk=SafeDisk)``,
     ``Index(disk=SafeDisk)`` or the ``DISK`` option of the Django backend.
 
-    Objects outside the allowlist, such as instances of your own classes,
-    Django model instances and responses, and the ``type`` objects that
-    ``memoize(typed=True)`` puts into keys, need either the default
-    :class:`Disk` or a disk from :meth:`with_types` that allows them.
+    Objects outside the allowlist, such as instances of your own classes and
+    Django model instances and responses, need either the default
+    :class:`Disk` or a disk from :meth:`with_types` that allows them. The
+    allowed classes themselves can be stored too, except ``NoneType`` and
+    ``bool``, so ``memoize(typed=True)`` works unless an argument is
+    ``None``, ``True`` or ``False``.
 
     """
 
@@ -868,7 +892,9 @@ def with_types(cls, *types, modules=()):
         do nothing beyond storing their fields, such as dataclasses, named
         tuples, enums and pydantic models. Reading looks classes up only in
         modules that are already imported, so import them before reading.
-        numpy classes are refused; use :meth:`with_numpy` to store arrays.
+        Classes of numpy, scipy, torch, bitstruct and pyahocorasick are
+        refused, because a crafted pickle of one can corrupt memory or run
+        code; use :meth:`with_numpy` to store arrays.
 
         >>> import collections
         >>> Point = collections.namedtuple('Point', 'x y')
@@ -882,17 +908,18 @@ def with_types(cls, *types, modules=()):
         :return: SafeDisk subclass to pass as ``disk=``
         :raises TypeError: if an entry of `types` is not a class
         :raises ValueError: if an entry of `modules` is not a module name, or
-            a type or module is numpy (arrays belong to :meth:`with_numpy`)
+            a type or module belongs to one of those packages
 
         """
         for entry in types:
             if not isinstance(entry, type):
                 raise TypeError('with_types takes classes, not %r' % (entry,))
-            if _in_modules(entry.__module__, _UNSAFE_MODULES):
-                raise ValueError(
-                    'with_types refuses the class %r; use with_numpy() to'
-                    ' store arrays' % (entry,)
-                )
+            for unsafe in _UNSAFE_MODULES:
+                if _in_modules(entry.__module__, (unsafe,)):
+                    raise ValueError(
+                        'with_types refuses the class %r%s'
+                        % (entry, _unsafe_reason(unsafe))
+                    )
 
         if isinstance(modules, str):
             modules = (modules,)
@@ -903,15 +930,14 @@ def with_types(cls, *types, modules=()):
                 part.isidentifier() for part in prefix.split('.')
             ):
                 raise ValueError('%r is not a module name' % (entry,))
-            if any(
-                _in_modules(prefix, (unsafe,))
-                or _in_modules(unsafe, (prefix,))
-                for unsafe in _UNSAFE_MODULES
-            ):
-                raise ValueError(
-                    'with_types refuses the module %r; use with_numpy() to'
-                    ' store arrays' % (entry,)
-                )
+            for unsafe in _UNSAFE_MODULES:
+                if _in_modules(prefix, (unsafe,)) or _in_modules(
+                    unsafe, (prefix,)
+                ):
+                    raise ValueError(
+                        'with_types refuses the module %r%s'
+                        % (entry, _unsafe_reason(unsafe))
+                    )
             prefixes.append(prefix)
 
         return _derive_disk(
@@ -933,7 +959,7 @@ def with_numpy(cls):
         value except in keys, and checks each header before it reads an
         array back. (Plain :class:`SafeDisk` refuses numpy pickles, because
         unpickling an array is not memory-safe.) numpy scalars keep their
-        type.
+        type, and arrays of ``datetime64`` and ``timedelta64`` are stored too.
         Object arrays, ``ndarray`` subclasses and other numpy scalars raise
         :exc:`TypeError` when written. It combines with :meth:`with_types`
         and :meth:`with_options` in any order.
@@ -966,22 +992,29 @@ def _unpickler(self, file):
             return arrays.unpickler(self, file)
         return self._plain_unpickler(file)
 
-    def _extended(self):
-        return bool(self.safe_types or self.safe_modules)
-
     def _check_object(self, obj, what):
-        """Raise TypeError unless the reader would accept `obj` back."""
+        """Raise TypeError unless the reader would accept `obj` back.
+
+        Outside SAFE_TYPES, an object is accepted when its pickle looks up
+        only names the reader resolves, such as a dict subclass that pickles
+        as a plain dict, which is then what the reader builds.
+
+        """
         try:
             self._check(obj, what)
         except TypeError:
-            if not self._extended():
-                raise
+            pass
         else:
             return
 
-        self._check_pickle(
-            pickle.dumps(obj, protocol=self.pickle_protocol), obj, what
-        )
+        try:
+            data = pickle.dumps(obj, protocol=self.pickle_protocol)
+        except (pickle.PicklingError, AttributeError) as error:
+            raise TypeError(
+                'SafeDisk cannot store %s: %s' % (what, error)
+            ) from error
+
+        self._check_pickle(data, obj, what)
 
     def _check_pickle(self, data, obj, what):
         """Raise TypeError unless the reader would accept the pickle `data`."""
@@ -1002,17 +1035,38 @@ def _check_pickle(self, data, obj, what):
             except pickle.UnpicklingError:
                 raise TypeError(
                     'SafeDisk cannot store %s of type %s.%s: it pickles'
-                    ' %s.%s, which the allowlist refuses; allow it with'
-                    ' SafeDisk.with_types()'
+                    ' %s.%s, which the allowlist refuses%s'
                     % (
                         what,
                         type(obj).__module__,
                         type(obj).__qualname__,
                         module,
                         name,
+                        self._refusal_hint(module, name),
                     )
                 ) from None
 
+    def _refusal_hint(self, module, name):
+        """Return how to store what pickles the refused `module`.`name`."""
+        if _in_modules(module, ('numpy',)):
+            if self.safe_numpy:
+                return '; with_numpy() stores only arrays and numpy scalars'
+            return '; store arrays with SafeDisk.with_numpy()'
+
+        for unsafe in _UNSAFE_MODULES:
+            if _in_modules(module, (unsafe,)):
+                return '; with_types() refuses %s classes' % unsafe
+
+        obj = sys.modules.get(module)
+        for part in name.split('.'):
+            obj = getattr(obj, part, None)
+
+        if not isinstance(obj, type):
+            return '; it is not a class, and with_types() allows only classes'
+        if module in ('builtins', '__builtin__'):
+            return ''
+        return '; allow it with SafeDisk.with_types()'
+
     def _check(self, obj, what):
         """Raise TypeError unless every object in `obj` is in SAFE_TYPES."""
         seen = set()
@@ -1023,6 +1077,8 @@ def _check(self, obj, what):
             type_item = type(item)
 
             if type_item not in SAFE_TYPES:
+                if isinstance(item, type) and item in _SAFE_CLASSES:
+                    continue
                 raise TypeError(
                     'SafeDisk cannot store %s of type %s.%s'
                     % (what, type_item.__module__, type_item.__qualname__)
@@ -1034,7 +1090,7 @@ def _check(self, obj, what):
                 seen.add(id(item))
                 if type_item is co.defaultdict:
                     factory = item.default_factory
-                    if factory is not None and factory not in SAFE_TYPES:
+                    if factory is not None and factory not in _SAFE_CLASSES:
                         raise TypeError(
                             'SafeDisk cannot store %s with defaultdict'
                             ' factory %r' % (what, factory)
@@ -1996,8 +2052,12 @@ def _cur(self):
         cur = getattr(self._local, 'cur', None)
 
         if cur is None:
-            self._connect()
-            cur = self._local.cur
+            # Through _con, so a subclass that overrides it, to open another
+            # database file say, gets a cursor on that connection.
+            con = self._con
+            cur = getattr(self._local, 'cur', None)
+            if cur is None or cur.connection is not con:
+                cur = self._local.cur = con.cursor()
 
         return cur
 

diskcache_community/core.pyi

@@ -8,6 +8,7 @@ from typing import (
     Final,
     Generator,
     Generic,
+    Iterable,
     Literal,
     NoReturn,
     overload,
@@ -192,7 +193,7 @@ class SafeDisk(Disk):
     safe_numpy: bool
     @classmethod
     def with_types(
-        cls, *types: type, modules: tuple[str, ...] = ...
+        cls, *types: type, modules: str | Iterable[str] = ...
     ) -> type[Self]: ...
     @classmethod
     def with_numpy(cls) -> type[Self]: ...

docs/api.rst

@@ -154,8 +154,9 @@ Read the :ref:`SafeDisk tutorial <tutorial-safedisk>` for details.
 .. data:: diskcache_community.core._UNSAFE_MODULES
 
    Packages whose classes :meth:`SafeDisk.with_types
-   <diskcache_community.SafeDisk.with_types>` refuses to allow, because
-   unpickling them is not memory-safe.
+   <diskcache_community.SafeDisk.with_types>` refuses to allow, because a
+   crafted pickle of one can corrupt memory or run code: numpy, scipy, torch,
+   bitstruct and pyahocorasick.
 
 SafeJSONDisk
 ------------