Merge branch 'main' into profiling-windows-venv

Author: edvilme

.github/CODEOWNERS

@@ -63,7 +63,7 @@
 .azure-pipelines/             @AA-Turner
 
 # GitHub & related scripts
-.github/                                    @ezio-melotti @hugovk @AA-Turner @webknjaz
+.github/                                    @ezio-melotti @hugovk @AA-Turner @webknjaz @itamaro
 Tools/build/compute-changes.py              @AA-Turner @hugovk @webknjaz
 Lib/test/test_tools/test_compute_changes.py @AA-Turner @hugovk @webknjaz
 Tools/build/verify_ensurepip_wheels.py      @AA-Turner @pfmoore @pradyunsg
@@ -73,19 +73,19 @@ Tools/build/verify_ensurepip_wheels.py      @AA-Turner @pfmoore @pradyunsg
 .ruff.toml                    @hugovk @AlexWaygood @AA-Turner
 
 # Patchcheck
-Tools/patchcheck/             @AA-Turner
+Tools/patchcheck/             @AA-Turner @itamaro
 
 
 # ----------------------------------------------------------------------------
 # Build System
 # ----------------------------------------------------------------------------
 
 # Autotools
-configure*                       @erlend-aasland @corona10 @AA-Turner @emmatyping
-Makefile.pre.in                  @erlend-aasland @AA-Turner @emmatyping
-Modules/makesetup                @erlend-aasland @AA-Turner @emmatyping
-Modules/Setup*                   @erlend-aasland @AA-Turner @emmatyping
-Tools/build/regen-configure.sh   @AA-Turner
+configure*                       @erlend-aasland @corona10 @AA-Turner @emmatyping @itamaro
+Makefile.pre.in                  @erlend-aasland @AA-Turner @emmatyping @itamaro
+Modules/makesetup                @erlend-aasland @AA-Turner @emmatyping @itamaro
+Modules/Setup*                   @erlend-aasland @AA-Turner @emmatyping @itamaro
+Tools/build/regen-configure.sh   @AA-Turner @itamaro
 
 # generate-build-details
 Tools/build/generate-build-details.py   @FFY00

Doc/c-api/threads.rst

@@ -736,10 +736,10 @@ Low-level APIs
 .. c:function:: PyObject* PyThreadState_GetDict()
 
    Return a dictionary in which extensions can store thread-specific state
-   information.  Each extension should use a unique key to use to store state in
+   information.  Each extension should use a unique key to store a state in
    the dictionary.  It is okay to call this function when no :term:`thread state`
    is :term:`attached <attached thread state>`. If this function returns
-   ``NULL``, no exception has been raised and the caller should assume no
+   ``NULL`` and no exception has been raised, then the caller should assume no
    thread state is attached.
 
 

Doc/conf.py

@@ -349,8 +349,6 @@
   \sphinxstrong{Python Software Foundation}\\
   Email: \sphinxemail{docs@python.org}
 }
-\let\Verbatim=\OriginalVerbatim
-\let\endVerbatim=\endOriginalVerbatim
 \setcounter{tocdepth}{2}
 ''',
     # The paper size ('letterpaper' or 'a4paper').

Doc/deprecations/pending-removal-in-3.21.rst

@@ -1,6 +1,15 @@
 Pending removal in Python 3.21
 ------------------------------
 
+* :mod:`abc`
+
+   * Soft-deprecated since Python 3.3 :class:`abc.abstractclassmethod`,
+     :class:`abc.abstractstaticmethod`, and :class:`abc.abstractproperty`
+     now raise a :exc:`DeprecationWarning`.
+     These classes will be removed in Python 3.21, instead
+     use :func:`abc.abstractmethod` with :func:`classmethod`,
+     :func:`staticmethod`, and :class:`property` respectively.
+
 * :mod:`ast`:
 
   * Classes ``slice``, ``Index``, ``ExtSlice``, ``Suite``, ``Param``,

Doc/howto/free-threading-python.rst

@@ -165,3 +165,132 @@ to false.  If the flag is true then the :class:`warnings.catch_warnings`
 context manager uses a context variable for warning filters.  If the flag is
 false then :class:`~warnings.catch_warnings` modifies the global filters list,
 which is not thread-safe.  See the :mod:`warnings` module for more details.
+
+
+Increased memory usage
+----------------------
+
+The free-threaded build will typically use more memory compared to the default
+build.  There are multiple reasons for this, mostly due to design decisions.
+
+
+All interned strings are immortal
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For modern Python versions (since version 2.3), interning a string (e.g. with
+:func:`sys.intern`) does not cause it to become immortal.  Instead, if the last
+reference to that string disappears, it will be removed from the interned
+string table.  This is not the case for the free-threaded build and any interned
+string will become immortal, surviving until interpreter shutdown.
+
+
+Non-GC objects have a larger object header
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The free-threaded build uses a different :c:type:`PyObject` structure.  Instead
+of having the GC related information allocated before the :c:type:`PyObject`
+structure, like in the default build, the GC related info is part of the normal
+object header.  For example, on the AMD64 platform, ``None`` uses 32 bytes on
+the free-threaded build vs 16 bytes for the default build.  GC objects (such as
+dicts and lists) are the same size for both builds since the free-threaded
+build does not use additional space for the GC info.
+
+
+QSBR can delay freeing of memory
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to safely implement lock-free data structures, a safe memory
+reclamation (SMR) scheme is used, known as quiescent state-based reclamation
+(QSBR).  This means that the memory backing data structures allowing lock-free
+access will use QSBR, which defers the free operation, rather than immediately
+freeing the memory.  Two examples of these data structures are the list object
+and the dictionary keys object.  See ``InternalDocs/qsbr.md`` in the CPython
+source tree for more details on how QSBR is implemented.  Running
+:func:`gc.collect` should cause all memory being held by QSBR to be actually
+freed.  Note that even when QSBR frees the memory, the underlying memory
+allocator may not immediately return that memory to the OS and so the resident
+set size (RSS) of the process might not decrease.
+
+
+mimalloc allocator vs pymalloc
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The default build will normally use the "pymalloc" memory allocator for small
+allocations (512 bytes or smaller).  The free-threaded build does not use
+pymalloc and allocates all Python objects using the "mimalloc" allocator.  The
+pymalloc allocator has the following properties that help keep memory usage
+low: small per-allocated-block overhead, effective memory fragmentation
+prevention, and quick return of free memory to the operating system.  The
+mimalloc allocator does quite well in these respects as well but can have some
+more overhead.
+
+In the free-threaded build, mimalloc manages memory in a number of separate
+heaps (currently four).  For example, all GC supporting objects are allocated
+from their own heap.  Using separate heaps means that free memory in one heap
+cannot be used for an allocation that uses another heap.  Also, some heaps are
+configured to use QSBR (quiescent-state based reclamation) when freeing the
+memory that backs up the heap (known as "pages" in mimalloc terminology).  The
+use of QSBR creates a delay between all memory blocks for a page being freed
+and the memory page being released, either for new allocations or back to the
+OS.
+
+The mimalloc allocator also defers returning freed memory back to the OS.  You
+can reduce that delay by setting the environment variable
+:envvar:`!MIMALLOC_PURGE_DELAY` to ``0``.  Note that this will likely reduce
+the performance of the allocator.
+
+
+Free-threaded reference counting can cause objects to live longer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the default build, when an object's reference count reaches zero, it is
+normally deallocated.  The free-threaded build uses "biased reference
+counting", with a fast-path for objects "owned" by the current thread and a
+slow path for other objects.  See :pep:`703` for additional details.  Any time
+an object's reference count ends up in a "queued" state, deallocation can be
+deferred.  The queued state is cleared from the "eval breaker" section of the
+bytecode evaluator.
+
+The free-threaded build also allows a different mode of reference counting,
+known as "deferred reference counting".  This mode is enabled by setting a flag
+on a per-object basis.  Deferred reference counting is enabled for the
+following types:
+
+* module objects
+* module top-level functions
+* class methods defined in the class scope
+* descriptor objects
+* thread-local objects, created by :class:`threading.local`
+
+When deferred reference counting is enabled, references from Python function
+stacks are not added to the reference count.  This scheme reduces the overhead
+of reference counting, especially for objects used from multiple threads.
+Because the stack references are not counted, objects with deferred reference
+counting are not immediately freed when their internal reference count goes to
+zero.  Instead, they are examined by the next GC run and, if no stack
+references to them are found, they are freed.  This means these objects are
+freed by the GC and not when their reference count goes to zero, as is typical.
+
+
+Per-thread reference counting can delay freeing objects
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To avoid contention on the reference count fields of frequently shared
+objects, the free-threaded build also uses "per-thread reference counting"
+for a few selected object types.  Rather than updating a single shared
+reference count, each thread maintains its own local reference count array,
+indexed by a unique id assigned to the object.  The true reference count is
+only computed by summing the per-thread counts when the object's local
+count drops to zero.  Per-thread reference counting is currently used for:
+
+* heap type objects (classes created in Python)
+* code objects
+* the ``__dict__`` of module objects
+
+Because the per-thread counts must be merged back to the object before it
+can be deallocated, objects using per-thread reference counting are
+typically freed later than they would be in the default build.  In
+particular, such an object is usually not freed until the thread that
+referenced it reaches a safe point (for example, in the "eval breaker"
+section of the bytecode evaluator) or exits.  Running :func:`gc.collect`
+will merge the per-thread counts and allow these objects to be freed.

Doc/library/abc.rst

@@ -237,7 +237,7 @@ The :mod:`!abc` module also supports the following legacy decorators:
 .. decorator:: abstractclassmethod
 
    .. versionadded:: 3.2
-   .. deprecated:: 3.3
+   .. deprecated-removed:: 3.3 3.21
        It is now possible to use :class:`classmethod` with
        :func:`abstractmethod`, making this decorator redundant.
 
@@ -258,7 +258,7 @@ The :mod:`!abc` module also supports the following legacy decorators:
 .. decorator:: abstractstaticmethod
 
    .. versionadded:: 3.2
-   .. deprecated:: 3.3
+   .. deprecated-removed:: 3.3 3.21
        It is now possible to use :class:`staticmethod` with
        :func:`abstractmethod`, making this decorator redundant.
 
@@ -278,7 +278,7 @@ The :mod:`!abc` module also supports the following legacy decorators:
 
 .. decorator:: abstractproperty
 
-   .. deprecated:: 3.3
+   .. deprecated-removed:: 3.3 3.21
        It is now possible to use :class:`property`, :meth:`property.getter`,
        :meth:`property.setter` and :meth:`property.deleter` with
        :func:`abstractmethod`, making this decorator redundant.

Doc/library/binascii.rst

@@ -367,9 +367,8 @@ The :mod:`!binascii` module defines the following functions:
    *ignorechars* should be a :term:`bytes-like object` containing characters
    to ignore from the input.
 
-   Similar functionality (accepting only text string arguments, but more
-   liberal towards whitespace) is also accessible using the
-   :meth:`bytes.fromhex` class method.
+   Similar functionality (but more liberal towards whitespace) is also accessible
+   using the :meth:`bytes.fromhex` class method.
 
    .. versionchanged:: 3.15
       Added the *ignorechars* parameter.

Doc/library/copy.rst

@@ -72,9 +72,13 @@ file, socket, window, or any similar types.  It does "copy" functions and
 classes (shallow and deeply), by returning the original object unchanged; this
 is compatible with the way these are treated by the :mod:`pickle` module.
 
-Shallow copies of dictionaries can be made using :meth:`dict.copy`, and
-of lists by assigning a slice of the entire list, for example,
-``copied_list = original_list[:]``.
+Shallow copies of many collections can be made using the corresponding
+:meth:`!copy` method (such as :meth:`list.copy`, :meth:`dict.copy` or
+:meth:`set.copy`), and of sequences (such as lists or bytearrays) by making
+a slice of the entire sequence (``sequence[:]``).
+However, these methods and slicing can create an instance of the base type
+when copying an instance of a subclass, whereas :func:`copy.copy` normally
+returns an instance of the same type.
 
 .. index:: pair: module; pickle
 

Doc/library/ctypes.rst

@@ -14,6 +14,14 @@ used to wrap these libraries in pure Python.
 
 .. include:: ../includes/optional-module.rst
 
+.. warning::
+
+   :mod:`!ctypes` provides low-level access to native libraries and the
+   process's memory, bypassing Python's safety mechanisms and allowing
+   execution of arbitrary native code.
+   Incorrect use can corrupt data and objects, reveal sensitive information,
+   cause crashes, or otherwise compromise the running process.
+
 
 .. _ctypes-ctypes-tutorial:
 
@@ -198,10 +206,8 @@ argument values::
    OSError: exception: access violation reading 0x00000020
    >>>
 
-There are, however, enough ways to crash Python with :mod:`!ctypes`, so you
-should be careful anyway.  The :mod:`faulthandler` module can be helpful in
-debugging crashes (e.g. from segmentation faults produced by erroneous C library
-calls).
+The :mod:`faulthandler` module can help debug crashes,
+such as segmentation faults produced by erroneous C library calls.
 
 ``None``, integers, bytes objects and (unicode) strings are the only native
 Python objects that can directly be used as parameters in these function calls.

Doc/library/email.headerregistry.rst

@@ -93,9 +93,10 @@ headers.
    ``kwds`` is a dictionary containing one pre-initialized key, ``defects``.
    ``defects`` is an empty list.  The parse method should append any detected
    defects to this list.  On return, the ``kwds`` dictionary *must* contain
-   values for at least the keys ``decoded`` and ``defects``.  ``decoded``
-   should be the string value for the header (that is, the header value fully
-   decoded to unicode).  The parse method should assume that *string* may
+   values for at least the keys ``decoded``, ``defects`` and ``parse_tree``.
+   ``decoded`` should be the string value for the header (that is, the header
+   value fully decoded to unicode). ``parse_tree`` is set to the parse tree obtained
+   from parsing the header. The parse method should assume that *string* may
    contain content-transfer-encoded parts, but should correctly handle all valid
    unicode characters as well so that it can parse un-encoded header values.