Treat bytes objects as immutable in C code

[vstinner]

Currently, in the CPython C code, multiple functions treat bytes objects as mutable. For example, they use char* to access their data, instead of const char*.

In the worst case, it can lead to bad bugs like issue gh-155702 in sqlite3 which modifies an immutable bytes object.

Some functions can be rewritten using PyBytesWriter C API to avoid that.

Linked PRs

• gh-155742: Use PyBytesWriter in io _textiowrapper_writeflush() #155743
• gh-155742: Use PyBytesWriter in json _match_number_unicode() #155744
• gh-155742: Use PyBytesWriter in _zstd.train_dict() #155745
• gh-155742: Use PyBytesWriter in CJK codecs #155746
• gh-155742: Use PyBytesWriter in Python/assemble.c #155747
• gh-155742: Use PyBytesWriter in marshal #155748
• gh-155742: Use PyBytesWriter in xml.etree #155749
• gh-155742: Use PyBytesWriter in codecs #155750
• gh-155742: Get PyBytes_AS_STRING() as const char* #155751
• gh-155742: Test PyBytesWriter_FinishWithSize() with negative size #155783
• [3.15] gh-155742: Test PyBytesWriter_FinishWithSize() with negative size (GH-155783) #155791
• gh-155742: Get singleton in PyBytesWriter_FinishWithSize() #155795
• gh-155742: Check singletons consistencty in the garbage collector #155801
• gh-155742: Add support.built_with_c_assertions() #156776
• gh-155742: Add tests on the PyMarshal C API #156890
• gh-155742: Optimize json float parsing #156897
• gh-155742: Remove dead code in _elementtree #157342
• gh-155742: Use PyBytesWriter in _zstd.finalize_dict() #157343
• gh-155742: Use PyBytesWriter in codeobject.c #157345
• gh-155742: Use PyBytesWriter in bytes.translate() #157346
• gh-155742: Use PyBytesWriter in zlib #157348
• gh-155742: Use PyBytesWriter in Python/assemble.c #157349

[vstinner]

I modified Python locally with following patch to replace char* with const char* in functions PyBytes_AS_STRING(), PyBytes_AsString() and PyBytes_AsStringAndSize(). So I can detect when a bytes object is treated as mutable.

diff --git a/Include/bytesobject.h b/Include/bytesobject.h
index c5a24195be6..4abd89469ad 100644
--- a/Include/bytesobject.h
+++ b/Include/bytesobject.h
@@ -36,7 +36,7 @@ PyAPI_FUNC(PyObject *) PyBytes_FromFormatV(const char*, va_list)
 PyAPI_FUNC(PyObject *) PyBytes_FromFormat(const char*, ...)
                                 Py_GCC_ATTRIBUTE((format(printf, 1, 2)));
 PyAPI_FUNC(Py_ssize_t) PyBytes_Size(PyObject *);
-PyAPI_FUNC(char *) PyBytes_AsString(PyObject *);
+PyAPI_FUNC(const char *) PyBytes_AsString(PyObject *);
 PyAPI_FUNC(PyObject *) PyBytes_Repr(PyObject *, int);
 PyAPI_FUNC(void) PyBytes_Concat(PyObject **, PyObject *);
 PyAPI_FUNC(void) PyBytes_ConcatAndDel(PyObject **, PyObject *);
@@ -50,7 +50,7 @@ PyAPI_FUNC(PyObject *) PyBytes_DecodeEscape(const char *, Py_ssize_t,
    cause an exception).  */
 PyAPI_FUNC(int) PyBytes_AsStringAndSize(
     PyObject *obj,      /* bytes object */
-    char **s,           /* pointer to buffer variable */
+    const char **s,           /* pointer to buffer variable */
     Py_ssize_t *len     /* pointer to length variable or NULL */
     );

diff --git a/Include/cpython/bytesobject.h b/Include/cpython/bytesobject.h
index 85bc2b827df..d7bdebca31e 100644
--- a/Include/cpython/bytesobject.h
+++ b/Include/cpython/bytesobject.h
@@ -20,7 +20,7 @@ PyAPI_FUNC(int) _PyBytes_Resize(PyObject **, Py_ssize_t);
 #define _PyBytes_CAST(op) \
     (assert(PyBytes_Check(op)), _Py_CAST(PyBytesObject*, op))

-static inline char* PyBytes_AS_STRING(PyObject *op)
+static inline const char* PyBytes_AS_STRING(PyObject *op)
 {
     return _PyBytes_CAST(op)->ob_sval;
 }
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
index ef35dad82e8..6bd53958ac1 100644
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -1338,7 +1338,7 @@ PyBytes_Size(PyObject *op)
     return Py_SIZE(op);
 }

-char *
+const char *
 PyBytes_AsString(PyObject *op)
 {
     if (!PyBytes_Check(op)) {
@@ -1351,7 +1351,7 @@ PyBytes_AsString(PyObject *op)

 int
 PyBytes_AsStringAndSize(PyObject *obj,
-                         char **s,
+                         const char **s,
                          Py_ssize_t *len)
 {
     if (s == NULL) {

[vstinner]

TODO: more code can/should use PyBytesWriter

• _elementtree: treebuilder_handle_data()
• _io.bytesio: self->buf
• _pickle: self->output_buffer, load_counted_binbytes()
• parser: decode_unicode_with_escapes()

I may propose PRs for these ones, once most of other PyBytesWriter PRs will be merged.

[serhiy-storchaka]

Note that the bug was introduced by rewriting using PyBytesWriter C API which can return a shared immutable bytes object. Before this, the code used PyBytes_FromStringAndSize(NULL, size) which returns unique bytes object. More rewrites can introduce more bugs.

[vstinner]

Note that the bug was introduced by rewriting using PyBytesWriter C API which can return a shared immutable bytes object. Before this, the code used PyBytes_FromStringAndSize(NULL, size) which returns unique bytes object. More rewrites can introduce more bugs.

The sqlite3 bug gh-155702 existed before the PyBytesWriter rewrite, but before it couldn't be reached since the old code didn't use the empty bytes singleton.

You're right that we should be careful about such bug when modifying more code to use PyBytesWriter.

[serhiy-storchaka]

It did not exist before the PyBytesWriter rewrite.