From da3ccbbceab8cfe4930bb9b62eb9e58fa6eea303 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Wed, 2 Sep 2026 14:04:08 +0300 Subject: [PATCH] gh-156821: Fix inaccuracies in the xml.parsers.expat documentation and docstrings GetInputContext() returns bytes, not a string, and its result extends to the end of the buffered input. Parse() accepts a bytes-like object as well as a string, and ignores the encoding declaration for a string. ParseFile() only supports binary files. XML_ERROR_XML_DECL was described as XML_ERROR_NO_ELEMENTS. UnparsedEntityDeclHandler is not restricted to Expat 1.2. GetSpecifiedAttributeCount() is only meaningful if ordered_attributes is true. Document the intern parameter of ParserCreate(), the intern and namespace_prefixes attributes, the SkippedEntityHandler handler, the XML_CTYPE_MIXED and XML_CTYPE_NAME content model types, and the EXPAT_VERSION, version_info and features data items, which describe the Expat library loaded by the interpreter. Remove obsolete notes about Expat versions 1.2 and 1.95.0, complete the list of attribute types in AttlistDeclHandler, and remove :noindex: from the content model constants, which have no other description and were therefore not referenceable. --- Doc/library/pyexpat.rst | 142 ++++++++++++++++++++++++++----------- Modules/clinic/pyexpat.c.h | 21 ++++-- Modules/pyexpat.c | 27 ++++--- 3 files changed, 133 insertions(+), 57 deletions(-) diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst index abd68c5006bd475..638b8f2862b7d19 100644 --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -35,7 +35,7 @@ the XML document. This module uses the :mod:`pyexpat` module to provide access to the Expat parser. Direct use of the :mod:`pyexpat` module is deprecated. -This module provides one exception and one type object: +This module provides the following exception, type object and data items: .. exception:: ExpatError @@ -53,6 +53,29 @@ This module provides one exception and one type object: The type of the return values from the :func:`ParserCreate` function. + +.. data:: EXPAT_VERSION + + The version string of the Expat library loaded by the interpreter, + like ``'expat_2.8.4'``. + + +.. data:: version_info + + The version of the Expat library loaded by the interpreter, + as a tuple of three integers: major, minor and micro version. + + +.. data:: features + + The list of the features with which the loaded Expat library + was compiled, as ``(name, value)`` pairs. + The value is only meaningful for features which have one, + like ``'XML_CONTEXT_BYTES'`` or the default protection limits + ``'XML_BLAP_ACT_THRES'`` and ``'XML_AT_MAX_AMP'``; + for other features, like ``'XML_DTD'`` and ``'XML_NS'``, + the value is ``0`` and only the presence of the name is significant. + The :mod:`!xml.parsers.expat` module contains two functions: @@ -61,7 +84,7 @@ The :mod:`!xml.parsers.expat` module contains two functions: Returns an explanatory string for a given error number *errno*. -.. function:: ParserCreate(encoding=None, namespace_separator=None) +.. function:: ParserCreate(encoding=None, namespace_separator=None, intern=None) Creates and returns a new :class:`xmlparser` object. *encoding* [1]_, if specified, must be a string naming the encoding @@ -122,6 +145,11 @@ The :mod:`!xml.parsers.expat` module contains two functions: http://www.python.org/ns/ elem1 elem2 + *intern*, if given, must be a dictionary. + It is used to intern the names of elements and attributes, + and is available as the :attr:`~xmlparser.intern` attribute. + By default a new empty dictionary is created for every parser. + Due to limitations in the ``Expat`` library used by :mod:`pyexpat`, the :class:`xmlparser` instance returned can only be used to parse a single XML document. Call ``ParserCreate`` for each document to provide unique @@ -143,18 +171,24 @@ XMLParser Objects .. method:: xmlparser.Parse(data[, isfinal]) - Parses the contents of the string *data*, calling the appropriate handler - functions to process the parsed data. *isfinal* must be true on the final call - to this method; it allows the parsing of a single file in fragments, + Parses the contents of *data*, + calling the appropriate handler functions to process the parsed data. + *data* can be a :term:`bytes-like object` or a string. + If it is a string, the encoding declaration in the XML data is ignored, + and the data is parsed as already decoded text. + *isfinal* must be true on the final call to this method; + it allows the parsing of a single file in fragments, not the submission of multiple files. - *data* can be the empty string at any time. + *data* can be empty at any time. .. method:: xmlparser.ParseFile(file) - Parse XML data reading from the object *file*. *file* only needs to provide - the ``read(nbytes)`` method, returning the empty string when there's no more - data. + Parse XML data reading from the object *file*. + *file* only needs to provide the ``read(nbytes)`` method, + which returns bytes, and an empty bytes object when there's no more data. + Text files are not supported; + use :meth:`Parse` for data which is already decoded. .. method:: xmlparser.SetBase(base) @@ -188,9 +222,15 @@ XMLParser Objects .. method:: xmlparser.GetInputContext() - Returns the input data that generated the current event as a string. The data is - in the encoding of the entity which contains the text. When called while an - event handler is not active, the return value is ``None``. + Returns the input data which generated the current event + as a :class:`bytes` object. + The data is in the encoding of the entity which contains the text. + It extends to the end of the currently buffered input, + therefore it can contain also the data of the following events, + and if the event was generated by a large amount of text, + not all of it may be available. + When called while an event handler is not active, + the return value is ``None``. .. method:: xmlparser.ExternalEntityParserCreate(context[, encoding]) @@ -437,6 +477,22 @@ against some common XML vulnerabilities. default, this attribute is false; it may be changed at any time. +.. attribute:: xmlparser.intern + + The dictionary used to intern the names of elements and attributes. + It is either the dictionary passed as the *intern* argument + of :func:`ParserCreate`, or a new dictionary created for this parser. + + +.. attribute:: xmlparser.namespace_prefixes + + If set to a true value, and namespace processing is enabled, + the namespace prefix is reported as the third part of the expanded name, + separated by the namespace separator. + Names which have no prefix are not affected. + By default, this attribute is false; it may be changed at any time. + + The following attributes contain values relating to the most recent error encountered by an :class:`xmlparser` object, and will only have correct values once a call to :meth:`Parse` or :meth:`ParseFile` has raised an @@ -500,8 +556,7 @@ otherwise stated. encoding of the document text, and an optional "standalone" declaration. *version* and *encoding* will be strings, and *standalone* will be ``1`` if the document is declared standalone, ``0`` if it is declared not to be standalone, - or ``-1`` if the standalone clause was omitted. This is only available with - Expat version 1.95.0 or newer. + or ``-1`` if the standalone clause was omitted. .. method:: xmlparser.StartDoctypeDeclHandler(doctypeName, systemId, publicId, has_internal_subset) @@ -510,14 +565,12 @@ otherwise stated. ...``). The *doctypeName* is provided exactly as presented. The *systemId* and *publicId* parameters give the system and public identifiers if specified, or ``None`` if omitted. *has_internal_subset* will be true if the document - contains an internal document declaration subset. This requires Expat version - 1.2 or newer. + contains an internal document declaration subset. .. method:: xmlparser.EndDoctypeDeclHandler() - Called when Expat is done parsing the document type declaration. This requires - Expat version 1.2 or newer. + Called when Expat is done parsing the document type declaration. .. method:: xmlparser.ElementDeclHandler(name, model) @@ -532,12 +585,16 @@ otherwise stated. declaration declares three attributes, this handler is called three times, once for each attribute. *elname* is the name of the element to which the declaration applies and *attname* is the name of the attribute declared. The - attribute type is a string passed as *type*; the possible values are - ``'CDATA'``, ``'ID'``, ``'IDREF'``, ... *default* gives the default value for + The attribute type is a string passed as *type*: + ``'CDATA'``, ``'ID'``, ``'IDREF'``, ``'IDREFS'``, ``'ENTITY'``, + ``'ENTITIES'``, ``'NMTOKEN'`` or ``'NMTOKENS'``, + an enumeration like ``'(x|y)'``, + or a notation list like ``'NOTATION(n1|n2)'``. + *default* gives the default value for the attribute used when the attribute is not specified by the document instance, or ``None`` if there is no default value (``#IMPLIED`` values). If the attribute is required to be given in the document instance, *required* will be - true. This requires Expat version 1.95.0 or newer. + true. .. method:: xmlparser.StartElementHandler(name, attributes) @@ -573,10 +630,10 @@ otherwise stated. .. method:: xmlparser.UnparsedEntityDeclHandler(entityName, base, systemId, publicId, notationName) - Called for unparsed (NDATA) entity declarations. This is only present for - version 1.2 of the Expat library; for more recent versions, use - :attr:`EntityDeclHandler` instead. (The underlying function in the Expat - library has been declared obsolete.) + Called for unparsed (NDATA) entity declarations. + If this handler is not set, such declarations are reported by + :attr:`EntityDeclHandler`, which is preferred for new code. + (The underlying function in the Expat library has been declared obsolete.) .. method:: xmlparser.EntityDeclHandler(entityName, is_parameter_entity, value, base, systemId, publicId, notationName) @@ -587,8 +644,7 @@ otherwise stated. ``None`` for parsed entities, and the name of the notation for unparsed entities. *is_parameter_entity* will be true if the entity is a parameter entity or false for general entities (most applications only need to be concerned with - general entities). This is only available starting with version 1.95.0 of the - Expat library. + general entities). .. method:: xmlparser.NotationDeclHandler(notationName, base, systemId, publicId) @@ -642,7 +698,7 @@ otherwise stated. .. method:: xmlparser.DefaultHandlerExpand(data) - This is the same as the :func:`DefaultHandler`, but doesn't inhibit expansion + This is the same as the :attr:`DefaultHandler`, but doesn't inhibit expansion of internal entities. The entity reference will not be passed to the default handler. @@ -686,6 +742,16 @@ otherwise stated. :attr:`DefaultHandler` callback, if provided. +.. method:: xmlparser.SkippedEntityHandler(entityName, is_parameter_entity) + + Called for entity references which are not expanded, + because the parser did not read the declaration of the entity. + This happens when the external DTD subset or an external parameter entity + is not parsed. + *is_parameter_entity* is true for a parameter entity + and false for a general entity. + + .. _expaterror-objects: ExpatError Exceptions @@ -786,35 +852,35 @@ The constants in the model type group are: .. data:: XML_CTYPE_ANY - :noindex: The element named by the model name was declared to have a content model of ``ANY``. .. data:: XML_CTYPE_CHOICE - :noindex: The named element allows a choice from a number of options; this is used for content models such as ``(A | B | C)``. .. data:: XML_CTYPE_EMPTY - :noindex: Elements which are declared to be ``EMPTY`` have this model type. .. data:: XML_CTYPE_MIXED - :noindex: + + The named element allows character data, optionally interspersed with + the named children; this is used for content models such as + ``(#PCDATA)`` and ``(#PCDATA | A | B)*``. .. data:: XML_CTYPE_NAME - :noindex: + + The model names a single element, as for ``A``. .. data:: XML_CTYPE_SEQ - :noindex: Models which represent a series of models which follow one after the other are indicated with this model type. This is used for models such as ``(A, B, C)``. @@ -823,25 +889,21 @@ The constants in the quantifier group are: .. data:: XML_CQUANT_NONE - :noindex: No modifier is given, so it can appear exactly once, as for ``A``. .. data:: XML_CQUANT_OPT - :noindex: The model is optional: it can appear once or not at all, as for ``A?``. .. data:: XML_CQUANT_PLUS - :noindex: The model must occur one or more times (like ``A+``). .. data:: XML_CQUANT_REP - :noindex: The model must occur zero or more times, as for ``A*``. @@ -925,7 +987,7 @@ The ``errors`` module has the following attributes: .. data:: XML_ERROR_NO_ELEMENTS The document contains no elements (XML requires all documents to contain exactly - one top-level element).. + one top-level element). .. data:: XML_ERROR_NO_MEMORY @@ -1028,7 +1090,7 @@ The ``errors`` module has the following attributes: .. data:: XML_ERROR_XML_DECL - The document contained no document element at all. + There was an error parsing the XML declaration. .. data:: XML_ERROR_TEXT_DECL diff --git a/Modules/clinic/pyexpat.c.h b/Modules/clinic/pyexpat.c.h index 9e32bb079c1ea68..959e06da7791145 100644 --- a/Modules/clinic/pyexpat.c.h +++ b/Modules/clinic/pyexpat.c.h @@ -62,6 +62,8 @@ PyDoc_STRVAR(pyexpat_xmlparser_Parse__doc__, "\n" "Parse XML data.\n" "\n" +"\'data\' can be a bytes-like object or a string. If it is a string,\n" +"the encoding declaration in the XML data is ignored.\n" "\'isfinal\' should be true at end of input."); #define PYEXPAT_XMLPARSER_PARSE_METHODDEF \ @@ -116,7 +118,9 @@ PyDoc_STRVAR(pyexpat_xmlparser_ParseFile__doc__, "ParseFile($self, file, /)\n" "--\n" "\n" -"Parse XML data from file-like object."); +"Parse XML data from a binary file-like object.\n" +"\n" +"Its read() method should return bytes."); #define PYEXPAT_XMLPARSER_PARSEFILE_METHODDEF \ {"ParseFile", _PyCFunction_CAST(pyexpat_xmlparser_ParseFile), METH_METHOD|METH_FASTCALL|METH_KEYWORDS, pyexpat_xmlparser_ParseFile__doc__}, @@ -222,7 +226,8 @@ PyDoc_STRVAR(pyexpat_xmlparser_GetSpecifiedAttributeCount__doc__, "attributes at lower indices in the list passed to\n" "StartElementHandler were given in the start tag. Each attribute\n" "takes two items in that list, its name and its value. Only\n" -"meaningful inside a StartElementHandler call."); +"meaningful inside a StartElementHandler call, and only if\n" +"ordered_attributes is true."); #define PYEXPAT_XMLPARSER_GETSPECIFIEDATTRIBUTECOUNT_METHODDEF \ {"GetSpecifiedAttributeCount", (PyCFunction)pyexpat_xmlparser_GetSpecifiedAttributeCount, METH_NOARGS, pyexpat_xmlparser_GetSpecifiedAttributeCount__doc__}, @@ -240,11 +245,13 @@ PyDoc_STRVAR(pyexpat_xmlparser_GetInputContext__doc__, "GetInputContext($self, /)\n" "--\n" "\n" -"Return the untranslated text of the input that caused the current event.\n" +"Return the input which generated the current event as bytes.\n" "\n" -"If the event was generated by a large amount of text (such as\n" -"a start tag for an element with many attributes), not all of the\n" -"text may be available."); +"The data is in the encoding of the entity which contains the text.\n" +"It extends to the end of the currently buffered input, therefore it\n" +"can contain also the data of the following events, and if the event\n" +"was generated by a large amount of text, not all of it may be\n" +"available."); #define PYEXPAT_XMLPARSER_GETINPUTCONTEXT_METHODDEF \ {"GetInputContext", (PyCFunction)pyexpat_xmlparser_GetInputContext, METH_NOARGS, pyexpat_xmlparser_GetInputContext__doc__}, @@ -863,4 +870,4 @@ pyexpat_ErrorString(PyObject *module, PyObject *arg) #ifndef PYEXPAT_XMLPARSER_SETALLOCTRACKERMAXIMUMAMPLIFICATION_METHODDEF #define PYEXPAT_XMLPARSER_SETALLOCTRACKERMAXIMUMAMPLIFICATION_METHODDEF #endif /* !defined(PYEXPAT_XMLPARSER_SETALLOCTRACKERMAXIMUMAMPLIFICATION_METHODDEF) */ -/*[clinic end generated code: output=d7e59d139fe45487 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=c68606a1fbc3a501 input=a9049054013a1b77]*/ diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c index fa8b0db60806233..9eb0dddb2488e23 100644 --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -876,13 +876,15 @@ pyexpat.xmlparser.Parse Parse XML data. +'data' can be a bytes-like object or a string. If it is a string, +the encoding declaration in the XML data is ignored. 'isfinal' should be true at end of input. [clinic start generated code]*/ static PyObject * pyexpat_xmlparser_Parse_impl(xmlparseobject *self, PyTypeObject *cls, PyObject *data, int isfinal) -/*[clinic end generated code: output=8faffe07fe1f862a input=053e0f047e55c05a]*/ +/*[clinic end generated code: output=8faffe07fe1f862a input=f8f3ba4eadf837e5]*/ { const char *s; Py_ssize_t slen; @@ -975,13 +977,15 @@ pyexpat.xmlparser.ParseFile file: object / -Parse XML data from file-like object. +Parse XML data from a binary file-like object. + +Its read() method should return bytes. [clinic start generated code]*/ static PyObject * pyexpat_xmlparser_ParseFile_impl(xmlparseobject *self, PyTypeObject *cls, PyObject *file) -/*[clinic end generated code: output=34780a094c8ca3ae input=ba4bc9c541684793]*/ +/*[clinic end generated code: output=34780a094c8ca3ae input=c96030cf5e4f577e]*/ { int rv = 1; PyObject *readmethod = NULL; @@ -1063,12 +1067,13 @@ Attributes defaulted from the DTD follow the specified ones, so attributes at lower indices in the list passed to StartElementHandler were given in the start tag. Each attribute takes two items in that list, its name and its value. Only -meaningful inside a StartElementHandler call. +meaningful inside a StartElementHandler call, and only if +ordered_attributes is true. [clinic start generated code]*/ static PyObject * pyexpat_xmlparser_GetSpecifiedAttributeCount_impl(xmlparseobject *self) -/*[clinic end generated code: output=f96b627de9393c0c input=4981c36cf99ebe9f]*/ +/*[clinic end generated code: output=f96b627de9393c0c input=f9f66f840b9dd16b]*/ { return PyLong_FromLong(XML_GetSpecifiedAttributeCount(self->itself)); } @@ -1077,16 +1082,18 @@ pyexpat_xmlparser_GetSpecifiedAttributeCount_impl(xmlparseobject *self) @permit_long_summary pyexpat.xmlparser.GetInputContext -Return the untranslated text of the input that caused the current event. +Return the input which generated the current event as bytes. -If the event was generated by a large amount of text (such as -a start tag for an element with many attributes), not all of the -text may be available. +The data is in the encoding of the entity which contains the text. +It extends to the end of the currently buffered input, therefore it +can contain also the data of the following events, and if the event +was generated by a large amount of text, not all of it may be +available. [clinic start generated code]*/ static PyObject * pyexpat_xmlparser_GetInputContext_impl(xmlparseobject *self) -/*[clinic end generated code: output=a88026d683fc22cc input=a672f48f09bb73d2]*/ +/*[clinic end generated code: output=a88026d683fc22cc input=13840373d8320ab6]*/ { if (self->in_callback) { int offset, size;