From bbc9ff7ba7ae92655fb3b91538fb258db9327227 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Wed, 2 Sep 2026 21:57:57 +0300 Subject: [PATCH] [3.15] gh-156821: Fix inaccuracies in the xml.parsers.expat documentation (GH-156822) Corrected, in the documentation and in the docstrings: * GetInputContext() returns bytes, and its result extends to the end of the buffered input; * Parse() accepts a bytes-like object as well as a string, and ignores the encoding declaration for a string; * ParseFile() only supports binary files; * XML_ERROR_XML_DECL was described as XML_ERROR_NO_ELEMENTS; * UnparsedEntityDeclHandler is not restricted to Expat 1.2; * GetSpecifiedAttributeCount() needs ordered_attributes. Documented the intern parameter and attribute, namespace_prefixes, SkippedEntityHandler, XML_CTYPE_MIXED and XML_CTYPE_NAME, and EXPAT_VERSION, version_info and features. Removed obsolete notes about Expat 1.2 and 1.95.0, and :noindex: from the content model constants, which are not documented elsewhere; completed the list of attribute types in AttlistDeclHandler. (cherry picked from commit d16a691111503798820a4b3a60ffae7b76c4d85a) Co-authored-by: Serhiy Storchaka --- Doc/library/pyexpat.rst | 142 ++++++++++++++++++++++++++----------- Modules/clinic/pyexpat.c.h | 18 +++-- Modules/pyexpat.c | 23 +++--- 3 files changed, 128 insertions(+), 55 deletions(-) diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst index 2e6938b5cf68602..8f076e86e957c2d 100644 --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -35,7 +35,7 @@ the XML document. This module uses the :mod:`pyexpat` module to provide access to the Expat parser. Direct use of the :mod:`pyexpat` module is deprecated. -This module provides one exception and one type object: +This module provides the following exception, type object and data items: .. exception:: ExpatError @@ -53,6 +53,29 @@ This module provides one exception and one type object: The type of the return values from the :func:`ParserCreate` function. + +.. data:: EXPAT_VERSION + + The version string of the Expat library loaded by the interpreter, + like ``'expat_2.8.4'``. + + +.. data:: version_info + + The version of the Expat library loaded by the interpreter, + as a tuple of three integers: major, minor and micro version. + + +.. data:: features + + The list of the features with which the loaded Expat library + was compiled, as ``(name, value)`` pairs. + The value is only meaningful for features which have one, + like ``'XML_CONTEXT_BYTES'`` or the default protection limits + ``'XML_BLAP_ACT_THRES'`` and ``'XML_AT_MAX_AMP'``; + for other features, like ``'XML_DTD'`` and ``'XML_NS'``, + the value is ``0`` and only the presence of the name is significant. + The :mod:`!xml.parsers.expat` module contains two functions: @@ -61,7 +84,7 @@ The :mod:`!xml.parsers.expat` module contains two functions: Returns an explanatory string for a given error number *errno*. -.. function:: ParserCreate(encoding=None, namespace_separator=None) +.. function:: ParserCreate(encoding=None, namespace_separator=None, intern=None) Creates and returns a new :class:`xmlparser` object. *encoding*, if specified, must be a string naming the encoding used by the XML data. Expat doesn't @@ -108,6 +131,11 @@ The :mod:`!xml.parsers.expat` module contains two functions: http://www.python.org/ns/ elem1 elem2 + *intern*, if given, must be a dictionary. + It is used to intern the names of elements and attributes, + and is available as the :attr:`~xmlparser.intern` attribute. + By default a new empty dictionary is created for every parser. + Due to limitations in the ``Expat`` library used by :mod:`pyexpat`, the :class:`xmlparser` instance returned can only be used to parse a single XML document. Call ``ParserCreate`` for each document to provide unique @@ -130,18 +158,24 @@ XMLParser Objects .. method:: xmlparser.Parse(data[, isfinal]) - Parses the contents of the string *data*, calling the appropriate handler - functions to process the parsed data. *isfinal* must be true on the final call - to this method; it allows the parsing of a single file in fragments, + Parses the contents of *data*, + calling the appropriate handler functions to process the parsed data. + *data* can be a :term:`bytes-like object` or a string. + If it is a string, the encoding declaration in the XML data is ignored, + and the data is parsed as already decoded text. + *isfinal* must be true on the final call to this method; + it allows the parsing of a single file in fragments, not the submission of multiple files. - *data* can be the empty string at any time. + *data* can be empty at any time. .. method:: xmlparser.ParseFile(file) - Parse XML data reading from the object *file*. *file* only needs to provide - the ``read(nbytes)`` method, returning the empty string when there's no more - data. + Parse XML data reading from the object *file*. + *file* only needs to provide the ``read(nbytes)`` method, + which returns bytes, and an empty bytes object when there's no more data. + Text files are not supported; + use :meth:`Parse` for data which is already decoded. .. method:: xmlparser.SetBase(base) @@ -161,9 +195,15 @@ XMLParser Objects .. method:: xmlparser.GetInputContext() - Returns the input data that generated the current event as a string. The data is - in the encoding of the entity which contains the text. When called while an - event handler is not active, the return value is ``None``. + Returns the input data which generated the current event + as a :class:`bytes` object. + The data is in the encoding of the entity which contains the text. + It extends to the end of the currently buffered input, + therefore it can contain also the data of the following events, + and if the event was generated by a large amount of text, + not all of it may be available. + When called while an event handler is not active, + the return value is ``None``. .. method:: xmlparser.ExternalEntityParserCreate(context[, encoding]) @@ -410,6 +450,22 @@ against some common XML vulnerabilities. default, this attribute is false; it may be changed at any time. +.. attribute:: xmlparser.intern + + The dictionary used to intern the names of elements and attributes. + It is either the dictionary passed as the *intern* argument + of :func:`ParserCreate`, or a new dictionary created for this parser. + + +.. attribute:: xmlparser.namespace_prefixes + + If set to a true value, and namespace processing is enabled, + the namespace prefix is reported as the third part of the expanded name, + separated by the namespace separator. + Names which have no prefix are not affected. + By default, this attribute is false; it may be changed at any time. + + The following attributes contain values relating to the most recent error encountered by an :class:`xmlparser` object, and will only have correct values once a call to :meth:`Parse` or :meth:`ParseFile` has raised an @@ -473,8 +529,7 @@ otherwise stated. encoding of the document text, and an optional "standalone" declaration. *version* and *encoding* will be strings, and *standalone* will be ``1`` if the document is declared standalone, ``0`` if it is declared not to be standalone, - or ``-1`` if the standalone clause was omitted. This is only available with - Expat version 1.95.0 or newer. + or ``-1`` if the standalone clause was omitted. .. method:: xmlparser.StartDoctypeDeclHandler(doctypeName, systemId, publicId, has_internal_subset) @@ -483,14 +538,12 @@ otherwise stated. ...``). The *doctypeName* is provided exactly as presented. The *systemId* and *publicId* parameters give the system and public identifiers if specified, or ``None`` if omitted. *has_internal_subset* will be true if the document - contains an internal document declaration subset. This requires Expat version - 1.2 or newer. + contains an internal document declaration subset. .. method:: xmlparser.EndDoctypeDeclHandler() - Called when Expat is done parsing the document type declaration. This requires - Expat version 1.2 or newer. + Called when Expat is done parsing the document type declaration. .. method:: xmlparser.ElementDeclHandler(name, model) @@ -505,12 +558,16 @@ otherwise stated. declaration declares three attributes, this handler is called three times, once for each attribute. *elname* is the name of the element to which the declaration applies and *attname* is the name of the attribute declared. The - attribute type is a string passed as *type*; the possible values are - ``'CDATA'``, ``'ID'``, ``'IDREF'``, ... *default* gives the default value for + The attribute type is a string passed as *type*: + ``'CDATA'``, ``'ID'``, ``'IDREF'``, ``'IDREFS'``, ``'ENTITY'``, + ``'ENTITIES'``, ``'NMTOKEN'`` or ``'NMTOKENS'``, + an enumeration like ``'(x|y)'``, + or a notation list like ``'NOTATION(n1|n2)'``. + *default* gives the default value for the attribute used when the attribute is not specified by the document instance, or ``None`` if there is no default value (``#IMPLIED`` values). If the attribute is required to be given in the document instance, *required* will be - true. This requires Expat version 1.95.0 or newer. + true. .. method:: xmlparser.StartElementHandler(name, attributes) @@ -546,10 +603,10 @@ otherwise stated. .. method:: xmlparser.UnparsedEntityDeclHandler(entityName, base, systemId, publicId, notationName) - Called for unparsed (NDATA) entity declarations. This is only present for - version 1.2 of the Expat library; for more recent versions, use - :attr:`EntityDeclHandler` instead. (The underlying function in the Expat - library has been declared obsolete.) + Called for unparsed (NDATA) entity declarations. + If this handler is not set, such declarations are reported by + :attr:`EntityDeclHandler`, which is preferred for new code. + (The underlying function in the Expat library has been declared obsolete.) .. method:: xmlparser.EntityDeclHandler(entityName, is_parameter_entity, value, base, systemId, publicId, notationName) @@ -560,8 +617,7 @@ otherwise stated. ``None`` for parsed entities, and the name of the notation for unparsed entities. *is_parameter_entity* will be true if the entity is a parameter entity or false for general entities (most applications only need to be concerned with - general entities). This is only available starting with version 1.95.0 of the - Expat library. + general entities). .. method:: xmlparser.NotationDeclHandler(notationName, base, systemId, publicId) @@ -615,7 +671,7 @@ otherwise stated. .. method:: xmlparser.DefaultHandlerExpand(data) - This is the same as the :func:`DefaultHandler`, but doesn't inhibit expansion + This is the same as the :attr:`DefaultHandler`, but doesn't inhibit expansion of internal entities. The entity reference will not be passed to the default handler. @@ -659,6 +715,16 @@ otherwise stated. :attr:`DefaultHandler` callback, if provided. +.. method:: xmlparser.SkippedEntityHandler(entityName, is_parameter_entity) + + Called for entity references which are not expanded, + because the parser did not read the declaration of the entity. + This happens when the external DTD subset or an external parameter entity + is not parsed. + *is_parameter_entity* is true for a parameter entity + and false for a general entity. + + .. _expaterror-objects: ExpatError Exceptions @@ -759,35 +825,35 @@ The constants in the model type group are: .. data:: XML_CTYPE_ANY - :noindex: The element named by the model name was declared to have a content model of ``ANY``. .. data:: XML_CTYPE_CHOICE - :noindex: The named element allows a choice from a number of options; this is used for content models such as ``(A | B | C)``. .. data:: XML_CTYPE_EMPTY - :noindex: Elements which are declared to be ``EMPTY`` have this model type. .. data:: XML_CTYPE_MIXED - :noindex: + + The named element allows character data, optionally interspersed with + the named children; this is used for content models such as + ``(#PCDATA)`` and ``(#PCDATA | A | B)*``. .. data:: XML_CTYPE_NAME - :noindex: + + The model names a single element, as for ``A``. .. data:: XML_CTYPE_SEQ - :noindex: Models which represent a series of models which follow one after the other are indicated with this model type. This is used for models such as ``(A, B, C)``. @@ -796,25 +862,21 @@ The constants in the quantifier group are: .. data:: XML_CQUANT_NONE - :noindex: No modifier is given, so it can appear exactly once, as for ``A``. .. data:: XML_CQUANT_OPT - :noindex: The model is optional: it can appear once or not at all, as for ``A?``. .. data:: XML_CQUANT_PLUS - :noindex: The model must occur one or more times (like ``A+``). .. data:: XML_CQUANT_REP - :noindex: The model must occur zero or more times, as for ``A*``. @@ -898,7 +960,7 @@ The ``errors`` module has the following attributes: .. data:: XML_ERROR_NO_ELEMENTS The document contains no elements (XML requires all documents to contain exactly - one top-level element).. + one top-level element). .. data:: XML_ERROR_NO_MEMORY @@ -1001,7 +1063,7 @@ The ``errors`` module has the following attributes: .. data:: XML_ERROR_XML_DECL - The document contained no document element at all. + There was an error parsing the XML declaration. .. data:: XML_ERROR_TEXT_DECL diff --git a/Modules/clinic/pyexpat.c.h b/Modules/clinic/pyexpat.c.h index 1a07726d303ecad..42e9a50c96c2b47 100644 --- a/Modules/clinic/pyexpat.c.h +++ b/Modules/clinic/pyexpat.c.h @@ -62,6 +62,8 @@ PyDoc_STRVAR(pyexpat_xmlparser_Parse__doc__, "\n" "Parse XML data.\n" "\n" +"\'data\' can be a bytes-like object or a string. If it is a string,\n" +"the encoding declaration in the XML data is ignored.\n" "\'isfinal\' should be true at end of input."); #define PYEXPAT_XMLPARSER_PARSE_METHODDEF \ @@ -116,7 +118,9 @@ PyDoc_STRVAR(pyexpat_xmlparser_ParseFile__doc__, "ParseFile($self, file, /)\n" "--\n" "\n" -"Parse XML data from file-like object."); +"Parse XML data from a binary file-like object.\n" +"\n" +"Its read() method should return bytes."); #define PYEXPAT_XMLPARSER_PARSEFILE_METHODDEF \ {"ParseFile", _PyCFunction_CAST(pyexpat_xmlparser_ParseFile), METH_METHOD|METH_FASTCALL|METH_KEYWORDS, pyexpat_xmlparser_ParseFile__doc__}, @@ -216,11 +220,13 @@ PyDoc_STRVAR(pyexpat_xmlparser_GetInputContext__doc__, "GetInputContext($self, /)\n" "--\n" "\n" -"Return the untranslated text of the input that caused the current event.\n" +"Return the input which generated the current event as bytes.\n" "\n" -"If the event was generated by a large amount of text (such as\n" -"a start tag for an element with many attributes), not all of the\n" -"text may be available."); +"The data is in the encoding of the entity which contains the text.\n" +"It extends to the end of the currently buffered input, therefore it\n" +"can contain also the data of the following events, and if the event\n" +"was generated by a large amount of text, not all of it may be\n" +"available."); #define PYEXPAT_XMLPARSER_GETINPUTCONTEXT_METHODDEF \ {"GetInputContext", (PyCFunction)pyexpat_xmlparser_GetInputContext, METH_NOARGS, pyexpat_xmlparser_GetInputContext__doc__}, @@ -839,4 +845,4 @@ pyexpat_ErrorString(PyObject *module, PyObject *arg) #ifndef PYEXPAT_XMLPARSER_SETALLOCTRACKERMAXIMUMAMPLIFICATION_METHODDEF #define PYEXPAT_XMLPARSER_SETALLOCTRACKERMAXIMUMAMPLIFICATION_METHODDEF #endif /* !defined(PYEXPAT_XMLPARSER_SETALLOCTRACKERMAXIMUMAMPLIFICATION_METHODDEF) */ -/*[clinic end generated code: output=270a0bfe3300e8a1 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=8da798fcf6adb09c input=a9049054013a1b77]*/ diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c index 7094927ad2058fc..8885a184e581a9b 100644 --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -875,13 +875,15 @@ pyexpat.xmlparser.Parse Parse XML data. +'data' can be a bytes-like object or a string. If it is a string, +the encoding declaration in the XML data is ignored. 'isfinal' should be true at end of input. [clinic start generated code]*/ static PyObject * pyexpat_xmlparser_Parse_impl(xmlparseobject *self, PyTypeObject *cls, PyObject *data, int isfinal) -/*[clinic end generated code: output=8faffe07fe1f862a input=053e0f047e55c05a]*/ +/*[clinic end generated code: output=8faffe07fe1f862a input=f8f3ba4eadf837e5]*/ { const char *s; Py_ssize_t slen; @@ -974,13 +976,15 @@ pyexpat.xmlparser.ParseFile file: object / -Parse XML data from file-like object. +Parse XML data from a binary file-like object. + +Its read() method should return bytes. [clinic start generated code]*/ static PyObject * pyexpat_xmlparser_ParseFile_impl(xmlparseobject *self, PyTypeObject *cls, PyObject *file) -/*[clinic end generated code: output=34780a094c8ca3ae input=ba4bc9c541684793]*/ +/*[clinic end generated code: output=34780a094c8ca3ae input=c96030cf5e4f577e]*/ { int rv = 1; PyObject *readmethod = NULL; @@ -1054,19 +1058,20 @@ pyexpat_xmlparser_GetBase_impl(xmlparseobject *self) } /*[clinic input] -@permit_long_summary pyexpat.xmlparser.GetInputContext -Return the untranslated text of the input that caused the current event. +Return the input which generated the current event as bytes. -If the event was generated by a large amount of text (such as -a start tag for an element with many attributes), not all of the -text may be available. +The data is in the encoding of the entity which contains the text. +It extends to the end of the currently buffered input, therefore it +can contain also the data of the following events, and if the event +was generated by a large amount of text, not all of it may be +available. [clinic start generated code]*/ static PyObject * pyexpat_xmlparser_GetInputContext_impl(xmlparseobject *self) -/*[clinic end generated code: output=a88026d683fc22cc input=a672f48f09bb73d2]*/ +/*[clinic end generated code: output=a88026d683fc22cc input=2d605aaa9edbb415]*/ { if (self->in_callback) { int offset, size;