gh-63102: Support custom targets in the pull parser (GH-156888)

The list of events was collected by the TreeBuilder in the C implementation,
so the pull parser only worked with the standard target.  The parser itself
now collects the events, and reports what the target returns.  XMLPullParser
and iterparse() get the target parameter, which makes it possible to parse a
large document incrementally without building a tree for it.

The namespace events no longer need a separate code path: the parser reports
the prefix and the uri if the target does not implement start_ns()/end_ns(),
as the Python implementation already did.

Author: serhiy-storchaka

Doc/library/xml.etree.elementtree.rst

@@ -640,22 +640,30 @@ Functions
    element instance.  Return ``True`` if this is an element object.
 
 
-.. function:: iterparse(source, events=None, parser=None)
+.. function:: iterparse(source, events=None, parser=None, *, target=None)
 
-   Parses an XML section into an element tree incrementally, and reports what's
-   going on to the user.  *source* is a filename or :term:`file object`
+   Parses an XML section incrementally, and reports what's going on to the
+   user.  Unless a custom target is used, an element tree is built.
+   *source* is a filename or :term:`file object`
    containing XML data.  *events* is a sequence of events to report back.  The
    supported events are the strings ``"start"``, ``"end"``, ``"comment"``,
    ``"pi"``, ``"start-ns"`` and ``"end-ns"``
    (the "ns" events are used to get detailed namespace
    information).  If *events* is omitted, only ``"end"`` events are reported.
    *parser* is an optional parser instance.
    If not given, the standard :class:`XMLParser` parser is used.
-   *parser* must be an instance of :class:`XMLParser` or its subclass
-   and can only use the default :class:`TreeBuilder` as a target.
-   Returns an :term:`iterator` providing ``(event, elem)`` pairs;
+   *parser* must be an instance of :class:`XMLParser` or its subclass.
+   *target* is the target of the standard parser,
+   as for :class:`XMLPullParser`;
+   it cannot be used together with *parser*.
+   Returns an :term:`iterator` providing ``(event, obj)`` pairs,
+   as described for :meth:`XMLPullParser.read_events`;
    it has a ``root`` attribute that references the root element of the
-   resulting XML tree once *source* is fully read.
+   resulting XML tree, or the value returned by the ``close()`` method
+   of a custom target, once *source* is fully read.
+   If a custom target is used, it is set to the value returned
+   by the :meth:`!close` method of the target.
+
    The iterator has the :meth:`!close` method that closes the internal
    file object if *source* is a filename.
 
@@ -691,6 +699,9 @@ Functions
       A :exc:`ResourceWarning` is now emitted if the iterator opened a file
       and is not explicitly closed.
 
+   .. versionchanged:: next
+      Added the *target* parameter.
+
 
 .. function:: parse(source, parser=None)
 
@@ -1524,7 +1535,7 @@ XMLParser Objects
 XMLPullParser Objects
 ^^^^^^^^^^^^^^^^^^^^^
 
-.. class:: XMLPullParser(events=None)
+.. class:: XMLPullParser(events=None, *, target=None)
 
    A pull parser suitable for non-blocking applications.  Its input-side API is
    similar to that of :class:`XMLParser`, but instead of pushing calls to a
@@ -1535,6 +1546,20 @@ XMLPullParser Objects
    are used to get detailed namespace information).  If *events* is omitted,
    only ``"end"`` events are reported.
 
+   *target* is the target object of the underlying :class:`XMLParser`.
+   If omitted, the standard :class:`TreeBuilder` is used,
+   and the reported objects are :class:`Element` instances.
+   With other targets the reported object is the value returned
+   by the corresponding method of the target,
+   so no tree is built if the target does not build one.
+   The target must implement the methods for all requested events,
+   except :meth:`!start_ns` and :meth:`!end_ns`:
+   if they are not implemented, a ``(prefix, uri)`` tuple and ``None``
+   are reported for the ``"start-ns"`` and ``"end-ns"`` events.
+
+   .. versionchanged:: next
+      Added the *target* parameter.
+
    .. method:: feed(data)
 
       Feed the given data to the parser.  *data* is a string
@@ -1567,9 +1592,10 @@ XMLPullParser Objects
 
       Return an iterator over the events which have been encountered in the
       data fed to the
-      parser.  The iterator yields ``(event, elem)`` pairs, where *event* is a
-      string representing the type of event (e.g. ``"end"``) and *elem* is the
-      encountered :class:`Element` object, or other context value as follows.
+      parser.  The iterator yields ``(event, obj)`` pairs, where *event* is a
+      string representing the type of event (e.g. ``"end"``) and *obj* is the
+      object returned by the corresponding method of the target.
+      With the standard :class:`TreeBuilder` it is as follows.
 
       * ``start``, ``end``: the current Element.
       * ``comment``, ``pi``: the current comment / processing instruction

Doc/whatsnew/3.16.rst

@@ -708,6 +708,13 @@ xml
   rather than defaulted from the DTD.
   (Contributed by Jason Orendorff and Serhiy Storchaka in :gh:`44871`.)
 
+* :class:`~xml.etree.ElementTree.XMLPullParser` and
+  :func:`~xml.etree.ElementTree.iterparse` now support the *target* parameter.
+  The reported object is the value returned by the corresponding method of
+  the target, so a large document can be parsed incrementally without
+  building a tree for it.
+  (Contributed by Serhiy Storchaka in :gh:`63102`.)
+
 zipfile
 -------
 

Lib/test/test_xml_etree.py

@@ -1656,6 +1656,43 @@ def test_unknown_events(self):
             del cm
             gc_collect()
 
+    class Target:
+        # a target which does not build a tree
+        def start(self, tag, attrib):
+            return tag
+        def end(self, tag):
+            return tag
+        def data(self, data):
+            pass
+
+    def test_target(self):
+        # gh-63102: a custom target reports its own objects
+        with open(SIMPLE_XMLFILE, 'rb') as f:
+            it = ET.iterparse(f, events=('start', 'end'), target=self.Target())
+            self.assertEqual(list(it), [
+                ('start', 'root'),
+                ('start', 'element'),
+                ('end', 'element'),
+                ('start', 'element'),
+                ('end', 'element'),
+                ('start', 'empty-element'),
+                ('end', 'empty-element'),
+                ('end', 'root'),
+            ])
+            self.assertIsNone(it.root)
+
+    def test_parser_with_target(self):
+        with open(SIMPLE_XMLFILE, 'rb') as f:
+            parser = ET.XMLParser(target=self.Target())
+            it = ET.iterparse(f, events=('start',), parser=parser)
+            self.assertEqual(next(it), ('start', 'root'))
+
+    def test_target_and_parser(self):
+        with self.assertRaisesRegex(ValueError,
+                                    "can't specify both parser and target"):
+            ET.iterparse(SIMPLE_XMLFILE, parser=ET.XMLParser(),
+                         target=self.Target())
+
     def test_non_utf8(self):
         source = io.BytesIO(
             b"<?xml version='1.0' encoding='iso-8859-1'?>\n"
@@ -2067,6 +2104,76 @@ def __next__(self):
         self._feed(parser, "<foo>bar</foo>")
         self.assert_event_tags(parser, [('start', 'foo'), ('end', 'foo')])
 
+    # gh-63102: the pull parser reports events from any target
+    class SimpleTarget:
+        def start(self, tag, attrib):
+            return ('start', tag)
+        def end(self, tag):
+            return ('end', tag)
+        def data(self, data):
+            pass
+        def comment(self, text):
+            return ('comment', text)
+        def pi(self, target, data=None):
+            return ('pi', target)
+        def close(self):
+            return 'closed'
+
+    def test_custom_target(self):
+        parser = ET.XMLPullParser(events=('start', 'end'),
+                                  target=self.SimpleTarget())
+        self._feed(parser, "<root><element/></root>")
+        self.assert_event_tuples(parser, [
+            ('start', ('start', 'root')),
+            ('start', ('start', 'element')),
+            ('end', ('end', 'element')),
+            ('end', ('end', 'root')),
+        ])
+
+    def test_custom_target_comment_pi(self):
+        parser = ET.XMLPullParser(events=('comment', 'pi'),
+                                  target=self.SimpleTarget())
+        self._feed(parser, "<root><!-- text --><?pitarget data?></root>")
+        self.assert_event_tuples(parser, [
+            ('comment', ('comment', ' text ')),
+            ('pi', ('pi', 'pitarget')),
+        ])
+
+    def test_custom_target_without_method(self):
+        class Target:
+            def close(self):
+                pass
+        for event in ('start', 'end', 'comment', 'pi'):
+            with self.subTest(event=event):
+                with self.assertRaisesRegex(TypeError,
+                        "the target does not support %r events" % event):
+                    ET.XMLPullParser(events=(event,), target=Target())
+        # the namespace events do not need methods of the target
+        parser = ET.XMLPullParser(events=('start-ns', 'end-ns'),
+                                  target=Target())
+        self._feed(parser, "<root xmlns='namespace' />")
+        self.assert_event_tuples(parser, [
+            ('start-ns', ('', 'namespace')),
+            ('end-ns', None),
+        ])
+
+    def test_custom_target_ns_events(self):
+        # the target does not implement start_ns()/end_ns(),
+        # so the prefix and the uri are reported
+        parser = ET.XMLPullParser(events=('start-ns', 'end-ns'),
+                                  target=self.SimpleTarget())
+        self._feed(parser, "<root xmlns='namespace' />")
+        self.assert_event_tuples(parser, [
+            ('start-ns', ('', 'namespace')),
+            ('end-ns', None),
+        ])
+
+    def test_custom_target_close(self):
+        parser = ET.XMLPullParser(events=('end',), target=self.SimpleTarget())
+        self._feed(parser, "<root/>")
+        parser.close()
+        self.assert_event_tuples(parser, [('end', ('end', 'root'))])
+
     def test_unknown_event(self):
         with self.assertRaises(ValueError):
             ET.XMLPullParser(events=('start', 'end', 'bogus'))

Lib/xml/etree/ElementTree.py

@@ -1239,7 +1239,7 @@ def parse(source, parser=None):
     return tree
 
 
-def iterparse(source, events=None, parser=None):
+def iterparse(source, events=None, parser=None, *, target=None):
     """Incrementally parse XML document into ElementTree.
 
     This class also reports what's going on to the user based on the
@@ -1250,14 +1250,14 @@ def iterparse(source, events=None, parser=None):
 
     *source* is a filename or file object containing XML data, *events* is
     a list of events to report back, *parser* is an optional parser
-    instance.
+    instance, *target* is an optional target of the standard parser.
 
     Returns an iterator providing (event, elem) pairs.
 
     """
     # Use the internal, undocumented _parser argument for now; When the
     # parser argument of iterparse is removed, this can be killed.
-    pullparser = XMLPullParser(events=events, _parser=parser)
+    pullparser = XMLPullParser(events=events, target=target, _parser=parser)
 
     if not hasattr(source, "read"):
         source = open(source, "rb")
@@ -1309,13 +1309,19 @@ def __del__(self, _warn=warnings.warn):
 
 class XMLPullParser:
 
-    def __init__(self, events=None, *, _parser=None):
+    def __init__(self, events=None, *, target=None, _parser=None):
         # The _parser argument is for internal use only and must not be relied
         # upon in user code. It will be removed in a future release.
         # See https://bugs.python.org/issue17741 for more details.
 
         self._events_queue = collections.deque()
-        self._parser = _parser or XMLParser(target=TreeBuilder())
+        if _parser is None:
+            if target is None:
+                target = TreeBuilder()
+            _parser = XMLParser(target=target)
+        elif target is not None:
+            raise ValueError("can't specify both parser and target")
+        self._parser = _parser
         # wire up the parser for event reporting
         if events is None:
             events = ("end",)
@@ -1611,6 +1617,10 @@ def _setevents(self, events_queue, events_to_report):
         parser = self._parser
         append = events_queue.append
         for event_name in events_to_report:
+            if (event_name in ("start", "end", "comment", "pi")
+                    and not hasattr(self.target, event_name)):
+                raise TypeError("the target does not support %r events"
+                                 % event_name)
             if event_name == "start":
                 parser.ordered_attributes = 1
                 def handler(tag, attrib_in, event=event_name, append=append,
@@ -1643,13 +1653,14 @@ def handler(prefix, event=event_name, append=append):
                         append((event, None))
                 parser.EndNamespaceDeclHandler = handler
             elif event_name == 'comment':
-                def handler(text, event=event_name, append=append, self=self):
-                    append((event, self.target.comment(text)))
+                def handler(text, event=event_name, append=append,
+                            comment=self.target.comment):
+                    append((event, comment(text)))
                 parser.CommentHandler = handler
             elif event_name == 'pi':
                 def handler(pi_target, data, event=event_name, append=append,
-                            self=self):
-                    append((event, self.target.pi(pi_target, data)))
+                            pi=self.target.pi):
+                    append((event, pi(pi_target, data)))
                 parser.ProcessingInstructionHandler = handler
             else:
                 raise ValueError("unknown event %r" % event_name)

Misc/NEWS.d/next/Library/2026-09-01-02-30-00.gh-issue-63102.Pz3wK8.rst

@@ -0,0 +1,6 @@
+:class:`~xml.etree.ElementTree.XMLPullParser` and
+:func:`~xml.etree.ElementTree.iterparse` now support the *target* parameter.
+The reported object is the value returned by the corresponding method
+of the target, so no tree is built if the target does not build one.
+Only the standard :class:`~xml.etree.ElementTree.TreeBuilder` was supported
+in the C implementation before.