gh-68475: Add DocumentBuilder instead of TreeBuilder.get_document_children()

TreeBuilder is unchanged: it discards comments and processing instructions
outside of the root element, and close() returns the root element.
The new DocumentBuilder subclass keeps them (when insert_comments or
insert_pis is true) and returns the list of the children of the document
from close(), the only result channel of the target protocol.
ElementTree.parse() loads such a list into children, XMLID() accepts it.

In C, the DocumentBuilder type shares the TreeBuilder struct and handlers;
the document list is NULL for a plain TreeBuilder.  The parser calls the
handlers directly for both exact types.

Author: serhiy-storchaka

Doc/library/xml.etree.elementtree.rst

@@ -609,15 +609,17 @@ Functions
    Parses an XML section from a string constant.  Same as :func:`XML`.  *text*
    is a string containing XML data.  *parser* is an optional parser instance.
    If not given, the standard :class:`XMLParser` parser is used.
-   Returns an :class:`Element` instance.
+   Returns an :class:`Element` instance
+   (the result of :meth:`XMLParser.close` with a custom *parser*).
 
 
 .. function:: fromstringlist(sequence, parser=None)
 
    Parses an XML document from a sequence of string fragments.  *sequence* is a
    list or other sequence containing XML data fragments.  *parser* is an
    optional parser instance.  If not given, the standard :class:`XMLParser`
-   parser is used.  Returns an :class:`Element` instance.
+   parser is used.  Returns an :class:`Element` instance
+   (the result of :meth:`XMLParser.close` with a custom *parser*).
 
    .. versionadded:: 3.2
 
@@ -815,7 +817,8 @@ Functions
    Parses an XML section from a string constant.  This function can be used to
    embed "XML literals" in Python code.  *text* is a string containing XML
    data.  *parser* is an optional parser instance.  If not given, the standard
-   :class:`XMLParser` parser is used.  Returns an :class:`Element` instance.
+   :class:`XMLParser` parser is used.  Returns an :class:`Element` instance
+   (the result of :meth:`XMLParser.close` with a custom *parser*).
 
 
 .. function:: XMLID(text, parser=None)
@@ -824,7 +827,11 @@ Functions
    which maps from element id:s to elements.  *text* is a string containing XML
    data.  *parser* is an optional parser instance.  If not given, the standard
    :class:`XMLParser` parser is used.  Returns a tuple containing an
-   :class:`Element` instance and a dictionary.
+   :class:`Element` instance (the result of :meth:`XMLParser.close` with
+   a custom *parser*) and a dictionary.
+
+   .. versionchanged:: next
+      Support a *parser* whose target is a :class:`DocumentBuilder`.
 
 
 .. _elementtree-xinclude:
@@ -1274,6 +1281,12 @@ ElementTree Objects
       name or :term:`file object`.  *parser* is an optional parser instance.
       If not given, the standard :class:`XMLParser` parser is used.  Returns the
       section root element.
+      If the target of the parser is a :class:`DocumentBuilder`,
+      the comments and processing instructions outside of the root element
+      are loaded too, into :attr:`children`.
+
+      .. versionchanged:: next
+         Added support for :class:`DocumentBuilder`.
 
 
    .. method:: write(file, encoding="us-ascii", xml_declaration=None, \
@@ -1388,12 +1401,8 @@ TreeBuilder Objects
    create comments and processing instructions.  When not given, the default
    factories will be used.  When *insert_comments* and/or *insert_pis* is true,
    comments/pis will be inserted into the tree if they appear within the root
-   element.  Those which appear outside of it are returned by
-   :meth:`get_document_children`.
-
-   .. versionchanged:: next
-      Comments and processing instructions outside the root element
-      are no longer discarded.
+   element.  Those which appear outside of it are discarded;
+   use :class:`DocumentBuilder` to keep them.
 
    .. method:: close()
 
@@ -1406,16 +1415,6 @@ TreeBuilder Objects
       Adds text to the current element.  *data* is a string.
 
 
-   .. method:: get_document_children()
-
-      Returns the children of the document:
-      the root element, and the comments and processing instructions
-      which were inserted outside of it.
-      Returns a list of :class:`Element` instances.
-
-      .. versionadded:: next
-
-
    .. method:: end(tag)
 
       Closes the current element.  *tag* is the element name.  Returns the
@@ -1473,6 +1472,27 @@ TreeBuilder Objects
       .. versionadded:: 3.8
 
 
+.. class:: DocumentBuilder(element_factory=None, *, comment_factory=None, \
+                           pi_factory=None, insert_comments=False, \
+                           insert_pis=False)
+
+   A :class:`TreeBuilder` which builds the whole document, not only the tree
+   of the root element.
+   The arguments are the same as for :class:`TreeBuilder`.
+   When *insert_comments* and/or *insert_pis* is true,
+   comments/pis which appear outside of the root element are kept
+   as the children of the document.
+
+   .. versionadded:: next
+
+   .. method:: close()
+
+      Flushes the builder buffers, and returns the children of the document:
+      the root element, and the comments and processing instructions
+      which were inserted outside of it.
+      Returns a list of :class:`Element` instances.
+
+
 .. class:: C14NWriterTarget(write, *, \
              with_comments=False, strip_text=False, rewrite_prefixes=False, \
              qname_aware_tags=None, qname_aware_attrs=None, \

Doc/whatsnew/3.16.rst

@@ -707,8 +707,8 @@ xml
   :class:`~xml.etree.ElementTree.ElementTree` now has the
   :attr:`~xml.etree.ElementTree.ElementTree.children` attribute,
   a sequence of the children of the document,
-  and :class:`~xml.etree.ElementTree.TreeBuilder` returns them
-  from the new :meth:`!get_document_children` method
+  and the new :class:`~xml.etree.ElementTree.DocumentBuilder` parser target
+  returns them from its ``close()`` method
   when *insert_comments* or *insert_pis* is true.
   (Contributed by Serhiy Storchaka in :gh:`68475`.)
 

Lib/test/test_xml_etree.py

@@ -4104,7 +4104,7 @@ class DocumentChildrenTest(unittest.TestCase):
     sample = ('<!--lead--><?pi data?><r><?in?><a/></r><?after?><!--tail-->')
 
     def parse(self, text=None):
-        builder = ET.TreeBuilder(insert_comments=True, insert_pis=True)
+        builder = ET.DocumentBuilder(insert_comments=True, insert_pis=True)
         tree = ET.ElementTree()
         tree.parse(io.StringIO(text if text is not None else self.sample),
                    ET.XMLParser(target=builder))
@@ -4117,6 +4117,14 @@ def test_only_the_root_by_default(self):
         self.assertEqual(len(tree.children), 1)
         self.assertIs(tree.children[0], tree.getroot())
 
+    def test_tree_builder_discards_the_prolog_and_the_epilog(self):
+        builder = ET.TreeBuilder(insert_comments=True, insert_pis=True)
+        tree = ET.ElementTree()
+        tree.parse(io.StringIO(self.sample), ET.XMLParser(target=builder))
+        self.assertEqual(summarize_list(tree.children), ['r'])
+        self.assertEqual(summarize_list(tree.getroot()),
+                         [ET.ProcessingInstruction, 'a'])
+
     def test_parse_keeps_the_prolog_and_the_epilog(self):
         tree = self.parse()
         self.assertEqual(summarize_list(tree.children),
@@ -4254,21 +4262,46 @@ def test_document_without_the_root(self):
         tree.write(file, encoding='unicode')
         self.assertEqual(file.getvalue(), '<!--a--><?p?>')
 
-    def test_builder_document(self):
-        builder = ET.TreeBuilder(insert_comments=True, insert_pis=True)
+    def test_document_builder(self):
+        builder = ET.DocumentBuilder(insert_comments=True, insert_pis=True)
         parser = ET.XMLParser(target=builder)
         parser.feed(self.sample)
-        root = parser.close()
-        self.assertEqual(root.tag, 'r')
-        self.assertEqual(len(builder.get_document_children()), 5)
-        self.assertIs(builder.get_document_children()[2], root)
+        document = parser.close()
+        self.assertIsInstance(document, list)
+        self.assertEqual(summarize_list(document),
+                         [ET.Comment, ET.ProcessingInstruction, 'r',
+                          ET.ProcessingInstruction, ET.Comment])
+        self.assertEqual(summarize_list(document[2]),
+                         [ET.ProcessingInstruction, 'a'])
 
-    def test_builder_document_without_inserting(self):
-        builder = ET.TreeBuilder()
+    def test_document_builder_without_inserting(self):
+        builder = ET.DocumentBuilder()
         parser = ET.XMLParser(target=builder)
         parser.feed(self.sample)
-        root = parser.close()
-        self.assertEqual(builder.get_document_children(), [root])
+        document = parser.close()
+        self.assertEqual(summarize_list(document), ['r'])
+        self.assertEqual(summarize_list(document[0]), ['a'])
+
+    def test_document_builder_subclass(self):
+        class Builder(ET.DocumentBuilder):
+            pass
+        builder = Builder(insert_comments=True)
+        parser = ET.XMLParser(target=builder)
+        parser.feed(self.sample)
+        self.assertEqual(summarize_list(parser.close()),
+                         [ET.Comment, 'r', ET.Comment])
+
+    def test_document_builder_xmlid(self):
+        parser = ET.XMLParser(target=ET.DocumentBuilder(insert_comments=True))
+        document, ids = ET.XMLID('<!--c--><r id="x"><a id="y"/></r>', parser)
+        self.assertEqual(summarize_list(document), [ET.Comment, 'r'])
+        self.assertEqual(sorted(ids), ['x', 'y'])
+        self.assertIs(ids['x'], document[1])
+
+    def test_document_builder_fromstring(self):
+        parser = ET.XMLParser(target=ET.DocumentBuilder(insert_comments=True))
+        document = ET.fromstring(self.sample, parser)
+        self.assertEqual(summarize_list(document), [ET.Comment, 'r', ET.Comment])
 
 class TreeBuilderTest(unittest.TestCase):
     sample1 = ('<!DOCTYPE html PUBLIC'

Lib/xml/etree/ElementTree.py

@@ -82,7 +82,7 @@
     "QName",
     "SubElement",
     "tostring", "tostringlist",
-    "TreeBuilder",
+    "TreeBuilder", "DocumentBuilder",
     "XML", "XMLID",
     "XMLParser", "XMLPullParser",
     "register_namespace",
@@ -698,15 +698,13 @@ def parse(self, source, parser=None):
                     return self._root
             while data := source.read(65536):
                 parser.feed(data)
-            # close() releases the target, ask it for the document first
-            document = getattr(getattr(parser, 'target', None),
-                               'get_document_children', None)
             result = parser.close()
-            # a custom target can return anything, even None
-            self._root = result
-            if document is not None:
-                self._children[:] = document()
+            if isinstance(result, list):
+                # a DocumentBuilder returns the children of the document
+                self.children[:] = result
             else:
+                # a custom target can return anything, even None
+                self._root = result
                 self._children = [result] if iselement(result) else []
             return self._root
         finally:
@@ -1546,10 +1544,13 @@ def XMLID(text, parser=None):
     parser.feed(text)
     tree = parser.close()
     ids = {}
-    for elem in tree.iter():
-        id = elem.get("id")
-        if id:
-            ids[id] = elem
+    # a DocumentBuilder returns the children of the document
+    nodes = tree if isinstance(tree, list) else [tree]
+    for node in nodes:
+        for elem in node.iter():
+            id = elem.get("id")
+            if id:
+                ids[id] = elem
     return tree, ids
 
 # Parse XML document from string constant.  Alias for XML().
@@ -1600,7 +1601,6 @@ def __init__(self, element_factory=None, *,
         self._elem = [] # element stack
         self._last = None # last element
         self._root = None # root element
-        self._document = [] # the children of the document
         self._tail = None # true if we're after an end tag
         if comment_factory is None:
             comment_factory = Comment
@@ -1649,7 +1649,6 @@ def start(self, tag, attrs):
             self._elem[-1].append(elem)
         elif self._root is None:
             self._root = elem
-            self._document.append(elem)
         self._elem.append(elem)
         self._tail = 0
         return elem
@@ -1692,19 +1691,39 @@ def _handle_single(self, factory, insert, *args):
             self._last = elem
             if self._elem:
                 self._elem[-1].append(elem)
-            else:
-                # outside the root element: the prolog or the epilog
-                self._document.append(elem)
             self._tail = 1
         return elem
 
-    def get_document_children(self):
-        """Return the children of the document.
 
-        These are the root element and the comments and processing
-        instructions which were inserted outside of it.
-        """
-        return list(self._document)
+class DocumentBuilder(TreeBuilder):
+    """Generic document structure builder.
+
+    This builder is like TreeBuilder, but its close() method returns
+    the list of the children of the document: the root element, and
+    the comments and processing instructions outside of it if
+    *insert_comments* or *insert_pis* is true.
+    """
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._document = [] # the children of the document
+
+    def close(self):
+        """Flush builder buffers and return the children of the document."""
+        assert len(self._elem) == 0, "missing end tags"
+        return self._document
+
+    def start(self, tag, attrs):
+        elem = super().start(tag, attrs)
+        if elem is self._root:
+            self._document.append(elem)
+        return elem
+
+    def _handle_single(self, factory, insert, *args):
+        elem = super()._handle_single(factory, insert, *args)
+        if insert and not self._elem:
+            # outside the root element: the prolog or the epilog
+            self._document.append(elem)
+        return elem
 
 
 # also see ElementTree and TreeBuilder

Misc/NEWS.d/next/Library/2026-08-31-12-00-00.gh-issue-68475.Nq7Vt2.rst

@@ -2,8 +2,8 @@
 the :attr:`~xml.etree.ElementTree.ElementTree.children` attribute,
 a sequence of the children of the document:
 the root element and the comments and processing instructions around it.
-When *insert_comments* or *insert_pis* is true,
-:class:`~xml.etree.ElementTree.TreeBuilder` no longer discards comments
-and processing instructions which occur outside of the root element;
-they are returned by the new :meth:`!get_document_children` method
-and are used by :func:`~xml.etree.ElementTree.parse`.
+The new :class:`~xml.etree.ElementTree.DocumentBuilder` parser target
+keeps comments and processing instructions which occur outside of the root
+element when *insert_comments* or *insert_pis* is true,
+and returns the children of the document from its ``close()`` method,
+which is used by :func:`~xml.etree.ElementTree.parse`.