Skip to content

Commit f5b4289

Browse files
[3.14] gh-93618: Document the memory usage of the incremental parsers (GH-156763) (GH-157420)
It was said that iterparse() can be useful for reading a large document without holding it wholly in memory, but the tree is only built incrementally, it is not freed incrementally. Document how to remove the processed elements, and that a custom target does not build a tree at all. (cherry picked from commit e7a3937) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
1 parent e7e092b commit f5b4289

1 file changed

Lines changed: 35 additions & 2 deletions

File tree

Doc/library/xml.etree.elementtree.rst

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -162,8 +162,37 @@ some storage device. In such cases, blocking reads are unacceptable.
162162
Because it's so flexible, :class:`XMLPullParser` can be inconvenient to use for
163163
simpler use-cases. If you don't mind your application blocking on reading XML
164164
data but would still like to have incremental parsing capabilities, take a look
165-
at :func:`iterparse`. It can be useful when you're reading a large XML document
166-
and don't want to hold it wholly in memory.
165+
at :func:`iterparse`.
166+
167+
Note that both parsers build the tree incrementally: it is not freed
168+
incrementally, so every parsed element is kept until the whole document is
169+
read. To keep the memory usage low, get rid of the data which is not needed
170+
any more.
171+
172+
If the processed elements are large, it is enough to clear them.
173+
This works wherever they are in the tree,
174+
but the emptied elements are left in it::
175+
176+
for event, elem in ET.iterparse(source):
177+
if elem.tag == 'record':
178+
process(elem)
179+
elem.clear()
180+
181+
If an element has a large number of children,
182+
remove the processed children from it::
183+
184+
for event, elem in ET.iterparse(source, events=('start', 'end')):
185+
if event == 'start' and elem.tag == 'parent':
186+
parent = elem
187+
elif event == 'end' and elem.tag == 'child':
188+
process(elem)
189+
parent.remove(elem)
190+
191+
These examples are not universal,
192+
they only give an idea for two common cases.
193+
If you do not need a tree at all,
194+
parse with :class:`XMLParser` and a custom target instead;
195+
it is not built then, and nothing has to be removed.
167196

168197
Where *immediate* feedback through events is wanted, calling method
169198
:meth:`XMLPullParser.flush` can help reduce delay;
@@ -637,6 +666,10 @@ Functions
637666
for applications where blocking reads can't be made. For fully non-blocking
638667
parsing, see :class:`XMLPullParser`.
639668

669+
The tree is only built incrementally, it is not freed incrementally:
670+
every parsed element is kept until the whole document is read.
671+
See :ref:`elementtree-pull-parsing` for how to keep the memory usage low.
672+
640673
.. note::
641674

642675
:func:`iterparse` only guarantees that it has seen the ">" character of a

0 commit comments

Comments
 (0)