Repository navigation
audiograph: tables in pages, so a large patch fits a fragmented heap - #328
Merged
Merged
Conversation
A graph used to ask for its node table, its control queue, the get() shadow and its buffers as four single blocks, each growing with the patch. On a MicroPython heap with megabytes free but no free block larger than 33 KB, a 711-node patch could not get its 74 KB node table. The nodes, the records and the shadow now live in pages of about 4 KB behind small arrays of page pointers, and each buffer has its own pointer, with the MicroPython and CircuitPython bindings allocating the buffers a few KB at a time. The shared code reaches them through audiodsp_graph_node(), audiodsp_graph_record() and audiodsp_graph_shadow(). CPython keeps one block for each table and points the pages into it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A graph's node table, control queue,
get()shadow and buffers used to be four single blocks, each sized by the patch. A large patch then needs one large block, which a fragmented MicroPython heap may not have even when plenty of memory is free.Now the nodes, the records and the shadow are kept in pages of about 4 KB, each behind a small array of page pointers, and every buffer has a pointer of its own. The MicroPython and CircuitPython bindings allocate the buffers a few KB at a time. The shared code reaches all of it through
audiodsp_graph_node(),audiodsp_graph_record()andaudiodsp_graph_shadow(). CPython keeps one block per table and points the pages into it.The case that prompted it is the PolyA6 patch in bdbarnett/vrack#52: 711 nodes, 104 buffers and a 5,688-record queue, built on MicroPython's default 8 MB unix heap. At
Graph()the heap had 5.0 MB free, but its largest free block was 33 KB, and the node table alone was 74 KB. Before this change it raisedMemoryError. With it, the patch compiles from the same heap state.Checks:
tests/parity/verify_graph.pywith CPython and MicroPython: runner, nodes and source agree with 0 samples differing. Its planted--fault drainstill fails.tests/test_cpython_audiograph*.py: 47 passed. The newtest_tables_that_span_many_pagescovers 101 nodes across four node pages and 1,400 records wrapping a 600-record queue twice. Pointing the shadow lookup at page 0 only makes it fail.The CircuitPython spike files are edited to match, but they aren't compiled here.