base64: Reduce module size and improve base32 decoding speed. - #1154
Open
agatti wants to merge 3 commits into
Open
base64: Reduce module size and improve base32 decoding speed.#1154agatti wants to merge 3 commits into
agatti wants to merge 3 commits into
Conversation
This commit moves the test code present in the `base64` module to the already existing test harness. The module is able to be executed as a stand-alone script or via the `-m base64` command line parameter passed to the interpreter, to act as a simple base64 encoder/decoder utility. This is following the behaviour of the CPython's equivalent module. However unlike the CPython implementation, this module also presents a `-t` argument that performs some basic encoding and decoding tests. Given the existence of a test harness, the usefulness of this option is rather limited as test code is better placed in test scripts. It is also rather unlikely that this module is executed by regular users as a standalone entity with the `-t` command line argument (this is even more relevant for embedded targets, as this functionality needs the `getopt` module being available to function). Moving the bits of code in question shortens the byte-compiled version of the module by 192 bytes, with the test code still being executed as part of CI jobs. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit updates the base32 encoding and decoding functions in order to reduce the footprint of the base64 module when byte-compiled, lower the amount of memory taken by the module once loaded, and finally to speed encoding and decoding operations up. Before these changes the base32 symbols were stored in a dictionary and forward and reverse lookup tables were built as lists upon module import. The symbols are just the letter A to Z and the numbers 2 to 7 in sequence with no gaps, and each symbol maps to an integer between 0 and 31. A forward lookup table can be trivially made by creating a bytes object with the symbols in sequence, as an index lookup is the same as accessing the n-th byte in the bytes object. The reverse lookup table can be also precomputed as a 256 bytes long bytes object filled with a sentinel value for representing a non-match or otherwise with the integer index in the forward lookup table. That brings down the byte-compiled size by 106 bytes. Regarding the memory footprint, besides the raw 288 bytes of data to store, the overhead is much lower as there's only two `bytes` objects being created behind the scenes. Finally, whilst forward lookups were still done by accessing a list via an integer, reverse lookups involved a dictionary search. Accessing a bytes object via an index is probably faster than a dictionary lookup, improving base32 decode speed. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit updates the version number of the `base64` package, bumping up the minor version. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR performs a few minor updates to the
base64module to reduce on its footprint (both on flash and in RAM) and improve Base32 decoding speed.Base32 encoding/decoding support was updated to use a different method for symbols forward- and reverse-lookup. Rather than using two dictionaries and a list, two
bytesobjects are used. The forward lookup table was replaced with a singlebytesobject containing all symbols in sequence (since they were already mapped to 0..31, which incidentally is also the symbols' position in the data chunk), and the reverse lookup table was precomputed off-line to provide a 1:1 map between a byte value and its matching 0..31 index (if one is there).That reduces the byte-compiled size by 106 bytes: one of the side-effects of that optimisation is the reduced number of objects being created to handle the Base32 encoding/decoding, reducing the overhead on the running interpreter and shrinking the objects table of the compiled version of the module. The other being Base32 decoding should be faster since there's no need for a full dictionary lookup to figure out which 5-bits sequence a byte maps to (which was incidentally done twice in the original code). The Base32 part of the test suite was also updated to feature the same non-ASCII buffer doing an encoding/decoding round-trip, as its Base64 counterpart.
As its CPython equivalent,
base64is supposed to be able to run as a simple base64 encode/decode tool via the-m base64command line argument passed to the Python interpreter. However, unlike the CPython implementation, this module also presents a-targument that performs some basic encoding and decoding tests.Given the existence of a test harness, the usefulness of this option is rather limited as test code is better placed in test scripts. It is also rather unlikely that this module is executed by regular users as a standalone entity with the
-tcommand line argument (this is even more relevant for embedded targets, as this functionality needs thegetoptmodule being available to function). Moving the test code from the module to the test harness and updating the help output reduced the byte-compiled size by 192 bytes.The combined size savings of the two proposed changes amount to 298 bytes. Maybe I could have made it to 300 with some more time spent on it, but that's quite good nonetheless :)
Testing
The updated
python-stdlib/base64/test_base64.pyscript was run successfully on Linux/x64 using the current MicroPythonmaster. The test's output was also checked by running the test on CPython 3.14 once the existingbase64.pyfile was renamed (to let CPython use its ownbase64module).Trade-offs and Alternatives
The module could be even shorter by replacing the reverse base32 lookup table with a call to
bytes.findinstead, however that's going to impact on the base32 decoding performance. If that's not a concern then I will update the PR to do so.Generative AI
I did not use generative AI tools when creating this PR.