[f-fix]

Add astpp, a self-inlining python macro preprocessor & runtime polyfill library (the AST parts run in big pythons, the runtime parts and the output both run in micropython too)

Summary

By adding astpp we make it possible to use exactly the same const(...) over in big python and in micropython, using the same syntax, and actually working at preprocessor-time. It also allows you to strip out debug code/assertions/docstrings.

The astpp preprocessor does a version of const compatible with mpy-cross / micropython, that also works on "big" pythons. In the preprocessed output the contants' values will be inlined. It also allows you to do "conditional omission" so that parts of your code won't end up in the output - this is great if you have some fluffier features that make sense for "big" python, but you want them gone from your micropython version.

It also supports -O/-O0/-O1/-O2/-O3/-OO optimization levels which do stuff like force __debug__ to be const(False), and strip out docstrings and such. Again, it's intended to reduce code size and token size.

It also allows you to optionally replace things, and I use this to rewrite @micropython.native and @micropython.viper to @keep - which then gets stripped off my the preprocessor - allowing to easily build a version which can be mpy-cross compiled generically without being target CPU specific.

In big python this module includes the preprocessor and a test suite too, but optimized builds omit the test suite and when you don't have the ast module the preprocessor itself won't be runnable. In any case, though, preprocessing your code actually gets rid of the astpp imports entirely and replaces the imported ports with small inlined replacements.

Testing

I only tested on the Linux i386 port of micropython, but on that one it seems to work fine:

micropython -c '
import mip
mip.install("github:f-fix/astpp/astpp.py")
from astpp import const, micropython
@micropython.native
def foo(bar):
    return dict(bar=bar)
_X=const(3)
print(foo(3))
'

Downloading github:f-fix/astpp/astpp.py to /home/bsittler/.micropython/lib
Copying: /home/bsittler/.micropython/lib/astpp.py
Done
{'bar': 3}

Trade-offs and Alternatives

The current alternative is to use micropython-specific imports and mpy-cross, but that provides equivalent benefits only on micropython. Using astpp those same benefits are realized on all pythons.

Generative AI

I used generative AI tools when creating this PR, but a human has checked the
code and is responsible for the code and the description above.

[bsittler]

The PyPI page is https://pypi.org/project/astpp/ and the github page is https://github.com/f-fix/astpp BTW. The README there has some more detailed examples.

[f-fix]

It's rebased to astpp 0.1.1 which means the micropython library version is now generated without manual editing:

astpp -O3 -D_OMIT_MAIN < ~/src/astpp/astpp.py > astpp.py &&
    black astpp.py

[f-fix]

it's rebased to 0.1.3 with more mpy-cross like constant replacement and fixed pass syntax.

[f-fix]

BTW I have been testing it with https://github.com/f-fix/xz_decompressor including on x86 micropython

$ LC_ALL=C wc -c < xz_decompressor.py
318878
$ python3 astpp.py -O3 < xz_decompressor.py  | LC_ALL=C wc -c
192210
$ python3 astpp.py -O3 -D_IS_DESKTOP_PY=False < xz_decompressor.py  | LC_ALL=C wc -c
177125
$ python3 astpp.py -O3 -D_IS_DESKTOP_PY=False -D_OMIT_MAIN < xz_decompressor.py  | LC_ALL=C wc -c
162023
$ python3 astpp.py -O3 -D_IS_DESKTOP_PY=False -D_OMIT_MAIN < xz_decompressor.py | mpy-cross -O3 -march=x86 - | LC_ALL=C wc -c
66922
$ python3 astpp.py -O3 -D_IS_DESKTOP_PY=False -D_OMIT_MAIN -Dmicropython.{native,viper}=keep < xz_decompressor.py | mpy-cross -O3 - | LC_ALL=C wc -c
47460