Mage: Text Analysis Made Easy

Mage is an experimental tool for performing text analysis. It does so by generating a lexer, parser and parse tree for you. Whether it is a piece of programming code or some tabular data in a fringe format, Mage has got you covered!

Features

• ✅ A simple yet expressive DSL to write your grammars in
• ✅ Full support for Python typings. Avoid runtime errors while building your language!
• 🚧 Lots of unit tests to enure your code does what you expect it to do
• 🚧 An intermediate language to very easily add support to any other programming language

👀 Mage is written in itself. Check out the generated code of part of our Python generator!

Implementation Status

Feature	Python	Rust	C	C++	JavaScript
CST	✅	⏳	⏳	⏳	⏳
AST	⏳	⏳	⏳	⏳	⏳
Lexer	🚧	⏳	⏳	⏳	⏳
Parser	⏳	⏳	⏳	⏳	⏳
Emitter	⏳	⏳	⏳	⏳	⏳

Installation

$ pip3 install --user -U magelang

Usage

Currently requires at least Python version 3.12 to run.

mage generate <lang> <filename>

Generate a parser for the given grammar in a language that you specify.

Example

mage generate python foo.mage --prefix foo --out-dir src/foolang

🚧 mage test <filename..>

Warning

This command is under construction.

Run all tests inside the documentation of the given grammar.

mage fuzz [filename]

Fuzz the given grammar using a pseudorandom number generator for replayable results.

mage fuzz grammars/magedown.mage

Warning

If you let mage fuzz --all run, Mage may write a lot of data to disk.

We recommend creating a temporary file system. A size of 64M should be more than enough. For example:

mkdir -p output/fuzz
sudo mount -t tmpfs tmpfs -o size=64m output/fuzz
mage fuzz --all

When you're done run sudo umount output/fuzz to unmount it.

Language Reference

The language reference can be found in docs/reference.md.

Python API

This section documents the API that is generated by taking a Mage grammar as input and specifying python as the output language.

In what follows, Node is the name of an arbitrary CST node (such as PyReturnStmt or MageRepeatExpr) and foo and bar are the name of fields of such a node. Examples of field names are expr, return_keyword, min, max,, and so on.

Node(...)

Construct a node with the fields specified in the ... part of the expression.

First go all elements that are required, i.e. they weren't suffixed with ? or * in the grammar or something similar. They may be specified as positional arguments or as keyword.

Next are all optional arguments. They must be specified as keyword arguments. When omitted, the corresponding fields are either set to None or a new empty token/node is created.

Examples

Creating a new CST node by providing positional arguments for required fields:

PyInfixExpr(
    PyNamedExpr('value'),
    PyIsKeyword(),
    PyNamedExpr('None')
)

The same example but now with keyword arguments:

PyInfixExpr(
    left=PyNamedExpr('value'),
    op=PyIsKeyword(),
    right=PyNamedExpr('None')
)

Omitting fields that are trivial to construct:

# Note that `return_keyword` is not specified
stmt = PyReturnStmt(expr=PyConstExpr(42))

# stmt.return_keyword was automatically created
assert(isinstance(stmt.return_keyword, ReturnKeyword()))

Node.count_foos() -> int

This member is generated when there was a repetition in field foo such as the Mage expression '.'+

It returns the amount of elements that are actually present in the CST node.

FAQ

What is a CST, AST, visitor, and so on?

A CST is a collection of structures and enumerations that completely represent the source code that needs to be parsed/emitted.

An AST is an abstract representation of the CST. Mage can automatically derive a good AST from a CST.

A visitor is (usually) a function that traverses the AST/CST in a particular way. It is useful for various things, such as code analysis and evaluation.

A rewriter is similar to a visitor in that it traverses that AST/CST but also creates new nodes during this traversal.

A lexer or scanner is at it core a program that splits the input stream into separate tokens that are easy to digest by the parser.

A parser converts a stream of tokens in AST/CST nodes. What parts of the input stream are converted to which nodes usually depends on how the parser is invoked.

How do I assign a list of nodes to another node in Python without type errors?

This is probably due to this feature in the Python type checker, which prevents subclasses from being assigned to a more general type.

For small lists, we recommend making a copy of the list, like so:

defn = PyFuncDef(body=list([ ... ]))

See also this issue in the Pyright repository.

Contributing

Run the following command in a terminal to link the mage command to your checkout:

pip3 install -e '.[dev]'

License

This code is generously licensed under the MIT license.