# How to assemble Python abstract syntax trees (AST) in Mathematica?

I would like to assemble Python 'ast.Module' objects inside of Mathematica and then submit them to Python via e.g. ExternalEvaluate["exec(astObject)"].

The Python ast, parse, eval, exec and compile functions can all operate on 'ast.Module' objects, either outputting them or taking them as inputs. However I don't know how to assemble this astObject within Mathematica/WL and then send it over to Python via ExternalEvaluate.

I am trying to programmatically generate Python code in MMA (for a genetic algorithm) and then submit it to Python for evaluation. I could assemble Python code strings, but then I have to handle all of the indentation, which seems like a pain.

For example, in Python it is possible to do the following:

import ast

pythonString="X3=X1*X2"
astObject=ast.parse(pythonString)
X1=3
X2=4
exec(compile(astObject,"","exec"))
print(X3)

-> 12


And of course from MMA it is possible to do:

session=StartExternalSession["Python"]
ExternalEvaluate[session, {"import ast","X1=3","X2=4",
"exec(compile(ast.parse(\"X3=X1*X2\"),\"\",\"exec\"))"} ]


to yield the same result (i.e. 12).

However I want to generate my bits of Python code ("X1=3","X2=4","X3=X1*X2") in Mathematica. These bits here are simple enough, but I intend to generate complete programs, i.e. statements and expressions, metaprogrammatically(!). To do that I then have to figure out how to parse Python's annoying indentations, which is of course how it distinguishes one set of expressions from the next and what their dependencies are. I am loath to do so, and figured that it might be easier to operate on the ast structure.

Originally I had thought I might be able to use an intermediate string-form from Python's ast.dump() function which looks like:

astString = ast.dump(pythonString)
-> "Module(Body=[Assign(targets=[Name(id='X3',ctx=Store())],value=BinOp(left=Name(id='X1',


and since this astString essentially serializes the astObject I could also generate this instead. However I cannot find any way of getting Python to do anything with this astString.

Is it possible to create this sort of Python object - like my astObject above - on the Mathematica side?

B

PS: Here is a description of the 'ast.Module' objects: https://greentreesnakes.readthedocs.io/en/latest/tofrom.html

PPS: I have cross-posted this on Wolfram Community: https://community.wolfram.com/groups/-/m/t/2070851

• "To do that I then have to figure out how to parse Python's annoying indentations [...]" -- I know you are asking AST to be used, but maybe answers that deal with Python's indentations are also of interest to you. Sep 10, 2020 at 16:02
• Yes, Anton, you are right, I am also looking at purely string-based solutions. And along those lines I was looking at your FunctionalParsers.m package as a way of automatically importing Python's EBNF grammar to perhaps allow me to generate pseudo-Python-with-explicit-bracketing, or to perhaps generate Hy code (which is Lisp written in Python) which of course has proper bracketing. I just can't understand why someone would create a language based on indentation, or why other's would then go on to use it... Sep 10, 2020 at 17:17
• "I just can't understand why someone would create a language based on indentation, or why other's would then go on to use it..." -- LOL ! Many people are mystified by that ! Sep 10, 2020 at 18:09
• Using "FunctionalParsers.m" is an interesting idea, but pursuing it might be far from an easy undertaking... Sep 10, 2020 at 18:13
• You might be better off programming two software monads, one in WL and another in Python, that have the same workflow design. Then you transfer between them with minor code changes; also no special formatting for Python is needed. Here is an example with a monad called LSAMon. (Scroll to the bottom.) Sep 10, 2020 at 19:25

I think that compiling Python code as a string is actually simpler than what you propose. But I also know that me just saying it won't convince anyone, so here's an example.

We define a couple of symbolic heads that will represent our Python program in Mathematica, and a function to render expressions with those symbolic heads:

ToPythonString[statements_List] := StringRiffle[ToPythonString /@ statements, "\n"]

ToPythonString[PyBlock[statement_, children_, indent_ : 0]] := StringJoin[
StringRepeat["    ", indent],
statement, ":\n",
ToPythonString[PyIndent /@ children]
]

ToPythonString[PyStatement[statement_, indent_ : 0]] := StringJoin[
StringRepeat["    ", indent],
statement
]

PyIndent[PyBlock[statement_, children_, indent_ : 0]] := PyBlock[
statement,
PyIndent /@ children,
indent + 1
]

PyIndent[PyStatement[statement_, indent_ : 0]] := PyStatement[
statement,
indent + 1
]


What these functions allow us to do is to write Python code in Mathematica without thinking about the indentation, it's a bit like building Python code with the ast module.

This is an example of rendering the symbolic expression as a string:

prog = {
PyStatement["a = 1"],
PyStatement["b = 2"],
PyBlock["If a > b", {
PyStatement["Print('a is larger than b')"]
}],
PyBlock["def f(x)", {
PyStatement["Print('executing f')"],
PyBlock["if x > 0", {
PyStatement["Print('x is larger than 0')"]
}]
}]
};

ToPythonString[prog]


Out:

a = 1
b = 2
If a > b:
Print('a is larger than b')
def f(x):
Print('executing f')
if x > 0:
Print('x is larger than 0')


We can easily build on this and make our symbolic representation of the Python program more descriptive.

PyAssign[lhs_, rhs_] := PyStatement[lhs <> " = " <> rhs]
PyPrint[text_] := PyStatement["Print(" <> text <> ")"]

PyFunction[name_, args_, statements_] := PyBlock[
"def " <> name <> "(" <> StringRiffle[args, ", "] <> ")",
statements
]

PyIf[cond_, statements_] := PyBlock[
"If " <> cond,
statements
]

PyIf[cond_, statements_, elseStatements_] := {
PyBlock[
"If " <> cond,
statements
],
PyBlock[
"else",
elseStatements
]
}


With these helper definitions, we can now write the following program in a very readable style.

prog = {
PyAssign["a", "1"],
PyAssign["b", "2"],
PyIf[
"a > b", {
PyPrint["a is larger than b"]
}],
PyFunction["f", {"x"},
PyIf[
"x > 0",
{PyPrint["x is larger than 0"]},
{PyPrint["x is not larger than 0"]}
]
]
};

ToPythonString[prog]


Out:

a = 1
b = 2
If a > b:
Print(a is larger than b)
def f(x):
If x > 0:
Print(x is larger than 0)
else:
Print(x is not larger than 0)


If you haven't yet, please look up "symbolic C" in the Mathematica documentation. It is basically a way to build an AST for a C language program in Mathematica which can then be converted into runnable C code. That is basically where we are headed with this code as well, although if I intended to make a complete implementation like that it would not look exactly like this (the ast module is certainly worth studying if one wants to go down the route).

Back to the point: what I want to convey with this answer is that you do not have to spend a lot of time to build a small framework that more or less solves the indentation problem that you mention in your question.

• Yes C.E. this looks very much like what I was imagining would be torture to implement, and you are right, you have made it quite straightforward. I will have to continue on in this fashion extending the language downwards towards the leaves to cover atoms and operators and lists and tuples and such, but this framework handles all of the indentation, and it should be doable. Sep 10, 2020 at 18:41
• @berniethejet In this example, I defined for example PyIf to something that evaluates into something else. You may consider to instead add the definition to ToPythonString, i.e. ToPythonString[PyIf[...]] :=. That way you can inspect your entire program in symbolic form, without having it being evaluated until you call ToPythonString. Sep 10, 2020 at 18:49
• Thanks, that is good to know. Your framework is certainly better than any I would have cobbled together (grumbling the whole way, as I would inevitably be, annoyed at Guido's indentation crusade). Sep 10, 2020 at 19:47

I am not sure what exactly you are looking for. I think the answer of your question:

I am trying to programmatically generate Python code in MMA [...] and then submit it to Python for evaluation. I could assemble Python code strings, but then I have to handle all of the indentation, which seems like a pain.

Is it possible to create this sort of Python object on the Mathematica side?

is pretty straightforward using ExternalEvaluate and Python's "ast" library.

Here is an example:

code = "'[i**2 for i in range(10)]'";

astTemplate =
StringTemplate["import ast; eval(compile(ast.parse(1, mode='eval'), '', 'eval'))"];

astTemplate[code]

(* "import ast; eval(compile(ast.parse('[i**2 for i in range(10)]', mode='eval'), '', 'eval'))" *)

ExternalEvaluate["Python", astTemplate[code]]

(* {0, 1, 4, 9, 16, 25, 36, 49, 64, 81} *)


(I used eval instead of exec, because eval returns a value.)

• Thanks Anton. Yes, I could have been more clear. Let me update my question a bit. Basically I want to generate the "'[i**2 for i in range(10)]'" part in your answer on the MMA-side, but not as a Python string, but as the ast.Module object that can plug into compile and eval and exec. Sep 10, 2020 at 2:24