How to assemble Python abstract syntax trees (AST) in Mathematica?

Question

I would like to assemble Python 'ast.Module' objects inside of Mathematica and then submit them to Python via e.g. ExternalEvaluate["exec(astObject)"].

The Python ast, parse, eval, exec and compile functions can all operate on 'ast.Module' objects, either outputting them or taking them as inputs. However I don't know how to assemble this astObject within Mathematica/WL and then send it over to Python via ExternalEvaluate.

I am trying to programmatically generate Python code in MMA (for a genetic algorithm) and then submit it to Python for evaluation. I could assemble Python code strings, but then I have to handle all of the indentation, which seems like a pain.

For example, in Python it is possible to do the following:

import ast

pythonString="X3=X1*X2"
astObject=ast.parse(pythonString)
X1=3
X2=4
exec(compile(astObject,"","exec"))
print(X3)

-> 12

And of course from MMA it is possible to do:

session=StartExternalSession["Python"]
ExternalEvaluate[session, {"import ast","X1=3","X2=4",
 "exec(compile(ast.parse(\"X3=X1*X2\"),\"\",\"exec\"))"} ]

to yield the same result (i.e. 12).

However I want to generate my bits of Python code ("X1=3","X2=4","X3=X1*X2") in Mathematica. These bits here are simple enough, but I intend to generate complete programs, i.e. statements and expressions, metaprogrammatically(!). To do that I then have to figure out how to parse Python's annoying indentations, which is of course how it distinguishes one set of expressions from the next and what their dependencies are. I am loath to do so, and figured that it might be easier to operate on the ast structure.

Originally I had thought I might be able to use an intermediate string-form from Python's ast.dump() function which looks like:

astString = ast.dump(pythonString)
-> "Module(Body=[Assign(targets=[Name(id='X3',ctx=Store())],value=BinOp(left=Name(id='X1',
ctx=Load()),op=Mult(),right=Name(id='X2',ctx=Load())))])"

and since this astString essentially serializes the astObject I could also generate this instead. However I cannot find any way of getting Python to do anything with this astString.

Is it possible to create this sort of Python object - like my astObject above - on the Mathematica side?

B

PS: Here is a description of the 'ast.Module' objects: https://greentreesnakes.readthedocs.io/en/latest/tofrom.html

PPS: I have cross-posted this on Wolfram Community: https://community.wolfram.com/groups/-/m/t/2070851

Answer 1

I think that compiling Python code as a string is actually simpler than what you propose. But I also know that me just saying it won't convince anyone, so here's an example.

We define a couple of symbolic heads that will represent our Python program in Mathematica, and a function to render expressions with those symbolic heads:

ToPythonString[statements_List] := StringRiffle[ToPythonString /@ statements, "\n"]

ToPythonString[PyBlock[statement_, children_, indent_ : 0]] := StringJoin[
  StringRepeat["    ", indent],
  statement, ":\n",
  ToPythonString[PyIndent /@ children]
  ]

ToPythonString[PyStatement[statement_, indent_ : 0]] := StringJoin[
  StringRepeat["    ", indent],
  statement
  ]

PyIndent[PyBlock[statement_, children_, indent_ : 0]] := PyBlock[
  statement,
  PyIndent /@ children,
  indent + 1
  ]

PyIndent[PyStatement[statement_, indent_ : 0]] := PyStatement[
  statement,
  indent + 1
  ]

What these functions allow us to do is to write Python code in Mathematica without thinking about the indentation, it's a bit like building Python code with the ast module.

This is an example of rendering the symbolic expression as a string:

prog = {
   PyStatement["a = 1"],
   PyStatement["b = 2"],
   PyBlock["If a > b", {
     PyStatement["Print('a is larger than b')"]
     }],
   PyBlock["def f(x)", {
     PyStatement["Print('executing f')"],
     PyBlock["if x > 0", {
       PyStatement["Print('x is larger than 0')"]
       }]
     }]
   };

ToPythonString[prog]

Out:

a = 1
b = 2
If a > b:
    Print('a is larger than b')
def f(x):
    Print('executing f')
    if x > 0:
        Print('x is larger than 0')

We can easily build on this and make our symbolic representation of the Python program more descriptive.

PyAssign[lhs_, rhs_] := PyStatement[lhs <> " = " <> rhs]
PyPrint[text_] := PyStatement["Print(" <> text <> ")"]

PyFunction[name_, args_, statements_] := PyBlock[
  "def " <> name <> "(" <> StringRiffle[args, ", "] <> ")",
  statements
  ]

PyIf[cond_, statements_] := PyBlock[
  "If " <> cond,
  statements
  ]

PyIf[cond_, statements_, elseStatements_] := {
  PyBlock[
   "If " <> cond,
   statements
   ],
  PyBlock[
   "else",
   elseStatements
   ]
  }

With these helper definitions, we can now write the following program in a very readable style.

prog = {
   PyAssign["a", "1"],
   PyAssign["b", "2"],
   PyIf[
    "a > b", {
     PyPrint["a is larger than b"]
     }],
   PyFunction["f", {"x"},
    PyIf[
     "x > 0",
     {PyPrint["x is larger than 0"]},
     {PyPrint["x is not larger than 0"]}
     ]
    ]
   };

ToPythonString[prog]

Out:

a = 1
b = 2
If a > b:
    Print(a is larger than b)
def f(x):
    If x > 0:
        Print(x is larger than 0)
    else:
        Print(x is not larger than 0)

If you haven't yet, please look up "symbolic C" in the Mathematica documentation. It is basically a way to build an AST for a C language program in Mathematica which can then be converted into runnable C code. That is basically where we are headed with this code as well, although if I intended to make a complete implementation like that it would not look exactly like this (the ast module is certainly worth studying if one wants to go down the route).

Back to the point: what I want to convey with this answer is that you do not have to spend a lot of time to build a small framework that more or less solves the indentation problem that you mention in your question.

Answer 2

I am not sure what exactly you are looking for. I think the answer of your question:

I am trying to programmatically generate Python code in MMA [...] and then submit it to Python for evaluation. I could assemble Python code strings, but then I have to handle all of the indentation, which seems like a pain.

Is it possible to create this sort of Python object on the Mathematica side?

is pretty straightforward using ExternalEvaluate and Python's "ast" library.

Here is an example:

code = "'[i**2 for i in range(10)]'";

astTemplate = 
  StringTemplate["import ast; eval(compile(ast.parse(`1`, mode='eval'), '', 'eval'))"];

astTemplate[code]

(* "import ast; eval(compile(ast.parse('[i**2 for i in range(10)]', mode='eval'), '', 'eval'))" *)

ExternalEvaluate["Python", astTemplate[code]]

(* {0, 1, 4, 9, 16, 25, 36, 49, 64, 81} *)

(I used eval instead of exec, because eval returns a value.)