# Is it possible to call back to Mathematica when using Python through ExternalEvaluate?

ExternalEvaluate makes it possible to call Python code from Mathematica. The Wolfram Client for Python makes it possible to call Mathematica from Python.

When using ExternalEvaluate, is it possible to call back to the same Mathematica kernel?

For the sake of example, suppose I have a Python function nint that does numerical integration. It takes another function as argument—the function to be integrated. Is there a way to use the Python function nint through ExternalEvaluate/ExternalFunction to integrate some function f that was defined in Mathematica?

To be clear, this is just an example to help explain the problem. I am not actually looking to do numerical integration, and at this moment performance (which would be a big deal for practical numerical integration) is not of concern to me.

• Do you mean something more complex than func = Echo; ExternalEvaluate["Python", "from wolframclient.language import wl; wl.Global.func(3)"]? Aug 28 '20 at 15:38
• @JasonB. This is not calling back to Mathematica. It is merely returning func[3], which evaluates after return. I basically want to evaluate stuff before the Python function returns, and I want to evaluate it in the calling kernel (not in a separate, independent kernel). Aug 28 '20 at 15:58
• @JasonB. For the sake of example, evaluate func[1] and func[2] while in the Python function, then compute the sum of results in Python, then return the result to Mathematica. Aug 28 '20 at 15:58
• My sense is that the main issue is connecting back to the evaluating kernel? It looks like this communication happens through ZMQ, so you'll likely need to open a ZMQ connection on this side, connect through this WolframKernelController object, and then talk through a session wrapped around that. It seems like an interesting little project, but probably gonna be a pain to get the bi-directionality working well. Aug 28 '20 at 20:48
• I want to evaluate it in the calling kernel ExternalEvaluate doesn't complete until python returns, so at the time of the callback wouldn't this just put the Mathematica function to call at the end of the evaluation queue? i.e it will not run until ExternalEvaluate has finished. I can't see a way around it without launching a new kernel, assuming ExternalEvaluate is atomic like this and Mathematica is not interruptable. Aug 29 '20 at 17:02

I'm Riccardo Di Virgilio, one of the developers of ExternalEvaluate and wolframclientforpython.

There is a way to do what you want, even if the function is not documented or exposed, so the code I'm giving to you might break in future versions.

try to run this in ExternalEvaluate:

from wolframclient.language.side_effects import wl_side_effect
from wolframclient.language import wl

wl_side_effect(wl.Print(2))
wl_side_effect(wl.Print(3))
wl_side_effect(wl.Print(4))

"actual result"


So basically we have a standard python logger that can send messages back to the original Kernel. The kernel receives the message and keeps waiting for the result of the computation.

If you are writing something complex I suggest you to take a look at the externalevaluate code, which is public.

https://github.com/WolframResearch/WolframClientForPython/blob/master/wolframclient/utils/externalevaluate.py

https://github.com/WolframResearch/WolframClientForPython/blob/master/wolframclient/language/side_effects.py

• Thanks for the answer! One issue is that wl_side_effect does not appear to return anything. Thus, we can use this to print message in WL, but not to use the WL kernel for computations. Aug 30 '20 at 10:59
• Right, as the name suggests the function is sending a pure side effect to the kernel, it doesn't wait for a reply, just dropping a message back. That is used internally to forward stdout to the kernel. If you need a response from the Kernel what is wrong with running ExternalEvaluate, return something, do a computation with it and call ExternalEvaluate again? Aug 31 '20 at 11:40
• There are things that cannot be accomplished this way. Assume we have a Python function which takes a callback as the argument. Integration is such a case. Aug 31 '20 at 16:02
• thanks for the feedback we will discuss a way to handle this usecase. Sep 1 '20 at 17:24
• Thanks for the answer! Sep 1 '20 at 17:57

Yes. It's mentioned in the document of ExternalEvaluate

The command "cmd" may contain an inline template expression <*expr*>. This evaluates expr before "cmd" is sent to the external evaluator.

that we could use TemplateExpression in the string passed to ExternalEvaluate. A basic example :

ExternalEvaluate["Python", "<* Range@5 *>"]


However, this method fails when you try to define a function in python session, because the <* expr *> is evaluated before the "cmd" is sent to the external evaluator.

A solution to this is defining the function in Mathematica with SetDelayed,and then call the function in python with <* *>. An example is demonstrated in this answer of mine.

Not an answer, but an extended comment since I am also very interested in this. The following example is inspired by the last example in this blog post on running julia and python together.

Consider a recursive formulation of the Fibonacci sequence as follows:

wlFib2[n_, fib_] := If[n <= 2, 1, fib[n - 1, wlFib2] + fib[n - 2, wlFib2]]


with the slight caveat that we're passing a second function as an argument to recurse with. We can then define a second function which does the same, and also prints either (W1 or (W2 when it's switching back and from the two functions. Purely in the WolframLanguage this (naturally) works:

wlFib1[n_, fib_] := Block[{result},
WriteString["stdout", "(W1"];
If[n <= 2,
result = 1,
WriteString["stdout", "(W2"];
result = fib[n - 1, wlFib1] + fib[n - 2, wlFib1];
WriteString["stdout", ")"];];
WriteString["stdout", ")"];
result]

In[1]:= wlFib1[6, wlFib2]
During evaluation of In[1]:=(W1(W2(W1(W2(W1)(W1)))(W1(W2))(W1(W2))(W1)))
Out[1]= 8


The question is, can we define one of these functions in python and communicate back and from? Here's an non-working attempt:

from wolframclient.language import wl
from wolframclient.language import wlexpr
from wolframclient.evaluation import WolframLanguageSession

wlSession = WolframLanguageSession()
wlSession.start()

def pyfib(n, wlfib):
print('(P', end='')
if n <= 2:
r = 1
else:
print('(W', end='')
# here we tell WL (wlfib) to recurse using Python
r =  wlfib(n-1, pyfib) + wlfib(n-2, pyfib)
print(')',end='')
print(')',end='')
return r

wlSession.evaluate(wlexpr('''
wlFib[n_, pyFib_] := If[n <= 2, 1, pyFib[n - 1, wlFib] + pyFib[n - 2, wlFib]]
'''))
wlfib = wlSession.function(wlexpr('wlFib'))

pyfib(6,wlfib)


which fails with the following exception:

NotImplementedError: Cannot serialize object of class <class 'function'>


the source code for which is found here. Looking through the source code a bit, it seems that maybe one could write an Encoder with allow_external_objects=True? but my attempts have so far been fruitless..

Maybe with this? https://reference.wolfram.com/language/WolframClientForPython/

Usage:

For functions defined in Mathematica:

Two ways:

1.

from wolframclient.evaluation import WolframLanguageSession
from wolframclient.language import wl, wlexpr
session = WolframLanguageSession()
Eval = session.evaluate

def sin(x):
return Eval(Eval(wl.Sin)(x))

>>> sin(wl.Plus(wl.Pi,2))
Times[-1, Sin[2]]


Notice FullForm is needed.

2.

def sinPlusCos(x):
return Eval(Eval(wlexpr("Sin@#+Cos@#&"))(x))


Use wlexpr and pure function.

In Mathematica, here is a small example:

ExternalEvaluate["Python", {"from wolframclient.evaluation import \
WolframLanguageSession",
"from wolframclient.language import wl, wlexpr",
"session = WolframLanguageSession()",
"Eval = session.evaluate",
"def sin(x): return Eval(Eval(wl.Sin)(x))", "sin(wl.Pi)"}]


{Null, Null, Null, Null, Null, 0}

"sin(wl.Pi)" can be replace by other things, for example: "sin(2.5)"

{Null, Null, Null, Null, Null, 0.598472}

• This does not call back to Mathematica. It starts a separate Mathematica kernel and evaluates something in that. Aug 29 '20 at 7:00