# Mathematica-Python interfacing: time lag in importing Python output in Mathematica

GOAL: I want to do a part of computation in Mathematica, write a text file (I call it INPUT.txt) containing the result from this partial calculation. Then I want to import this text file (INPUT.txt) in Python to do second part of computation, and write the result getting from Python to another text file (I call it OUTPUT.txt). This new text file (OUTPUT.txt) is then exported back to Mathematica to do further calculation.

EXAMPLE: Of course I want to do something fairly complicated thing, but for demonstration, I give a trivial example in the following.

Here is my simple Mathematica code of the process just explained above:

a = RandomReal[{-1, 1}];
b = RandomReal[{-2, 2}];
x = a + b;
y = a - b;
Export["~/Desktop/INPUT.txt", {x, y}, "Table",
"LineSeparators" -> " "];
process = StartProcess[\$SystemShell];
WriteLine[process, "cd ~/Desktop/"];
WriteLine[process, "python TEST.py"];
WriteLine[process, "Exit"];
fromPYTHON = Import["~/Desktop/OUTPUT.txt", "Table"];
{u, v, w} = fromPYTHON[[1]];
FINALresult = u + v + w


In this code, one can see that I am also running part of the computation in Python (which is saved as TEST.py). For now I keep the Python script as simple as possible:

SET=[]
with open('INPUT.txt') as LINE:
for line in LINE:
DATA = line.split()

vec=[float(DATA[0]),float(DATA[1])]
[x,y]=vec
u=x**2 -3*y
v=-x+1.4*y**3
w=x**3-y**3
SET.append( (float(u),float(v),float(w)) )

FILE = open("OUTPUT.txt", "w")
for A,B,C in SET:
FILE.write("{0}\t{1}\t{2}\n".format(A,B,C))
FILE.close()


ISSUE: Everything looks simple and straightforward. But the issue is, before the OUTPUT.txt is available, Mathematica already tries to import it, and fails. Since it takes some time (I do not know why) for the file to be available in the memory, there seems to be a time lag. Due to which Mathematica fails to import it back for the rest of the computation.

UNWANTED SOLUTION: I myself have a solution, which is, before importing the file, if I ask Mathemtica to Pause for a while, so that the memory has the file OUTPUT.txt ready, the issue goes away. Of course for realistic calculation, this long pause is not accepted.

So the modified code looks exactly the same, except the addition of Pause[1]:

....(rest of the code as before)....
WriteLine[process, "Exit"];
Pause[1];
fromPYTHON = Import["~/Desktop/OUTPUT.txt", "Table"];
....(rest of the code as before)....


QUESTION: (1) is this an issue with Python, or Mathematica? (2) why there is such a long time lag between writing the file by Python and importing back in Mathematica? (3) how can I resolve the issue without pausing my calculation? (4) please note that, when Python is importing the file generated by Mathematica, no such time lag seems to be happening. Python immediately imports, and do the calculation (as you can also verify running my codes).

CAUTION: do not get fooled by running the Mathematica file twice. The first time you run, it already generates the OUTPUT.txt file, which is now accessible to Mathematica if you hit the run button again. But note, this is the OUTPUT.txt file from the previous run. Be sure to delete the OUTPUT.txt file before you quit kernel, and re-run Mathematica.

• You probably want to send some command to 'process' and ask if it is done with the computation. Then, you continue with the mathematica program. Something like reference.wolfram.com/language/ref/TaskWait.html Jul 2, 2021 at 14:55
• I think you might be using the wrong tool for the job. If the only "interaction" with the process is to change the working directory, I would rather use RunProcess with the ProcessDirectory option and directly run the python executable instead of the shell. RunProcess will automatically wait for the started process to return. Use StartProcess only, when you explicitly do not want to wait for the process to finish... Jul 4, 2021 at 8:04
• Hi Albert Retey, do you mind give an example how to modify the relevant part of my code to use "RunProcess", instead of "StartProcess". I did try with Runprocess at first, but could not implement, hence moved to Startprocess. Jul 4, 2021 at 9:41

I would also recommend using ExternalEvaluate for this kind of interaction with python. But you can keep everything the way you have it with just a small change.

First I add a print statement at the end of the python script to signal when it has finished writing the file, just something like print("finished")

Then call ReadLine[process] after the last WriteLine call and before Import. ReadLine will block the kernel until it has received something.

• Thanks. Following your method, I see that the issue is resolved. But I also see that there is an unnecessarily long time it takes to write the file and to import back to Mathematica (which confirms my earlier claim). Do you have any idea why this time lag? Naively I would expect writing a txt file should be done immediately. Jul 2, 2021 at 16:58
• I am accepting your answer. The answer provided by George Varnavides also works. But for some reason ExternalEvaluate[] is taking much longer compared to the original code I wrote, combined with your proposed solution of adding the print option. Thank you again. Jul 2, 2021 at 18:34
• Glad I could help. Have you run timing on the python script from outside of Mathematica? In your code, without a Pause or ReadLine, Mathematica would evaluate Import immediately after calling the python script. If running python TEST.py takes longer than microseconds I would expect the behavior you are seeing here. Jul 2, 2021 at 18:40

Look at ExternalEvaluate if you have v12.0? The python computations will by design be 'blocking' using this approach, i.e. the Wolfram Kernel will wait on the ExternalEvaluate to finish.

Here's your simple example using TemplateExpression:

session = StartExternalSession["Python"];
ExternalEvaluate[session, "
x = <* x *>
y = <* y *>

u=x**2 -3*y
v=-x+1.4*y**3
w=x**3-y**3
[u,v,w]"]

• Thanks. A naive question: how do I re-use this set [u,v,w] to perform further computation inside Mathematica, meaning outside this session? Jul 2, 2021 at 16:52
• Assign the result to a variable res=ExternalEvaluate[..] or deconstruct directly {u,v,w}=ExternalEvaluate[..] Jul 2, 2021 at 17:29