Using ExternalEvaluate within ParallelDo

Question

I have had great success using Python in Mathematica (for some computations for which there are very well optimised Python packages but not for Mathematica). I have some Mathematica functions that are calls of Python functions, similar to the example given in Applications section of this page: https://reference.wolfram.com/language/ref/ExternalEvaluate.html.

I would now like to run these functions within a ParallelDo function, unfortunately it isn't working. Here is a MWE and the output:

session = StartExternalSession["Python-NumPy"];
ExternalEvaluate[session, "def double(x):
    return x*2"];
doublePython[arg_] := ExternalEvaluate[session, "double(" <> ToString[arg] <> ")"]

Do[Pause[1]; Print[doublePython[i]], {i, Range[4]}] // AbsoluteTiming
ParallelDo[Pause[1]; Print[doublePython[i]], {i, Range[4]}] // AbsoluteTiming

Answer 1

For these sorts of objects you can create a session per kernel. Ideally you would want separate Python session per kernel to prevent one parallel process from being blocked by another when the ExternalEvaluate calls are being made. More can be found in the Parallel Computing Tools User Guide.

Launch the kernels

LaunchKernels[]

Create a Python instance on each kernel and setup the environment.

ParallelEvaluate[
 session = StartExternalSession["Python-NumPy"];
 ExternalEvaluate[session, "def double(x):
      return x*2"];
 ]

Now you may run your process in parallel.

ParallelDo[Pause[1]; 
  Print[doublePython[i]], {i, Range[4]}] // AbsoluteTiming

Clean up with

ParallelEvaluate[DeleteObject[session]]
CloseKernels[]

Hope this helps.