# Using virtualenv and ExternalEvaluate

When doing development work with python, it's often recommended that one use virtualenv. It provides an environment separate from one's standard python evaluator.

Now that 11.2 has support for simple python sessions, how can I get a session running with venv?

### For those comfortable with venv / pip

If you're already used to using pip and venv on the command line, the easiest thing to do is just activate your virtual environment of choice, install zmq with pip, and pass the venv binary to RegisterExternalEvaluator.

### General structure for getting venv / pip set up with ZMQ

If you're not used to that, here's a quick way to do it.

First we define a function for building a new virtual env in a standard location (note that if you're not on Mac you may need to tweak ProcessEnvironment in different ways):

$pyVenvDir = FileNameJoin@{$HomeDirectory, "virtualenvs"};
pyVenvDir[
dir : _String?DirectoryQ | Automatic : Automatic,
name : _String | _Real : 2.7
] :=
FileNameJoin@{
Replace[dir, Automatic :> $pyVenvDir], Replace[name, v_Real :> "python" <> ToString[v]] }; pyVenvBin[ dir : _String?DirectoryQ | Automatic : Automatic, name : _String | _Real : 2.7 ] := FileNameJoin@{#, "bin", FileNameTake[#]} &@ pyVenvDir[dir, name]; pyVenvNew[ dir : _String?DirectoryQ | Automatic : Automatic, name : _String | _Real : 2.7, version : _String | _Real : Automatic ] := With[{d = Replace[dir, Automatic :>$pyVenvDir]},
If[! DirectoryQ[d],
CreateDirectory[d]
];
With[{
proc =
RunProcess[{
"virtualenv",
Replace[
Replace[version, {
v_Real :> "python" <> ToString[v],
Automatic :>
Replace[name, {
v_Real :> "python" <> ToString[v]
}]
}], {
s_String :> "--python=" <> s,
_ -> Nothing
}],
Replace[name, v_Real :> "python" <> ToString[v]]
},
ProcessDirectory -> d,
ProcessEnvironment ->
<|

"PATH" -> "/usr/local/bin:" <> Environment["PATH"]
|>
]
},
If[proc["ExitCode"] === 0,
FileNameJoin@{d,
Replace[name, v_Real :> "python" <> ToString[v]]},
proc
]
]
];


And then we'll build a new one (I'm gonna change the base dir first though):

(* $pyVenvDir = FileNameJoin@{$TemporaryDirectory, "virtualenvs"} (* because I already did this in my base venv directory *)
$pyVenvDir = FileNameJoin@{$TemporaryDirectory, "virtualenvs"};
pyVenvNew[3.4](* give it whatever name and version you like *)

"/private/var/folders/9t/tqc70b7d61v753jkdbjkvd640000gp/T/virtualenvs/\
python3.4"


Then we define some functions for working with a activated venv as a ProcessObject:

If[! AssociationQ@$pyVenv,$pyVenv = None];
pyVenvStart[
dir : _String?DirectoryQ | Automatic : Automatic,
name : _String | _Real : 2.7
] :=

With[{d =
FileNameJoin@{#, "bin", FileNameTake[#]} &@
pyVenvDir[dir, name]},
If[AssociationQ@$pyVenv,$pyVenv["Process"] // KillProcess
];
If[FileExistsQ@d,
$pyVenv = <| "Process" -> With[{ p = StartProcess[$SystemShell]
},
WriteLine[p,
StringRiffle@{
"source",
FileNameJoin@{pyVenvDir[dir, name], "bin", "activate"}
}
];
Pause[.05];
p
],
"Venv" ->
pyVenvDir[dir, name]
|>
]
];
pyVenvRun[
venvDir : _String?DirectoryQ | Automatic : Automatic,
name : _String | _Real : 2.7,
cmds : {__String}
] :=
With[{vd = pyVenvDir[venvDir, name]},
If[! $pyVenv["Venv"] === vd || ! TrueQ[ProcessStatus[$pyVenv["Process"], "Running"]],
pyVenvStart[venvDir, name]
];
With[{po = $pyVenv["Process"]}, If[TrueQ@ProcessStatus[po, "Running"], WriteLine[po, #] & /@ cmds; Pause[.1]; Merge[ Reap[ Block[{read}, read = AssociationMap[ ReadString[ProcessConnection[po, #], EndOfBuffer] &, { "StandardOutput", "StandardError" } ]; While[AnyTrue[Values[read], StringLength[#] > 0 &], Sow[read]; Pause[.1]; read = AssociationMap[ ReadString[ProcessConnection[po, #], EndOfBuffer] &, { "StandardOutput", "StandardError" } ]; ] ] ][[2]], StringJoin ],$Failed
]
]
];


Test that it works:

pyVenvRun[3.4,
{"python --version"}
]

<|"StandardOutput" -> "Python 3.4.4
", "StandardError" -> ""|>


And finally a function that uses this to set up the venv so it can be used with ExternalEvaluate:

pyVenvRegister[
venvDir : _String?DirectoryQ | Automatic : Automatic,
name : _String | _Real : 2.7
] :=
(
If[! DirectoryQ@pyVenvDir[venvDir, name],
pyVenvNew[venvDir, name]
];
pyVenvRun[venvDir, name,
{"pip install zmq"}
];
RegisterExternalEvaluator[
"Python",
FileNameJoin@{#, "bin", FileNameTake[#]} &@
pyVenvDir[venvDir, name]
]
)


Test this:

venvKey = pyVenvRegister[3.4]

"fea636c5-5f9b-4ffb-8db8-a1302378991e"

venvData =
FindExternalEvaluators[]["Python",
"fea636c5-5f9b-4ffb-8db8-a1302378991e"] // Normal

<|"Executable" ->
"/private/var/folders/9t/tqc70b7d61v753jkdbjkvd640000gp/T/\
virtualenvs/python3.4/bin/python3.4", "Version" -> "3.4.4",
"Registered" -> True|>


And now we can start a session in our venv-ed set up:

$python = StartExternalSession[ Prepend["System" -> "Python"]@ KeyDrop[venvData, "Registered"] ] ExternalSessionObject["d8c9a48a-b130-42e9-9c02-044c0e631857"]  Note that we may to use pyVenvRun in the future. For instance, a fresh venv doesn't have PIL installed: ExternalEvaluate[$python,
"import PIL"
]

Failure["EvaluationFailure",
Association[
"MessageTemplate" :> MessageName[ExternalEvaluate, "error"],
"MessageParameters" -> Association["Type" -> "ImportError"],
"StackTrace" -> Panel[
Style["Traceback (most recent call last):\n  File \
\"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/\
code.py\", line 90, in runcode\n    exec(code, self.locals)\n  File \
\"<input>\", line 1, in <module>\nImportError: No module named 'PIL'\n\
", "Program"], ImageSize -> {All, {1, 500}},
BaselinePosition -> Baseline, ContentPadding -> False,
FrameMargins -> 0, StripOnInput -> True],
"ExceptionType" -> "ImportError", "Input" -> "import PIL",
"Output" -> Missing["KeyAbsent", "output"]]]


But we can install that:

pyVenvRun[3.4,
{"pip install pillow"}
]

Running setup.py \
(path:/private/var/folders/9t/tqc70b7d61v753jkdbjkvd640000gp/T/\
virtualenvs/python3.4/build/olefile/setup.py) egg_info for package \
olefile

Installing collected packages: pillow, olefile
Running setup.py install for olefile

Successfully installed pillow olefile
Cleaning up...
", "StandardError" -> ""|>


And now this works:

ExternalEvaluate[$python, "from PIL import Image" ]; ExternalEvaluate[$python,
"im = Image.new('RGB', [512, 512], 'white'); im.show()"
]


• I am moderately familiar with pip and venv but apparently not familiar enough. Would you mind explaining what zmq is for and what (on Ubuntu) "the binary" that should be passed to ExternalEvaluate is. Is it the activate file that normally activates the environment? This does not run python, you would have to follow it with python file.py. Or is it something else that gets created in the /bin directory (my tries with calling the python3 link placed here seems to indicate this does not automatically run in the virtual environment). Apr 22 at 9:53
• Also, following your steps, after creating the virtual environment (which looks okay), when I run pyVenvRun[3.8, {"python --version"} ] I get <|"StandardOutput" -> "Python 3.8.5 ", "StandardError" -> "/bin/sh: 1: source: not found "|>. Apr 22 at 10:03
• Okay I fixed this problem by replacing source with "." which I understand to be the equivalent but more robust, i.e. runs in more types of shells. (I don't know why MMA is running it in a shell environment that does not support source.) Apr 22 at 10:05
• I think I sorted it out, see the supplementary answer I posted. Apr 22 at 10:59
• Would you happen to know how to quickly create a cell that automatically gets evaluated using such a newly registered environment, just like I can create a regular python cell by typing > at the start. (Asked as question here:mathematica.stackexchange.com/questions/244998/…) Apr 22 at 13:16

This is just a supplement to the excellent answer by b3m2a1. I will answer how to register an existing virtual environment in MMA.

First install the package pyzmq in virtual environment. This package is required to run python in ExternalEvaluate (https://reference.wolfram.com/language/workflow/ConfigurePythonForExternalEvaluate.html).

$$source venv/bin/activate pyzmq$$ pip install --user pyzmq


Next find an executable located in venv/bin that runs python in the virtual environment, it will for example be called python3.8. Note this is slightly different from really running in a virtual environment for example os.environ['VIRTUAL_ENV'] will now give KeyError: 'VIRTUAL_ENV'. However, it will correctly use the packages from your virtual environment.

Next register it and save it to some name with the following (The code string that the first line outputs is the way to find your environment):

RegisterExternalEvaluator["Python",
"/path_to_executable_in_venv/python3.8"]
venvName = FindExternalEvaluators[]["Python", %] // Normal;
pythonSession =
StartExternalSession[
Prepend["System" -> "Python"]@KeyDrop[venvName, "Registered"]]


Now you can use that and it will use the packages in the virtual environment.

ExternalEvaluate[pythonSession, "2+2"]