4
$\begingroup$

When doing development work with python, it's often recommended that one use virtualenv. It provides an environment separate from one's standard python evaluator.

Now that 11.2 has support for simple python sessions, how can I get a session running with venv?

$\endgroup$
3
$\begingroup$

For those comfortable with venv / pip

If you're already used to using pip and venv on the command line, the easiest thing to do is just activate your virtual environment of choice, install zmq with pip, and pass the venv binary to RegisterExternalEvaluator.


General structure for getting venv / pip set up with ZMQ

If you're not used to that, here's a quick way to do it.

First we define a function for building a new virtual env in a standard location (note that if you're not on Mac you may need to tweak ProcessEnvironment in different ways):

$pyVenvDir = FileNameJoin@{$HomeDirectory, "virtualenvs"};
pyVenvDir[
   dir : _String?DirectoryQ | Automatic : Automatic,
   name : _String | _Real : 2.7
   ] :=
  FileNameJoin@{
    Replace[dir, Automatic :> $pyVenvDir],
    Replace[name, v_Real :> "python" <> ToString[v]]
    };
pyVenvBin[
  dir : _String?DirectoryQ | Automatic : Automatic,
  name : _String | _Real : 2.7
  ] :=
 FileNameJoin@{#, "bin", FileNameTake[#]} &@
  pyVenvDir[dir, name];
pyVenvNew[
   dir : _String?DirectoryQ | Automatic : Automatic,
   name : _String | _Real : 2.7,
   version : _String | _Real : Automatic
   ] :=
  With[{d = Replace[dir, Automatic :> $pyVenvDir]},
   If[! DirectoryQ[d],
    CreateDirectory[d]
    ];
   With[{
     proc =
      RunProcess[{
        "virtualenv",
        Replace[
         Replace[version, {
           v_Real :> "python" <> ToString[v],
           Automatic :>
            Replace[name, {
              v_Real :> "python" <> ToString[v]
              }]
           }], {
          s_String :> "--python=" <> s,
          _ -> Nothing
          }],
        Replace[name, v_Real :> "python" <> ToString[v]]
        },
       ProcessDirectory -> d,
       ProcessEnvironment ->
        <|

         "PATH" -> "/usr/local/bin:" <> Environment["PATH"]
         |>
       ]
     },
    If[proc["ExitCode"] === 0,
     FileNameJoin@{d, 
       Replace[name, v_Real :> "python" <> ToString[v]]},
     proc
     ]
    ]
   ];

And then we'll build a new one (I'm gonna change the base dir first though):

(* $pyVenvDir = FileNameJoin@{$TemporaryDirectory, "virtualenvs"} (* because I already did this in my base venv directory *)
$pyVenvDir = FileNameJoin@{$TemporaryDirectory, "virtualenvs"};
pyVenvNew[3.4](* give it whatever name and version you like *)

"/private/var/folders/9t/tqc70b7d61v753jkdbjkvd640000gp/T/virtualenvs/\
python3.4"

Then we define some functions for working with a activated venv as a ProcessObject:

If[! AssociationQ@$pyVenv, $pyVenv = None];
pyVenvStart[
   dir : _String?DirectoryQ | Automatic : Automatic,
   name : _String | _Real : 2.7
   ] :=

  With[{d = 
     FileNameJoin@{#, "bin", FileNameTake[#]} &@
      pyVenvDir[dir, name]},
   If[AssociationQ@$pyVenv,
    $pyVenv["Process"] // KillProcess
    ];
   If[FileExistsQ@d,
    $pyVenv =
     <|
      "Process" ->
       With[{
         p =
          StartProcess[$SystemShell]
         },
        WriteLine[p,
         StringRiffle@{
           "source",
           FileNameJoin@{pyVenvDir[dir, name], "bin", "activate"}
           }
         ];
        Pause[.05];
        ReadString[p, EndOfBuffer];
        p
        ],
      "Venv" ->
       pyVenvDir[dir, name]
      |>
    ]
   ];
pyVenvRun[
   venvDir : _String?DirectoryQ | Automatic : Automatic,
   name : _String | _Real : 2.7,
   cmds : {__String}
   ] :=
  With[{vd = pyVenvDir[venvDir, name]},
   If[! $pyVenv["Venv"] === vd || ! 
      TrueQ[ProcessStatus[$pyVenv["Process"], "Running"]],
    pyVenvStart[venvDir, name]
    ];
   With[{po = $pyVenv["Process"]},
    If[TrueQ@ProcessStatus[po, "Running"],
     WriteLine[po, #] & /@ cmds;
     Pause[.1];
     Merge[
      Reap[
        Block[{read},
         read =
          AssociationMap[
           ReadString[ProcessConnection[po, #], EndOfBuffer] &,
           {
            "StandardOutput",
            "StandardError"
            }
           ];
         While[AnyTrue[Values[read], StringLength[#] > 0 &],
          Sow[read];
          Pause[.1];
          read =
           AssociationMap[
            ReadString[ProcessConnection[po, #], EndOfBuffer] &,
            {
             "StandardOutput",
             "StandardError"
             }
            ];
          ]
         ]
        ][[2]],
      StringJoin
      ],
     $Failed
     ]
    ]
   ];

Test that it works:

pyVenvRun[3.4,
 {"python --version"}
 ]

<|"StandardOutput" -> "Python 3.4.4
  ", "StandardError" -> ""|>

And finally a function that uses this to set up the venv so it can be used with ExternalEvaluate:

pyVenvRegister[
  venvDir : _String?DirectoryQ | Automatic : Automatic,
  name : _String | _Real : 2.7
  ] :=
 (
  If[! DirectoryQ@pyVenvDir[venvDir, name],
   pyVenvNew[venvDir, name]
   ];
  pyVenvRun[venvDir, name,
   {"pip install zmq"}
   ];
  RegisterExternalEvaluator[
   "Python",
   FileNameJoin@{#, "bin", FileNameTake[#]} &@
    pyVenvDir[venvDir, name]
   ]
  )

Test this:

venvKey = pyVenvRegister[3.4]

"fea636c5-5f9b-4ffb-8db8-a1302378991e"

venvData =
 FindExternalEvaluators[]["Python", 
   "fea636c5-5f9b-4ffb-8db8-a1302378991e"] // Normal

<|"Executable" -> 
  "/private/var/folders/9t/tqc70b7d61v753jkdbjkvd640000gp/T/\
virtualenvs/python3.4/bin/python3.4", "Version" -> "3.4.4", 
 "Registered" -> True|>

And now we can start a session in our venv-ed set up:

$python =
 StartExternalSession[
  Prepend["System" -> "Python"]@
   KeyDrop[venvData, "Registered"]
  ]

ExternalSessionObject["d8c9a48a-b130-42e9-9c02-044c0e631857"]

session

Note that we may to use pyVenvRun in the future. For instance, a fresh venv doesn't have PIL installed:

ExternalEvaluate[$python,
 "import PIL"
 ]

Failure["EvaluationFailure", 
Association[
 "MessageTemplate" :> MessageName[ExternalEvaluate, "error"], 
  "MessageParameters" -> Association["Type" -> "ImportError"], 
  "StackTrace" -> Panel[
Style["Traceback (most recent call last):\n  File \
\"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/\
code.py\", line 90, in runcode\n    exec(code, self.locals)\n  File \
\"<input>\", line 1, in <module>\nImportError: No module named 'PIL'\n\
", "Program"], ImageSize -> {All, {1, 500}}, 
    BaselinePosition -> Baseline, ContentPadding -> False, 
    FrameMargins -> 0, StripOnInput -> True], 
  "ExceptionType" -> "ImportError", "Input" -> "import PIL", 
  "Output" -> Missing["KeyAbsent", "output"]]]

But we can install that:

pyVenvRun[3.4,
 {"pip install pillow"}
 ]

<|"StandardOutput" -> "Downloading/unpacking pillow
  Downloading/unpacking olefile (from pillow)
    Running setup.py \
(path:/private/var/folders/9t/tqc70b7d61v753jkdbjkvd640000gp/T/\
virtualenvs/python3.4/build/olefile/setup.py) egg_info for package \
olefile

  Installing collected packages: pillow, olefile
    Running setup.py install for olefile

  Successfully installed pillow olefile
  Cleaning up...
  ", "StandardError" -> ""|>

And now this works:

ExternalEvaluate[$python,
  "from PIL import Image"
  ];
ExternalEvaluate[$python,
 "im = Image.new('RGB', [512, 512], 'white'); im.show()"
 ]

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.