I am writing a Mathematica program that tries to run 100+ Python programs, via StartProcess[], continuously. It seems to work when I run < 100 programs, but, at the current number of 500 programs that I am attempting to run, my computer crashes every time.

Is what I am trying to accomplish realistic? And, if so, is there another way I can run these scripts that will not crash my computer?


Here is the code:

(*This function takes in the creatureFolder pathway and starts the \
AlpacaInterfacer.py application*)
executePythonFile[creatureFolder_] := Module[{process},
  process = StartProcess[$SystemShell];
  (*Change the directory to the current one*)
  WriteLine[process, StringJoin["cd ", creatureFolder]];
  (*Execute the file*)
  WriteLine[process, "AlpacaInterfacer.py"];

And I am running executePythonFile[] in a Table, where the Python scripts' locations are saved to a list, so that the same program will not be started twice.

  • 2
    $\begingroup$ Why not just use subprocess within python to manage this? Seems simpler. Also why not use ExternalEvaluate/StartExternalSession? $\endgroup$ – b3m2a1 Feb 21 '19 at 19:52
  • $\begingroup$ I think StartExternalSession might be what I was looking for! What would the differences be between StartExternalSession and StartProcess, in terms of memory usage? Thanks! $\endgroup$ – Jmeeks29ig Feb 21 '19 at 19:57
  • 1
    $\begingroup$ I'm not sure if memory is your issue so much as having a whole pile of open processes running. The CPU isn't really designed to have so many processes. A better way to do this is probably using threading.Thread within python, assuming each process itself isn't terribly memory hungry. Otherwise create a multiprocessing.Pool object to process your inputs. As I noted, the process management should really be on the python side, rather than trying to fake something yourself. $\endgroup$ – b3m2a1 Feb 21 '19 at 20:01
  • $\begingroup$ Ok, I will look that up then, thanks for the help! $\endgroup$ – Jmeeks29ig Feb 21 '19 at 20:04
  • 1
    $\begingroup$ Glad to be help. Hopefully you can stave off any further crashes. $\endgroup$ – b3m2a1 Feb 21 '19 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.