Error when implementing StartProcess[] in loops (Do, While, Table)

Question

I'm using Mathematica to control another program (Surface Evolver) through the terminal (ie using StartProcess[]; and WriteLine[];). The programs communicate by writing and reading text files (Export[]; and Import[]; respectively). The script I wrote is packaged in Block[]; (Module[]; yields the same results) and run iteratively in a While[] loop (I've also used Do[] and Table[], same results). Everything works great when I run small batches of iterations. I don't think I'm running out of memory (I clear the sys cache and wipe variables within the module every iteration to prevent memory leaks because I've seen some chatter about that in other threads). However, at about n=100 I'll get an error

OpenWrite::noopen: Cannot open in:100

EDIT

Okay, this should be more manageable, I've isolated the source of the error to the repeated creation of processes. It seems like the error is not MTHM per se, but an inability to open a new system process.

Dynamic[n]
Do[Module[{TestProcess},
TestProcess = StartProcess[$SystemShell];
WriteLine[TestProcess, "echo example line"];
KillProcess[TestProcess];
Pause[0.1];
Clear[TestProcess];
ClearSystemCache[];
];,{n, 1, 1000}];

END EDIT

System Info:

1. Mac OS X Mavericks
2. Mathematica 10
3. Screen Saver is off

Answer 1

The problem is that the stdin stream for each process is being left open and needs to be closed.

To demonstrate the issue, we create a single process:

$process = StartProcess[$SystemShell];
WriteLine[$process, "echo example line"];
KillProcess[$process];

The call to WriteLine implicitly creates a stream object. Even though the process has been killed, the stdin stream object remains (named in:0 in this case):

Streams[]

We need to manually close the stream (see the sidebar below for why we have to use "StandardOutput" to get the stdin stream):

Close[ProcessConnection[$process, "StandardOutput"]];

Streams[]

Also, had we obtained any explicit references to any of the other standard process streams, we would need to close them manually as well.

This would be a good thing for the documentation to mention in the Possible Issues section :)

---

Sidebar (EDIT: the problem described below has been fixed in version 10.0.2)

I will mention in passing that ProcessConnection has the stream names for stdin and stdout reversed. We must reference stdin using the stream name "StandardOutput" and stdout using "StandardInput". This confusion is evident in examples given in the documentation:

Note how the example is writing the string "echo line 1" to the shell's stdin, but the stream name used was "StandardOutput". Beware.

Answer 2

We had a bug like this on v. 10: it specifically stopped running new processes after you created 100 of them. This was fixed in version 10.0.1. I recommend testing it there, as version 10.0.1 has a number of improvements with respect to StartProcess.

I can't test your example in MacOS right now, so please let me know if this isn't working on v. 10.0.1.

Regarding other comments here:

• it is true the streams are kept open after killing the process, and this is a bug. I will report it in our internal system. I don't think this is what causes the issue reported here.

• regarding the stream names, we spent a while discussing this point when designing the StartProcess features. Wolfram Language connects with the external processes using streams, and what is stdin for WL is stdout for the process. So both naming schemes could theoretically make sense, and it can be a bit confusing. We decided to call WL's stdin StandardInput, because is it a stream you can read from.

• I am also reporting the issue that ybeltukov mentions to our internal bug system

Thanks