9
$\begingroup$

I'm using Mathematica to control another program (Surface Evolver) through the terminal (ie using StartProcess[]; and WriteLine[];). The programs communicate by writing and reading text files (Export[]; and Import[]; respectively). The script I wrote is packaged in Block[]; (Module[]; yields the same results) and run iteratively in a While[] loop (I've also used Do[] and Table[], same results). Everything works great when I run small batches of iterations. I don't think I'm running out of memory (I clear the sys cache and wipe variables within the module every iteration to prevent memory leaks because I've seen some chatter about that in other threads). However, at about n=100 I'll get an error

OpenWrite::noopen: Cannot open in:100

EDIT

Okay, this should be more manageable, I've isolated the source of the error to the repeated creation of processes. It seems like the error is not MTHM per se, but an inability to open a new system process.

Dynamic[n]
Do[Module[{TestProcess},
TestProcess = StartProcess[$SystemShell];
WriteLine[TestProcess, "echo example line"];
KillProcess[TestProcess];
Pause[0.1];
Clear[TestProcess];
ClearSystemCache[];
];,{n, 1, 1000}];

END EDIT

System Info:

  1. Mac OS X Mavericks
  2. Mathematica 10
  3. Screen Saver is off
$\endgroup$
  • 1
    $\begingroup$ You second code-block is not complete. The start seems to be missing and the ,{n,1,1000}]; belongs to nothing. Otherwise, nice that you could break the problem down. It is more likely that someone can help you with this. $\endgroup$ – halirutan Oct 10 '14 at 0:08
  • $\begingroup$ Fixed, thanks - that is the hope! $\endgroup$ – Nickel Oct 10 '14 at 1:41
  • $\begingroup$ Mathematica 10 under Linux: bunch of errors after ~900 cycles. General::stream: -- Message text not found -- ($Failed) and so on. $\endgroup$ – ybeltukov Oct 10 '14 at 1:44
10
$\begingroup$

The problem is that the stdin stream for each process is being left open and needs to be closed.

To demonstrate the issue, we create a single process:

$process = StartProcess[$SystemShell];
WriteLine[$process, "echo example line"];
KillProcess[$process];

The call to WriteLine implicitly creates a stream object. Even though the process has been killed, the stdin stream object remains (named in:0 in this case):

Streams[]

stream list screenshot

We need to manually close the stream (see the sidebar below for why we have to use "StandardOutput" to get the stdin stream):

Close[ProcessConnection[$process, "StandardOutput"]];

Streams[]

stream list screenshot

Also, had we obtained any explicit references to any of the other standard process streams, we would need to close them manually as well.

This would be a good thing for the documentation to mention in the Possible Issues section :)


Sidebar (EDIT: the problem described below has been fixed in version 10.0.2)

I will mention in passing that ProcessConnection has the stream names for stdin and stdout reversed. We must reference stdin using the stream name "StandardOutput" and stdout using "StandardInput". This confusion is evident in examples given in the documentation:

documentation screenshot

Note how the example is writing the string "echo line 1" to the shell's stdin, but the stream name used was "StandardOutput". Beware.

$\endgroup$
  • $\begingroup$ Note: I tested this on Windows 7, which exhibited the problem as well albeit at a much higher iteration count. I expect the solution to be the same on MacOSX, but I am not in a position to test it directly myself at the moment. $\endgroup$ – WReach Oct 11 '14 at 13:10
  • $\begingroup$ The problem with swapped stream names has been fixed in version 10.0.2. $\endgroup$ – WReach Dec 10 '14 at 22:33
9
$\begingroup$

We had a bug like this on v. 10: it specifically stopped running new processes after you created 100 of them. This was fixed in version 10.0.1. I recommend testing it there, as version 10.0.1 has a number of improvements with respect to StartProcess.

I can't test your example in MacOS right now, so please let me know if this isn't working on v. 10.0.1.

Regarding other comments here:

  • it is true the streams are kept open after killing the process, and this is a bug. I will report it in our internal system. I don't think this is what causes the issue reported here.

  • regarding the stream names, we spent a while discussing this point when designing the StartProcess features. Wolfram Language connects with the external processes using streams, and what is stdin for WL is stdout for the process. So both naming schemes could theoretically make sense, and it can be a bit confusing. We decided to call WL's stdin StandardInput, because is it a stream you can read from.

  • I am also reporting the issue that ybeltukov mentions to our internal bug system

Thanks

$\endgroup$
  • $\begingroup$ +1. With respect to the stream names... I would agree if the stream names were simply "Input" and "Output". But I would argue that the qualifier Standard in "StandardInput" implies that it is being seen from the perspective of the process. After all, Mathematica's own stdin and stdout streams play no role here (i.e. brand new streams are created for the connection). However, what is done is done. It would be harmful to change it now. Perhaps it would make a fine addition to the Possible Issues section of the documentation :) $\endgroup$ – WReach Oct 13 '14 at 17:47
  • $\begingroup$ @WReach I agree these are the worst possible names. I will see if we can change them and take the hit of incompatibility now in exchange for potentially many years of confusion down the line. $\endgroup$ – Taliesin Beynon Oct 13 '14 at 21:17
  • $\begingroup$ @WReach yes, you are right. Tali and I are talking about this now. Thanks for your comment. $\endgroup$ – Hernan Moraldo Oct 13 '14 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.