Using Read for calling external command does not release file handle

Question

The standard and robust method for piping to/from external commands under Windows is to create a temporary file, save the input into it, pipe it to the command and save its output to a second file. Here is a crippled version of the built-in RunThrough (that is the standard way for piping), that omits the output, for testing purposes, and prevents the command window to pop up (you can check the definition of RunThrough by removing its ReadProtected attribute).

runThrough[comm_String, s_String] := Block[{in = OpenWrite[], out = OpenWrite[]},
   WriteString[in, s <> "\n"]; Close /@ {in, out};
   (* If[i > 2040, Print[{i, in, out}]]; *)
   Read["!" <> comm <> " < " <> First@in <> " > " <> First@out];
   DeleteFile@First@# & /@ {in, out}];

In the following example I called the Windows ver command that simply returns the OS version, so it's pretty harmless to run. The problem comes when this runThrough is run iteratively. Always precisely and reproducibly at i=2045 on my machine, error messages start to rain.

Dynamic@i
Table[runThrough["ver", "x"], {i, 10000}]

OpenRead::noopen: -- Message text not found -- (C:\...\Mathematica 9.0.1\Documentation\English\PacletDB.m)
General::stream: -- Message text not found -- ($Failed)
OpenWrite::noopen: -- Message text not found -- (C:\Users\...\AppData\Local\Temp\m-49f8c51e-d090-4dc3-87b4-0a5310380f70)
General::stream: $Failed is not a string, InputStream[ ], or OutputStream[ ].
...

If the commented line is uncommented (I left it for everyone to figure out the platform-dependent time of breakdown), I can see that the out file is not created by OpenWrite, as the result is $Failed. It happens to any external command, even for those that properly accept input from a pipe, not just OS natives like ver. Furthermore, somehow the kernel gets messed up, as any attempt after the messages to access the Documentation Center (for example F1-ing a function) brings up a dialog:

The file you tried to open was not found or could not be opened.

Is it a problem of OpenWrite or I simply try to do too many things too fast and the OS cannot keep up with Mathematica? If so, how can I overcome such unwanted effects?

(Fresh kernel, no package was loaded, Win7 64bit, Mathematica 9.0.1)

Answer 1

The problem has nothing to do with OpenWrite. You never Close the stream you open in your call to Read.

Read, unlike ReadList, does not automatically close a stream (file, pipe, etc.) that's given as its first argument string. (That's because the purpose of Read is to be able to read from the same source in pieces, unlike ReadList which does it all at once.)

You could certainly Close the stream simply by passing the same string to Close that you did to Read. Mathematica remembers the names you implicitly create streams with, so you can use the file name again and again without having to separately create a "stream object" the way you would in other languages.

But since the "name" of this stream is pretty complicated and you had to assemble it out of parts, it's probably easier here to use a stream object directly, by calling OpenRead.

Add strm to the list of variables in your Block list, then replace your call to Read with the following 3 lines:

strm = OpenRead["!" <> comm <> " < " <> First@in <> " > " <> First@out];
Read[strm];
Close[strm];

Alternatively -- and you may find this easier to deal with -- since ReadList does auto-close its auto-opened stream, you can just replace your Read with the line:

First@ReadList["!" <> comm <> " < " <> First@in <> " > " <> First@out, Expression, 1]

I added First because ReadList returns a list, and 1 because you really just want a single line. (Since the count is the third argument I had to be explicit about the item type; Expression is the default.)

Answer 2

The number 2045 is suspicious. Add three to it to include stdin, stdout, and stderr, and you get 2048, which I suspect is total number of file descriptors available to you. I conclude your problem is caused by eating up all the available file descriptors. This is usually caused by doing too many file opens without doing any file closings to return some file descriptors to the pool of available descriptors. Not an error peculiar to Mathematica -- can happen with any badly written I/O code. I've seen this many times in my 55 years of programming.

Edit

I could not reproduce your problem on OS X and I don't have access to Mathematica on Windows. Here is the code I used to run tests with Mathematica V9.0.1 on OS X 10.6.8

test[cmnd_String, s_String] :=
  Module[{in = OpenWrite[], out = OpenWrite[]},
    WriteString[in, s <> "\n"];
    Close /@ {in, out};
    Run[cmnd <> " < " <> First @ in <> " > " <> First @ out];
    (*FilePrint[First @ in];
      FilePrint[First @ out];*)
    DeleteFile @ First @ # & /@ {in, out}
    (*; Streams[]*)]

k = 0;
Do[test["cat", "quick brown fox"]; k = i, {i, 5000}];
Dynamic @ k

5000

Note that I changed Read to Run. I thought Read must be a typo.