1
$\begingroup$

I'm trying to open a file to read and do some processing indefinitely until I decide to abort:

MyInputStream = OpenRead["data.csv"]
While[True, 
  rec = Read[MyInputStream, Record];
  MyProcessing[rec]
]

or, say:

Dynamic[MyProcessing[Read[MyInputStream, Record]]]

In other words, I want any of Read, ReadString, ReadList to be blocking and wait for input. Currently, it stops on EndOfFile even though the file I opened keeps growing. I cannot do anything to make it blocking with what the official documentation provides.

I also found that if I stream my data through a socket rather than a file, I am able to achieve what I want to do using SocketListen (the documentation has an explicit answer), but I want to stick to simple file I/O for now. Here is how to solve the problem if I send my data over local socket on port 5555:

SocketListen[SocketOpen[5555], (Data = #Data) &];
Dynamic[MyProcessing[Data]]

I would also like to avoid polling solutions.

Can someone help, please?

$\endgroup$
8
  • 1
    $\begingroup$ ReadString is actually blocking. I believe that the problem is that file you are trying to read is already complete. There is no reason for ReadString to wait for input because there is no indication that the file is going to change. Once the file is read, you get an EOF marker. $\endgroup$
    – Szabolcs
    Jul 8, 2021 at 14:49
  • $\begingroup$ Okay, thanks, so it is a question about file I/O not ReadString? @Szabolcs $\endgroup$
    – Dan Oak
    Jul 8, 2021 at 15:06
  • $\begingroup$ My file is updated in a shell script like cmd >> data.csv. cmd itself is a never-ending process. I suppose I should go somewhere else with a question about shell scripting, then? $\endgroup$
    – Dan Oak
    Jul 8, 2021 at 15:08
  • 1
    $\begingroup$ I did not say anything about the question being off topic. I am not sure why you interpreted it that way. My point was that if you insist on using a file, polling is likely unavoidable. Try not to use a file. Read directly form the output of cmd (e.g. with RunProcess, see all the ReadString examples, or just open "!cmd" as if it were a file. If this is not possible, e.g. because cmd was already running before you started Mathematica, then use tail -f as an intermediary. $\endgroup$
    – Szabolcs
    Jul 8, 2021 at 15:12
  • $\begingroup$ Amazing! Thank you. I found the solution using StartProcess and tail -f. I'll add it as an answer. $\endgroup$
    – Dan Oak
    Jul 8, 2021 at 15:45

2 Answers 2

3
$\begingroup$

If your system has tail or alternatives:

TailProcess = StartProcess[{"tail", "-f", "data.csv"}];

and use Read/ReadLine/... on TailProcess. Credits to @Szabolcs.

For now, it is an appealing solution to me, although it makes me dependent on one additional link in the processing chain.

$\endgroup$
1
$\begingroup$

enter image description here

To demonstrate, there is a python script that adds a new line of text every 1.5 seconds, 4 times, and a counter variable to run the loop only 12 times.

First, we create a stream with OpenRead, and by using SetStreamPosition, it'll set to zero first. After reading, lastPosition will be updated, the stream will be closed, the stream position will be checked, if it doesn't change (lastPosition == newPosition) then we'll wait 1.5 seconds otherwise the content will be printed and ...

If your case involves removing data, this code may return an error because in SetStreamPosition it may exceed the file limits. The code only covers appending new data.

Block[{counter = 0, newPosition = 0, lastPosition = 0, content, stream},
 While[True,
  If[counter > 12, Break[]];
  stream = OpenRead["D:\\test.txt"];
  SetStreamPosition[stream, lastPosition];
  content = Read[stream, String];
  newPosition = StreamPosition[stream];
  Close[stream];
  If[lastPosition != newPosition, Print[content], Pause[1.5]];
  lastPosition = newPosition;
  counter += 1;
  ]
 ]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.