9
$\begingroup$

I'm writing a log to a simple .txt file and want to read the last few records using Mathematica. But there are some problems.

enter image description here

f = OpenRead["logs.txt"];

Skip[f, ConstantArray[Number, 2], 1];

Read[f, ConstantArray[Number, 2]]

{3, 4}

Add lines and save.

enter image description here

Read[f, ConstantArray[Number, 2]]

EndOfFile

Of course, I can closing file every time, open again, to do Skip and read updated data. But this is very slow for the large log file.

It should be noted that all lines in my log are aligned to have the specific size. In this example they have size 7 bytes. So I can check the number of records:

FileByteCount["logs.txt"]/7

4

ADDENDUM

SetStreamPosition is fast. But first run is slower than consequent. For example, on the real data:

f = OpenRead["all_trades.txt"];

(* first run *)
SetStreamPosition[f, 0];
SetStreamPosition[f, 3899999]; // AbsoluteTiming
(* {0.02188, Null} *)

(* second run *)
SetStreamPosition[f, 0];
SetStreamPosition[f, 3899999]; // AbsoluteTiming
{0.000301056, Null}

SetStreamPosition[f, 0];
Skip[f, ConstantArray[Number, 10], 100000]; // AbsoluteTiming
(* {0.162914, Null} *)

StreamPosition[f]
(* 3899999 *)

Close[f];

But this does not work with updatable data!

enter image description here

f = OpenRead["logs.txt"];

Read[f, ConstantArray[Number, 2]]
(* {1, 2} *)

StreamPosition[f]
(* 5 *)

SetStreamPosition[f, 6];
Read[f, ConstantArray[Number, 2]]
(* {3, 4} *)

StreamPosition[f]
(* 11 *)

Add lines and save.

enter image description here

SetStreamPosition[f, 12];
Read[f, ConstantArray[Number, 2]]
(* EndOfFile *)

It looks like a bug. Is not it?

There are no problems in Python.

f = open('logs.txt', 'r')

# Add lines and save.
f.seek(14)
f.read(2)

'05'

|improve this question
$\endgroup$
  • 2
    $\begingroup$ I don't think C or C++ streams dynamically update to accommodate live-action file changes (not an answer, and of questionable relevancy, but perhaps worth noting) $\endgroup$ – user6014 Sep 7 '18 at 21:09
  • 1
    $\begingroup$ The file's initial data is likely sent to some sort of buffer as soon as the stream is opened. The buffer won't dynamically update, as the data's already been sent. I would be surprised if this were considered a bug, but who knows. $\endgroup$ – user6014 Sep 7 '18 at 21:14
  • $\begingroup$ @user6014 You're right - this is not a bug. See my answer. $\endgroup$ – Alexey Golyshev Sep 26 '18 at 8:23
3
$\begingroup$

This might be OS-dependent, but I would assume that:

oldPos = StreamPosition[f];
Close[f];
f = OpenRead["f:\\imgs\\logs.txt"];
SetStreamPosition[f, p];
Read[f, ConstantArray[Number, 2]]

should be faster than Skip, since it just moves the low-level file pointer.

|improve this answer
$\endgroup$
  • $\begingroup$ Good (+1). But it does not work. Please see the updated question. $\endgroup$ – Alexey Golyshev Sep 6 '18 at 17:34
  • $\begingroup$ Hm, it looks as if you're using SetStreamPosition wrong. Generally, with text files you should assume that only positions returned by StreamPosition are valid. For example, if the file is unicode-encoded, you might seek to a position "in the middle" of a character, and you will only read garbage from then on. I'm not sure why it isn't working, though. If you close the file, re-open it and seek to the previous position, you should see the new data? $\endgroup$ – Niki Estner Sep 6 '18 at 18:33
  • $\begingroup$ Yes, I see the new data when close and re-open the file. First run of SetStreamPosition is 7x faster than Skip. But second run of SetStreamPosition is yet more faster than the first (73x relative to the first run or 543x relative to Skip). When I close file, first run of SetStreamPosition will be again slower that the second, even in the same kernel session. Also there are small overheads on closing and re-opening the file. The ideal solution would be to avoid closing the file as in Python. $\endgroup$ – Alexey Golyshev Sep 7 '18 at 6:29
  • $\begingroup$ Can you reproduce EndOfFile error with my example? I am thinking whether should I report this case to the developers. $\endgroup$ – Alexey Golyshev Sep 7 '18 at 6:29
  • $\begingroup$ And yes, I can read file in Python directly from Mathematica using ExternalEvaluate ;-) $\endgroup$ – Alexey Golyshev Sep 7 '18 at 6:44
3
$\begingroup$

Wolfram Support Answer

The input stream is "buffered" by C, which is to say the file's data is read into memory when the stream is opened. Then all "read" operations work on the buffered data in memory, so if the file changes in the meantime, those changes are not reflected. This is a lot faster than constantly checking the file on disk.

You could remember the current stream position (using StreamPosition), Close the file, re-open the file again (using OpenRead), and then move back to the saved stream position (using SetStreamPosition). Closing and re-opening would get rid of the in-memory buffer.

This is, of course, not dynamic and unfortunately, there is not a documented or supported method of doing this.

However, OpenRead with the option AppendCheck->True may be able to fill the purpose. The option will reset the end of file marker in most cases allowing new information from the file to be read. This option is not documented or supported, so please consider this when using it at your discretion as we would not be able to provide further information on it.

Benchmarking

f = OpenRead["all_trades.txt", AppendCheck -> True];

AbsoluteTiming[
 SetStreamPosition[f, 0];
 Skip[f, ConstantArray[Number, 10], 163658];
 ]

{0.246617, Null}

StreamPosition[f]

6382661

AbsoluteTiming[
 SetStreamPosition[f, 0];
 SetStreamPosition[f, 6382661];
 ]

{0.00020363, Null}

Add a new line to the file and save.

Read[f, ConstantArray[Number, 10]]; // AbsoluteTiming

{0.0000380343, Null}

So OpenRead[..., AppendCheck -> True] is the fastest way.

|improve this answer
$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.