1
$\begingroup$

[MMA 11.0, Win7 64bit]

I have a number of CVS files of ~1-2GB, millions of rows, so I want to read them in records, chunks building on Albert Retey's answer to this question

I assumed, and the documentation confirmed, that

There is always a "current point" maintained for any stream. When you read an object from a stream, the current point is left after the input you read. Successive calls to Read can therefore be used to read successive objects in a stream such as a file.

But, as this trivial example showed, StreamPosition is stubbornly 0 after a read

Close[openedReadStream];
readPath = "C:\\aCSV.csv";
openedReadStream := OpenRead[readPath];
StreamPosition[openedReadStream]
Read[openedReadStream, {Record}, WordSeparators -> {",", "\t"},RecordSeparators -> {"\r\n", "\n", "\r"}]
StreamPosition[openedReadStream]
Read[openedReadStream, {Record}, WordSeparators -> {",", "\t"},RecordSeparators -> {"\r\n", "\n", "\r"}]

Output:

0
{"Date,Time,Bid Price,Ask Price,Contributor,Region,City"}
0
{"Date,Time,Bid Price,Ask Price,Contributor,Region,City"}

A plethora of questions arise, of which the most pressing is: how do I avoid this problem?

TIA!

$\endgroup$

1 Answer 1

5
$\begingroup$

Having written it up, I might as well answer having double-checked my thinking before posting, just in case I was about to make a fool of myself...

The problem lies on this line:

openedReadStream := OpenRead[readPath];

which, as a SetDelayed causes the file to be reopened everytime I read from it.

The correct, i.e. appropriate, input of course uses Set (immediate), so the stream is opened just once:

openedReadStream = OpenRead[readPath];

after which the output is now as expected:

0
{"Date,Time,Bid Price,Ask Price,Contributor,Region,City"}
53
{"04/02/2017,00:00.100,1.00266,1.00282,DCFX,ASI,AKL"}

So, rookie error. (On the other hand, it would perhaps have been nice if MMA had issued a warning about reopening an already open file.)

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.