Import numbers from CSV with OpenRead

Question

I have tried to use OpenRead for my application as it appears to be less of a burden on the memory.

I OpenRead a .csv file to extract coordinates in the form {x,y} in the following way

ClearAll[f];
f = OpenRead["mathfile.csv"];
g = ReadList[f, String, RecordLists -> True];
Close[f];

I can access each record seperately by just calling g[[1]]. However the format is not as numbers and so a function like ListPlot[g[[1]]] doesn't work.

If I use "Number" or "Real" as an option within ReadList I get an error:

"Read:readn: Invalid real number found when reading from 'f'."

Using "String" or "Record" within ReadList doesn't cause a problem. However I would have to convert the "String" or "Record" into a number format.

What am I doing wrong ? I would like to call g[[1]] and have the co-ordinate in a form that can be read by ListPlot

Thanks

Answer 1

Mathematica doesn't have configurable separators for Read and ReadList, except when reading Words and Records, so importing a CSV of numbers is trickier. Some workarounds:

1. Use TSV format, which will "just work."

2. Read a Character to get rid of the comma separator:

   ReadList[stream, {Number, Character, Number}, RecordLists -> True][[All, {1,3}]]
   

3. Set WordSeparators -> {","}, the read Words and apply ToExpression. This is maybe the slowest.

---

BTW, there are a few nice (albeit maybe hefty) answers in this related question that you might want to look at.

Answer 2

If you use OpenRead as described above you'll have to iterate over your records a second time in order to convert strings to pairs of numbers. This is not memory-efficient. Also, if your input stream will no longer be needed after you load the records, it's better to invoke With[{f=OpenRead["mathfile.csv"]},...] so as to localize the variable as a constant.

If you have a CSV with records of the form:

x1, y1
x2, y2
...

you can simply use the Import function's CSV support, like so:

g = Import["mathfile.csv","CSV"]

This will return a two-dimensional list of records of the form {{x1,y1},{x2,y2},...}. Then you get

g[[1]] == {x1,y1}

Use the Import option HeaderLines->n to skip n lines if you have named columns etc.

In my opinion, the capabilities of Import are quite robust and efficient-- even for large files-- and should be considered before falling back to low-level streams, particularly for simple record formats like 2-tuples.

Answer 3

I got some good performance in the following manner. My data was originally an XLSX. I saved it as a MSDOS txt file. Unfortunately, the {x,y} data had long runs punctuated by a text saying 'Segment 1:' e.t.c.. This was stopping the ReadList from working..So after removing the text and whitespace within Excel, it was saved as .txt.I didn't use Import, as it just takes too long and eats up memory (I have a total 300 000 rows of {x,y} data that I want to build a dynamic animation with) with a CSV....

So I basically open the txt for reading and then use ReadList to extract a number of entries in blocks... this is quite quick as my data has a high resolution and so not every data point is necessary for a good looking plot.

ClearAll[str];
str = OpenRead["mathfile1.txt"];
g = Table[ReadList[str, {Real, Number}, 1000], {100}];
Close[str];

Entries can be accessed as g[[1,x]]

I found that if ReadList reads over 15000 entries, my computer starts to seriously hang. I have 8 GB of RAM