Importing large csv files [duplicate]

Question

I'm trying to import a large csv file (around 1.5GB) so I can multiply one of the columns, and plot it against another. There are seven columns in the file when I try and open it in excel, and I need the 2nd,7th and 11th ones. The 2nd one is numbers ranging from 0.5 to 149.5, the 7th is number in the format 0.0000000571 (although not all of them are that specific number). NA is also present in the 7th column, and the 11th column is dates and times in the format dd/mm/yyyy min:hr. When I'm simply using Import it takes forever, and was trying to figure out a faster way to get an output in the same format. I tried looking into ReadList but have no idea how I would make it work. I'm trying to plot the 7th column on the x axis and the 2nd on the y, after selecting parts of it by date. I've already figured out that part but if I could speed up the import it would be really helpful.

the whole file is 14 columns and it also has a row of titles which I don't need to include.

edit: the answers included here How can I import a huge CSV file quickly? wont work, since they don't allow me to separate out specific rows and only import the entire file.

Answer 1

You could consider importing it as text and then using StringSplit.

Example using randomly generated numerical data:

In[11]:= csv = "C:\\Users\\User\\Desktop\\data.csv";
FileByteCount[csv]/10.^6

Out[12]= 1009.82

Importing the csv directly isn't bad in this example.

In[30]:= AbsoluteTiming[temp = Import[csv];]
Clear[temp]

Out[30]= {140.857, Null}

Using ReadList and StringSplit provides a 3 fold improvement.

In[9]:= readColumns[csvPath_, nColumns_, selectColumns_] := With[
  {str = OpenRead[csvPath]},
  temp = ReadList[str, String];
  data = Transpose[
     Flatten[Partition[StringSplit[temp, ","], nColumns], 
      1]][[selectColumns]];
  Close[str];
  Clear[temp];
  data
  ]

AbsoluteTiming[output = readColumns[csv, 11, {2, 7, 11}];]

Out[10]= {48.3006, Null}