-1
$\begingroup$

This question already has an answer here:

I'm trying to import a large csv file (around 1.5GB) so I can multiply one of the columns, and plot it against another. There are seven columns in the file when I try and open it in excel, and I need the 2nd,7th and 11th ones. The 2nd one is numbers ranging from 0.5 to 149.5, the 7th is number in the format 0.0000000571 (although not all of them are that specific number). NA is also present in the 7th column, and the 11th column is dates and times in the format dd/mm/yyyy min:hr. When I'm simply using Import it takes forever, and was trying to figure out a faster way to get an output in the same format. I tried looking into ReadList but have no idea how I would make it work. I'm trying to plot the 7th column on the x axis and the 2nd on the y, after selecting parts of it by date. I've already figured out that part but if I could speed up the import it would be really helpful.

the whole file is 14 columns and it also has a row of titles which I don't need to include.

edit: the answers included here How can I import a huge CSV file quickly? wont work, since they don't allow me to separate out specific rows and only import the entire file.

$\endgroup$

marked as duplicate by m_goldberg, gwr, Henrik Schumacher, Yves Klett, Carl Lange Jan 24 at 16:24

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ You have figured out how to import the Excel sheet and only need help with the importing of the file speed? $\endgroup$ – Jose Enrique Calderon Jan 22 at 21:49
  • $\begingroup$ I strongly suggest to look at my solution here over other large CSV posts: mathematica.stackexchange.com/questions/183599/… $\endgroup$ – GenericAccountName Jan 23 at 17:59
  • $\begingroup$ @JoseECalderon exactly, the solution that b-shields left seems to work but now my data isn't being read as numbers $\endgroup$ – nikkibell Jan 23 at 18:58
0
$\begingroup$

You could consider importing it as text and then using StringSplit.

Example using randomly generated numerical data:

In[11]:= csv = "C:\\Users\\User\\Desktop\\data.csv";
FileByteCount[csv]/10.^6

Out[12]= 1009.82

Importing the csv directly isn't bad in this example.

In[30]:= AbsoluteTiming[temp = Import[csv];]
Clear[temp]

Out[30]= {140.857, Null}

Using ReadList and StringSplit provides a 3 fold improvement.

In[9]:= readColumns[csvPath_, nColumns_, selectColumns_] := With[
  {str = OpenRead[csvPath]},
  temp = ReadList[str, String];
  data = Transpose[
     Flatten[Partition[StringSplit[temp, ","], nColumns], 
      1]][[selectColumns]];
  Close[str];
  Clear[temp];
  data
  ]

AbsoluteTiming[output = readColumns[csv, 11, {2, 7, 11}];]

Out[10]= {48.3006, Null}
$\endgroup$
  • 1
    $\begingroup$ This does not take into account the RFC rules for CSV so only works if there are no " characters, newlines, or commas in the data.Take a peek at mathematica.stackexchange.com/questions/183599/… to get an idea of an approach that chunks the data but still uses Import to parse CSV-isms $\endgroup$ – GenericAccountName Jan 23 at 18:06
  • $\begingroup$ This seems to work but now my data points aren't being read as numbers. how do I fix that? $\endgroup$ – nikkibell Jan 23 at 18:56
  • $\begingroup$ ToExpression will convert the number strings to numbers. $\endgroup$ – b-shields Jan 23 at 21:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.