1
$\begingroup$

Consider some large ".csv" file. For instance:

file = Export[NotebookDirectory[] <> "file.csv", RandomChoice[Alphabet[], 1000000], "TABLE"]

enter image description here

What is an efficient way to read an arbitrary line from this file? (Import uses up all my RAM and stalls my computer.)

Attempt

I tried the following code involving ReadList but it wastes computation building a giant list. It is too slow for my (actual) purposes.

lineNumber = 987654;
Last[ReadList[file, String, lineNumber] ]

Question

Write a program that outputs the $987654$th line from file

$\endgroup$
2
  • 5
    $\begingroup$ Have you tried opening it as a stream and using Skip then Read ? See also SetStreamPosition $\endgroup$
    – flinty
    Apr 6, 2021 at 17:28
  • 3
    $\begingroup$ By the way - it's a lot faster to generate that file like this Export["file.csv", RandomChoice[Alphabet[], 1000000], "TABLE"]; instead of calling RandomChoice multiple times in a Table. $\endgroup$
    – flinty
    Apr 6, 2021 at 17:33

1 Answer 1

2
$\begingroup$

What version of Mathematica are you using?

First off, the dataset recommended imports in 1.1 seconds on my machine:

In[62]:= file = 
  Export["file.csv", RandomChoice[Alphabet[], 1000000], "CSV"];

In[63]:= Import["file.csv", "Dimensions"]

Out[63]= {1000000, 1}

In[68]:= Import["file.csv", {"Data"}] // MaxMemoryUsed // AbsoluteTiming

Out[68]= {1.14441, 176206064}

But that said, if you have a much larger file than this example and still wish to extract only a specific row of data, this does so with minimal memory overhead and a very fast import:

In[67]:= Import["file.csv", {"Data", 987654}] // MaxMemoryUsed // AbsoluteTiming

Out[67]= {0.34555, 23767024}

It will properly address the CSV spec while only keeping rows requested in memory. Once the data is found it stops parsing. This syntax supports integers, lists, and spans.

$\endgroup$
1
  • $\begingroup$ Brilliant! Import["file.csv", {"Data", 987654}] works on my actual 17 GB dataset. Thanks. Btw I am using Mathematica 12.2 $\endgroup$
    – Conor
    Apr 8, 2021 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.