15
$\begingroup$

I use

f = OpenRead["largefile.m"]
x = Read[f];
Close[f];

to read in a Mathematica expression from a large file largefile.m (about 3 GB) and assign it to x. The Read[f] step takes very long time. How to monitor its progress? BTW, if the file largefile.m is larger than 4.1 GB (as ls -la displays), will this approach work?

|improve this question
$\endgroup$
  • $\begingroup$ Out of curiosity: have you tried using Get? $\endgroup$ – Mr.Wizard♦ Apr 23 '12 at 6:29
  • $\begingroup$ Also, what does your expression look like and how was it generated? Could you write it in a format that is more easily read in parts rather than all at once? $\endgroup$ – Mr.Wizard♦ Apr 23 '12 at 6:35
  • $\begingroup$ I get Get:bigfile error using Get[]. $\endgroup$ – qazwsx Apr 23 '12 at 6:44
  • $\begingroup$ I could if that's the best approach. Right now it is {{...}, {...}} where ... is a long sequence of expressions. $\endgroup$ – qazwsx Apr 23 '12 at 6:45
  • $\begingroup$ Last year at the Wolfram Tech Conference, I attended a presentation about the various ways to deal with files. You might want to look at the presentation that can be found here $\endgroup$ – Sjoerd C. de Vries Apr 23 '12 at 8:42
10
$\begingroup$

I am fairly sure that for files of this size, you'll have problems loading them into Mathematica (unless you have truly huge amount of RAM on your machine, but perhaps even in this case). If your file contains newline-terminated strings, or if you otherwise know its structure (types stored there), consider using ReadList or BinaryReadList. I gave one example in my post here. In this way, you can attach something like ProgressIndicator to it, and monitor the progress. Also, I would recommend to use something similar to large data framework from this answer, to convert the contents of your file / parts of resulting expression, to a file-backed form, since operating on the entire dataset in-memory might not be possible and / or efficient for such huge amounts of data.

|improve this answer
$\endgroup$
6
$\begingroup$

I've been using BinaryReadList to load reasonably large 100-500MB files and its been working well.

However, I've never had a way to watch the progress. Based @Leonid's answer I hacked together a ProgressIndicator for the function.

In my case my large data files consist of sets of 4 32bit (big endian) floats. Here I read the file in 100 steps. Timing is similiar to reading the file all at once.

ImportProgress[filename_] := Module[{str, data, n},
  (*open stream*)
  str = OpenRead[filename, BinaryFormat -> True];
  data = {}; n = 1;
  (*display the dynamic progress indicator*)
  Print[ProgressIndicator[Dynamic[n/100]]];
  (*read data*)
  While[n < 100,
   AppendTo[data, 
    BinaryReadList[str, {"Real32", "Real32", "Real32", "Real32"}, 
     Ceiling[FileByteCount[filename]/16/100], ByteOrdering -> +1]];
   n++];
  (*close stream*)
  Close[str];
  (*return the data*)
  data
]

(*read the data*)
data = ImportProgress["R06_14539-v01.pos"];

Here is a more elegant alternative using Monitor where the ProgressIndicator disappears after the file finishes loading:

ImportProgressMonitor[filename_] := Monitor[
  Module[{str, data},
   (*open stream*)
   str = OpenRead[filename, BinaryFormat -> True];
   data = {};
   (*read data*)
   n = 1;
   While[n < 100,
    AppendTo[data, 
     BinaryReadList[str, {"Real32", "Real32", "Real32", "Real32"}, 
      Ceiling[FileByteCount[filename]/16/100], ByteOrdering -> +1]];
    n++];
   (*close stream*)
   Close[str];
   data
   ],
  (*monitor the progress*)
  ProgressIndicator[n/100]
]
|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.