I have a notebook that Imports files from a folder and processes and stores outputs into another folder.

It works well with small number of files, however, when there's a large number of input files, it crashes halfway.

Just to be clear, I'm using the code below to import all my data

allData = Table[Import[...],{i,1,ndatasets}]

Following this, the variable allData is manipulated on.

I have a feeling allData may be too big when the number of files imported is large.

Is there a way to run the notebook in batches, over say, 10 or so files at a time?

Or is there another solution instead of using big variables like allData? In other programming languages, I would use a FOR loop over datasets, and smaller variables that can be "cleared" within my loop.

  • 5
    $\begingroup$ If you do not need to have the entire dataset in memory, you can still use Do to process files one by one. Do in Mathematica is comparate for for or foreach in other languages. $\endgroup$
    – Szabolcs
    Oct 1 '19 at 15:25

Why feel when you can measure?

I usually approach this problem through a Do[] loop. I also have the function MemorySummary[] defined as

MemorySummary[]:=Print["Memory currently in use : ",MemoryInUse[]/1024.^2, "Mb, Maximum memory used in session : ",MaxMemoryUsed[]/1024.^2, "Mb"]

One suggestion would be to preallocate your allData array. You know its length as ndatasets. What else do you know about it? Does each file have the same number of variables?

With this in mind, one basic algorithm looks like

alldata = Range@ndatasets;
 tmpdata = Import[dataset[[jf]], "table"];
 (* here we perform any cleaning or other data manipulation that might be required on the tmpdata array *)
 alldata[[jf]] = tmpdata;
 (* should memory be an issue, here is where you'd Clear[] or Remove[] if you need to *)
 Print[DateString[], ": File "<>ToString@jf<>" using "<>MemorySummary[]],
 {jf, ndatasets}

so we get time taken to read each data set as well as the memory in use.

Extension to parallel processing is fairly straightforward through either ParallelDo[] or manually through Do[] with optional step if you choose.

Your mileage may vary.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.