12
$\begingroup$

There are somewhat similar questions on here, but I didn't find a satisfying answer for this case: I import a 1.7 GB binary data file using the following command:

BinaryReadList["/path/to/file", "Real64"]

It runs fine but the memory usage peaks at 18 GB during the import process, which is about 10 times the file size. Is there a more efficient way to do this? Why does Mathematica consume that much memory?

edit: It turned out, that almost all the memory is consumed by displaying the output. When I add a semicolon after the command, the consumed memory is close to the file size. This might not be a bug, but I still wonder why Mathematica behaves like this.

here is a test case:

  1. generate a file with:
BinaryWrite["test.dat", ConstantArray[99, 1*^8], "Integer8"];
  1. quit the kernel to reset MaxMemoryUsed[] and execute the following:
membefore = MemoryInUse[];
BinaryReadList["test.dat", "Real64"]
(MaxMemoryUsed[] - membefore)/(UnitConvert[FileSize["test.dat"], "Bytes"][[1]])
  1. don't forget to
DeleteFile["test.dat"];

this basically divides the MaxMemoryUsed by the FileSize and outputs the factor. You will notice a big difference depending on whether you add a semicolon after BinaryReadList["test.dat", "Real64"] or not.

(I use Mathematica 12 on Ubuntu)

$\endgroup$
  • 7
    $\begingroup$ Interesting. Can you please create a minimal example that other people can run on their systems to see whether it is OS-dependent? Filing a big report at Wolfram Support is a good idea anyways. $\endgroup$ – Henrik Schumacher Nov 13 '19 at 15:44
  • 4
    $\begingroup$ A BIG bug report :D @Henrik $\endgroup$ – Szabolcs Nov 14 '19 at 6:47
  • 3
    $\begingroup$ The problem is that the full data are converted into boxes for display in the FE, which causes the massive memory usage. If I recall correctly, this was discussed in a recent(ish) live-stream (in relation to Short), not sure what the conclusion was however (I think they were talking about a way to only typeset stuff if necessary) $\endgroup$ – Lukas Lang Nov 14 '19 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.