There are somewhat similar questions on here, but I didn't find a satisfying answer for this case: I import a 1.7 GB binary data file using the following command:
It runs fine but the memory usage peaks at 18 GB during the import process, which is about 10 times the file size. Is there a more efficient way to do this? Why does Mathematica consume that much memory?
edit: It turned out, that almost all the memory is consumed by displaying the output. When I add a semicolon after the command, the consumed memory is close to the file size. This might not be a bug, but I still wonder why Mathematica behaves like this.
here is a test case:
- generate a file with:
BinaryWrite["test.dat", ConstantArray[99, 1*^8], "Integer8"];
- quit the kernel to reset MaxMemoryUsed and execute the following:
membefore = MemoryInUse; BinaryReadList["test.dat", "Real64"] (MaxMemoryUsed - membefore)/(UnitConvert[FileSize["test.dat"], "Bytes"][])
- don't forget to
this basically divides the MaxMemoryUsed by the FileSize and outputs the factor. You will notice a big difference depending on whether you add a semicolon after
BinaryReadList["test.dat", "Real64"] or not.
(I use Mathematica 12 on Ubuntu)