1
$\begingroup$

Hardware: PC with 256GB RAM

OS: Windows 10

Mathematica: Version 13.0.1

One of my projects has led to the creation of a SparseArray whose ByteCount is ~ 35GB in RAM. At the time of creation the SparseArray is functional and returns information when appropriately queried.

When I Export the SparseArray using the .mx format, the data stores on my hard drive with a memory footprint of ~ 17GB. No errors are generated.

When I attempt to Import the same SparseArray using a fresh kernel, the computer spins for about 2 minutes - during this time I see RAM spike to roughly 90GB, and then slowly the RAM usages drops and the kernel just quits. I hear a beep - no error messages are generated, and I can't retrieve my data.

I wish I could say this behavior has not been seen before, but while working in Mathematica v12, I attempted to use BinarySerialize to store a large Association - only to discover I couldn't retrieve it either. Attempts to get help from Wolfram tech support were fruitless as they claimed they didn't have access to a machine with enough RAM to explore the problem. Sigh.

Questions I can't help but ponder:

  1. Has anyone else noticed such erratic behavior trying to retrieve serialized data from Mathematica? (I seem to notice the behavior when the ByteCount of the object reaches a few tens of gigabytes in size.)

  2. Any advice for resolving this? Avoiding BinarySerialize and mx formats and sticking with Put/Get does seem to work, but then my read/write times are glacially slow. Almost a deal breaker kind of slow.

Finally, if there are any adventurous souls with sufficient hardware, here is a link to my SparseArray object. (Will it load on your computer without quitting the kernel?)

https://www.dropbox.com/s/2hk2e6ubbi8ivuq/InformativeKmersSparseArray-A.mx?dl=0

If by some miracle you get it to load, the following should be a successful query: (assuming you store the SparseArray in a variable called "data".)

data[[2322312231212121]]

Thanks for reading.

$\endgroup$
7
  • $\begingroup$ Can you share the code used to serialize the matrix. Sounds like it might have unpacked from SparseArray during serialization. $\endgroup$
    – Edmund
    Mar 18 at 17:46
  • $\begingroup$ @Edmund Based on your suggestion, I tried setting On["Packing"] before importing the SparseArray and no warnings were generated. As far as the serialization code you mention, I am relying on the internal code of Export that understands MX files. I am not doing anything outside of that to provide additional serialization. The literal command is: Export["InformativeKmersSparseArray.mx", informativeKmersSparseArray]; If this is an undetectable unpacking issue, I may need to resort to storing the components needed to build the SparseArray and recreate on the fly.... Thanks! $\endgroup$
    – Todd Allen
    Mar 18 at 18:30
  • 1
    $\begingroup$ You could try DumpSave instead of Export. $\endgroup$
    – Roman
    Mar 18 at 18:33
  • 2
    $\begingroup$ I was able to import without any issues, see here. Mma 13.0.1 on Win 10. Maybe restart your kernel or session. $\endgroup$
    – Edmund
    Mar 18 at 18:33
  • 1
    $\begingroup$ @Edmund Thank you and wow! This is the first evidence I have that the problem is unique to my hardware/software. I have tried restarting the kernel and rebooting the machine many times. I guess I can try resetting Mathematica and reinstalling it. I have a bit of a sinking feeling..... $\endgroup$
    – Todd Allen
    Mar 18 at 18:44

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.