1
$\begingroup$

Hardware: PC with 256GB RAM

OS: Windows 10

Mathematica: Version 13.0.1

One of my projects has led to the creation of a SparseArray whose ByteCount is ~ 35GB in RAM. At the time of creation the SparseArray is functional and returns information when appropriately queried.

When I Export the SparseArray using the .mx format, the data stores on my hard drive with a memory footprint of ~ 17GB. No errors are generated.

When I attempt to Import the same SparseArray using a fresh kernel, the computer spins for about 2 minutes - during this time I see RAM spike to roughly 90GB, and then slowly the RAM usages drops and the kernel just quits. I hear a beep - no error messages are generated, and I can't retrieve my data.

I wish I could say this behavior has not been seen before, but while working in Mathematica v12, I attempted to use BinarySerialize to store a large Association - only to discover I couldn't retrieve it either. Attempts to get help from Wolfram tech support were fruitless as they claimed they didn't have access to a machine with enough RAM to explore the problem. Sigh.

Questions I can't help but ponder:

  1. Has anyone else noticed such erratic behavior trying to retrieve serialized data from Mathematica? (I seem to notice the behavior when the ByteCount of the object reaches a few tens of gigabytes in size.)

  2. Any advice for resolving this? Avoiding BinarySerialize and mx formats and sticking with Put/Get does seem to work, but then my read/write times are glacially slow. Almost a deal breaker kind of slow.

Finally, if there are any adventurous souls with sufficient hardware, here is a link to my SparseArray object. (Will it load on your computer without quitting the kernel?)

https://www.dropbox.com/s/2hk2e6ubbi8ivuq/InformativeKmersSparseArray-A.mx?dl=0

If by some miracle you get it to load, the following should be a successful query: (assuming you store the SparseArray in a variable called "data".)

data[[2322312231212121]]

Thanks for reading.

Improve this question
$\endgroup$
7
  • $\begingroup$ Can you share the code used to serialize the matrix. Sounds like it might have unpacked from SparseArray during serialization. $\endgroup$
    – Edmund
    Mar 18 at 17:46
  • $\begingroup$ @Edmund Based on your suggestion, I tried setting On["Packing"] before importing the SparseArray and no warnings were generated. As far as the serialization code you mention, I am relying on the internal code of Export that understands MX files. I am not doing anything outside of that to provide additional serialization. The literal command is: Export["InformativeKmersSparseArray.mx", informativeKmersSparseArray]; If this is an undetectable unpacking issue, I may need to resort to storing the components needed to build the SparseArray and recreate on the fly.... Thanks! $\endgroup$
    – Todd Allen
    Mar 18 at 18:30
  • 1
    $\begingroup$ You could try DumpSave instead of Export. $\endgroup$
    – Roman
    Mar 18 at 18:33
  • 2
    $\begingroup$ I was able to import without any issues, see here. Mma 13.0.1 on Win 10. Maybe restart your kernel or session. $\endgroup$
    – Edmund
    Mar 18 at 18:33
  • 1
    $\begingroup$ @Edmund Thank you and wow! This is the first evidence I have that the problem is unique to my hardware/software. I have tried restarting the kernel and rebooting the machine many times. I guess I can try resetting Mathematica and reinstalling it. I have a bit of a sinking feeling..... $\endgroup$
    – Todd Allen
    Mar 18 at 18:44

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.