3
$\begingroup$

Reformulating my question in simpler terms.

Take two lists

list1 = RandomInteger[{1, 100000000}, 25000000];
list2 = RandomReal[{1, 10}, 25000000];

Make an Association out of them

as = AssociationThread[list1, list2]; 
Export["somewhere", as, "WDX"];)

And I want to do this for even larger lists. At the moment I can't manage it - the reason being that Mathematica eats all my RAM. I have noticed that when I try creating the Association there is a spike in RAM usage (in steps of a few hundred Megabytes). If a make the lists a bit smaller, so that the command can be evaluated, after the spike is gone, I can go on using as in other calculations with far less RAM being taken. This spike in memory usage is also seen when I Import the exported file (the file is around 200mb).

My question is whether it is possible to avoid this somehow - maybe create (and later Import) the Association in pieces or something else? I also tried using a SparseArray instead, replacing the AssociationThread code by

sa = SparseArray[Table[list1[[i]] -> list2[[i]], {i, 1, Length[list1]}]];

The spkie is still there, even more prominent than with the Association. Not only that but another problem comes up, even if I reduce the number of elements significantly, but keep the large values in the first list, which means that the dimension of the SparseArray will be enormous. When I try using it in a calculation, my Mathematica just resets (that is - clears all definitions - like when it runs out of memory). Is this expected behaviour or am I doing something wrong.

Thank you

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ You will have better chances for a good answer, if you strip off all the specifics of your particular case, and reduce this to a minimal and simple example illustrating the problem. In the form it is now, it takes too much effort to get to the heart of the matter, for the readers / potential answerers. $\endgroup$
    – Leonid Shifrin
    Jun 18, 2015 at 13:49
  • $\begingroup$ @LeonidShifrin Editted my question $\endgroup$
    – ThunderBiggi
    Jun 19, 2015 at 10:53
  • 1
    $\begingroup$ I'm somewhat surprised that On["Packing"] does in this case not trigger an unpack message. It looks like Associations are somewhat inefficient to construct with AssociationThread for that many keys even though no message is shown. The alternatives I know will show similar problems. If working with data of that size Mathematica will usually only perform well if you can avoid unpacking and it seems impossible or at least difficult to avoid unpacking here. I even suspect that it is just the final Association which is huge. Can you work with the packed arrays directly? $\endgroup$
    – Albert Retey
    Jun 19, 2015 at 13:48
  • 1
    $\begingroup$ As for storing data: I have not checked recent versions but WDX has a tradition to be slow and memory demanding, since MX nowadays will even work cross platform in many ways I'd suggest to export as that... $\endgroup$
    – Albert Retey
    Jun 19, 2015 at 13:48

1 Answer 1

Reset to default
3
$\begingroup$

This is likely primarily due to array unpacking. See here:

list1 takes only 

ByteCount[list1]
(* 200000144 *)

space when packed, but this increases to 

ByteCount@Developer`FromPackedArray[list1]
(* 600000080 *)

when unpacking. It's the same for list2, so the total rises to 1.2 GB. Make a rule list out of these and we reach double that:

rules = Thread[list1 -> list2];

ByteCount[rules]
(* 2400000080 *)

The association adds a bit of overhead over that 2.4 GB, reaching 2.6 GB.

One way to create a sparse array is SparseArray[rules], which involves all this unpacking, so we know where the unpacking comes from. You might wonder if the SparseArray[list1 -> list2] syntax fares better, but unfortunately it doesn't seem to be the case. Maybe internally it still creates rules? The result is fortunately only 350 MB in size though.

There's yet another way to create the sparse array:

sa = SparseArray[{}, {100000000}];
sa[[list1]] = list2;

This still creates a large memory spike, but the spike is only half the size of the other creation methods.

I realize this is not a full solution but I hope it can still help.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.