Part of my code takes in list of lists of around 200 integers and produce a new list of lists. Following the approach in this post and this post, I created the new list using the linked list approach and Flatten
it at the end.
So far the code runs fine, until I hit the mark when my list is of size $10^8$. There, the linked list is still produced (so the 32GB RAM is enough to at least store the list), however, flattening it caused Mathematica to run out of memory and the kernel quits.
Is there a better alternative to deal with this memory overhead? I tried Sow
/Reap
but they are extremely slow to even reach this progress. I am not sure if dividing the creation of the list into blocks will help (the data is created sequentially from the previous list)
Edit: The theoretical maximum of the list size is about $2∗10^8$, so it will be great if the solution can deal with that size as well.
The list of lists is not a matrix. Each row has different length. Also the whole list has to be sorted (with some custom defined function) so I at least need to keep the whole list in memory?
Edit: Sample data starts like:
{{49},
{49, 50},
{34, 49, 50},
{34, 49, 50, 69},
{34, 49, 50, 53, 69},
{34, 37, 49, 50, 53, 69},
{21, 34, 37, 49, 50, 53, 69},
{5, 21, 34, 37, 49, 50, 53, 69},
{34, 37, 49, 50, 53, 69, 118},
{21, 34, 37, 49, 50, 53, 69, 118},
{5, 21, 34, 37, 49, 50, 53, 69, 118},
{21, 34, 37, 38, 49, 50, 53, 69, 118},
{5, 21, 34, 37, 38, 49, 50, 53, 69, 118},
......
}
and I need to create a list that is roughly the size of the same order of magnitude, with some rows removed and some rows added, sequentially from the rows of the current list, and then sort it at the end.
AppendTo
? $\endgroup$Developer`PackedArrayQ
applied to that list returnTrue
? $\endgroup$Join@@list
. This should need less memory for ragged lists. If each of the sublists is an packed array, then alsoJoin@@list
should be a packed array. $\endgroup$