Flattening large list of lists

Question

Part of my code takes in list of lists of around 200 integers and produce a new list of lists. Following the approach in this post and this post, I created the new list using the linked list approach and Flatten it at the end.

So far the code runs fine, until I hit the mark when my list is of size $10^8$. There, the linked list is still produced (so the 32GB RAM is enough to at least store the list), however, flattening it caused Mathematica to run out of memory and the kernel quits.

Is there a better alternative to deal with this memory overhead? I tried Sow/Reap but they are extremely slow to even reach this progress. I am not sure if dividing the creation of the list into blocks will help (the data is created sequentially from the previous list)

Edit: The theoretical maximum of the list size is about $2∗10^8$, so it will be great if the solution can deal with that size as well.

The list of lists is not a matrix. Each row has different length. Also the whole list has to be sorted (with some custom defined function) so I at least need to keep the whole list in memory?

Edit: Sample data starts like:

{{49}, 
{49, 50}, 
{34, 49, 50}, 
{34, 49, 50, 69}, 
{34, 49, 50, 53, 69}, 
{34, 37, 49, 50, 53, 69}, 
{21, 34, 37, 49, 50, 53, 69}, 
{5, 21, 34, 37, 49, 50, 53, 69}, 
{34, 37, 49, 50, 53, 69, 118}, 
{21, 34, 37, 49, 50, 53, 69, 118}, 
{5, 21, 34, 37, 49, 50, 53, 69, 118}, 
{21, 34, 37, 38, 49, 50, 53, 69, 118},
{5, 21, 34, 37, 38, 49, 50, 53, 69, 118},
......
}

and I need to create a list that is roughly the size of the same order of magnitude, with some rows removed and some rows added, sequentially from the rows of the current list, and then sort it at the end.

Answer 1

Here is an approach that might or might not work, but at least the memory impact is small. I'm using Internal`Bag to store your linked list. With Internal`StuffBag you can put your sublists into the bag; one after another. When you have done your computation, use Internal`BagPart[bag,all] to extract your ragged list as normal Mathematica list. It seems in this step, no copying is done and the memory footprint is small.

Creating a function for returning random integer lists of different length around 200 and initializing your bag:

rand[] := With[{n = RandomInteger[{150, 250}]},
  RandomInteger[1000, n]
];
bag = Internal`Bag[];

Measuring the current memory-in-use in MB with MemoryInUse[]/2^20. gives about 50MB. Now fill your bag with 10^5 lists

Do[Internal`StuffBag[bag, rand[]], {10^5}]

After this, Mathematica has 220MB of memory in use. Getting the bag out as a normal list of lists

list = Internal`BagPart[bag, All];

That adds only one MB of memory to the footprint. Without a concrete example, it is hard to give further advice, except that you should use Henrik's advice and ensure that your list of integers are indeed packed.