# Speed up Flatten[] of a large nested list

I have a large jagged list, that is each sub-list has a different length. I would like to Flatten this list for Histogram purposes, but it seems to be taking an inordinate amount of time and memory

jaggedList=Table[RandomReal[1,RandomSample[Range[400000,800000],1]],{n,100}];


Just to illustrate, length of each of elements of the main list

ListPlot[Length/@jaggedList]


Full Flatten takes a long time, my real data is several times larger, it gets painfully slow

fullFlatten=Flatten@jaggedList;//AbsoluteTiming
{10.0055,Null}


I noticed flattening non-jagged sub-lists is not a problem

partialFlatten=Flatten/@jaggedList;//AbsoluteTiming
{0.289219,Null}


Memory usage is huge on the final result of the full list, even though number of elements is the same:

ByteCount/@{fullFlatten,partialFlatten,jaggedList}
{1460378864,486808224,486808224}


Would super appreciate any tips on what I can change to make this faster / more memory compact !

## 2 Answers

Applying Join is much faster than Flatten:

SeedRandom[1]
jaggedList = Table[RandomReal[1, RandomSample[Range[400000, 800000], 1]], {n, 100}];

fullFlatten = Flatten@jaggedList; // AbsoluteTiming // First


8.2375848

fullFlatten2 = Join @@ jaggedList; // AbsoluteTiming // First


0.29729

fullFlatten2 == fullFlatten


True

ByteCount /@ {fullFlatten, fullFlatten2, jaggedList}


{1462957016, 487652456, 487667608}

• Super appreciate it, that's exactly what I needed ! Now just to speed up the Histogram! Commented Oct 24, 2018 at 22:56

The difference between using Flatten and using Join as in @kglr's answer is that Flatten unpacks. Here is a smaller example:

SeedRandom[1]
list = Table[RandomReal[1, RandomSample[2;;5, 1]], 3]


{{0.269558, 0.445678, 0.158104, 0.751213, 0.965444}, {0.0518202, 0.675946, 0.698472}, {0.344389, 0.830322, 0.556863}}

Turn on packing messages:

On["Packing"]


Then, using Flatten:

Flatten[list]


DeveloperFromPackedArray::unpack: Unpacking array in call to HoldForm.

DeveloperFromPackedArray::punpack: Unpacking array with dimensions {5} in call to Flatten.

DeveloperFromPackedArray::unpack: Unpacking array in call to HoldForm.

DeveloperFromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.

DeveloperFromPackedArray::unpack: Unpacking array in call to HoldForm.

General::stop: Further output of DeveloperFromPackedArray::unpack will be suppressed during this calculation.

DeveloperFromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.

General::stop: Further output of DeveloperFromPackedArray::punpack will be suppressed during this calculation.

{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}

and using Join:

Join @@ list


{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}

As you can see, using Join generates no unpacking messages, which is why it is much faster.

• Appreciate the point about packing, this is helping me troubleshoot more! Commented Oct 26, 2018 at 18:32