Speed up Flatten[] of a large nested list

Question

I have a large jagged list, that is each sub-list has a different length. I would like to Flatten this list for Histogram purposes, but it seems to be taking an inordinate amount of time and memory

jaggedList=Table[RandomReal[1,RandomSample[Range[400000,800000],1]],{n,100}];

Just to illustrate, length of each of elements of the main list

ListPlot[Length/@jaggedList]

Full Flatten takes a long time, my real data is several times larger, it gets painfully slow

fullFlatten=Flatten@jaggedList;//AbsoluteTiming
{10.0055,Null}

I noticed flattening non-jagged sub-lists is not a problem

partialFlatten=Flatten/@jaggedList;//AbsoluteTiming
{0.289219,Null}

Memory usage is huge on the final result of the full list, even though number of elements is the same:

ByteCount/@{fullFlatten,partialFlatten,jaggedList}
{1460378864,486808224,486808224}

Would super appreciate any tips on what I can change to make this faster / more memory compact !

Answer 1

Applying Join is much faster than Flatten:

SeedRandom[1]
jaggedList = Table[RandomReal[1, RandomSample[Range[400000, 800000], 1]], {n, 100}];

fullFlatten = Flatten@jaggedList; // AbsoluteTiming // First

8.2375848

fullFlatten2 = Join @@ jaggedList; // AbsoluteTiming // First

0.29729

fullFlatten2 == fullFlatten

True

ByteCount /@ {fullFlatten, fullFlatten2, jaggedList}

{1462957016, 487652456, 487667608}

Answer 2

The difference between using Flatten and using Join as in @kglr's answer is that Flatten unpacks. Here is a smaller example:

SeedRandom[1]
list = Table[RandomReal[1, RandomSample[2;;5, 1]], 3]

{{0.269558, 0.445678, 0.158104, 0.751213, 0.965444}, {0.0518202, 0.675946, 0.698472}, {0.344389, 0.830322, 0.556863}}

Turn on packing messages:

On["Packing"]

Then, using Flatten:

Flatten[list]

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

Developer`FromPackedArray::punpack: Unpacking array with dimensions {5} in call to Flatten.

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

Developer`FromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

General::stop: Further output of Developer`FromPackedArray::unpack will be suppressed during this calculation.

Developer`FromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.

General::stop: Further output of Developer`FromPackedArray::punpack will be suppressed during this calculation.

{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}

and using Join:

Join @@ list

{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}

As you can see, using Join generates no unpacking messages, which is why it is much faster.