12
$\begingroup$

I have a large jagged list, that is each sub-list has a different length. I would like to Flatten this list for Histogram purposes, but it seems to be taking an inordinate amount of time and memory

jaggedList=Table[RandomReal[1,RandomSample[Range[400000,800000],1]],{n,100}];

Just to illustrate, length of each of elements of the main list

ListPlot[Length/@jaggedList]

list lengths

Full Flatten takes a long time, my real data is several times larger, it gets painfully slow

fullFlatten=Flatten@jaggedList;//AbsoluteTiming
{10.0055,Null}

I noticed flattening non-jagged sub-lists is not a problem

partialFlatten=Flatten/@jaggedList;//AbsoluteTiming
{0.289219,Null}

Memory usage is huge on the final result of the full list, even though number of elements is the same:

ByteCount/@{fullFlatten,partialFlatten,jaggedList}
{1460378864,486808224,486808224}

Would super appreciate any tips on what I can change to make this faster / more memory compact !

$\endgroup$
12
$\begingroup$

Applying Join is much faster than Flatten:

SeedRandom[1]
jaggedList = Table[RandomReal[1, RandomSample[Range[400000, 800000], 1]], {n, 100}];

fullFlatten = Flatten@jaggedList; // AbsoluteTiming // First

8.2375848

fullFlatten2 = Join @@ jaggedList; // AbsoluteTiming // First

0.29729

fullFlatten2 == fullFlatten

True

ByteCount /@ {fullFlatten, fullFlatten2, jaggedList}

{1462957016, 487652456, 487667608}

$\endgroup$
  • $\begingroup$ Super appreciate it, that's exactly what I needed ! Now just to speed up the Histogram! $\endgroup$ – Anatoly Oct 24 '18 at 22:56
17
$\begingroup$

The difference between using Flatten and using Join as in @kglr's answer is that Flatten unpacks. Here is a smaller example:

SeedRandom[1]
list = Table[RandomReal[1, RandomSample[2;;5, 1]], 3]

{{0.269558, 0.445678, 0.158104, 0.751213, 0.965444}, {0.0518202, 0.675946, 0.698472}, {0.344389, 0.830322, 0.556863}}

Turn on packing messages:

On["Packing"]

Then, using Flatten:

Flatten[list]

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

Developer`FromPackedArray::punpack: Unpacking array with dimensions {5} in call to Flatten.

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

Developer`FromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

General::stop: Further output of Developer`FromPackedArray::unpack will be suppressed during this calculation.

Developer`FromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.

General::stop: Further output of Developer`FromPackedArray::punpack will be suppressed during this calculation.

{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}

and using Join:

Join @@ list

{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}

As you can see, using Join generates no unpacking messages, which is why it is much faster.

$\endgroup$
  • $\begingroup$ Appreciate the point about packing, this is helping me troubleshoot more! $\endgroup$ – Anatoly Oct 26 '18 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.