GroupBy with multiple conditions running very slowly

Question

I have a list lst matching the pattern {{_String, _String, _?NumericQ, _?NumericQ}..}. The outer list has about 150,000 elements (inner lists), so it is certainly not very long. When I run:

lst // GroupBy[(#[[2]] &) -> (Drop[#, {2}] &)] // Map[GroupBy[First -> Rest]]

the code takes about 0.4 seconds to return the result. But when I run

lst // GroupBy[{(#[[2]] &) -> (Drop[#, {2}] &), First -> Rest}]

the code takes about five minutes to return the result!

Is the second approach not recommended? There is nothing in the official documentation to discourage its use, but it is about three orders of magnitude worse in terms of performance.

Answer 1

I suspect this has to do with the number of times you're changing the lengths of lists and the efficiency of Map. In general, I would discourage an approach that gradually reduces the lengths of lists. Your code uses Drop, Rest and First in sequence. It's probably more efficient to retain the sublists all through the calculation and only drop whatever you don't need at the end. Compare:

list = With[{n = 10000},
  Transpose@Join[
    RandomChoice[Alphabet[], {2, n}],
    RandomReal[{0, 1}, {2, n}]
    ]
  ];

RepeatedTiming[
 list // GroupBy[(#[[2]] &) -> (Drop[#, {2}] &)] // 
   Map[GroupBy[First -> Rest]];]

with

Part[
 GroupBy[First] /@ GroupBy[list, #[[2]] &],
 All,
 All,
 All,
 {3, 4}
 ]; // RepeatedTiming

On my machine this runs a bit faster than your code. So instead of dropping elements while going through the calculation, just do it all at the end by using Part (which is a very efficient function).

Answer 2

lst = With[{c = CharacterRange["a", "z"], r = RandomReal[{-2, 2}, 20]},
  Flatten[{RandomChoice[c, {#, 2}], RandomChoice[r, {#, 2}]}, {{2}, {1, 3}}] &[150000]];

As a workaround you can use the 3rd argument of GroupBy which correspond to the functionality of Map:

a = lst // GroupBy[(#[[2]] &) -> (Drop[#, {2}] &)] // Map[GroupBy[First -> Rest]];
b = GroupBy[lst[[All, {2, 1, 3, 4}]], #, GroupBy[#]] &[First -> Rest];
a === b

True