13
$\begingroup$
GroupBy[{1, 1, 1, 1, 1, 1}, # > RandomReal[{0, 2}] &]

sometimes returns something like

<|False -> {1, 1, 1, 1}, True -> {1, 1}|>

but then again it might just return

<|True -> {1, 1, 1, 1}|>

Very strange. Apparently this issue can arise without duplicates though. The following two always have the same results.

a = 0.7;
GroupBy[{1, 1, 1, 1, 1, 1}, # > (a += 0.1) &]
(* <|False -> {1, 1, 1, 1}|> *)

a = 0.7;
GroupBy[{1.01, 1.02, 1.03, 1.04, 1.05, 1.06}, # > (a += 0.1) &]
(* <|False -> {1.04, 1.05, 1.06}|> *)

Is there a good reason for this? Seems like a bug.

$\endgroup$
11
  • 1
    $\begingroup$ Very interesting. Look also at a = 0.7; GroupBy[{1.01, 1.02, 1.03, 1.04, 1.05, 1.06}, Echo[#, "Check"] > (Echo[a += 0.1, "a ="]) &]. It doesn't seem to check elements sequentially. $\endgroup$
    – thorimur
    May 3, 2021 at 22:23
  • 1
    $\begingroup$ Also, apparently GroupBy[Table[i, {i,6}], Echo[#] > Echo[RandomReal[{0, 2}], "random"] &] sometimes checks some list elements multiple times? weird... $\endgroup$
    – thorimur
    May 3, 2021 at 22:26
  • 1
    $\begingroup$ oh! it re-checks the first element that gives each key, i think. so, it'll always re-check the first element of the list, and then the first element that produced whatever the next key is, etc.. one can check this with other functions, e.g. GroupBy[Table[i, {i, 8}], Floor[Echo[#]/3] &]. $\endgroup$
    – thorimur
    May 3, 2021 at 22:36
  • 1
    $\begingroup$ This has nothing to do with duplicates, nor is it a bug IMO. It is doing what it should. It effectively does a GatherBy using the criteria, then checks the same against the first member(s) of the result to build the association. When the latter test(s) are the same, only the latter is retained, as in AssociationThread... $\endgroup$
    – ciao
    May 3, 2021 at 22:36
  • 2
    $\begingroup$ that's a bug in my view. it's not using the correct key-value pair $\endgroup$
    – thorimur
    May 3, 2021 at 22:37

2 Answers 2

13
$\begingroup$

The problem is that GroupBy uses the key function once to map over all of the elements, and then it uses it again for each distinct key. As an example, consider:

SeedRandom[3];

f[x_] := x > RandomReal[2]

TraceScan[
    Identity,
    GroupBy[{1, 1, 1, 1, 1, 1}, f],
    _Map,
    Print @* Rule,
    TraceInternal -> True
];

f/@{1,1,1,1,1,1}->{True,True,True,True,True,False}

Note that there are 5 trues and 1 false when mapping. From this information, GroupBy constructs the following association:

<|f[1] -> {1, 1, 1, 1, 1}, f[1] -> {1}|>

This is where the last two function calls of f occurs.

SeedRandom[3]

TraceScan[
    Identity,
    GroupBy[{1, 1, 1, 1, 1, 1}, f],
    _f,
    Print @* Rule,
    TraceOff -> Map,
    TraceInternal->True
];

f[1]->False

f[1]->False

Note that both function calls return False, so the final association is:

<|False -> {1, 1, 1, 1, 1}, False -> {1}|>

<|False -> {1}|>

When an association has duplicate keys, only the last duplicate is retained. This explains the behavior you see.

An alternative to using GroupBy is to use the ResourceFunction "GroupByList":

SeedRandom[3]

ResourceFunction["GroupByList"][{1, 1, 1, 1, 1, 1}, f /@ {1, 1, 1, 1, 1, 1}]

<|True -> {1, 1, 1, 1, 1}, False -> {1}|>

$\endgroup$
5
  • 1
    $\begingroup$ nice use of TraceScan! would you consider it a bug? although this explains the result, shouldn't GroupBy keep around the keys it finds "along the way" to build the resulting association, instead of producing them again (given cases like this)? $\endgroup$
    – thorimur
    May 3, 2021 at 22:46
  • $\begingroup$ (I suppose one could hack it to do so by using a memoizing f manually, but still...) $\endgroup$
    – thorimur
    May 3, 2021 at 22:48
  • 2
    $\begingroup$ @thorimur I think it should be reported at least as an unexpected behavior. $\endgroup$ May 4, 2021 at 9:38
  • 4
    $\begingroup$ @AlexeyPopkov FYI and others, this has been discussed internally back in 2018, and the bug has been reported. It wasn't deemed the top priority one however. Perhaps it is time to remind ourselves about it. $\endgroup$ May 4, 2021 at 12:58
  • 2
    $\begingroup$ IMO this is at best a documentation bug if it is not mentioned. $\endgroup$
    – Carsten S
    May 4, 2021 at 15:51
3
$\begingroup$

Per my comment, observe:

myGB[lst_, tst_] := With[{gb = GatherBy[lst, tst]},
   If[Length@gb == 1, AssociationThread[{tst@gb[[1, 1]]} -> gb],
    AssociationThread[{tst@gb[[1, 1]], tst@gb[[2, 1]]} -> gb]]];

Then test with the following. The results will be the same.

xxx = {1, 1, 1, 1, 1, 1};
seed = RandomInteger[500];

SeedRandom[seed];
GroupBy[xxx, # > RandomReal[{0, 2}] &]

SeedRandom[seed];
myGB[xxx, (# > RandomReal[{0, 2}] &)]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.