10
$\begingroup$
GroupBy[{1, 1, 1, 1, 1, 1}, # > RandomReal[{0, 2}] &]

sometimes returns something like

<|False -> {1, 1, 1, 1}, True -> {1, 1}|>

but then again it might just return

<|True -> {1, 1, 1, 1}|>

Very strange. Apparently this issue can arise without duplicates though. The following two always have the same results.

a = 0.7;
GroupBy[{1, 1, 1, 1, 1, 1}, # > (a += 0.1) &]
(* <|False -> {1, 1, 1, 1}|> *)

a = 0.7;
GroupBy[{1.01, 1.02, 1.03, 1.04, 1.05, 1.06}, # > (a += 0.1) &]
(* <|False -> {1.04, 1.05, 1.06}|> *)

Is there a good reason for this? Seems like a bug.

$\endgroup$
10
  • 1
    $\begingroup$ Very interesting. Look also at a = 0.7; GroupBy[{1.01, 1.02, 1.03, 1.04, 1.05, 1.06}, Echo[#, "Check"] > (Echo[a += 0.1, "a ="]) &]. It doesn't seem to check elements sequentially. $\endgroup$ – thorimur May 3 at 22:23
  • 1
    $\begingroup$ Also, apparently GroupBy[Table[i, {i,6}], Echo[#] > Echo[RandomReal[{0, 2}], "random"] &] sometimes checks some list elements multiple times? weird... $\endgroup$ – thorimur May 3 at 22:26
  • 1
    $\begingroup$ oh! it re-checks the first element that gives each key, i think. so, it'll always re-check the first element of the list, and then the first element that produced whatever the next key is, etc.. one can check this with other functions, e.g. GroupBy[Table[i, {i, 8}], Floor[Echo[#]/3] &]. $\endgroup$ – thorimur May 3 at 22:36
  • 1
    $\begingroup$ This has nothing to do with duplicates, nor is it a bug IMO. It is doing what it should. It effectively does a GatherBy using the criteria, then checks the same against the first member(s) of the result to build the association. When the latter test(s) are the same, only the latter is retained, as in AssociationThread... $\endgroup$ – ciao May 3 at 22:36
  • 1
    $\begingroup$ that's a bug in my view. it's not using the correct key-value pair $\endgroup$ – thorimur May 3 at 22:37
13
$\begingroup$

The problem is that GroupBy uses the key function once to map over all of the elements, and then it uses it again for each distinct key. As an example, consider:

SeedRandom[3];

f[x_] := x > RandomReal[2]

TraceScan[
    Identity,
    GroupBy[{1, 1, 1, 1, 1, 1}, f],
    _Map,
    Print @* Rule,
    TraceInternal -> True
];

f/@{1,1,1,1,1,1}->{True,True,True,True,True,False}

Note that there are 5 trues and 1 false when mapping. From this information, GroupBy constructs the following association:

<|f[1] -> {1, 1, 1, 1, 1}, f[1] -> {1}|>

This is where the last two function calls of f occurs.

SeedRandom[3]

TraceScan[
    Identity,
    GroupBy[{1, 1, 1, 1, 1, 1}, f],
    _f,
    Print @* Rule,
    TraceOff -> Map,
    TraceInternal->True
];

f[1]->False

f[1]->False

Note that both function calls return False, so the final association is:

<|False -> {1, 1, 1, 1, 1}, False -> {1}|>

<|False -> {1}|>

When an association has duplicate keys, only the last duplicate is retained. This explains the behavior you see.

An alternative to using GroupBy is to use the ResourceFunction "GroupByList":

SeedRandom[3]

ResourceFunction["GroupByList"][{1, 1, 1, 1, 1, 1}, f /@ {1, 1, 1, 1, 1, 1}]

<|True -> {1, 1, 1, 1, 1}, False -> {1}|>

$\endgroup$
5
  • 1
    $\begingroup$ nice use of TraceScan! would you consider it a bug? although this explains the result, shouldn't GroupBy keep around the keys it finds "along the way" to build the resulting association, instead of producing them again (given cases like this)? $\endgroup$ – thorimur May 3 at 22:46
  • $\begingroup$ (I suppose one could hack it to do so by using a memoizing f manually, but still...) $\endgroup$ – thorimur May 3 at 22:48
  • 2
    $\begingroup$ @thorimur I think it should be reported at least as an unexpected behavior. $\endgroup$ – Alexey Popkov 2 days ago
  • 3
    $\begingroup$ @AlexeyPopkov FYI and others, this has been discussed internally back in 2018, and the bug has been reported. It wasn't deemed the top priority one however. Perhaps it is time to remind ourselves about it. $\endgroup$ – Leonid Shifrin 2 days ago
  • 1
    $\begingroup$ IMO this is at best a documentation bug if it is not mentioned. $\endgroup$ – Carsten S 2 days ago
2
$\begingroup$

Per my comment, observe:

myGB[lst_, tst_] := With[{gb = GatherBy[lst, tst]},
   If[Length@gb == 1, AssociationThread[{tst@gb[[1, 1]]} -> gb],
    AssociationThread[{tst@gb[[1, 1]], tst@gb[[2, 1]]} -> gb]]];

Then test with the following. The results will be the same.

xxx = {1, 1, 1, 1, 1, 1};
seed = RandomInteger[500];

SeedRandom[seed];
GroupBy[xxx, # > RandomReal[{0, 2}] &]

SeedRandom[seed];
myGB[xxx, (# > RandomReal[{0, 2}] &)]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.