4
$\begingroup$

In my program I noticed that a line using Position is the performance bottleneck because it is executed very often. Using some dummy data and adding timing functions, it is as follows:

lst = Table[<|"Class" -> RandomChoice[{"A", "B"}], "Distance" -> RandomReal[], "Sample" -> RandomReal[1, 2]|>, 1000];
Mean@Table[AbsoluteTiming[
    idxs = Position[
        lst,
        a_ /; a["Class"] == "A" && 
            a["Distance"] < 0.1 &&
            a["Sample"] != {0.1, 0.31}
    ] // Flatten;], 100]
(*{0.00929925, Null}*)

Is there a way to get the same results more efficiently?

Related? 1, Related? 2

$\endgroup$
  • 1
    $\begingroup$ One thing you might try is to use GroupBy[Key["Class"]] so that you only need to look for elements in part of the association. Of course, you'd need to store the index inside the items first. For keys like "Distance" you could try to sort the items by value and do a binary search for the cutoff. $\endgroup$ – Lukas Lang Sep 26 '17 at 14:16
2
$\begingroup$

A very simple and fairly effective way to improve the performance of Position is to include a level specification and setting the Option Heads → False. By default, the level specification is {0, Infinity} so when using Position on a list of Associations the condition is also checked for every Value in the Associations. So when you're not interested in nested occurrences you can use {1} as a level specification.

The following leads to a ~6.5 times speedup.

lst = Table[<|"Class" -> RandomChoice[{"A", "B"}], "Distance" -> RandomReal[], "Sample" -> RandomReal[1, 2]|>, 1000];
Mean@Table[AbsoluteTiming[
    idxs = Position[
        lst,
        a_ /; a["Class"] == "A" && 
            a["Distance"] < 0.1 &&
            a["Sample"] != {0.1, 0.31},
        {1}, Heads -> False
    ] // Flatten;], 100]
(*{0.00139966, Null}*)

Update to add suggestion by Mathe172:
Even just adding the indexes is already slower than the method above.

Mean@Table[AbsoluteTiming[indexed = MapIndexed[Append[#1, "Idx" -> First@#2] &, lst];], 100]
(*{0.00171767, Null}*)

However, grouping and then selecting is a bit faster than selecting on the whole list, and on its own faster than the method above, so may be worth it if you already need to iterate over the list elements at an earlier point anyways.

Mean@Table[AbsoluteTiming[
    grouped = GroupBy[indexed, #Class &];
    idxs = #Idx & /@ Select[grouped["A"], #Distance < 0.1 && #Sample != {0.1, 0.31} &];],
    1000]
(*{0.000905544, Null}*)
Mean@Table[AbsoluteTiming[
    idxs = #Idx & /@ Select[indexed, #Class == "A" && #Distance < 0.1 && #Sample != {0.1, 0.31} &];],
    1000]
(*{0.00120144, Null}*)
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.