10
$\begingroup$

Suppose I have a list called mask composed of 1,2,3,...n. n is different in different situation. Let me takes n=3 for demonstration

mask=RandomInteger[{1,3},1000000]

and another list

list = RandomReal[{0, 1}, 1000000];

I want to pick those element corresponding not equal to 1.

Pick[list, mask, _?(# != 1 &)]; // Timing

This takes 1.125 sec

But If I already know mask only composed of 1,2,3, then this

Pick[list, mask, 2 | 3]; // Timing

is faster, it takes 0.25 sec

But the problem is I am not sure that is in mask, so this is not general.

So the question is there more efficient way than this _?(# != 1 &) pattern? Why is it slower then pattern 2|3?

$\endgroup$
  • $\begingroup$ As for why 2|3 is faster than _?(# != 1 &): it's because the latter involves evaluating Mathematica code (evaluating the pure function) for each test. The former doesn't. $\endgroup$ – Szabolcs Oct 25 '15 at 17:25
  • $\begingroup$ I tried Pick[list, mask, Except[1]] but it fails because the Except matches the whole list. Pick[list, mask, Except[1, _Integer]] works and is the same speed as 2 | 3. $\endgroup$ – 2012rcampion Oct 25 '15 at 23:49
  • $\begingroup$ @Szabolcs I don't understand. 2|3 doesn't evaluate for each test? Then how can it know which one to pick? $\endgroup$ – matheorem Oct 26 '15 at 0:30
  • $\begingroup$ @2012rcampion good observation! Thank you! But in this particular case, Szabolcs's method is faster. $\endgroup$ – matheorem Oct 26 '15 at 5:57
  • $\begingroup$ @matheorem Think about how Pick might be implemented in C. For _?(# != 1 &) you'd need a callback to the main evaluator (i.e. run Mathematica code) for each test. For 2|3 you don't. You just need to test for equality between 2 (or 3) and the given list element, but this test doesn't involve running Mathematica code. It can be done only in C. $\endgroup$ – Szabolcs Oct 26 '15 at 7:02
9
$\begingroup$

Since I think version 8, Pick is optimized for the case when the pattern is a single element (i.e. 1 or 2 but not 1|2), and when the inputs are packed arrays.

If you need performance, make sure that you hit this special case. Use vectorized arithmetic operations to transform the lists into a suitable form.

Pick[list, mask, _?(# != 1 &)]; // AbsoluteTiming
(* {0.547087, Null} *)

Pick[list, Unitize[mask - 1], 1]; // AbsoluteTiming
(* {0.019021, Null} *)

My BoolEval package tries to automate this process for more complicated cases, at the cost of only a little performance.

<< BoolEval`

BoolPick[list, mask == 1]; // AbsoluteTiming
(* {0.029157, Null} *)
$\endgroup$
  • $\begingroup$ Thank you so much. What do you mean by vectorized arithmetic operations, why is it so fast? Except Unitize, are there other vectorized arithmetic operations useful? $\endgroup$ – matheorem Oct 26 '15 at 0:36
  • $\begingroup$ @matheorem The rule of thumb is that arithmetic on packed arrays of machine numbers tends to be fast. That's because it can be implemented very efficiently in terms of SIMD instructions, it's easy to parallelize (think adding two arrays) and packed array storage is very efficient. $\endgroup$ – Szabolcs Oct 26 '15 at 7:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.