I have a huge list longList that I want to split into two parts depending on whether an element of the list satisfies the condition: f[#]< 0.1 or not. My question is what is the most efficient way of doing this? The output should be two lists, the first one containing all elements of longList that satisfy the condition and the second one all the rest.

So far I've tried:

temp = GroupBy[longList, f[#] < 0.1 &];
{list1, list2} = {temp[True], temp[False]};

But it's not fast enough and if I wrap GroupBy in Parallelize I get an error saying that it cannot be parallelized.

  • $\begingroup$ Maybe Map[f[#] < 0.1 &, longList] and ParallelMap? $\endgroup$ – Rohit Namjoshi May 7 '20 at 19:42
  • 1
    $\begingroup$ Can you provide more information on what f is and what the list elements are? $\endgroup$ – Szabolcs May 7 '20 at 20:23

Parallelize the application of the selector function, then use Pick to choose the elements for which the selector function returns Trueor False.

I made up a slow function f and a "long list":

f = (Pause[0.1]; # < 50) &;
longlist = RandomReal[{0, 100}, {10}];

You can appreciate the difference in application time between serial and parallel execution:

AbsoluteTiming[Map[f, longlist];]                      (* {1.01678, Null} *)
AbsoluteTiming[result = ParallelMap[f, longlist];]     (* {0.31652, Null} *)

You can then use Pick to split your list:

true =  Pick[longlist, result, True]
false = Pick[longlist, result, False] 

You could also use Complement to achieve the same, but it seems much slower than Pick:

false = Complement[longlist, true]
  • $\begingroup$ Thanks, this allowed me to parallelize and get the speed improvement I was looking for. $\endgroup$ – Cantor May 8 '20 at 6:17

If the list contains numbers, I suggest vectorization instead of parallelization. Try to write f so that it is vectorizable.

My BoolEval package (partially available as a resource function) is also convenient.

For example, if f is Sin (which is vectorized) then you can do

arr = RandomReal[100, 100000000];

<< BoolEval`

mask = BoolEval[Sin[arr] > 0.5]; // RepeatedTiming
(* {1.5, Null} *)

Pick[arr, mask, 1] // Length (* elements satisfying the condition *)
(* 33521172 *)

Pick[arr, mask, 0] // Length (* elements NOT satisfying the condition *)
(* 66478828 *)

If you only want one set of elements, a shorthand is

BoolPick[arr, Sin[arr] > 0.5]

I expect that no Parallel* function will be able to compete with this approach for as long as f is vectorizable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.