Considerations for Compiling and Parallelizing Code Outside or Inside Map[]

Question

Assume f as well as Map[f,list] are parallelizable and compilable, such that parallelization and compilation give significant speedup for both.

What considerations should I make when determining which of the following paradigms is best to use and gives best performance?

---

1. Map[(*compiled form of f with Parallelization->True*),list]

(compilation and parallelization inside)

2. ParallelMap[(*compiled form of f*),list]

(parallelization outside, compilation inside)

3. compiled form of entire Map[f,list] with Parallelization->True

(compilation and parallelization outside)

---

What are general use cases for each of these three paradigms?

Answer 1

In these cases, one typically has to try a variety of things and measure.

Here I try to give a simple example:

Block[{x, xx, y, yy},
  xx = Table[Compile`GetElement[x, i], {i, 1, 12}];
  
  With[{code = Power[Total[Power[x, 24]], 1/24]},
   
   f = Function @@ {x, code /. Compile`GetElement -> Indexed};
   
   cf = Compile[{{x, _Real, 1}},
     code,
     CompilationTarget -> "C",
     RuntimeAttributes -> {Listable},
     RuntimeOptions -> "Speed"
     ];
   
   cpf = Compile[{{x, _Real, 1}},
     code,
     CompilationTarget -> "C",
     RuntimeAttributes -> {Listable},
     Parallelization -> True,
     RuntimeOptions -> "Speed"
     ];
   
   ];
  ];

Now the test results (from my Intel Haswell 4 Core CPU):

n = 100000; X = RandomReal[{-1, 1}, {n, 12}];

a1 = Map[f, X]; // AbsoluteTiming // First 

a2 = Map[cf, X]; // AbsoluteTiming // First 

a3 = Map[cpf, X]; // AbsoluteTiming // First

a4 = ParallelMap[f, X]; // AbsoluteTiming // First 

a5 = ParallelMap[cf, X]; // AbsoluteTiming // First 

a6 = ParallelMap[cpf, X]; // AbsoluteTiming // First

a7 = cf[X]; // AbsoluteTiming // First 

a8 = cpf[X]; // AbsoluteTiming // First

a1 == a2 == a3 == a4 == a5 == a6 == a7 == a8

0.271289

0.025701

0.032341

2.30748

1.98798

1.98874

0.024378

0.010684

True

As you see, ParallelMap is very bad in this case. The major reason is that the loop iterates are just too small to justify the (immense!) overhead of Parallel.

We can also see from this that Parallelization->True is only active in conjuction with RunTimeAttributes->{Listable} and the latter is activle only if the function is called in the Listable way.

But with both activated, it is almost as fast as built-in vectorization

a9 = Power[Total[Power[X, 24], {2}], 1/24]; // AbsoluteTiming // First
a8 == a9

0.009269

True

This is a very simple loop and the ratio between memory accesses and floating point operations is really bad. So other loops may behave differently.