Parallel compiled functions running on parallel sub-kernels

Question

Question

Why is it, that compiled, Listable, parallel functions which work perfectly fine on the main kernel, do not run in parallel on sub-kernels?

Details

First Example

Let me give a first example. I compile a function $f:\mathbb{R}\to\mathbb{R}$ which is a simple sum of sine-functions with the options CompilationTarget -> "C", RuntimeAttributes -> {Listable} and Parallelization -> True. Due to the Listable attribute I can now call the function with tensor parameters and due to the parallelization, the values in the tensor are processed in parallel. If you are on a slow machine, adjust n to be not that high:

f = Compile[{{t, _Real, 0}},
     #, CompilationTarget -> "C", RuntimeAttributes -> {Listable}, 
     Parallelization -> True] &@
   Sum[Sin[2.0 Pi k t]/k, {k, 1000}];
data = With[{n = 1000000}, Table[t, {t, 0, 1, 1/(n - 1)}]];
f[data];

Looking at the system monitor during the calculation shows, that all processors are running at 100%

If you like you can compare the speed of this execution with for instance f /@ data.

Under the hood

This may be only correct for Linux and OS X!

What happens when you use Compile with the "C" option is, that a shared library is created from your Mathematica-code which contains a function that can be called. The libraries are stored in a folder which is specific to the process id of a specific kernel. Let's make a short function, to print the important stuff of such a compiled function. I extract from a CompiledFunction the information, where the shared library is placed and what type the function is. Additionally, I add $KernelID and $ProcessID:

printCFuncLibrary[HoldPattern[CompiledFunction[__, lib_]]] := 
 StringJoin["{KernelID: ", ToString[$KernelID], ", ProcessID: ", 
  ToString[$ProcessID], "} -> ", ToString[lib, InputForm]]

Using this on f and I get

printCFuncLibrary[f]

(*
  {KernelID: 0, ProcessID: 3809} -> 
  LibraryFunction["/home/patrick/.Mathematica/ApplicationData/\
    CCompilerDriver/BuildFolder/lenerd-3809/compiledFunction0.so",
  "compiledFunction0", {{Real, 0, "Constant"}}, Real]
*)

Please note that that the build-folder has the process id of my main-kernel in it: "lenerd-3809". If you try now to execute this on different sub-kernels, you see that shared library function which is used stays the same. This is kind of expected:

ParallelTry[printCFuncLibrary, {f, f, f, f}, 4]

What is kind of unexpected is, that when I call f even on only 1 sub-kernel, I lose the vector-parallelization completely and only one processor is working on the task

ParallelTry[f, {data}, 1];

I would have expected, that when calling the compiled function (compiled on the main-kernel) in sub-kernels, that there are some clashes whatsoever.

Compiling the function on the sub-kernels

Since I could not explain the above behavior, but I could surely imagine, that having only one shared library function which is maybe only loaded one time, is not the best situation when several processes want to access it.

But why not compile the function on all sub-kernels. With this every sub-kernel gets its own copy of the shared library and loads its own version of the function:

ParallelEvaluate[
  fsub = Compile[{{t, _Real, 0}},
       #, CompilationTarget -> "C", RuntimeAttributes -> {Listable}, 
       Parallelization -> True] &@
     Sum[Sin[2.0 Pi k t]/k, {k, 1000}];
  ];

ParallelEvaluate[printCFuncLibrary[fsub]]

I skip my output here, but what you should see is, that every sub-kernel gets its own copy of the shared library, place in a folder which is named like the process id of the sub-kernel. Additionally, on our main-kernel there doesn't exist a function fsub and therefore calling it with a numeric value stays unevaluated. On the other hand, ParallelEvaluate[fsub[.1]] calculates the correct results.

If you now try to supply the vector data to the compiled function on the sub-kernel you see, that this is not processed parallel

ParallelTry[fsub, {data}];

I tried several other things to get some insight in the behavior, but nothing really helped me to understand, what's going on.

You might ask...

... when your compiled function is parallelized so nicely, isn't it pretty useless, to take a second layer of parallelization? The answer is, yes, but for my real problem this is not the case. Assume you have a minimization problem and you parallelize your target-function only. Still, since the minimization method runs serially and only the calls to the target-function are parallel, there is still much cpu-time left. In such a cases, it would be reasonable to run two or more minimizations at the same time.

Answer 1

This issue has nothing to do with the compiling/loading/calling process in parallel sub-kernels. I was so worried that it has something to do with the parallel usage of a library-function that I didn't check whether the issue has a very simple reason. Luckily, Szabolcs gave me the important hint.

The number of used threads in parallel compiled code is set by one of the SystemOptions:

SystemOptions["ParallelOptions"]

(*
{"ParallelOptions" -> {"AbortPause" -> 2., "BusyWait" -> 0.01,
 "MathLinkTimeout" -> 15., "ParallelThreadNumber" -> 8, 
 "RecoveryMode" -> "ReQueue", "RelaunchFailedKernels" -> False}}
*)

Here "ParallelThreadNumber" defines how many hardware-threads are used. Unexpectedly for me, this setting is different for a sub-kernel which is started by a call to one of the parallel Mathematica functions:

ParallelEvaluate[
 OptionValue["ParallelOptions" /. SystemOptions["ParallelOptions"], 
  "ParallelThreadNumber"]]
(*
{0, 0, 0, 0}
*)

If this option is set to a different value (e.g. $ProcessorCount) on the sub-kernels everything works as expected.

Test Example

Let's create a small test environment consisting of a parallel compiled function fc, a function f which nests fc many times and a list of values we want to process:

createTestEnvironment[] := (fc = 
   Compile[{{t, _Real, 0}}, Re[#], CompilationTarget -> "C", 
      RuntimeAttributes -> {Listable}, Parallelization -> True] &[
    FourierSeries[t, t, 6]];
  f[x_] := Nest[fc, x, 5000];
  data = Table[x, {x, 0, 1, 0.0001}];
  )

If we are running f now on one main kernel, we see that there is still cpu-time left

createTestEnvironment[];
First[AbsoluteTiming[f[data]]]

Therefore, lets now compile fc for each parallel sub-kernel, set the "ParallelThreadNumber" option on each sub-kernel to a higher value and execute it with ParallelEvaluate

DistributeDefinitions[createTestEnvironment];
ParallelEvaluate[
  createTestEnvironment[]
  ];
ParallelEvaluate[
  SetSystemOptions[
   "ParallelOptions" -> "ParallelThreadNumber" -> $ProcessorCount]
  ];
First[AbsoluteTiming[
  ParallelEvaluate[f[data]]]]

The run of the first example on the main kernel took about 8 seconds here. Therefore executing it 4 times (because I have 4 sub-kernels) serially would require 32 seconds. Running it instead in parallel takes only 22 seconds which is about 70% of the serial execution time. The system monitor shows additionally, that all cores are rocking