18
$\begingroup$

Question

Why is it, that compiled, Listable, parallel functions which work perfectly fine on the main kernel, do not run in parallel on sub-kernels?

Details

First Example

Let me give a first example. I compile a function $f:\mathbb{R}\to\mathbb{R}$ which is a simple sum of sine-functions with the options CompilationTarget -> "C", RuntimeAttributes -> {Listable} and Parallelization -> True. Due to the Listable attribute I can now call the function with tensor parameters and due to the parallelization, the values in the tensor are processed in parallel. If you are on a slow machine, adjust n to be not that high:

f = Compile[{{t, _Real, 0}},
     #, CompilationTarget -> "C", RuntimeAttributes -> {Listable}, 
     Parallelization -> True] &@
   Sum[Sin[2.0 Pi k t]/k, {k, 1000}];
data = With[{n = 1000000}, Table[t, {t, 0, 1, 1/(n - 1)}]];
f[data];

Looking at the system monitor during the calculation shows, that all processors are running at 100%

enter image description here

If you like you can compare the speed of this execution with for instance f /@ data.

Under the hood

This may be only correct for Linux and OS X!

What happens when you use Compile with the "C" option is, that a shared library is created from your Mathematica-code which contains a function that can be called. The libraries are stored in a folder which is specific to the process id of a specific kernel. Let's make a short function, to print the important stuff of such a compiled function. I extract from a CompiledFunction the information, where the shared library is placed and what type the function is. Additionally, I add $KernelID and $ProcessID:

printCFuncLibrary[HoldPattern[CompiledFunction[__, lib_]]] := 
 StringJoin["{KernelID: ", ToString[$KernelID], ", ProcessID: ", 
  ToString[$ProcessID], "} -> ", ToString[lib, InputForm]]

Using this on f and I get

printCFuncLibrary[f]

(*
  {KernelID: 0, ProcessID: 3809} -> 
  LibraryFunction["/home/patrick/.Mathematica/ApplicationData/\
    CCompilerDriver/BuildFolder/lenerd-3809/compiledFunction0.so",
  "compiledFunction0", {{Real, 0, "Constant"}}, Real]
*)

Please note that that the build-folder has the process id of my main-kernel in it: "lenerd-3809". If you try now to execute this on different sub-kernels, you see that shared library function which is used stays the same. This is kind of expected:

ParallelTry[printCFuncLibrary, {f, f, f, f}, 4]

What is kind of unexpected is, that when I call f even on only 1 sub-kernel, I lose the vector-parallelization completely and only one processor is working on the task

ParallelTry[f, {data}, 1];

enter image description here

I would have expected, that when calling the compiled function (compiled on the main-kernel) in sub-kernels, that there are some clashes whatsoever.

Compiling the function on the sub-kernels

Since I could not explain the above behavior, but I could surely imagine, that having only one shared library function which is maybe only loaded one time, is not the best situation when several processes want to access it.

But why not compile the function on all sub-kernels. With this every sub-kernel gets its own copy of the shared library and loads its own version of the function:

ParallelEvaluate[
  fsub = Compile[{{t, _Real, 0}},
       #, CompilationTarget -> "C", RuntimeAttributes -> {Listable}, 
       Parallelization -> True] &@
     Sum[Sin[2.0 Pi k t]/k, {k, 1000}];
  ];

ParallelEvaluate[printCFuncLibrary[fsub]]

I skip my output here, but what you should see is, that every sub-kernel gets its own copy of the shared library, place in a folder which is named like the process id of the sub-kernel. Additionally, on our main-kernel there doesn't exist a function fsub and therefore calling it with a numeric value stays unevaluated. On the other hand, ParallelEvaluate[fsub[.1]] calculates the correct results.

If you now try to supply the vector data to the compiled function on the sub-kernel you see, that this is not processed parallel

ParallelTry[fsub, {data}];

I tried several other things to get some insight in the behavior, but nothing really helped me to understand, what's going on.

You might ask...

... when your compiled function is parallelized so nicely, isn't it pretty useless, to take a second layer of parallelization? The answer is, yes, but for my real problem this is not the case. Assume you have a minimization problem and you parallelize your target-function only. Still, since the minimization method runs serially and only the calls to the target-function are parallel, there is still much cpu-time left. In such a cases, it would be reasonable to run two or more minimizations at the same time.

$\endgroup$
4
  • 4
    $\begingroup$ Have you looked at the appropriate SystemOption on the parallel kernels? ParallelEvaluate@SystemOptions["ParallelOptions"] $\endgroup$
    – Szabolcs
    Jun 19, 2012 at 7:00
  • 1
    $\begingroup$ How the hell could I assume, that $ProcessorCount is used on the sub-kernels too?? Unbelievable! Can you write up an answer? $\endgroup$
    – halirutan
    Jun 19, 2012 at 7:17
  • $\begingroup$ I'll try to answer when I get to try this on a multicore machine. Usually I can test parallel stuff on a single core one too, but here it's important to use multicore ... if you have solved it based on my comment, feel free to post your own answer! $\endgroup$
    – Szabolcs
    Jun 19, 2012 at 14:31
  • $\begingroup$ Ok, when you don't care then I write up what's happening. Thanks. $\endgroup$
    – halirutan
    Jun 20, 2012 at 1:50

1 Answer 1

12
$\begingroup$

This issue has nothing to do with the compiling/loading/calling process in parallel sub-kernels. I was so worried that it has something to do with the parallel usage of a library-function that I didn't check whether the issue has a very simple reason. Luckily, Szabolcs gave me the important hint.

The number of used threads in parallel compiled code is set by one of the SystemOptions:

SystemOptions["ParallelOptions"]

(*
{"ParallelOptions" -> {"AbortPause" -> 2., "BusyWait" -> 0.01,
 "MathLinkTimeout" -> 15., "ParallelThreadNumber" -> 8, 
 "RecoveryMode" -> "ReQueue", "RelaunchFailedKernels" -> False}}
*)

Here "ParallelThreadNumber" defines how many hardware-threads are used. Unexpectedly for me, this setting is different for a sub-kernel which is started by a call to one of the parallel Mathematica functions:

ParallelEvaluate[
 OptionValue["ParallelOptions" /. SystemOptions["ParallelOptions"], 
  "ParallelThreadNumber"]]
(*
{0, 0, 0, 0}
*)

If this option is set to a different value (e.g. $ProcessorCount) on the sub-kernels everything works as expected.

Test Example

Let's create a small test environment consisting of a parallel compiled function fc, a function f which nests fc many times and a list of values we want to process:

createTestEnvironment[] := (fc = 
   Compile[{{t, _Real, 0}}, Re[#], CompilationTarget -> "C", 
      RuntimeAttributes -> {Listable}, Parallelization -> True] &[
    FourierSeries[t, t, 6]];
  f[x_] := Nest[fc, x, 5000];
  data = Table[x, {x, 0, 1, 0.0001}];
  )

If we are running f now on one main kernel, we see that there is still cpu-time left

createTestEnvironment[];
First[AbsoluteTiming[f[data]]]

enter image description here

Therefore, lets now compile fc for each parallel sub-kernel, set the "ParallelThreadNumber" option on each sub-kernel to a higher value and execute it with ParallelEvaluate

DistributeDefinitions[createTestEnvironment];
ParallelEvaluate[
  createTestEnvironment[]
  ];
ParallelEvaluate[
  SetSystemOptions[
   "ParallelOptions" -> "ParallelThreadNumber" -> $ProcessorCount]
  ];
First[AbsoluteTiming[
  ParallelEvaluate[f[data]]]]

The run of the first example on the main kernel took about 8 seconds here. Therefore executing it 4 times (because I have 4 sub-kernels) serially would require 32 seconds. Running it instead in parallel takes only 22 seconds which is about 70% of the serial execution time. The system monitor shows additionally, that all cores are rocking

enter image description here

$\endgroup$
3
  • $\begingroup$ Thanks for the summary! My machine has, unfortunately, a single core CPU. $\endgroup$
    – Szabolcs
    Jun 21, 2012 at 14:30
  • $\begingroup$ I've tried to launch the commands you provided. In mathematica 10, the supposed slower method runs faster. $\endgroup$
    – Fabio
    Jan 30, 2016 at 7:26
  • $\begingroup$ I'm not sure I understand. Or maybe you misunderstand what I'm doing. In the first example, I run f[data] on the main kernel exactly one time. Takes 9.1 seconds on V10.3.1. Then, I'm calling it 8 times because I have 8 subkernel and it only needs 24 seconds which tells you that one single run indeed only needed 24/8=3 seconds. We are talking about two different parallelizations here. Parallel threads and parallel sub-kernels and I combine both to squeeze the last bit out of my CPU. Ping me in Mathematica Chat if you have further questions. $\endgroup$
    – halirutan
    Jan 30, 2016 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.