# Launching remote slave kernels on HPC causes slow down

I had a post earlier regarding how I can run Mathematica at a HPC cluster (running PBSpro) by having a single MathKernel launch say 190 slaves remotely, in order so that I only utilise one MathKernel license. I now have this working and the schematic of my notebook looks like:

(*First launch all the remote kernels*)

Needs["SubKernelsRemoteKernels"];
numCore=10; (*change to num cores per node*)
PBSKernel[host_String]:=
LaunchKernels[RemoteMachine[host,"ssh -x -f -l 3 1 /path/to/math -mathlink -linkmode Connect 4 -linkname '2' -subkernel -noinit",numCore]];
hostfile = Environment["PBS_NODEFILE"];

If[hostfile =!= $Failed, (* if we're running in a batch job *) hosts = Import[hostfile, "List"];$ConfiguredKernels = Join[$ConfiguredKernels, PBSKernel /@ hosts]; ] Export["kernels.txt", ParallelTable[{$MachineName, $KernelID}, {$KernelCount}], "Table"];

(*now they are launched evaluate something over them, first creating some output files,
one for each node, postfixed by the node name*)
$runningLogFile = "log.data"; ParallelEvaluate[If[FileExistsQ[$runningLogFile <> $MachineName], Get[$runningLogFile <> $MachineName],Export[$runningLogFile <> $MachineName, "", "Text"];];] DistributeDefinition[...]; ParallelMap[func, Tuples[{Range[1/10, 1, 1/10], Range[0, 59]}]];  where func[{x_?NumericQ, y_?IntegerQ}] := Block[{}, g[r_,s_,t_]:= (*do calc, essentially an integral,altho it does depend on some prev defined helper functions external*); (*func to write this to file in certain format*) gwrite[r_, s_,t_] := gwrite[r,s,t] = g[r, s, t] /. v_:>(PutAppend[Unevaluated[g[r, s, t] = v;],$runningLogFile <> $MachineName]; v); (*write to file using CriticalSection to stop clashing*) CriticalSection[{logfileLock}, Do[gwrite[tf, x, y], {tf, taufStart, taufEnd, intv}]]; ]  Now this all works fine, in the sense of giving me the expected data, but I have noticed a massive slowdown in computation speed this way, versus the previous method I was using whereby I launched 19 MathKernels, one for each compute node and then in turn had each launch 10 slaves locally. This is despite the fact that I have been assigned exactly the same resource, i.e. 19 nodes each with 10 cores and the same memory. The only difference is I am now only using one MathKernel instead of 19 to launch the slaves. The old method did my test portion of paramater space in 20mins vs over 4hrs for the current method, which is a drastic slow down that is too much for my longer jobs. What could be causing this slow down? Ideas: 1)Remote slave kernels need to communicate and this is causing some bottleneck. I can't see why this would be a problem for me as my func above is just an integral that should not depend on anything the other kernels are doing. 2) Somehow all the remote cores are not getting used? I know they are being Launched as they appear in the Kernels.txt file at the start of the calculation and also $KernelCount=190, but is my method of employing ParallelMap successfully using them all? Is there a way I can check this? I also know all the nodes are being used as I append the $MachineName to each data file and get 19 of them as expected. 3) Anything else? Mathlink just can't handle this many cores successfully? If I keep exactly the same script and clone it 19 times, then split the list of Tuples equally between the 19 new scripts, and feed these into 19 nodes with a MathKernel running on each (which launches 10 local cores) in serial then things massively speed up and I am back to 20mins. EDIT: after reviewing the output logs there is one odd message, although I'm not sure how relevant it is: StringForm["From 1:", ParallelKernelskernel[ParallelKernelsPrivatebk[remoteKernel[SubKernelsRemoteKernelsPrivatelk[LinkObject["49195@10.141.0.3,46878@10.141.0.3", 97, 97], "node012", {"node012", "ssh -x -f -l 3 1 math -mathlink -linkmode Connect 4 -linkname '2' -subkernel -noinit"}, SubKernelsRemoteKernelsPrivatespeed$1327]], ParallelKernelsPrivateid$1376, ParallelKernelsPrivatename$1376], ParallelKernelsPrivateek[ParallelKernelsPrivatenev$1377, ParallelKernelsPrivatepb$1377, ParallelKernelsPrivaterd$1377], ParallelKernelsPrivatesk[ParallelKernelsPrivateq$1378, ParallelKernelsPrivaten0$1378, ParallelKernelsPrivaten1$1378]]]

-
If you would use OSC mpiexec or some other TM-based launcher instead of ssh you would be able to see the CPU time in the job accounting statistics, which would answer the question over whether all of the kernels are actually being used. Personally I think it's likely that you are seeing MathLink limitations, but I have no way to prove this. – Oleksandr R. Mar 9 '13 at 21:13
This might be a bit too obvious, but have you tried using Method -> "CoarsestGrained" or Method -> "EvaluationsPerKernel" -> 2 with ParallelMap`? – Ajasja Mar 9 '13 at 22:34
@Ajasja I haven't tried it, but it can't hurt so I will give that a go next, thanks. – fpghost Mar 10 '13 at 0:11

An extended comment follows.

At a high level distributed/parallel processing can get managed in a couple of different ways.

Some approaches have used a single communication channel between a master kernel and each of its slave kernels. These single channels handle both data passed back and forth as well as monitoring and management functions.

A master kernel may monitor how long jobs take or if a job has completed or if a slave kernel has become available for a new job or if a job fails. Depending on the application, it may also need to monitor errors or precision in calculations such that if expected results fall out of some expected range it gets declared as invalid and the computation package may get reissued. This essentially tries to error correct for losses in the transmission of information, which more likely happens when a channel hasn't been filled to capacity (less room for errors).

Other approaches establish two or more communications channels between a master kernel and each slave kernel. Data and instructions travel over dedicated channel(s) and management and monitoring over others.

Lots of things affect all of this. Bandwidth, noise in the channels, counter measures to noise, as well as the level of the channel. We used to do this kind of thing over TCP (for all I know people still do, I haven't kept up because we have so many commodity solutions like Mathematica now).

Multiple channels doesn't necessarily mean fast processing or even fewer errors. A lot depends of the problem.

So given all of this your thought that:

1) Remote slave kernels need to communicate and this is causing some bottleneck. I can't see why this would be a problem for me as my func above is just an integral that should not depend on anything the other kernels are doing.

seems closest to the mark, but I'd guess that you've experienced a process management and monitoring bottleneck.

-
You are probably right. What a shame if this is the case, I could have lived with it being a little bit slower, but not this much slower. – fpghost Mar 10 '13 at 0:13