# Right syntax to launch multiple kernels on SLURM cluster

Important:

sometimes you may need to modify the kernel location. In my case I had to find the kernel location explicitly by executing which wolfram command in the shell after loading the mathematica module.

Therefore in my case the line below was modified from

kernelLaunch = "wolfram -wstp -linkmode connect 4 -linkname 2 -subkernel -noinit >&/dev/null &";

to

kernelLocation = "/share/apps/mathematica/12.0.0/Scripts/wolfram -wstp -linkmode connect 4 -linkname 2 -subkernel -noinit >& /dev/null &";

I could only manage to ssh wolfram across nodes when the full kernel location was specified. Again use which wolfram to check.

Original Post

I have access to a SLURM based cluster where I allocate 2 nodes, each with 12 processors. I am looking for a way to launch all 24 kernels to utilize the resources to the maximum for parallel computation.

nodes = ReadList["!scontrol show hostname $SLURM_JOB_NODELIST",String]; cores = ConstantArray[ToExpression@Environment["SLURM_CPUS_PER_TASK"], Length@nodes]; dir = Directory[]; Block[{$ContextPath}, Needs["SubKernelsRemoteKernels"]];

kernelLaunch = "wolfram -wstp -linkmode connect 4 -linkname 2 -subkernel -noinit >&/dev/null &";

LaunchKernels[RemoteMachine[#1,"ssh -x -f "<>#1<>" \""<>kernelLaunch<>"\"",#2]]&,
{nodes,cores}];

res = HPCKernel[nodes,cores];
corecount = $KernelCount; Save["results.wl", {nodes,cores,dir,res,corecount}]  In the results.wl I get the following outputs: nodes = {"wm8", "wm9"} cores = {12, 12} dir = "/work/ah1" res = {$$Failed,$$Failed} corecount = 0  We can see that res and hence HPCKernel has failed to launch any kernel. In fact the corecount remains 0. I have success in allocating appropriate resources, however I have no idea how to launch all available kernels. Where am I going wrong with this? Any help will be highly appreciated. ## 1 Answer I always struggle with clusters and I can't give you a full solution, but I'll share the setup code I use on our slurm cluster. Perhaps you can adapt it. It works even when you get a different number of cores on each node. I added some comments to it. This solution assumes that you set up things so that you can run arbitrary commands on any node using ssh nodename command. I set up passwordless authentication for this. I am certain that there are much better ways with slurm, but this works, and I did not have more time to spend on it. (* This is SlurmSetup.m, placed in$UserBaseDirectory/Applications.
It is the first thing I load in every script meant to run on the cluster. *)

If[
Environment["SLURMD_NODENAME"] === $Failed, Print["NOT RUNNING IN A SLURM JOB! ABORTING!!"]; Abort[] ] Begin["SlurmSetup"] (* Have decent printing from subkernels, see e.g. https://mathematica.stackexchange.com/q/208124/12 *) SetOptions[$Output, FormatType -> OutputForm]

Needs["SubKernelsRemoteKernels"];

(* I used this when I needed M12.0 but it wasn't the default version. Thus I needed to launch kernels with math1200. *)
(*$RemoteCommand = "ssh -x -f -l 3 1 -a math1200 -wstp -linkmode Connect 4 -linkname '2' -subkernel -noinit -nopaclet";*) (* Parse SLURM_TASKS_PER_NODE to find out how many tasks we can launch on each node. Look up the syntax in the slurm docs. It's tricky. *) tasksPerNode = StringSplit[Environment["SLURM_TASKS_PER_NODE"], ","] // Map[StringCases[{a : (DigitCharacter ..) ~~ "(x" ~~ b : (DigitCharacter ..) ~~ ")" :> ConstantArray @@ FromDigits /@ {a, b}, a : (DigitCharacter ..) :> FromDigits[a]}]] // Flatten; headNode = Environment["SLURMD_NODENAME"]; nodes = ReadList["!scontrol show hostname$SLURM_NODELIST", String];

(* Set up the correct number of subkernels on each nodes that slurm allocated for us.
These will be automatically launched by LaunchKernels[] *)
$ConfiguredKernels = MapThread[RemoteMachine, {nodes, tasksPerNode}]; Print["Parallel run information: "] Print[{tasksPerNode, headNode, nodes,$ConfiguredKernels}]
Print[""]

(* Subkernels don't load Kernel/init.m, so I have an alternative init files that sets up $Path and other things. I load at here for the subkernels. *) (* ParallelNeeds["MyInit"] *) (* Increase timeout because on our cluster kernel launching tends to time out ... *) ParallelSettings$MathLinkTimeout = 30.;

(* I print this for diagnostic purposes ... sometimes people hog the licenses. *)
Print["Available process licenses: " <> IntegerString[$$MaxLicenseProcesses -$$LicenseProcesses] <> "."]
Print["Available subprocess licenses: " <> IntegerString[$$MaxLicenseSubprocesses -$$LicenseSubprocesses] <> "."]

Print["Launching " <> IntegerString@Total[tasksPerNode] <>  " kernels..."]
LaunchKernels[] (* use only LaunchKernels[], never LaunchKernels[n] *)
Print["Finished launching " <> IntegerString@Length@Kernels[] <> " kernels."]

Print[""]

End[]

• thanks. I am going to take a look at your code and give this a try. Jan 17, 2020 at 16:20
• the kernel count is again 0. Do you think that is because i should have more than 1 tasksPerNode to launch any kernel on both machines? Is this the issue? Jan 17, 2020 at 17:18
• also, from your answer: "This solution assumes that you set up things so that you can run arbitrary commands on any node using ssh nodename command. I set up passwordless authentication for this." can you please let me know how do i ensure that i have this pre-requisite of passwordless authentication? does this mean that mathematica sessions should be loaded on both nodes and that if i am on one node i should be able to do ssh nameofothernode math - script filename.m to run some file on the other node. Sorry but i am new to clusters and have no idea how to proceed with this. Jan 17, 2020 at 17:27
• I'm not terribly experienced with clusters either. Are you sure you requested more than 1 task per node in the job file? Jan 18, 2020 at 10:51
• I think with Mathematica you want 1 CPU/task because every subkernel uses 1 core and each subkernel is counted as one task. Jan 18, 2020 at 11:02