6
$\begingroup$

I'm trying to run Mathematica parallel on multiple nodes on a HPC, and I use the remote kernel tab in the "Parallel Kernel Configuration" settings to set the communication between different nodes(detailed setup here). Since every time I would get different nodes of the HPC, each time I have to manually change the setting in the "Parallel Kernel Configuration". So is it possible to change the settings within the code?

Here is the detail where I want to change:

enter image description here

For example, how can I enable 16 kernels on a remote host named "mike123" and 10 kernels on a remote host named "mike234"

$\endgroup$
6
  • $\begingroup$ Do you have access to download Lightweight Grid? It's much more simple. $\endgroup$
    – Murta
    Sep 2, 2013 at 3:01
  • $\begingroup$ @Murta unfortunately not, but I'm wondering can one use Lightweight Grid on a HPC machine? Does it compatible with the PBS system? $\endgroup$ Sep 2, 2013 at 3:08
  • $\begingroup$ i don't know, but I have tested in normal PCs, and you just install LWG in the remote machines that you want to use the kernels, and the machines just appear in the LWG Menu of your picture if they are in the same network. Very simple. $\endgroup$
    – Murta
    Sep 2, 2013 at 3:16
  • $\begingroup$ It seems that we can do that in code by LaunchKernel, as explained in the question here: mathematica.stackexchange.com/questions/20948/… $\endgroup$ Sep 4, 2013 at 14:10
  • $\begingroup$ @xslittlegrass I disagree with your (and others') votes to close. It's true that the solution to your problem is the code in the other question (not even the answer), but that doesn't make it a duplicate. In fact, that question is entirely about the slowdowns and someone redirected to that from here would be thoroughly confused. I don't recall a question on changing the kernel configuration, so I would suggest that you answer your own question with an explanation of how to do that using LaunchKernel. $\endgroup$
    – rm -rf
    Sep 7, 2013 at 18:05

1 Answer 1

6
$\begingroup$

It turns out that we can actually launch subKernels in multiple nodes by using the LaunchKernels function. The solution is presented by fpghost in the question here about the slow down of the subKernels. In order not to confuse the reader by redirecting to that question, I will just explain how to use the code in that question to solve my problem here, as suggested by rm -rf. (This solution only apply to PBS system.)

Firstly, we should know how to launch subKernels locally and remotely. Basically, we can use LaunchKernels[] to launch subkernels both on the local system and on remote system. The default will launch kernels on the local system,

LaunchKernels[]

will launch subkernels on my local system, in my case it will launch 16 subKernels.

It can be also used to launch subKernels on remote system,

LaunchKernels[RemoteMachine[`remote`, 
         "ssh -x -f -l `3` `1` math -mathlink -linkmode Connect `4` -linkname '`2`' -subkernel -noinit", 16]]

will launch 16 subKernels on remote system remote, for the detailed explanation of the arguments, see here.

Secondly, we should know which nodes have been assigned to our job. This information can be easily get from PBS environmental variables:

PBS_NODEFILE : File containing list of allocated nodes
PBS_O_WORKDIR: Directory from where job is submitted
PBS_O_QUEUE: Queue job was submitted to
PBS_JOBID: Job ID number
PBS_JOBNAME: The name of the job.
PBS_NP: Number of processes requested
PBS_NUM_PPN: Number of processors per node requested

Put everything together, here is the function that can launch all the subKernels on the nodes assigned to our job.

Needs["SubKernels`RemoteKernels`"];
mikeLaunchKernels[] := 
 Module[{numCore, PBSKernel, hostfile, local, , remote},
  numCore = 16;(*change to num cores per node*)
  PBSKernel[host_String] := 
   LaunchKernels[RemoteMachine[host, 
     "ssh -x -f -l `3` `1` math -mathlink -linkmode Connect `4` -linkname '`2`' -subkernel -noinit", numCore]];
  hostfile = Environment["PBS_NODEFILE"];(*get all the nodes name*)
  local = ToString@Get["!echo $HOSTNAME"];(*get local node name*)
  remote = Complement[DeleteDuplicates@Import[hostfile, "List"], {local}];(*get remote node name*)
  If[hostfile =!= $Failed,(*if we're running in a batch job*)
$ConfiguredKernels = Join[$ConfiguredKernels, LaunchKernels[], PBSKernel /@ remote];];
  Kernels[]
  ]

Example:

Here is an example on our cluster:

First I request 4 interactive nodes

qsub -V -I -l walltime=02:00:00,nodes=4:ppn=16

it returned me with the node information and the shell

--------------------------------------
PBS has allocated the following nodes:

mike381
mike382
mike383
mike384

A total of 64 processors on 4 nodes allocated
---------------------------------------------
[abc@mike381 ~]$

so we have four nodes mike381 to mike384, and the shell is on mike381.

After calling to mikeLaunchKernels[], we launched 16 subKernels on each nodes we have:

{KernelObject[1, "local"], KernelObject[2, "local"], 
 KernelObject[3, "local"], KernelObject[4, "local"], 
 KernelObject[5, "local"], KernelObject[6, "local"], 
 KernelObject[7, "local"], KernelObject[8, "local"], 
 KernelObject[9, "local"], KernelObject[10, "local"], 
 KernelObject[11, "local"], KernelObject[12, "local"], 
 KernelObject[13, "local"], KernelObject[14, "local"], 
 KernelObject[15, "local"], KernelObject[16, "local"], 
 KernelObject[17, "mike382"], KernelObject[18, "mike382"], 
 KernelObject[19, "mike382"], KernelObject[20, "mike382"], 
 KernelObject[21, "mike382"], KernelObject[22, "mike382"], 
 KernelObject[23, "mike382"], KernelObject[24, "mike382"], 
 KernelObject[25, "mike382"], KernelObject[26, "mike382"], 
 KernelObject[27, "mike382"], KernelObject[28, "mike382"], 
 KernelObject[29, "mike382"], KernelObject[30, "mike382"], 
 KernelObject[31, "mike382"], KernelObject[32, "mike382"], 
 KernelObject[33, "mike383"], KernelObject[34, "mike383"], 
 KernelObject[35, "mike383"], KernelObject[36, "mike383"], 
 KernelObject[37, "mike383"], KernelObject[38, "mike383"], 
 KernelObject[39, "mike383"], KernelObject[40, "mike383"], 
 KernelObject[41, "mike383"], KernelObject[42, "mike383"], 
 KernelObject[43, "mike383"], KernelObject[44, "mike383"], 
 KernelObject[45, "mike383"], KernelObject[46, "mike383"], 
 KernelObject[47, "mike383"], KernelObject[48, "mike383"], 
 KernelObject[49, "mike384"], KernelObject[50, "mike384"], 
 KernelObject[51, "mike384"], KernelObject[52, "mike384"], 
 KernelObject[53, "mike384"], KernelObject[54, "mike384"], 
 KernelObject[55, "mike384"], KernelObject[56, "mike384"], 
 KernelObject[57, "mike384"], KernelObject[58, "mike384"], 
 KernelObject[59, "mike384"], KernelObject[60, "mike384"], 
 KernelObject[61, "mike384"], KernelObject[62, "mike384"], 
 KernelObject[63, "mike384"], KernelObject[64, "mike384"]}
$\endgroup$
5
  • $\begingroup$ If I have qsub two nodes and got node name fu01 and fu02 each has 24 cores. The shell is on fu01. Then as I understand, I can just type LaunchKernels[RemoteMachine[fu02, "ssh -x -f -l 3 1 math -mathlink -linkmode Connect 4 -linkname '2' -subkernel -noinit", 16]], right? But it didn't work. It just gives ` $Failed` $\endgroup$
    – matheorem
    Nov 12, 2013 at 9:03
  • $\begingroup$ @matheorem I think fu02 should be "fu02", since the argument should be a string. Could you try that? $\endgroup$ Nov 12, 2013 at 15:34
  • $\begingroup$ Yeah! string works! Thank you very Much! To notice before Launch, Needs["SubKernelsRemoteKernels"] is essential. But why I can't find "SubKernelsRemoteKernels" in mathematica doc? $\endgroup$
    – matheorem
    Nov 13, 2013 at 1:54
  • $\begingroup$ @matheorem glad it works. I find they are in the document here. $\endgroup$ Nov 13, 2013 at 17:52
  • $\begingroup$ yeah,my fault, I didn't read the doc carefully:) $\endgroup$
    – matheorem
    Nov 14, 2013 at 2:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.