Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I'm trying to run Mathematica parallel on multiple nodes on a HPC, and I use the remote kernel tab in the "Parallel Kernel Configuration" settings to set the communication between different nodes(detailed setup here). Since every time I would get different nodes of the HPC, each time I have to manually change the setting in the "Parallel Kernel Configuration". So is it possible to change the settings within the code?

Here is the detail where I want to change:

enter image description here

For example, how can I enable 16 kernels on a remote host named "mike123" and 10 kernels on a remote host named "mike234"

share|improve this question
    
Do you have access to download Lightweight Grid? It's much more simple. –  Murta Sep 2 '13 at 3:01
    
@Murta unfortunately not, but I'm wondering can one use Lightweight Grid on a HPC machine? Does it compatible with the PBS system? –  xslittlegrass Sep 2 '13 at 3:08
    
i don't know, but I have tested in normal PCs, and you just install LWG in the remote machines that you want to use the kernels, and the machines just appear in the LWG Menu of your picture if they are in the same network. Very simple. –  Murta Sep 2 '13 at 3:16
    
It seems that we can do that in code by LaunchKernel, as explained in the question here: mathematica.stackexchange.com/questions/20948/… –  xslittlegrass Sep 4 '13 at 14:10
    
@xslittlegrass I disagree with your (and others') votes to close. It's true that the solution to your problem is the code in the other question (not even the answer), but that doesn't make it a duplicate. In fact, that question is entirely about the slowdowns and someone redirected to that from here would be thoroughly confused. I don't recall a question on changing the kernel configuration, so I would suggest that you answer your own question with an explanation of how to do that using LaunchKernel. –  rm -rf Sep 7 '13 at 18:05

1 Answer 1

up vote 2 down vote accepted

It turns out that we can actually launch subKernels in multiple nodes by using the LaunchKernels function. The solution is presented by fpghost in the question here about the slow down of the subKernels. In order not to confuse the reader by redirecting to that question, I will just explain how to use the code in that question to solve my problem here, as suggested by rm -rf. (This solution only apply to PBS system.)

Firstly, we should know how to launch subKernels locally and remotely. Basically, we can use LaunchKernels[] to launch subkernels both on the local system and on remote system. The default will launch kernels on the local system,

LaunchKernels[]

will launch subkernels on my local system, in my case it will launch 16 subKernels.

It can be also used to launch subKernels on remote system,

LaunchKernels[RemoteMachine[`remote`, 
         "ssh -x -f -l `3` `1` math -mathlink -linkmode Connect `4` -linkname '`2`' -subkernel -noinit", 16]]

will launch 16 subKernels on remote system remote, for the detailed explanation of the arguments, see here.

Secondly, we should know which nodes have been assigned to our job. This information can be easily get from PBS environmental variables:

PBS_NODEFILE : File containing list of allocated nodes
PBS_O_WORKDIR: Directory from where job is submitted
PBS_O_QUEUE: Queue job was submitted to
PBS_JOBID: Job ID number
PBS_JOBNAME: The name of the job.
PBS_NP: Number of processes requested
PBS_NUM_PPN: Number of processors per node requested

Put everything together, here is the function that can launch all the subKernels on the nodes assigned to our job.

Needs["SubKernels`RemoteKernels`"];
mikeLaunchKernels[] := 
 Module[{numCore, PBSKernel, hostfile, local, , remote},
  numCore = 16;(*change to num cores per node*)
  PBSKernel[host_String] := 
   LaunchKernels[RemoteMachine[host, 
     "ssh -x -f -l `3` `1` math -mathlink -linkmode Connect `4` -linkname '`2`' -subkernel -noinit", numCore]];
  hostfile = Environment["PBS_NODEFILE"];(*get all the nodes name*)
  local = ToString@Get["!echo $HOSTNAME"];(*get local node name*)
      remote = Complement[DeleteDuplicates@Import[hostfile, "List"], {local}];(*get remote node name*)
      If[hostfile =!= $Failed,(*if we're running in a batch job*)
$ConfiguredKernels = Join[$ConfiguredKernels, LaunchKernels[], PBSKernel /@ remote];];
  Kernels[]
  ]

Example:

Here is an example on our cluster:

First I request 4 interactive nodes

qsub -V -I -l walltime=02:00:00,nodes=4:ppn=16

it returned me with the node information and the shell

--------------------------------------
PBS has allocated the following nodes:

mike381
mike382
mike383
mike384

A total of 64 processors on 4 nodes allocated
---------------------------------------------
[abc@mike381 ~]$

so we have four nodes mike381 to mike384, and the shell is on mike381.

After calling to mikeLaunchKernels[], we launched 16 subKernels on each nodes we have:

{KernelObject[1, "local"], KernelObject[2, "local"], 
 KernelObject[3, "local"], KernelObject[4, "local"], 
 KernelObject[5, "local"], KernelObject[6, "local"], 
 KernelObject[7, "local"], KernelObject[8, "local"], 
 KernelObject[9, "local"], KernelObject[10, "local"], 
 KernelObject[11, "local"], KernelObject[12, "local"], 
 KernelObject[13, "local"], KernelObject[14, "local"], 
 KernelObject[15, "local"], KernelObject[16, "local"], 
 KernelObject[17, "mike382"], KernelObject[18, "mike382"], 
 KernelObject[19, "mike382"], KernelObject[20, "mike382"], 
 KernelObject[21, "mike382"], KernelObject[22, "mike382"], 
 KernelObject[23, "mike382"], KernelObject[24, "mike382"], 
 KernelObject[25, "mike382"], KernelObject[26, "mike382"], 
 KernelObject[27, "mike382"], KernelObject[28, "mike382"], 
 KernelObject[29, "mike382"], KernelObject[30, "mike382"], 
 KernelObject[31, "mike382"], KernelObject[32, "mike382"], 
 KernelObject[33, "mike383"], KernelObject[34, "mike383"], 
 KernelObject[35, "mike383"], KernelObject[36, "mike383"], 
 KernelObject[37, "mike383"], KernelObject[38, "mike383"], 
 KernelObject[39, "mike383"], KernelObject[40, "mike383"], 
 KernelObject[41, "mike383"], KernelObject[42, "mike383"], 
 KernelObject[43, "mike383"], KernelObject[44, "mike383"], 
 KernelObject[45, "mike383"], KernelObject[46, "mike383"], 
 KernelObject[47, "mike383"], KernelObject[48, "mike383"], 
 KernelObject[49, "mike384"], KernelObject[50, "mike384"], 
 KernelObject[51, "mike384"], KernelObject[52, "mike384"], 
 KernelObject[53, "mike384"], KernelObject[54, "mike384"], 
 KernelObject[55, "mike384"], KernelObject[56, "mike384"], 
 KernelObject[57, "mike384"], KernelObject[58, "mike384"], 
 KernelObject[59, "mike384"], KernelObject[60, "mike384"], 
 KernelObject[61, "mike384"], KernelObject[62, "mike384"], 
 KernelObject[63, "mike384"], KernelObject[64, "mike384"]}
share|improve this answer
    
If I have qsub two nodes and got node name fu01 and fu02 each has 24 cores. The shell is on fu01. Then as I understand, I can just type LaunchKernels[RemoteMachine[fu02, "ssh -x -f -l 3 1 math -mathlink -linkmode Connect 4 -linkname '2' -subkernel -noinit", 16]], right? But it didn't work. It just gives ` $Failed` –  matheorem Nov 12 '13 at 9:03
    
@matheorem I think fu02 should be "fu02", since the argument should be a string. Could you try that? –  xslittlegrass Nov 12 '13 at 15:34
    
Yeah! string works! Thank you very Much! To notice before Launch, Needs["SubKernelsRemoteKernels`"]` is essential. But why I can't find "SubKernels`RemoteKernels`" in mathematica doc? –  matheorem Nov 13 '13 at 1:54
    
@matheorem glad it works. I find they are in the document here. –  xslittlegrass Nov 13 '13 at 17:52
    
yeah,my fault, I didn't read the doc carefully:) –  matheorem Nov 14 '13 at 2:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.