19
$\begingroup$

I'm currently using Mathematica on a High Performance Cluster (HPC) which consists of many compute nodes each with around 16 cores. I currently run my Mathematica script on 20 of the nodes that invokes 10 cores and 10 of the subkernels licenses in Parallel, meaning I use 20 Mathkernel licenses and 200 subkernel licenses.

The problem is we have limited Mathkernel licenses (36, and for me to be using 20 of them is unfair on everyone else!) although ample subkernel licenses (288). Is there a way I can just use a single (or at least fewer) Mathkernel licenses to invoke the 200 subkernels I need?

Currently in each of the 20 scripts I just have

LaunchKernels[10];
ParellelTable[....];

which launches the 10 local subkernels on each node, but could I specify different nodes to launch subkernels on perhaps? Thereby I would only need to launch one Mathkernel which could invoke the 200 subkernels spread across the compute nodes.

$\endgroup$
7
  • 6
    $\begingroup$ Are you asking or boasting? :) $\endgroup$ Feb 27, 2013 at 13:54
  • $\begingroup$ Yes, this is possible, and I am doing this right now. But how it can be done is dependent on the grid engine your HPC cluster is using as well as details of the local setup. $\endgroup$
    – Szabolcs
    Feb 27, 2013 at 14:00
  • $\begingroup$ @belisarius hehe @Szabolcs The grid engine is PBS Pro. $\endgroup$
    – fpghost
    Feb 27, 2013 at 14:09
  • $\begingroup$ @fpghost I spent a day and a half on just figuring this out, so I uploaded a solution to bitbucket. It's not for PBS, but looking at it might save you some time. See my answer. $\endgroup$
    – Szabolcs
    Feb 27, 2013 at 14:17
  • $\begingroup$ @Szabolcs thanks very much, I'll have a play around and see how I get on. $\endgroup$
    – fpghost
    Feb 27, 2013 at 14:30

3 Answers 3

14
$\begingroup$

What you need to launch subkernels across several nodes on a HPC cluster is the following:

  1. Figure out how to request several compute nodes for the same job
  2. Find the names of the nodes that have been allocated for your job
  3. Find out how to launch subkernels on these nodes from within the main kernel

All of these depend on the grid engine your cluster is using, as well as your local setup, and you'll need to check its docs and ask your administrator about the details. I have an example for our local setup (complete with a jobfile), which might be helpful for you to study:

https://bitbucket.org/szhorvat/crc/src

Our cluster uses the Sun Grid Engine. The names of the nodes (and information about them) are listed in a "hostfile" which you can find by retrieving the value of the PE_HOSTFILE environment variable. (I think this works the same way with PBS, except the environment variable is called something else.)

Note that if you request multiple nodes in a single job file, the job script will be run on only one of the nodes, and you'll be launching the processes across all nodes manually (at least on SGE and PBS).

Launching processes on different nodes is usually possible with ssh: just run ssh nodename command to run command. You may also need to set up passphraseless authentication if it is not set up by default. To launch subkernels, you'll need to pass the -f option to ssh to let it return immediately after it has launched the remote process.

Some setups use rsh instead of ssh. To launch a command in the background using rsh, you'll need to do

rsh -n nodename "command >& /dev/null &"

To run the remote process in the background, it important to redirect the output (both stdout and stderr) because there's a bug in rsh (also described in its man page) that won't let it return immediately otherwise.

Another thing to keep in mind about rsh is that you can't rsh to the local machine, so you'll need to launch the subkernels which will run on the same machine as the main kernel without rsh.

See my example for details.

$\endgroup$
23
  • $\begingroup$ How's the performance on the cluster? I'd be very wary of scaling problems with this many subkernels... also, last I tried, MathLink begins to crap out with hundreds of simultaneous connections. $\endgroup$ Feb 27, 2013 at 14:34
  • $\begingroup$ @OleksandrR. I haven't run more than 60 kernels like this yet, and for my application the communication was minimal between the kernels, so I didn't notice any performance problems. I'll try a very big job soon, just out of curiosity. $\endgroup$
    – Szabolcs
    Feb 27, 2013 at 14:41
  • $\begingroup$ So to request several compute nodes for the same job I simplet do qsub -l select=10:ncpus=10:mem=2gb subscript.sh for example which would give me 10 nodes each with 10 cores. So far so good. $\endgroup$
    – fpghost
    Feb 27, 2013 at 15:08
  • $\begingroup$ @fpghost Szabolcs's suggestion of using ssh should work, but given that you're using PBS, you might find OSC mpiexec preferable. I find it easier to use and much more reliable than the ssh-based method (N.B.: for other programs; Mathematica isn't available on any of the clusters I have access to). Despite the name, the programs you launch do not have to be compiled against an MPI library. $\endgroup$ Feb 27, 2013 at 15:13
  • 2
    $\begingroup$ @OleksandrR. I think you are right MathLink does seem to be dying for me with 190 Kernels. Despite getting there in the end it now takes vastly longer than when I had 19 single MathKernels with 10 local slaves each. $\endgroup$
    – fpghost
    Mar 9, 2013 at 21:13
6
$\begingroup$

Update

The node names in a job can be access through environment variables such as PBS_NODEFILE and HOSTNAME, so that launching subkernels on the correct nodes can be automated.


I'm also trying the run more subKernels from a main kernel on HPC. I usually apply an interaction job on the HPC and run math kernels on it, and then connect back to the front end on may laptop. My waiting time for the queue of the interactive job is very short so it is convenient for me to do the work in the interactive way. Here is how I did, it may not be the same, but hope it would help.

Apply a interative job

qsub -V -I -l walltime=01:00:00,nodes=2:ppn=16 -A hpc_atistartup

it will return something like this:

qsub: waiting for job 48488.mike3 to start
qsub: job 48488.mike3 ready

--------------------------------------
Running PBS prologue script
--------------------------------------
PBS has allocated the following nodes:

mike054
mike067

A total of 32 processors on 2 nodes allocated
---------------------------------------------
Check nodes and clean them of stray processes
---------------------------------------------
Checking node mike054 15:43:46 
Checking node mike067 15:43:48 
Done clearing all the allocated nodes
------------------------------------------------------
Concluding PBS prologue script - 01-Sep-2013 15:43:48
------------------------------------------------------
[aaa@mike054 ~]$ 

We can see I get nodes mike054 and mike067, and the shell is on node mike054.

Start remote master kernel

From the menu of the local front end(my laptop), Evaluation ==> Kernel Configuration Options , add a remote Kernel, here I added one called superMike. Select "Advanced Options" and fill it with "-LinkMode Listen -LinkProtocol TCPIP".

enter image description here

Then run a command in a notebook, for example $Version. It would pop out a window like this:

enter image description here

The port and IP address should be different than mine.

With this pop up window opened, go to your shell at the HPC we just get, run the command math to launch command line mathematica. After I get the mathematica shell, enter

$ParentLink = LinkConnect["[email protected],[email protected]", LinkProtocol->"TCPIP"]

and hit enter. Then hit the "OK" key of that pop up window. If it successfully connected, it would pop up a message window with

Out[1]:=LinkObject[[email protected],[email protected], 59, 2]

and the $Version command should return the results:

enter image description here

For details of the remote kernel connection, see the post here.

Start subKernels

Open the Remote Kernels tab in Evaluation ==> Parallel Kernel Configuration, clink "Add Host" to add other nodes we get in the interactive job. In this case I get nodes mike054 and mike067, and the shell I get is on node mike054. So I will add mike067 by fill the Hostname, set the number of kernels and check "Enable".

enter image description here

After that we can go to Evaluation ==> Parallel Kernel status, and check whether the subKernel are working. If everything went successfully we can see something like this

enter image description here

We can see that we've launched 16 subKernels on node mike054 and 16 subKernels on node mike067.

Hope it will help.

$\endgroup$
6
  • $\begingroup$ Hi,xslittlegrass! Do you know how to do it without front end? The network of the HPC which I use is not fast enough to use a front end. And I didn't understand Szabolcs's method. Does he achieve the same effect as yours(to parallel computation across different nodes)? $\endgroup$
    – matheorem
    Nov 11, 2013 at 14:52
  • $\begingroup$ @matheorem Here is the (better) way to do in the code :) $\endgroup$ Nov 11, 2013 at 19:47
  • $\begingroup$ Thank you for the information! I am reading it. I have a question. I can't understand the step Start remote master kernel. In the Mathematica doc "ParallelTools/tutorial/ConnectionMethods", it didn't say that we have to start remote master kernel. And I can understand why your method can launch a master kernel on remote node. Can explain it a little? $\endgroup$
    – matheorem
    Nov 12, 2013 at 6:46
  • $\begingroup$ @matheorem that remote master kernel is just the default remote kernel we are in. Say if we get 2 nodes "mike1" and "mike2" from qsub and we are on "mike1". Then when you type "math" to launch mathematica kernel, it will launch a master kernel. This master kernel is what I was referring to. It is a "remote" kernel in the sense that the kernel is running on the hpc and the front end is running on my laptop. That's all I meant. After launching the master kernel, we can then launch the sub-kernels using the linked codes. $\endgroup$ Nov 12, 2013 at 15:47
  • $\begingroup$ Now, I am sure that the first step Start remote master kernel is unnecessary. Just Add host in the remote Kernels tab, and then first CloseKernels[], second LauchKernels[], all the Kernels will be launched. This works on my HPC. $\endgroup$
    – matheorem
    Nov 13, 2013 at 0:07
0
$\begingroup$

It depends. In our framework it is like that: We have a Mathematica licence server hosting say 10 licences. I lately saw a screen dump from the server status and there, used Kernels and Subkernels were handled independently. This means, if everyone else is using Mathematica in a non-parallel way, I can take me all the Subkernels by going to the preferences and adjusting the number of used Subkernels in the Parallel tab

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.