# Mathematica Parellelization on HPC

I'm currently using Mathematica on a High Performance Cluster (HPC) which consists of many compute nodes each with around 16 cores. I currently run my Mathematica script on 20 of the nodes that invokes 10 cores and 10 of the subkernels licenses in Parallel, meaning I use 20 Mathkernel licenses and 200 subkernel licenses.

The problem is we have limited Mathkernel licenses (36, and for me to be using 20 of them is unfair on everyone else!) although ample subkernel licenses (288). Is there a way I can just use a single (or at least fewer) Mathkernel licenses to invoke the 200 subkernels I need?

Currently in each of the 20 scripts I just have

LaunchKernels[10];
ParellelTable[....];


which launches the 10 local subkernels on each node, but could I specify different nodes to launch subkernels on perhaps? Thereby I would only need to launch one Mathkernel which could invoke the 200 subkernels spread across the compute nodes.

-
Are you asking or boasting? :) –  belisarius is forth Feb 27 '13 at 13:54
Yes, this is possible, and I am doing this right now. But how it can be done is dependent on the grid engine your HPC cluster is using as well as details of the local setup. –  Szabolcs Feb 27 '13 at 14:00
@belisarius hehe @Szabolcs The grid engine is PBS Pro. –  fpghost Feb 27 '13 at 14:09
@fpghost I spent a day and a half on just figuring this out, so I uploaded a solution to bitbucket. It's not for PBS, but looking at it might save you some time. See my answer. –  Szabolcs Feb 27 '13 at 14:17
@Szabolcs thanks very much, I'll have a play around and see how I get on. –  fpghost Feb 27 '13 at 14:30

What you need to launch subkernels across several nodes on a HPC cluster is the following:

1. Figure out how to request several compute nodes for the same job
2. Find the names of the nodes that have been allocated for your job
3. Find out how to launch subkernels on these nodes from within the main kernel

All of these depend on the grid engine your cluster is using, as well as your local setup, and you'll need to check its docs and ask your administrator about the details. I have an example for our local setup (complete with a jobfile), which might be helpful for you to study:

https://bitbucket.org/szhorvat/crc/src

Our cluster uses the Sun Grid Engine. The names of the nodes (and information about them) are listed in a "hostfile" which you can find by retrieving the value of the PE_HOSTFILE environment variable. (I think this works the same way with PBS, except the environment variable is called something else.)

Note that if you request multiple nodes in a single job file, the job script will be run on only one of the nodes, and you'll be launching the processes across all nodes manually (at least on SGE and PBS).

Launching processes on different nodes is usually possible with ssh: just run ssh nodename command to run command. You may also need to set up passphraseless authentication if it is not set up by default. To launch subkernels, you'll need to pass the -f option to ssh to let it return immediately after it has launched the remote process.

Some setups use rsh instead of ssh. To launch a command in the background using rsh, you'll need to do

rsh -n nodename "command >& /dev/null &"


To run the remote process in the background, it important to redirect the output (both stdout and stderr) because there's a bug in rsh (also described in its man page) that won't let it return immediately otherwise.

Another thing to keep in mind about rsh is that you can't rsh to the local machine, so you'll need to launch the subkernels which will run on the same machine as the main kernel without rsh.

See my example for details.

-
How's the performance on the cluster? I'd be very wary of scaling problems with this many subkernels... also, last I tried, MathLink begins to crap out with hundreds of simultaneous connections. –  Oleksandr R. Feb 27 '13 at 14:34
@OleksandrR. I haven't run more than 60 kernels like this yet, and for my application the communication was minimal between the kernels, so I didn't notice any performance problems. I'll try a very big job soon, just out of curiosity. –  Szabolcs Feb 27 '13 at 14:41
So to request several compute nodes for the same job I simplet do qsub -l select=10:ncpus=10:mem=2gb subscript.sh for example which would give me 10 nodes each with 10 cores. So far so good. –  fpghost Feb 27 '13 at 15:08
@fpghost Szabolcs's suggestion of using ssh should work, but given that you're using PBS, you might find OSC mpiexec preferable. I find it easier to use and much more reliable than the ssh-based method (N.B.: for other programs; Mathematica isn't available on any of the clusters I have access to). Despite the name, the programs you launch do not have to be compiled against an MPI library. –  Oleksandr R. Feb 27 '13 at 15:13
@OleksandrR. I think you are right MathLink does seem to be dying for me with 190 Kernels. Despite getting there in the end it now takes vastly longer than when I had 19 single MathKernels with 10 local slaves each. –  fpghost Mar 9 '13 at 21:13

Update

The node names in a job can be access through environment variables such as PBS_NODEFILE and HOSTNAME, so that launching subkernels on the correct nodes can be automated.

I'm also trying the run more subKernels from a main kernel on HPC. I usually apply an interaction job on the HPC and run math kernels on it, and then connect back to the front end on may laptop. My waiting time for the queue of the interactive job is very short so it is convenient for me to do the work in the interactive way. Here is how I did, it may not be the same, but hope it would help.

Apply a interative job

qsub -V -I -l walltime=01:00:00,nodes=2:ppn=16 -A hpc_atistartup


it will return something like this:

qsub: waiting for job 48488.mike3 to start

--------------------------------------
Running PBS prologue script
--------------------------------------
PBS has allocated the following nodes:

mike054
mike067

A total of 32 processors on 2 nodes allocated
---------------------------------------------
Check nodes and clean them of stray processes
---------------------------------------------
Checking node mike054 15:43:46
Checking node mike067 15:43:48
Done clearing all the allocated nodes
------------------------------------------------------
Concluding PBS prologue script - 01-Sep-2013 15:43:48
------------------------------------------------------
[aaa@mike054 ~]$ We can see I get nodes mike054 and mike067, and the shell is on node mike054. Start remote master kernel From the menu of the local front end(my laptop), Evaluation ==> Kernel Configuration Options , add a remote Kernel, here I added one called superMike. Select "Advanced Options" and fill it with "-LinkMode Listen -LinkProtocol TCPIP". Then run a command in a notebook, for example $Version. It would pop out a window like this:

The port and IP address should be different than mine.

With this pop up window opened, go to your shell at the HPC we just get, run the command math to launch command line mathematica. After I get the mathematica shell, enter

$ParentLink = LinkConnect["50013@127.0.0.1,50014@127.0.0.1", LinkProtocol->"TCPIP"]  and hit enter. Then hit the "OK" key of that pop up window. If it successfully connected, it would pop up a message window with Out[1]:=LinkObject[50013@127.0.0.1,50014@127.0.0.1, 59, 2]  and the $Version command should return the results:

For details of the remote kernel connection, see the post here.

Start subKernels

Open the Remote Kernels tab in Evaluation ==> Parallel Kernel Configuration, clink "Add Host" to add other nodes we get in the interactive job. In this case I get nodes mike054 and mike067, and the shell I get is on node mike054. So I will add mike067 by fill the Hostname, set the number of kernels and check "Enable".

After that we can go to Evaluation ==> Parallel Kernel status, and check whether the subKernel are working. If everything went successfully we can see something like this

We can see that we've launched 16 subKernels on node mike054 and 16 subKernels on node mike067.

Hope it will help.

-
Hi,xslittlegrass! Do you know how to do it without front end? The network of the HPC which I use is not fast enough to use a front end. And I didn't understand Szabolcs's method. Does he achieve the same effect as yours(to parallel computation across different nodes)? –  matheorem Nov 11 '13 at 14:52
@matheorem Here is the (better) way to do in the code :) –  xslittlegrass Nov 11 '13 at 19:47
Thank you for the information! I am reading it. I have a question. I can't understand the step Start remote master kernel. In the Mathematica doc "ParallelTools/tutorial/ConnectionMethods", it didn't say that we have to start remote master kernel. And I can understand why your method can launch a master kernel on remote node. Can explain it a little? –  matheorem Nov 12 '13 at 6:46
@matheorem that remote master kernel is just the default remote kernel we are in. Say if we get 2 nodes "mike1" and "mike2" from qsub and we are on "mike1". Then when you type "math" to launch mathematica kernel, it will launch a master kernel. This master kernel is what I was referring to. It is a "remote" kernel in the sense that the kernel is running on the hpc and the front end is running on my laptop. That's all I meant. After launching the master kernel, we can then launch the sub-kernels using the linked codes. –  xslittlegrass Nov 12 '13 at 15:47
Now, I am sure that the first step Start remote master kernel is unnecessary. Just Add host in the remote Kernels tab, and then first CloseKernels[], second LauchKernels[], all the Kernels will be launched. This works on my HPC. –  matheorem Nov 13 '13 at 0:07

It depends. In our framework it is like that: We have a Mathematica licence server hosting say 10 licences. I lately saw a screen dump from the server status and there, used Kernels and Subkernels were handled independently. This means, if everyone else is using Mathematica in a non-parallel way, I can take me all the Subkernels by going to the preferences and adjusting the number of used Subkernels in the Parallel tab

-