gridMathematica on cluster with PBS Pro

Question

I'm running my script (say "cluster.m") on the University cluster with PBS Pro installed. The script uses simple Parallelize like this:

Parallelize[Table[myfunction[otherparams, i], {i, number of points}]];

i.e. all calculations are independent in each point and the parallelization is trivial. Then I run it on the 8-cores processor by the following .sh script:

#!/bin/sh
#PBS -l walltime=24:00:00
#PBS -l select=1:ncpus=8:mem=16000m
cat cluster.m | /opt/shared/wolfram/bin/MathKernel -noprompt

My script can be parallelized by more that 8 process (number of points > 10^5). Our cluster has licence for gridMathematica 9 (88 "Sub MathKernel" process), but I cannot find any guide for the launch gridMathematica with PBS Pro commands. I found only something like

#PBS -l select=2:ncpus=8:mem=16000m -l gridmath9=16

Here I'm trying to get two different 8-cores processors and parallelize the script into 16 processes in total. But it does not work, the cluster says "Unknown resource Resource_List.gridmath9".

I also have tried

module add gridmathematica-9

but the answer is "module: command not found".

Could anyone give me a working example of such .sh script or advise correct way to launch? Many thanks!

Answer 1

Setting up Mathematica to work well on a cluster was painful for me. I don't know how to get it working on PBS, but you may find the following general information useful.

Do look at the ClusterIntegration package. If it doesn't work as it is, you will have to customize things.

Your Mathematica program should be in an .m file which should be run from the job file as

math -script myprogram.m

This script is responsible for launching the correct number of subkernels on all the compute nodes that were allocated to you. The job file is launched in an environment where information about these nodes is available in some form (different for each job schedulnig system). You will need to retrieve this information from the environment (see the PBS documentation) and use it to construct subkernel specifications. These will include the commands to launch the subkernels. On many systems this is done through ssh. Again, see the PBS documentation.

To construct a subkernel specification, load

Needs["SubKernels`RemoteKernels`"]

then write each specification as

SubKernels`RemoteKernels`RemoteMachine[host, command, cores]

where host must be the hostname, command must be the launch command in a form suitable for execution within the job script, and cores is the number of CPU cores on that host that you can use (i.e. how many subkernels will be launched there).

The launch command will have templates in it. They are of the form `1`, which will be automatically filled out. Probably the only one you need to be concerned about is 1, as this will expand to the host name. You'll need to pass this to ssh.

The subkernel must be launched with a command similar to

math -mathlink -linkmode Connect `4` -linkname '`2`' -subkernel -noinit >& /dev/null

where math is the Mathematica kernel executable.

Using ssh to launch that may look something like

ssh `1` mathcommand &

where mathcommand is the subkernel launch command from above and `1` will have the host name auto-inserted. This part will be specific to your cluster, so ask your admin.

When you have your subkernel specifications, put them in a list and assign that to $ConfiguredKernels. Then use LaunchKernels[] without any arguments to launch the subkernels.

An example you can refer to is the solution I used on our cluster (SGE, not PBS) a long time ago: https://bitbucket.org/szhorvat/crc/src The details (e.g. how to retrieve the available nodes, how to launch a process on another node, etc.) will be different for your cluster.