5
$\begingroup$

I'm running my script (say "cluster.m") on the University cluster with PBS Pro installed. The script uses simple Parallelize like this:

Parallelize[Table[myfunction[otherparams, i], {i, number of points}]];

i.e. all calculations are independent in each point and the parallelization is trivial. Then I run it on the 8-cores processor by the following .sh script:

#!/bin/sh
#PBS -l walltime=24:00:00
#PBS -l select=1:ncpus=8:mem=16000m
cat cluster.m | /opt/shared/wolfram/bin/MathKernel -noprompt

My script can be parallelized by more that 8 process (number of points > 10^5). Our cluster has licence for gridMathematica 9 (88 "Sub MathKernel" process), but I cannot find any guide for the launch gridMathematica with PBS Pro commands. I found only something like

#PBS -l select=2:ncpus=8:mem=16000m -l gridmath9=16

Here I'm trying to get two different 8-cores processors and parallelize the script into 16 processes in total. But it does not work, the cluster says "Unknown resource Resource_List.gridmath9".

I also have tried

module add gridmathematica-9

but the answer is "module: command not found".

Could anyone give me a working example of such .sh script or advise correct way to launch? Many thanks!

$\endgroup$
  • $\begingroup$ Have you already tried asking your system administrators? $\endgroup$ – MarcoB Dec 6 '16 at 6:28
  • $\begingroup$ Yes. Mathematica is a new software on our cluster and I'm one of the first who learns how to launch it correctly... $\endgroup$ – Andrey Gelash Dec 6 '16 at 6:40
  • $\begingroup$ PBS is supported by gridMathematica's Cluster Integration Package. See reference.wolfram.com/language/ClusterIntegration/ref/PBS.html $\endgroup$ – sakra Dec 6 '16 at 8:08
  • $\begingroup$ I'm not sure it was time to accept the answer. If it helped—great. But perhaps you could use it to write a real, PBS specific answer. Self-answers are always encouraged. $\endgroup$ – Szabolcs Dec 6 '16 at 17:57
  • $\begingroup$ Thanks, right. Your answer is helpful to learn more about Mathematica cluster integration. I need some time to realize it. But indeed, it is still unclear for me how to manage Grid Mathematica from the .sh file.. $\endgroup$ – Andrey Gelash Dec 6 '16 at 18:55
5
$\begingroup$

Setting up Mathematica to work well on a cluster was painful for me. I don't know how to get it working on PBS, but you may find the following general information useful.

Do look at the ClusterIntegration package. If it doesn't work as it is, you will have to customize things.

Your Mathematica program should be in an .m file which should be run from the job file as

math -script myprogram.m

This script is responsible for launching the correct number of subkernels on all the compute nodes that were allocated to you. The job file is launched in an environment where information about these nodes is available in some form (different for each job schedulnig system). You will need to retrieve this information from the environment (see the PBS documentation) and use it to construct subkernel specifications. These will include the commands to launch the subkernels. On many systems this is done through ssh. Again, see the PBS documentation.

To construct a subkernel specification, load

Needs["SubKernels`RemoteKernels`"]

then write each specification as

SubKernels`RemoteKernels`RemoteMachine[host, command, cores]

where host must be the hostname, command must be the launch command in a form suitable for execution within the job script, and cores is the number of CPU cores on that host that you can use (i.e. how many subkernels will be launched there).

The launch command will have templates in it. They are of the form `1`, which will be automatically filled out. Probably the only one you need to be concerned about is 1, as this will expand to the host name. You'll need to pass this to ssh.

The subkernel must be launched with a command similar to

math -mathlink -linkmode Connect `4` -linkname '`2`' -subkernel -noinit >& /dev/null

where math is the Mathematica kernel executable.

Using ssh to launch that may look something like

ssh `1` mathcommand &

where mathcommand is the subkernel launch command from above and `1` will have the host name auto-inserted. This part will be specific to your cluster, so ask your admin.

When you have your subkernel specifications, put them in a list and assign that to $ConfiguredKernels. Then use LaunchKernels[] without any arguments to launch the subkernels.

An example you can refer to is the solution I used on our cluster (SGE, not PBS) a long time ago: https://bitbucket.org/szhorvat/crc/src The details (e.g. how to retrieve the available nodes, how to launch a process on another node, etc.) will be different for your cluster.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.