How to change front end setting "Parallel Kernel Configuration" in the code?

Question

I'm trying to run Mathematica parallel on multiple nodes on a HPC, and I use the remote kernel tab in the "Parallel Kernel Configuration" settings to set the communication between different nodes(detailed setup here). Since every time I would get different nodes of the HPC, each time I have to manually change the setting in the "Parallel Kernel Configuration". So is it possible to change the settings within the code?

Here is the detail where I want to change:

For example, how can I enable 16 kernels on a remote host named "mike123" and 10 kernels on a remote host named "mike234"

Answer 1

It turns out that we can actually launch subKernels in multiple nodes by using the LaunchKernels function. The solution is presented by fpghost in the question here about the slow down of the subKernels. In order not to confuse the reader by redirecting to that question, I will just explain how to use the code in that question to solve my problem here, as suggested by rm -rf. (This solution only apply to PBS system.)

Firstly, we should know how to launch subKernels locally and remotely. Basically, we can use LaunchKernels[] to launch subkernels both on the local system and on remote system. The default will launch kernels on the local system,

LaunchKernels[]

will launch subkernels on my local system, in my case it will launch 16 subKernels.

It can be also used to launch subKernels on remote system,

LaunchKernels[RemoteMachine[`remote`, 
         "ssh -x -f -l `3` `1` math -mathlink -linkmode Connect `4` -linkname '`2`' -subkernel -noinit", 16]]

will launch 16 subKernels on remote system remote, for the detailed explanation of the arguments, see here.

Secondly, we should know which nodes have been assigned to our job. This information can be easily get from PBS environmental variables:

PBS_NODEFILE : File containing list of allocated nodes
PBS_O_WORKDIR: Directory from where job is submitted
PBS_O_QUEUE: Queue job was submitted to
PBS_JOBID: Job ID number
PBS_JOBNAME: The name of the job.
PBS_NP: Number of processes requested
PBS_NUM_PPN: Number of processors per node requested

Put everything together, here is the function that can launch all the subKernels on the nodes assigned to our job.

Needs["SubKernels`RemoteKernels`"];
mikeLaunchKernels[] := 
 Module[{numCore, PBSKernel, hostfile, local, , remote},
  numCore = 16;(*change to num cores per node*)
  PBSKernel[host_String] := 
   LaunchKernels[RemoteMachine[host, 
     "ssh -x -f -l `3` `1` math -mathlink -linkmode Connect `4` -linkname '`2`' -subkernel -noinit", numCore]];
  hostfile = Environment["PBS_NODEFILE"];(*get all the nodes name*)
  local = ToString@Get["!echo $HOSTNAME"];(*get local node name*)
  remote = Complement[DeleteDuplicates@Import[hostfile, "List"], {local}];(*get remote node name*)
  If[hostfile =!= $Failed,(*if we're running in a batch job*)
$ConfiguredKernels = Join[$ConfiguredKernels, LaunchKernels[], PBSKernel /@ remote];];
  Kernels[]
  ]

---

Example:

Here is an example on our cluster:

First I request 4 interactive nodes

qsub -V -I -l walltime=02:00:00,nodes=4:ppn=16

it returned me with the node information and the shell

--------------------------------------
PBS has allocated the following nodes:

mike381
mike382
mike383
mike384

A total of 64 processors on 4 nodes allocated
---------------------------------------------
[abc@mike381 ~]$

so we have four nodes mike381 to mike384, and the shell is on mike381.

After calling to mikeLaunchKernels[], we launched 16 subKernels on each nodes we have:

{KernelObject[1, "local"], KernelObject[2, "local"], 
 KernelObject[3, "local"], KernelObject[4, "local"], 
 KernelObject[5, "local"], KernelObject[6, "local"], 
 KernelObject[7, "local"], KernelObject[8, "local"], 
 KernelObject[9, "local"], KernelObject[10, "local"], 
 KernelObject[11, "local"], KernelObject[12, "local"], 
 KernelObject[13, "local"], KernelObject[14, "local"], 
 KernelObject[15, "local"], KernelObject[16, "local"], 
 KernelObject[17, "mike382"], KernelObject[18, "mike382"], 
 KernelObject[19, "mike382"], KernelObject[20, "mike382"], 
 KernelObject[21, "mike382"], KernelObject[22, "mike382"], 
 KernelObject[23, "mike382"], KernelObject[24, "mike382"], 
 KernelObject[25, "mike382"], KernelObject[26, "mike382"], 
 KernelObject[27, "mike382"], KernelObject[28, "mike382"], 
 KernelObject[29, "mike382"], KernelObject[30, "mike382"], 
 KernelObject[31, "mike382"], KernelObject[32, "mike382"], 
 KernelObject[33, "mike383"], KernelObject[34, "mike383"], 
 KernelObject[35, "mike383"], KernelObject[36, "mike383"], 
 KernelObject[37, "mike383"], KernelObject[38, "mike383"], 
 KernelObject[39, "mike383"], KernelObject[40, "mike383"], 
 KernelObject[41, "mike383"], KernelObject[42, "mike383"], 
 KernelObject[43, "mike383"], KernelObject[44, "mike383"], 
 KernelObject[45, "mike383"], KernelObject[46, "mike383"], 
 KernelObject[47, "mike383"], KernelObject[48, "mike383"], 
 KernelObject[49, "mike384"], KernelObject[50, "mike384"], 
 KernelObject[51, "mike384"], KernelObject[52, "mike384"], 
 KernelObject[53, "mike384"], KernelObject[54, "mike384"], 
 KernelObject[55, "mike384"], KernelObject[56, "mike384"], 
 KernelObject[57, "mike384"], KernelObject[58, "mike384"], 
 KernelObject[59, "mike384"], KernelObject[60, "mike384"], 
 KernelObject[61, "mike384"], KernelObject[62, "mike384"], 
 KernelObject[63, "mike384"], KernelObject[64, "mike384"]}