16
$\begingroup$

On a HPC, there are many computation nodes, each node corresponds to an independent machine. Every node has several cores, for example 8 per node.

Mathematica can launch remote kernels on different nodes (see method here). After all the kernels were launched, Mathematica seems to regard all the kernels as the same thing. We can use ParallelMap and different options "FinestGrained" or "CoarsestGrained" to distribute computation tasks to different kernels regardless of which node a kernel belongs to.

But I am in a case where I want to distribute my computation task according to machines instead of kernels, because my computation involves matrix eigenvalues. When the matrix is large, the memory usage is considerable while each node has limited memory resources. So I can not parallelize the computation to "FinestGrained" to kernels, as that will exhaust the memory and probably would cause the node to break down.

One way to solve this is to launch a limited number of kernels on each node, for example, only launch one kernel per node, and the one kernel on each node won't eat up all the memory. But in this way, I will lose the multi-thread feature of Eigenvalues computation (see here), because memory usage is determined by the dimension of the matrix. Using 1 core or 8 cores to compute the same matrix needs the same amount of memory, but obviously 8 cores will be faster. So if I launch only one kernel on each node I am wasting the remaining 7 cores on each node.

So here is the question. How to distribute the computation tasks according to machines instead of kernels, and preserve multi-thread feature of built-in functions? I mean to create a new option "FinestGrainedToNodes" which will distribute computation tasks evenly to different node.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Interesting question although it seems to me that you have completed your prototype and you are now optimizing to get as fast computations as possible. Maybe compiling the computationally intensive part of your code could offer you a boost, or maybe it is time to transfer your code in a lower level language as C++. $\endgroup$
    – tchronis
    Nov 29, 2013 at 10:52
  • 1
    $\begingroup$ @tchronis well, the most time consuming part is eigenvalues computation, it can't be compiled. Eigenvalues is already a built-in numeric function, it should not be much slower than C or fortran $\endgroup$
    – matheorem
    Nov 29, 2013 at 13:13
  • 1
    $\begingroup$ Eigenvalues will use all cores even if just one Mathematica kernel is running. The parallelism takes place in Intel MKL which Mathematica uses for much of its numeric calculations. $\endgroup$
    – ssch
    Dec 1, 2013 at 11:52
  • 2
    $\begingroup$ @ssch Thank you ssch! You enlightened me, though your comment is not 100% right. Try ParallelTable[Eigenvalues[Randomreal[{1,2},{6000,6000}],{1}], this won't automatic multi-thread. But I figured out just now that ParallelTable[SetSystemOptions["MKLThreads" -> 8];Eigenvalues[Randomreal[{1,2},{6000,6000}],{1}]will use 8 cores. So problem solved. When we want to use multi-thread feature, we have to "parallel" SetSystemOptions["MKLThreads" -> 8] to each kernels. $\endgroup$
    – matheorem
    Dec 1, 2013 at 12:36
  • $\begingroup$ @matheorem Hi very interesting question!!Could you please also post the answer completely and in a bit elaborated version that all can use also $\endgroup$
    – Alex
    Dec 7, 2013 at 10:19

1 Answer 1

Reset to default
2
$\begingroup$

Since no one make the answer, I tried to answer my own question.

Now, Suppose your HPC has nodes named as hp001,hp002,hp003,..., and you have successfully requested three of them, that is hp001,hp002,hp003. And suppose that you are performing parallel calculation that will consume memory resources heavily. So you can't not parallel the evaluation to each kernel, for if you do that, the physical memory on each node will be depleted. So you can only launch limited kernels on each node in order to avoid memory depletion.

What's more, if the calculation which you carried out contains some built-in multi-thread featured function (such as Eigenvalues), then we can manually configure the "MKLThreads" option to get idle cpu resources which have not been launched to work.

Now the code part:

the following PBSKernel function serve to launch remote Kernels.

Needs["SubKernels`RemoteKernels`"];
Clear[PBSKernel];
PBSKernel[host_String, numcore_] := 
  LaunchKernels[
   RemoteMachine[host, 
    "ssh -x -f -l `3` `1` math -mathlink -linkmode Connect `4` \
-linkname '`2`' -subkernel -noinit", numcore]];

We can launch 1 kernel for each node

Map[PBSKernel[#,1]&,{"hp001","hp002","hp003"}]

and then we set the MKLThread option in each kernel as the number of processors in each node.

ParallelEvaluate[SetSystemOptions["MKLThreads" -> $ProcessorCount]]

OK, all is done. You can now add your own code after this.

Notice that my method is not a direct answer to my title question, but the effect is similar.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.