On a HPC, there are many computation nodes, each node corresponds to an independent machine. Every node has several cores, for example 8 per node.
Mathematica can launch remote kernels on different nodes (see method here). After all the kernels were launched, Mathematica seems to regard all the kernels as the same thing. We can use ParallelMap
and different options "FinestGrained"
or "CoarsestGrained"
to distribute computation tasks to different kernels regardless of which node a kernel belongs to.
But I am in a case where I want to distribute my computation task according to machines instead of kernels, because my computation involves matrix eigenvalues. When the matrix is large, the memory usage is considerable while each node has limited memory resources. So I can not parallelize the computation to "FinestGrained"
to kernels, as that will exhaust the memory and probably would cause the node to break down.
One way to solve this is to launch a limited number of kernels on each node, for example, only launch one kernel per node, and the one kernel on each node won't eat up all the memory. But in this way, I will lose the multi-thread feature of Eigenvalues
computation (see here), because memory usage is determined by the dimension of the matrix. Using 1 core or 8 cores to compute the same matrix needs the same amount of memory, but obviously 8 cores will be faster. So if I launch only one kernel on each node I am wasting the remaining 7 cores on each node.
So here is the question. How to distribute the computation tasks according to machines instead of kernels, and preserve multi-thread feature of built-in functions? I mean to create a new option "FinestGrainedToNodes"
which will distribute computation tasks evenly to different node.
Eigenvalues
is already a built-in numeric function, it should not be much slower than C or fortran $\endgroup$Eigenvalues
will use all cores even if just one Mathematica kernel is running. The parallelism takes place in Intel MKL which Mathematica uses for much of its numeric calculations. $\endgroup$ParallelTable[Eigenvalues[Randomreal[{1,2},{6000,6000}],{1}]
, this won't automatic multi-thread. But I figured out just now thatParallelTable[SetSystemOptions["MKLThreads" -> 8];Eigenvalues[Randomreal[{1,2},{6000,6000}],{1}]
will use 8 cores. So problem solved. When we want to use multi-thread feature, we have to "parallel"SetSystemOptions["MKLThreads" -> 8]
to each kernels. $\endgroup$