Eigensystem on HPC

Question

I am submitting Mathematica jobs to an HPC. I have a large matrix that is a function of (x,y) and I am diagonalizing it for around 500 points in the plane.

My initial idea was to request a node with say 32 processors. Then I use ParallelMap so that each processor diagonalizes at different points.

However, running tests on my own 4-core desktop, I've noticed a problem. Eigensystem, even called in series, uses all four cores. When called in parallel, it appears to use the four cores more efficiently (i.e. CPU usage is on average higher), but not drastically so. Thus, there is not very much speed up (maybe around 10%). I conclude Eigensystem already has some built-in way to take advantage of multiple processors(?).

The question is, if I reserve a node on the HPC and use Eigensystem in series, will it: (1) automatically use all the available processors on the node, (2) only the processors I reserved in the PBS script request, or (3) something else?

I am doing tests myself, but I wanted to see if anyone had any experience with this. And I have to wait a long time in the queue to do anything.

Answer 1

It depends on how PBS and the cluster environment are set up. Ideally, if cpuset support has been compiled in to PBS, and if you start Mathematica directly inside the PBS job, you should find that it uses all processors allocated by PBS (on that node--Eigensystem is not MPI-parallelized). If cpuset support isn't provided, then you risk starting as many threads as there are (physical) processors on that node--which will annoy the other users and administrators, and will not produce a good outcome for anyone.

Furthermore, suppose that you are given four logical processors across two physical processors: here you will get two MKL threads, whereas if those processes were split across four logical processors, you could have had four threads, which will result in different execution characteristics. Different clusters may have SMT either enabled or disabled depending on the administrators' decision based on typical job characteristics. If you are allocated logical processors scattered randomly over the node, the threading library probably will not know what to do about this. And even if cpuset support is available, and PBS allocates processes in a favorable way, whether the execution proceeds efficiently or not still depends on CPU affinity settings (both in PBS and in the threading library). You may need to play with the environment variables controlling CPU affinity in the MKL to get the best results.

Overall, it is too complicated to say, as an outsider without access to your system, what will give the best results (or even work at all). But, a general piece of advice that would somewhat sidestep all of these issues is to request an entire node (including all CPUs), rather than a subset thereof. This way, no matter what PBS and the threading library do, the job is not going to be presented with a strange non-uniform physical architecture; this should hopefully reduce the likelihood of performance issues. And, if it does all go wrong nonetheless, at least only your job will perform poorly--you are not harming other users' work by inadvertently claiming their execution resources.

I'm afraid there is not much you can do apart from waiting a long time in the queue to do your own tests--at least not unless all of this is helpfully documented somewhere by your administrators. But, you can at least do most of that testing by starting some very short interactive jobs in which you figure out first of all how the PBS behavior, node environment, and threading library interact with one another on your cluster. These types of jobs should (hopefully) not have to spend too long queueing.