# Mathematica, MKL_NUM_THREADS, and GNU/Linux Process Limits

I am using Mathematica 10.3 on a large GNU/Linux system (CentOS 6.6) with 40 processors and 250 GB of RAM. It is a delight to watch it chew through linear algebra problems using all the processors.

However, it seems that the system becomes unstable, and perhaps even the Mathematica Kernel crashes, after evaluating expressions that use all of these resources. (A built-in example is to evaluate the NDEigensystem help notebook on the 40-core machine.) It seems as though the system has run out of permissible user processes.

When I issue (zsh):

 ~> limit

cputime         unlimited
filesize        unlimited
datasize        unlimited
stacksize       10MB
coredumpsize    0kB
memoryuse       unlimited
maxproc         1024
descriptors     1024
memorylocked    64kB
maxfilelocks    unlimited
sigpending      256546
msgqueue        819200
nice            0
rt_priority     0


which leads me to believe that as many settings as are relevant are unlimited.

I have also explored various settings for OMP_NUM_THREADS and MKL_NUM_THREADS--setting these to smaller settings, e.g. MKL_NUM_THREADS=8, does not seem to exhibit the instability. However, the other cores remain unused, and these really need to be in play for Mathematica to remain competitive with other technologies such as R linked with MKL.

There are other settings for OMP and MKL, such as OMP_DYNAMIC, but I have not found a magic recipe to get this to work.

My question: have other users seen such (mis)behavior, and are there settings that can be invoked in order to get around these issues?

• I think maxproc counts each thread in every process separately, so 1024 may not be enough. Is there any difference if you set it to unlimited? – ilian Nov 22 '15 at 23:26
• Can you let me know which example of the NDEigensystem page crashes for you? How many threads can you use until you see the crash? – user21 Nov 23 '15 at 8:22
• @ilian i think this may be the solution, thanks! i'm going to test this for a bit and then write back. – Eric Brown Nov 23 '15 at 8:22