1
$\begingroup$

At the moment I continously encounter the problem that while using ParallelTable the process kswap needs all the memory, while the four running MathKernels receive almost zero %Memory, which slows down the calculation incredibly.

I am parallelizing a two dimensional integral with one external variable as

Intx0x1[x0_?NumericQ,x1_?NumericQ]:=Intx0x1[x0,x1]=NIntegrate[f[x0,x1,x2],{x2,x2min[x0,x1],x2max[x0,x1]},Method->{Automatic,"SymbolicProcessing"->0}]

Intx0[x0_?NumericQ]:=Intx0[x0]=NIntegrate[Intx0x1[x0,x1],{x1,x1min,x1max[x0]},Method->{Automatic,"SymbolicProcessing"->0}]

TabInt=ParallelTable[Intx0[x0],{x0,x0min,x0max,dx0}]

The x0, x1, x2 dependence in f is quite complicated, however one single integration needs only approximately 0.30 s. Nevertheless for taking 100 points in TabInt the evaluation needs up to 5 minutes. Now the x1,x2 dependence will be replaced by more complicated functions f1[x0,x1],f2[x0,x2],... and more. All are based on further integrals and a similar structure as the one above. Which renders one calculaton to last 20 minutes. On top of that there are many more parameters that I want to keep as variables in the definition of Intx0x1, Intx0, and TablInt, while the x0,x1,x2 dependences will become increasingly complicated and some will be interpolated functions that are based on the same structure as above.

So I understand that it is probably a problem of the limited working memory and the distribution of results among the different kernels.

I have only very limited knowledge about Linux, but i found out that if I kill one of the running kernels with

"kill -9 PID"

Then memory is set free and the compution runs as usual. However, during an evaluation I always need to monitor the processes using top in a shell and then manually kill one or two of the kernels, several times. Moreover, it also happens that I pick the "wrong" kernel (=main kernel?) and then the kernel fully quits and all results are lost and I have to start the whole calculation new.

Now my question is, how can I avoid the calculation to be "stuck"? Is there a command in mathematica that allows me to automatize the kernel-killing such that e.g. I do not kill the wrong kernel and such that e.g. I can let it run over night, without the need of manually monitoring and killing kernels?

Since my knowledge is limited, could someone explain to me what exactly is happening?

$\endgroup$
  • 2
    $\begingroup$ It's very hard to answer without seeing the actual expressions you are integrating. But I must say that killing the subkernels seems like a very bad idea and also something that will probably lead to wrong results. Also, why are you defining Intx0x1 and Intx0 if you never call them? Can you also double check the syntax in your code? There are some parenthesis issues, like NIntegrate[[f[... and missing {}. $\endgroup$ – Marius Ladegård Meyer Dec 5 '16 at 13:24
  • $\begingroup$ I corrected the syntax mistakes. I used a different notation than in my original code and was not careful enough about the notation. As far as I can see the results are stable. In the shell I can see that there are more than four MathKernels with different PIDs running. At the moment I don't see any other option. I started one calculation yesterday. This morning the process was still running after 8 hours. All MathKernels were using approx. 2%CPU, kswap taking the rest. I killed one kernel and then the others were taking again their 100%CPU and the result was there ten minutes later. $\endgroup$ – user3526475 Dec 5 '16 at 14:05
  • $\begingroup$ @MariusLadegårdMeyer The function f itself is a long expression. I would not know how to post it. The purpose of it is the calculation of a squared amplitude involving three resonances including their real part of the self energy in the mass, involving again another integration over an "internal" x0 by a dispersion integral plus subtraction constants, and all kind of additional x0,x1,x2 dependencies because of cut-off functions. I really don't know how I could produce something like a minimal working example and hoped that the above structure would be enough. $\endgroup$ – user3526475 Dec 5 '16 at 14:12
2
$\begingroup$

As Marius has mentioned killing the subkernels seems a very bad idea, it just is a last resort to recover the machine when a calculation already has gone out of control, you will usually loose results and what you don't loose might be corrupted.

It looks like your poblem is that for some cases NIntegrate needs too much memory, which could either be an inidication of an error in NIntegrate or it could be a result of a "difficult" but still correctly evaluating case, which I think is possible for the algorithms that NIntegrate internally uses.

In any case when memory is the limit it might be counterproductive to parallelize, as then the parallel kernels will share the physical available memory. If in general your expressions can be integrated with low memory consumption (< phys_mem/num_procs) and only some cases don't work or will need excessive amounts of memory you could do the following to run your problem more reliably:

  • use MemoryConstrained (and maybe also TimeConstrained) for a single calculation step.
  • save every intermediate result once it is available to disk
  • before starting a calculation, check whether the corresponding result already exists on disk.

Doing so, you will avoid loosing what already was done, with checking for already calculated results you can savely restart your calculation. Using the constrains might help to not even run into the problems you see. There might still be cases where you still have to kill the kernels, but whatever has been written to disk will survive that. Of course when the constrains are violated that will leave some of the intergrals unevaluated, but if these are just a minority, you can try to tackle them after all easy ones have been finished. A first try would be to do do the difficult ones without parallelization, which leaves more memory per process, if that also fails you will need to have a closer look at why they don't work.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.