# Remote parallel processing

(The problem has been narrowed down, and solved to a more simple one described at the end of the post beside UPDATE 3)

I'm doing some numerical work that takes quite a long time to run on my computer (which has 4 kernels). I got access to a cluster which has a Mathematica license that lets me use up to 80 kernels.

I made a few simple tests to find out how efficient it turns out to be. The first one being

ParallelTable[Pause[1]; i, {i, 1, 80}];// AbsoluteTiming
{1.171367, Null}


which tells me that all 80 kernels are running in parallel.

The next test I ran on my computer (not the cluster) was

Table[Sqrt[i], {i, 1, 10^6}]; // AbsoluteTiming
{11.3152, Null}


and

ParallelTable[Sqrt[i], {i, 1, 10^6}]; // AbsoluteTiming
{13.5914, Null}


which somehow tells me that 4 kernels in parallel run slower than 1 alone (maybe because it takes time to rejoin all information sent to the kernels?).

The same test using only 1 kernel from the cluster returns {13.661838, Null} from which I deduce that their processor is less powerful than mine. More interestingly, when I run all 80 kernels, I get {19.057895, Null}. I'm a bit disappointed with this result, but I'm sure that I'm missing a detail which explains this discrepancy.

Moreover, what I'm really interested is to create a table (using ParallelTable) of the function (for 0<=w<=1000 and 0<=b<=300; the options used give me an overall error of ~0.1% ,which is acceptable):

FAuR[w_, b_] := If[b == 0, 0, NIntegrate[1/u Sqrt[u^2 - (w/gAu)^2]
BesselJ[1, b Sqrt[u^2 - (w/gAu)^2]] FFAuR[u^2], {u, w/gAu,
40}, MinRecursion -> 3, MaxRecursion -> 100,
WorkingPrecision -> 20, PrecisionGoal -> 10,
AccuracyGoal -> Infinity] ]


where gAu=19700/183 and FFAuR is an interpolated function which can be imported using the command FFAuR = << "FF-Au.m"(the file FF_Au.m can be downloaded from the link provided at the end).

Running this function on my computer I get

FAuR[1, 1] // AbsoluteTiming
NIntegrate::slwcon: Numerical integration converging too slowly; suspect one of the following: singularity, value of the integration is 0, highly oscillatory integrand, or WorkingPrecision too small. >>
{5.26011, 0.086004396132821223043}


while running it on the cluster I get

FAuR[1, 1] // AbsoluteTiming
NIntegrate::slwcon: Numerical integration converging too slowly; suspect one of the following: singularity, value of the integration is 0, highly oscillatory integrand, or WorkingPrecision too small. >>
{431.500101, 0.086004396132821223043}


It takes 80x more time to run on the cluster, what is happening? I was really hoping to have a great improvement and being able to get the most out of the cluster but these results are telling me otherwise. Any idea?

If anyone is interested in running the code, I uploaded the FF_Au.m (32 MB) file to my Google Drive:

My computer is running Mathematica 11.0 on Windows 10, and the cluster is running Mathematica 10.0.

UPDATE 1 (21/08): I found out that the problem is caused by the interpolated function FFAuR. Integrating it numerically alone takes 435 seconds on the cluster and 5 seconds on my computer. However, plotting takes the same amount of time on both. Something might have changed inside NIntegrate when updating version 10 to 11.

UPDATE 2 (21/08): I narrowed the problem to a power. Evaluating the following code

NIntegrate[FFAuR[Power[x, 1]], {x, 0, 10}] // AbsoluteTiming
{0.990320, 0.353331}


takes 1s on the cluster and 1.1s on my on my computer. Squaring the argument of the function FFAuR somehow wrecks everything:

NIntegrate[FFAuR[Power[x, 2]], {x, 0, 10}] // AbsoluteTiming
{461.942459,0.580889}


It takes 435s to evaluate this code on the cluster while it takes only 3s on my PC! I still don't know why this happens.

UPDATE 3 (21/08): I found the solution. Mathematica 10.0 has a different symbolic preprocessing which makes the evaluation of NIntegrate take much longer. Adding Method->{"SymbolicProcessing" -> 0} at the end of NIntegrate solves the problem!

Thanks!

• The integration is a sequentially solved and, therefore, parallelization will not give you an acceleration of your job. It consumes too much time because the cluster has a bit weaker processors plus it needs time for data distribution to remote kernel (through LAN that is slowly). – Rom38 Aug 22 '17 at 3:59
After spending some time isolating the problem, I found out that it was caused by the way symbolic preprocessing is done by NIntegrate algorithm in different Mathematica versions. The solution was to add Method -> {Automatic, "SymbolicProcessing" -> 0} at the end of NIntegrate. The post where I found this solution, from where you can read more about, is: