3
$\begingroup$

I have a code in which I create a few very large matrices with which I need to do long calculations. I have parallelised my code and, in the end, the thing that slows me the most is the time it takes to distribute the definitions of the matrices, which I am using in the calculations, to the subkernels. I was wondering whether it is somehow possible to avoid this distribution in the sense that - it is possible to just create all those matrices on each subkernel as well and then somehow let the master kernel know that they are there, so it doesn't need to distribute anything. What I am talking about is the following:

LaunchKernels[6];
$MinPrecision = 50; $MaxPrecision = Infinity;
ParallelEvaluate[$MinPrecision = 50];
ParallelEvaluate[$MaxPrecision = Infinity];
a[j_, data_] := 
  Product[data[[j]] - data[[k]], {k, 
    Range[1, Length@data, 1]~Delete~j}];
Doff[i_, j_, data_] := a[i, data]/(a[j, data] (data[[i]] - data[[j]]));
Don[j_, data_] := 
  Sum[1/(data[[j]] - data[[k]]), {k, 
    Range[1, Length@data, 1]~Delete~j}];
Dm[i_, j_, data_] := If[i == j, Don[j, data], Doff[i, j, data]];

AbsoluteTiming[NumX = 45; NumZ = 35; xmin = 0; xmax = 1; zmin = 0; 
 zmax = 1; 
 xdata = Reverse[
   Table[N[(xmax + xmin)/
      2 + (xmax - xmin)/2 Cos[(k*\[Pi])/NumX], $MinPrecision], {k, 0, 
     NumX}]]; 
 zdata = Reverse[
   Table[N[(zmax + zmin)/
      2 + (zmax - zmin)/2 Cos[(k*\[Pi])/NumZ], $MinPrecision], {k, 0, 
     NumZ}]];

 dMatX = Table[
   Dm[i, j, xdata], {i, 1, Length@xdata}, {j, 1, Length@xdata}]; 
 dMatZ = Table[
   Dm[i, j, zdata], {i, 1, Length@zdata}, {j, 1, Length@zdata}];
 d2MatX = MatrixPower[dMatX, 2];
 d2MatZ = MatrixPower[dMatZ, 2];
 d2zzMat = KroneckerProduct[IdentityMatrix[NumX + 1], d2MatZ];
 d2xxMat = KroneckerProduct[d2MatX, IdentityMatrix[NumZ + 1]];
 dzMat = KroneckerProduct[IdentityMatrix[NumX + 1], dMatZ];
 dxMat = KroneckerProduct[dMatX, IdentityMatrix[NumZ + 1]];
 d2xzMat = KroneckerProduct[dMatX, dMatZ];
 xxdata = Flatten@Transpose@ConstantArray[xdata, NumZ + 1];
 zzdata = Flatten@ConstantArray[zdata, NumX + 1];]

which takes

{0.968119, Null}

to evaluate. However, when I distribute the whole thing (another complication is that it is arbitrary precision, so I can't use packed arrays) it takes a lot of time

AbsoluteTiming[DistributeDefinitions[dMatX, dMatZ, d2MatX, d2MatZ]; 
 DistributeDefinitions[xxdata, zzdata, dxMat, dzMat, d2xxMat, d2xzMat,
   d2zzMat];]

producing

{123.718, Null}

I thought that I can just ParallelEvaluate the code that creates the matrices above and then they will be present on each subkernel (I am also creating them on the master kernel). Unfortunately, when running my calculations afterwards, the code still first distributes definitions for the same amount of time at the beginning (which I can check by looking at evaluation times and also the activity of the cores, as it is always only 2 kernels wokring when definitions are being distributed). So my question is - is it possible to somehow create the matrices on the subkernels and then tell the master kernel that they are there.

$\endgroup$
  • $\begingroup$ I would experiment with putting these matrices in a different context than Global` and then prevent the distribution of that context by either explicitly setting the DistributedContexts option on the relevant functions, or by paying attention to the value of $DistributedContexts. I do not have time to try this, but I advise paying attention to: 1. since some recent M version, ParallelEvaluate also has the DistributedContexts option, and will auto-distribute just like ParallelTable, etc. $\endgroup$ – Szabolcs Sep 30 '17 at 13:26
  • $\begingroup$ In earlier versions, it did not auto-distribute. 2. Check what is the default value of DistributedContexts for each function you use. I think that when it's Automatic (as in DistributeDefinitions itself), you'd need to set it to something else (e.g. :> $DistributedContexts) to prevent picking up all contexts appearing in your expression. Otherwise: Your approach seems sensible to me. And yes, arbitrary precision numbers are very slow to distribute, or in general to pass through MathLink. $\endgroup$ – Szabolcs Sep 30 '17 at 13:27
  • $\begingroup$ @Szabolcs Thank you very much for the input, I was hoping that you will see this. I am not familiar with DistributeContexts, but I will look into it now $\endgroup$ – ThunderBiggi Sep 30 '17 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.