4
$\begingroup$

Why does DistributeDefinitions take so long to distribute variables that are 100-300 Mb in size?

I am setting up ParallelTable to compute values with a very high precision on a dual-core machine with 16 GB RAM. The precision range in the table is variable so I pass to the Kernels, constants like Log[2] and Log[10] with the upper range of precision, 100 million for example, and then just SetPrecision to the desired (lower) working precision to reduce execution time. I'm surprised this takes so long to distribute the definitions considering this is being done in RAM. Below is a sample of what I'm trying to do. The constants myLog2=Log[2] and myLog10=Log[10] are first computed to 100 million digits of precision.

I was wondering if I'm not setting up the kernels correctly or if there is a way to speed up the process. Note if I do not use DistributeDefinitions, the kernels seem to do this automatically and take just as long and I need to distribute more than just the two in the example below.

LaunchKernels[]; $KernelCount
AbsoluteTiming[
  DistributeDefinitions[myLog2, myLog10];
]

2

{218.259, Null}

Now ParallelTable executes quickly:

(*
  time using DistriburteDefinitions[myLog2, myLog10]
*)

AbsoluteTiming[theTable = ParallelTable[
    val1 = SetPrecision[myLog2, 5000000];
    val2 = SetPrecision[myLog10, 5000000];
    val3 = val1 val2;
    val3,
    {n, 1, 2}];]

{2.91649, Null}

$\endgroup$

2 Answers 2

2
$\begingroup$

I don't know why it takes so long, but you can get around it by dumping the variables in an mx file and then Getting the file on the parallel kernels:

n = 7;
file = "path/to/log2.mx";
myLog2 = N[Log[2], 10^n];
myLog10 = N[Log[10], 10^n];
{ByteCount[myLog2], ByteCount[myLog10]}

DumpSave[file, {myLog2, myLog10}];
LaunchKernels[];
DistributeDefinitions[file];
ParallelEvaluate[Get[file]; {ByteCount[myLog2], ByteCount[myLog10]}]
$\endgroup$
3
  • $\begingroup$ Thanks Sjoerd. Afraid though that's not improving the time when the precision is 100 million. I noticed you did 10 million and I know it takes Mathematica a long time to compute Log[x] for high precision. I used y-cruncher to get 100 million and then just imported into Mathematica. But the DumpSave/Get approach at 100 million still took about 220 seconds to run ParallelTable the first time. But then only 3 seconds to re-run it. Still though equivalent to using DistributeDefinitions on the actual constants when running the first time. $\endgroup$
    – Dominic
    Nov 11, 2020 at 16:11
  • 1
    $\begingroup$ @Dominic So why don't you just load the data directly into the parallel kernels the same way you load it into the master kernel? Just run the loading code inside of ParallelEvaluate then. Maybe you can give some more details about what you're doing exactly? $\endgroup$ Nov 11, 2020 at 17:39
  • $\begingroup$ Turns out DumpSave is context sensitive and I'm not so good with this. Others on the web have recommended not using it. This reference: mathematica.stackexchange.com/questions/3378/… suggests using Import/Export. These use the same binary format (cuts the file size in half) and loading into the Kernels do seem to be much faster. Will do some more testing. $\endgroup$
    – Dominic
    Nov 11, 2020 at 19:56
1
$\begingroup$

Import/Export solves the problem as I see it. Here's the steps I used to import three 100-million digit numbers into the kernels and there was no appreciable wait time. I realize Mathematica can compute these values to high precision but I will eventually need them much higher and y-crusher does it much quicker.

Perhaps someone can improve my method?

Compute Log(2), Log(10), and Pi via y-cruncher to 100 million digits. y-cruncher saves the results to a text file. Next import these text files into Mathematica and save to expressions:

    log2FileName = "c:\\Users\\Dominic\\Desktop\\yCruncher\\log2File.txt";
log10FileName = 
  "c:\\Users\\Dominic\\Desktop\\yCruncher\\log10File.txt";
piFileName = "c:\\Users\\Dominic\\Desktop\\yCruncher\\piFile.txt";
myLog2String = Import[log2FileName, "String"];
myLog10String = Import[log10FileName, "String"];
myPiString = Import[piFileName, "String"];
    
    AbsoluteTiming[
     myLog2 = ToExpression[myLog2String];
     myLog10 = ToExpression[myLog10String];
     myPi = ToExpression[myPiString];
     ]

Now export the three Mathematica expressions to a .mx file:

myDataSaveFileName = 
  "c:\\Users\\Dominic\\Desktop\\yCruncher\\dataSave.mx";
Export[myDataSaveFileName, {myLog2, myLog10, myPi}, 
  "MX"];

Once this is done, do not have to re-do the set-up steps above.

Start kernels, distribute only the file name and import data:

In[31]:= If[$KernelCount != 2, 
  LaunchKernels[];
  ];
myDataSaveFileName = 
  "c:\\Users\\Dominic\\Desktop\\yCruncher\\dataSave.mx";
DistributeDefinitions[myDataSaveFileName];

AbsoluteTiming[theTable = ParallelTable[
    Print["doing: ", n];
    {theLog2, theLog10, thePi} = Import[myDataSaveFileName];
    val1 = SetPrecision[theLog2, 5000000];
    val2 = SetPrecision[theLog10, 5000000];
    valPi = SetPrecision[thePi, 5000000];
    val3 = val1 valPi;
    val3,
    {n, 1, 5}];]



(kernel 2) doing: 1

(kernel 1) doing: 3

(kernel 2) doing: 2

(kernel 1) doing: 4

(kernel 2) doing: 5

Out[32]= {7.15045, Null}
$\endgroup$
1
  • $\begingroup$ Glad to see you managed to figure it out. One note: I recommend using ParallelEvaluate instead of ParallelTable in the last code block. ParallelEvaluate is specifically designed for these sort of initialization procedures. $\endgroup$ Nov 13, 2020 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.