Why is DistributeDefinitions taking so long with 100-300 Mb constants?

Question

Why does DistributeDefinitions take so long to distribute variables that are 100-300 Mb in size?

I am setting up ParallelTable to compute values with a very high precision on a dual-core machine with 16 GB RAM. The precision range in the table is variable so I pass to the Kernels, constants like Log[2] and Log[10] with the upper range of precision, 100 million for example, and then just SetPrecision to the desired (lower) working precision to reduce execution time. I'm surprised this takes so long to distribute the definitions considering this is being done in RAM. Below is a sample of what I'm trying to do. The constants myLog2=Log[2] and myLog10=Log[10] are first computed to 100 million digits of precision.

I was wondering if I'm not setting up the kernels correctly or if there is a way to speed up the process. Note if I do not use DistributeDefinitions, the kernels seem to do this automatically and take just as long and I need to distribute more than just the two in the example below.

LaunchKernels[]; $KernelCount
AbsoluteTiming[
  DistributeDefinitions[myLog2, myLog10];
]

2

{218.259, Null}

Now ParallelTable executes quickly:

(*
  time using DistriburteDefinitions[myLog2, myLog10]
*)

AbsoluteTiming[theTable = ParallelTable[
    val1 = SetPrecision[myLog2, 5000000];
    val2 = SetPrecision[myLog10, 5000000];
    val3 = val1 val2;
    val3,
    {n, 1, 2}];]

{2.91649, Null}

Answer 1

I don't know why it takes so long, but you can get around it by dumping the variables in an mx file and then Getting the file on the parallel kernels:

n = 7;
file = "path/to/log2.mx";
myLog2 = N[Log[2], 10^n];
myLog10 = N[Log[10], 10^n];
{ByteCount[myLog2], ByteCount[myLog10]}

DumpSave[file, {myLog2, myLog10}];
LaunchKernels[];
DistributeDefinitions[file];
ParallelEvaluate[Get[file]; {ByteCount[myLog2], ByteCount[myLog10]}]

Answer 2

Import/Export solves the problem as I see it. Here's the steps I used to import three 100-million digit numbers into the kernels and there was no appreciable wait time. I realize Mathematica can compute these values to high precision but I will eventually need them much higher and y-crusher does it much quicker.

Perhaps someone can improve my method?

Compute Log(2), Log(10), and Pi via y-cruncher to 100 million digits. y-cruncher saves the results to a text file. Next import these text files into Mathematica and save to expressions:

    log2FileName = "c:\\Users\\Dominic\\Desktop\\yCruncher\\log2File.txt";
log10FileName = 
  "c:\\Users\\Dominic\\Desktop\\yCruncher\\log10File.txt";
piFileName = "c:\\Users\\Dominic\\Desktop\\yCruncher\\piFile.txt";
myLog2String = Import[log2FileName, "String"];
myLog10String = Import[log10FileName, "String"];
myPiString = Import[piFileName, "String"];
    
    AbsoluteTiming[
     myLog2 = ToExpression[myLog2String];
     myLog10 = ToExpression[myLog10String];
     myPi = ToExpression[myPiString];
     ]

Now export the three Mathematica expressions to a .mx file:

myDataSaveFileName = 
  "c:\\Users\\Dominic\\Desktop\\yCruncher\\dataSave.mx";
Export[myDataSaveFileName, {myLog2, myLog10, myPi}, 
  "MX"];

Once this is done, do not have to re-do the set-up steps above.

Start kernels, distribute only the file name and import data:

In[31]:= If[$KernelCount != 2, 
  LaunchKernels[];
  ];
myDataSaveFileName = 
  "c:\\Users\\Dominic\\Desktop\\yCruncher\\dataSave.mx";
DistributeDefinitions[myDataSaveFileName];

AbsoluteTiming[theTable = ParallelTable[
    Print["doing: ", n];
    {theLog2, theLog10, thePi} = Import[myDataSaveFileName];
    val1 = SetPrecision[theLog2, 5000000];
    val2 = SetPrecision[theLog10, 5000000];
    valPi = SetPrecision[thePi, 5000000];
    val3 = val1 valPi;
    val3,
    {n, 1, 5}];]



(kernel 2) doing: 1

(kernel 1) doing: 3

(kernel 2) doing: 2

(kernel 1) doing: 4

(kernel 2) doing: 5

Out[32]= {7.15045, Null}