I am trying to do a computation that takes a lot of CPU time and it is quite computationally intensive. I want to make it Parallel instead of serial to compute the results faster.

Things I tried:

Step 1. I launched 2 sub kernels on which I would like to do the computation. The $DistributedContexts is set to None. This is because of the vast number of functions and definitions that need to be distributed. Instead, I load all the definitions and rules into a script file.m and evaluate the script across all the sub-kernels and in the main kernel as well.

Step 2 I define a list CachedQntity as a list of inputs to be evaluated for the function DerivCache that has been defined in the file.m and is present in all kernels. In reality, the list is very large and I have presented only a subset here.

DerivCache[stf_[inds___], derivorder_, pnorder_] := 
 DerivCache[stf[inds], derivorder, pnorder] = 
  Evaluate[PNExpUntilFixed[DerivCacheHelper[stf[inds], derivorder, pnorder],pnorder]]

This is the function that is supposed to be evaluated on the CachedQntity list. It tries to calculate the derivatives of a tensor(stf[inds]) at some fixed post-Newtonian order. pnorder is a fraction or integer, and derivorder is a whole number representing the number of derivatives. Also, it caches the expression obtained after calculating the derivative.

$DistributedContexts = None;

CachedQntity := {{\[ScriptCapitalI][], 5, 8}, {\[ScriptCapitalX][], 3,4}};
STFCacheHelper := ParallelMap[DerivCache @@ # &, CachedQntity];

DumpSave["/directory/file.mx", {STFCacheHelper, STFDerivCache}];

Moving ahead: The problem that I am having is the following. Consider that I have some number of kernel (ex -3) and the length of the list CachedQntity is 12. The ParallelMap distributes the list over the kernels to be evaluated on the function DerivCache and gives the result in a list.

But I know that when DerivCache tries to compute something it also caches that result into its definition. I would like to retrieve those cached definitions from each sub-kernel and combine them into a single definition of DerivCache with all the cached expressions in the main kernel. Then the last step is to save that definition as a final result into a .mx file.

How to achieve this? I don't know how to retrieve definitions from the sub kernels and get them into the main kernel. Is there another way to achieve the intended result?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.