4
$\begingroup$

I tried to find a solution to my problem in several answers posted in MSE but due to lack of my knowledge on parallelization territory, I couldn't figure out solution to my problem.

I have a big code with several Packages (with private contexts in each package) in addition to call some of MMA packages. The computations takes around 10 hours (which is too long time for my goal) and it gets worse when run this code tens of times !!!

The general structure of the code is as follows:

enter image description here

I am trying now to use the parallelization technique to reduce the processing time. I thought first to use ParallelDo but it did not work. After reading through documentation and several answers in MSE I realized that ParallelDo is not the right way to parallelize the code. According to what I understand, I thought to parallelize the parts that are inside the red rectangles in the picture.

I first tried Parallelization on the first rectangle as follows:

Do[
 Parallelize[{"SQL data1+Processing", "SQL data2+processing", 
   "SQL data3+processing"}];
 Rest of the code ...,
 {Do Variable}]

I got tons of errors with all indicate that Kernels did not recognized the call of the Packages and their functions.

My question is how to use Parallelization in this case?

Thank you

Update:

Here is a minimal example of my problem:

BeginPackage["Test`"];
f::usage="Test function 1";
g::usage="Test function 2";
n::usage="Test Number";
str::usage="Test string"
Begin["`Private`"];
f[x_]:=Module[{},
n=IntegerPart[x^2]];
g[list_]:=Total[list];
str:=RandomSample[{"a","b","c","d","e","f"},3];
End[];
EndPackage[];

Now if I use Do as follows it works:

Get["Test.m"]
result1 = {};
result2 = {};
result3 = {};
Do[
 r11 = f[#] & /@ Range[i];
 r12 = g[r11];
 str1 = StringMatchQ[StringJoin@str, "*a*"];
 AppendTo[result1, {r12, str1}];

 r21 = f[#] & /@ Range[i + 1];
 r22 = g[r21];
 str2 = StringMatchQ[StringJoin@str, "*a*"];
 AppendTo[result2, {r22, str2}];

 r31 = f[#] & /@ Range[i + 3];
 r32 = g[r31];
 str3 = StringMatchQ[StringJoin@str, "*a*"];
 AppendTo[result3, {r32, str3}];
 ,
 {i, 1, 5}]

But this does not work:

Get["Test.m"]
result1 = {};
result2 = {};
result3 = {};
Do[
 Parallelize[{
   r11 = f[#] & /@ Range[i];
   r12 = g[r11];
   str1 = StringMatchQ[StringJoin@str, "*a*"];
   AppendTo[result1, {r12, str1}];,

   r21 = f[#] & /@ Range[i + 1];
   r22 = g[r21];
   str2 = StringMatchQ[StringJoin@str, "*a*"];
   AppendTo[result2, {r22, str2}];,

   r31 = f[#] & /@ Range[i + 3];
   r32 = g[r31];
   str3 = StringMatchQ[StringJoin@str, "*a*"];
   AppendTo[result3, {r32, str3}];
   }]
 ,
 {i, 1, 5}]
$\endgroup$
  • 2
    $\begingroup$ What portions of the code take most of the time? Also, remember that DistributeDefinitions or ParallelNeeds sometimes must be used. $\endgroup$ – bbgodfrey May 25 '16 at 2:40
  • $\begingroup$ @bbgodfrey The computation inside the first two red rectangular take most of the time (each Do cycle takes around 4 sec and the total number of the Do cycles are around 9000). Does DistributeDefinitions means declare all symbols that has to be known to all kernels, I have hundreds of them including those in the privet contexts !!! $\endgroup$ – Algohi May 25 '16 at 2:59
  • $\begingroup$ @bbgodfrey I just tried using ParallelNeeds but still symbols belong to loaded packages are not recognized by sub kernels $\endgroup$ – Algohi May 25 '16 at 3:15
  • 1
    $\begingroup$ I do not have good answers to your questions. Is it possible to provide code for the initial loop? Despite your diagram, I do not have a good feel for the structure of the code. $\endgroup$ – bbgodfrey May 25 '16 at 4:19
  • 2
    $\begingroup$ Can you try to create a minimal example with some package? Did you call ParallelNeeds before or after launching the kernels? Call if afterwards only because otherwise any error messages that might be printed will be suppressed. ParallelNeeds is absolutely necessary for the package to be usable on the subkernels. $\endgroup$ – Szabolcs May 25 '16 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.