I'm programming an algorithm where I have to evaluate, at each iteration, a characteristic of a subgroup of data. An evaluation of a subgroup takes 2-3 secondes and I have 200 sugroups to test per iteration (I expect 600 iteration approximatively). So the parallelisation of the program is clearly necessary. I can launch between 4 and 8 kernels (when it will be ok, 24). But I have issues that a don't know how to fix it...
The characteristic is computed thanks to a function wich call packages. Before the main loop I have to run some pre-calculation. My program (too long to be copied here I think) is organized that way:
Importation of datas;
Preparation of the subgroups;
Thanks to the functions into the packages a calculate some constant I need
Definition of the functions for the main loop, and the one a I will parallelize:
List=list of 200 subgroups, DistributeDefinitions["Global`"], DistributeDefinitions[Rfun],
R=ParallelMap[RFun @@ #&, List],
RFun comes from a package I called before)
Call of test in a For loop...
The problem is when I run it I have many error message asking me a package that I don't use in the function Get::noopen: Cannot open blabla...and wich I get like the others...Is this a problem of distribution of defintions to the slaves kernels? because I tried many things, I tried to move my DistributeDeifinitions nearly everywhere in my algorithm,I tried ParallelMap[RFun @@ #&, List,DistributedContexts->Automatic]...Sometimes it seems to be ok the programme run but when I close mathematica and re-open it the errors come back... When I run the program with one kernel (and just Map[...]) everything works fine...
Someone have an hint on whats going on?
Edit: My program is ok When I use:
But the first time a run it I have the error: "Get::noopen: Cannot open..."
So I run a lighter version of the program launching kernels, using the parallelMap function only one time and I close the kernels. When a rerun it it's ok and I can do all the calculation i want. But It's not a very good solution. Someone has an idea?