1
$\begingroup$

I would like to give some context: I need to analyse many data sets stored in files using different kind of analyses. To keep track I like to tag each results of my analysis for example like result["analysis1"]["file1"]. For this I am used to write code like this:

analyse[params_][file_]:=Module[{data},
  data=accessdata[file];
  result["analysis1"][file]=doanalysis1[params, data];
  result["analysis2"][file]=doanalysis2[params, data];
]

(*analalyse many files*)
files={"file1", "file2", ...};
Map[analyse[params],files];

(* aggregate results e.g. in plots or histograms*)
ListPlot[result["analysis1"]/@files];
Histogram[result["analysis2"]/@files]

This works fine and gives me great flexibility. For example if I need to re-analise e.g. "file17", maybe with some other parameters, I can just call analyse[otherparam]["file17"] and plot the updated results again. or I can save the results with Save["resultfile", result].

My problem arises, if I try to parallelize this.

ParallelMap[analyse[params],files];

does not work, because each kernel will have its own version of result. I tried:

SetSharedFunction[result];
ParallelMap[analyse[params],files];

But I got error messages from all kernels like:

Set::write: Tag Null in Null["file1"] is protected.

Also I tried:

SetSharedFunction[result["analysis1"]]

which apparently does not even get executed. How can one share result such that the code above works with ParallelMap?

$\endgroup$
1
$\begingroup$

I found a workaround that avoids the use of tagged functions inside the ParallelMapbut allows its use outside.

analyse[params_][file_]:=Module[{data},
  data=accessdata[file];
  result["analysis1", file]=doanalysis1[params, data];
  result["analysis2", file]=doanalysis2[params, data];
]
result[tag_][file_]:= result[tag, file]
SetSharedFunction[result];
ParallelMap[analyse[params],files];
ListPlot[result["analysis1"]/@files];

Still I would be interested in other ideas!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.