I would like to evaluate a Dataset query in parallel, my latest approach being:

PQuery[d_, q__] := If[Head@d === Dataset, Dataset, # &]@
  Replace[Normal@Query@q, Map[f_] :> ParallelMap[f, Normal@d]]

The idea is to let Query create the query operator and then replace the topost Map with a ParallelMap. Also, I remove any wrapper like Dataset around the data, since ParallelMap doesn't seem to like Dataset as parameter.

My question now is: Is there any simpler and/or more robust way of doing this (this breaks for example if the head is e.g. MapAt)? Also, something seems to be extremely slow for some kinds of datasets (could this be caused by Normal?)

  • $\begingroup$ Can you show an example of usage for your PQuery function, and a toy dataset as well? As this stands, it is somewhat unclear to me what you want to accomplish. $\endgroup$ – MarcoB Jun 15 '16 at 17:31
  • $\begingroup$ @MarcoB Sorry for the extremely late reply. The idea is very simple: Query[All,f][Dataset[{1,2}] essentially maps f over {1,2}. Now, if f takes a long time, it would be good if the Map (you can see this Map by evaluating Normal@Query[All,f]) could be replaced by a ParallelMap (at least the topmost instance). The approach used for PQuery is a very primitive implementation of this idea, and also rather slow (as mentionned). $\endgroup$ – Lukas Lang Jul 20 '16 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.