I would like to evaluate a
Dataset query in parallel, my latest approach being:
PQuery[d_, q__] := If[Head@d === Dataset, Dataset, # &]@ Replace[Normal@Query@q, Map[f_] :> ParallelMap[f, Normal@d]]
The idea is to let
Query create the query operator and then replace the topost
Map with a
ParallelMap. Also, I remove any wrapper like
Dataset around the data, since
ParallelMap doesn't seem to like
Dataset as parameter.
My question now is: Is there any simpler and/or more robust way of doing this (this breaks for example if the head is e.g.
MapAt)? Also, something seems to be extremely slow for some kinds of datasets (could this be caused by