8
$\begingroup$

I have a simple map I would like to parallelize (4x speedup for my 4 cores):

testData = ResourceData["MNIST", "TestDataset"] // Normal;
encode[i_Image] := BaseEncode @ ExportByteArray[i, "PNG"];
AbsoluteTiming[result = MapAt[encode, testData, {All, "Image"}];]
(* => {57.9009, Null} *)

Unfortunately, wrapping MapAt with Parallelize doesn't work:

LaunchKernels[];
DistributeDefinitions[encode];
AbsoluteTiming[result = Parallelize @ MapAt[encode, testData, {All, "Image"}];]

enter image description here

Is there a simple workaround I'm missing?

$\endgroup$
  • $\begingroup$ Not sure about MapAt. Is testData is available to the parallel kernels? Check ParallelEvaluate[testData[[1]]]. $\endgroup$ – Theo Tiger Apr 4 '18 at 22:55
8
+100
$\begingroup$

You can replace your MapAt with a ParallelMap. I haven't investigated, why your Parallelize doesn't work, but this version here runs only some seconds on 12 cores:

testData = ResourceData["MNIST", "TestDataset"] // Normal;

encode[i_Image] := BaseEncode@ExportByteArray[i, "PNG"];
encode[ass_Association] := Association[ass, "Image" -> encode[ass["Image"]]]

ParallelMap[encode, testData];

The trick, if you like to call it so, is to provide a version of encode that directly converts an Association into the form you want by replacing the image with its encoded form.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.