8
$\begingroup$

I have a simple map I would like to parallelize (4x speedup for my 4 cores):

testData = ResourceData["MNIST", "TestDataset"] // Normal;
encode[i_Image] := BaseEncode @ ExportByteArray[i, "PNG"];
AbsoluteTiming[result = MapAt[encode, testData, {All, "Image"}];]
(* => {57.9009, Null} *)

Unfortunately, wrapping MapAt with Parallelize doesn't work:

LaunchKernels[];
DistributeDefinitions[encode];
AbsoluteTiming[result = Parallelize @ MapAt[encode, testData, {All, "Image"}];]

enter image description here

Is there a simple workaround I'm missing?

$\endgroup$
1
  • $\begingroup$ Not sure about MapAt. Is testData is available to the parallel kernels? Check ParallelEvaluate[testData[[1]]]. $\endgroup$
    – Theo Tiger
    Apr 4, 2018 at 22:55

1 Answer 1

8
+100
$\begingroup$

You can replace your MapAt with a ParallelMap. I haven't investigated, why your Parallelize doesn't work, but this version here runs only some seconds on 12 cores:

testData = ResourceData["MNIST", "TestDataset"] // Normal;

encode[i_Image] := BaseEncode@ExportByteArray[i, "PNG"];
encode[ass_Association] := Association[ass, "Image" -> encode[ass["Image"]]]

ParallelMap[encode, testData];

The trick, if you like to call it so, is to provide a version of encode that directly converts an Association into the form you want by replacing the image with its encoded form.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.