I'm have a simple piece of code:

Do[If[!FailedQ[g = grab[i]], AppendTo[results, g]], {i, 2000000, 3000000}]

I had thought that the simplest way to parallelize this would be


And then to rerun the former with ParallelDo[] instead of Do[].

But this doesn't work. What is the correct and simplest way to do parallelize a trivial accumulation loop like this?

Here is the function for testing:

FailedQ[expr_] := FailedQ[expr, 0]
FailedQ[expr_, d : _Integer | \[Infinity]] := !FreeQ[expr, $FailedSymbols, {0, d}]
grab[n_] := Quiet @ Module[{u,r,i},
     u = "http://photo.net/photodb/photo?photo_id="<>ToString[n];
       r = First@StringCases[Import[u,"HTML"],"ratings, "~~Shortest[s__]~~" average":>ToExpression[StringDrop[s,-2]]];
       i = Import["http://gallery.photo.net/photo/" <> ToString[n] <> "-lg.jpg", "Image"], Return @ $Failed];
  • $\begingroup$ Note: I didn't post the code for grab because I hope that the solution will be agnostic to the internals of the custom function in the loop. $\endgroup$
    – M.R.
    Jul 23, 2015 at 22:29
  • $\begingroup$ Doesn't work why? Of course, if grab is stateful or has other side-effects, it might be far from trivial. $\endgroup$ Jul 23, 2015 at 22:38
  • $\begingroup$ What is FailedQ? Note that Reap and Sow are far more efficient at list accumulation problems than AppendTo, since AppendTo's performance degrades as the list gets larger (I think). $\endgroup$ Jul 24, 2015 at 1:26
  • $\begingroup$ @OleksandrR. I added the function so you can try it $\endgroup$
    – M.R.
    Jul 24, 2015 at 1:53
  • 1
    $\begingroup$ @M.R. You won't need to use several cores because that's not what's taking up time. You're not doing any heavy processing. The reason your code without any parallelizing is slow is because you fetch images synchronously; you start downloading one image, and when that's done you start on the next. All your time is spent waiting for the files to transfer, not waiting for your kernels to do computations. Now what if you could download ten images simultaneously on one kernel? That's what asynchronous fetching does. $\endgroup$
    – C. E.
    Jul 24, 2015 at 2:17

3 Answers 3


This may not be the fastest way, but it's a way:

ParallelSow[expr_] := Sow[expr]
Reap[ParallelDo[If[countedQ[i], ParallelSow[f[i]]], {i, 1, 10^7}]]

where countedQ[i] is some Boolean function that determines whether f[i] gets added to the accumulated list. Feel free to make improvements.

Note that AppendTo scales as $O(n^2)$ where $n$ is accumulated list size, as is documented in the "Possible Issues" section of the documentation page. For small lists, AppendTo is more convenient, but for larger lists Reap and Sow are asymptotically better (I think they're $O(n)$).

  • 1
    $\begingroup$ It isn't going to work; you have to Sow in the master kernel, like this. $\endgroup$
    – C. E.
    Jul 24, 2015 at 1:46
  • $\begingroup$ And the point of using Append is to save the data when you have to abort the loop. $\endgroup$
    – M.R.
    Jul 24, 2015 at 1:59
  • $\begingroup$ @DumpsteDoofus Didn't they fix that bug in 10? $\endgroup$
    – M.R.
    Jul 24, 2015 at 2:00
  • 1
    $\begingroup$ @Pickett: Oops, should have viewed the output LOL. Thanks, let me fix that. $\endgroup$ Jul 24, 2015 at 2:55
  • $\begingroup$ Your question doesn't say anything about aborting. $\endgroup$
    – george2079
    Jul 28, 2015 at 14:12

How about

     ParallelTable[If[!FailedQ[g = grab[i]], g],
          {i, 2000000, 3000000}],Except[Null]]

At some point you'll run into a memory issue with all the Nulls, but I think with only 10^6 you are ok.


Here's a simple approach:

ParallelTable[With[{g = grab[i]}, If[FreeQ[g, $Failed], g, Nothing]], {i, 2000000, 3000000}]

Note: Nothing requires version 10.2


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.