# How to parallelize a Do loop containing URLFetchAsynchronous?

I'm have a simple piece of code:

results={};
Do[If[!FailedQ[g = grab[i]], AppendTo[results, g]], {i, 2000000, 3000000}]


I had thought that the simplest way to parallelize this would be

LaunchKernels[]
SetSharedVariable[results]


And then to rerun the former with ParallelDo[] instead of Do[].

But this doesn't work. What is the correct and simplest way to do parallelize a trivial accumulation loop like this?

Here is the function for testing:

FailedQ[expr_] := FailedQ[expr, 0]
FailedQ[expr_, d : _Integer | \[Infinity]] := !FreeQ[expr, $FailedSymbols, {0, d}] grab[n_] := Quiet @ Module[{u,r,i}, u = "http://photo.net/photodb/photo?photo_id="<>ToString[n]; Check[ r = First@StringCases[Import[u,"HTML"],"ratings, "~~Shortest[s__]~~" average":>ToExpression[StringDrop[s,-2]]]; i = Import["http://gallery.photo.net/photo/" <> ToString[n] <> "-lg.jpg", "Image"], Return @$Failed];
Return[{n,i,r}]]

• Note: I didn't post the code for grab because I hope that the solution will be agnostic to the internals of the custom function in the loop. – M.R. Jul 23 '15 at 22:29
• Doesn't work why? Of course, if grab is stateful or has other side-effects, it might be far from trivial. – Oleksandr R. Jul 23 '15 at 22:38
• What is FailedQ? Note that Reap and Sow are far more efficient at list accumulation problems than AppendTo, since AppendTo's performance degrades as the list gets larger (I think). – DumpsterDoofus Jul 24 '15 at 1:26
• @OleksandrR. I added the function so you can try it – M.R. Jul 24 '15 at 1:53
• @M.R. You won't need to use several cores because that's not what's taking up time. You're not doing any heavy processing. The reason your code without any parallelizing is slow is because you fetch images synchronously; you start downloading one image, and when that's done you start on the next. All your time is spent waiting for the files to transfer, not waiting for your kernels to do computations. Now what if you could download ten images simultaneously on one kernel? That's what asynchronous fetching does. – C. E. Jul 24 '15 at 2:17

This may not be the fastest way, but it's a way:

SetSharedFunction[ParallelSow]
ParallelSow[expr_] := Sow[expr]
Reap[ParallelDo[If[countedQ[i], ParallelSow[f[i]]], {i, 1, 10^7}]]


where countedQ[i] is some Boolean function that determines whether f[i] gets added to the accumulated list. Feel free to make improvements.

Note that AppendTo scales as $O(n^2)$ where $n$ is accumulated list size, as is documented in the "Possible Issues" section of the documentation page. For small lists, AppendTo is more convenient, but for larger lists Reap and Sow are asymptotically better (I think they're $O(n)$).

• It isn't going to work; you have to Sow in the master kernel, like this. – C. E. Jul 24 '15 at 1:46
• And the point of using Append is to save the data when you have to abort the loop. – M.R. Jul 24 '15 at 1:59
• @DumpsteDoofus Didn't they fix that bug in 10? – M.R. Jul 24 '15 at 2:00
• @Pickett: Oops, should have viewed the output LOL. Thanks, let me fix that. – DumpsterDoofus Jul 24 '15 at 2:55
• Your question doesn't say anything about aborting. – george2079 Jul 28 '15 at 14:12

 results=Cases[
ParallelTable[If[!FailedQ[g = grab[i]], g],
{i, 2000000, 3000000}],Except[Null]]


At some point you'll run into a memory issue with all the Nulls, but I think with only 10^6 you are ok.

Here's a simple approach:

ParallelTable[With[{g = grab[i]}, If[FreeQ[g, \$Failed], g, Nothing]], {i, 2000000, 3000000}]


Note: Nothing requires version 10.2