Is there a better way to download images in parallel?

Question

Is this the best way to batch save a list of image urls in parallel?

save[url_, path_:"~/Downloads/"] := Module[{imgName},
        imgName = FileBaseName[url]<>"."<>FileExtension[url];
        URLSaveAsynchronous[url, FileNameJoin[{path, imgName}], Beep];
    ]

urls = Cases[Import["http://www.ebay.com/sch/i.html?_from=R40&_trksid=p2050601.m570.l1313.TR11.TRC1.A0.H0.Xguitar.TRS0&_nkw=guitar&ghostText=&_sacat=0","ImageLinks"], x_ /; StringMatchQ[x,___~~".jpg"]]

Table[save[u], {u,urls}]

I would like to address the following issues:

1. When I run this on only 6k urls, the kernel crashes immediately and I don't know why. Update: when I switched Identity to Beep in the above function this solved the immediate crash, I'd rather not have the Beep though...
2. I need to monitor the progress of all downloads. I tried to print a cell that monitors Length[AsynchronousTasks[]] with a ProgressIndicator but since my function is inside a package, the $CellContext is causing the Dynamic box expressions to fail.
3. Many times some of the Asynchronous tasks silently fail, which isn't optimal. I need a way to resume and/or retry the downloads that fail and see why.
4. I've had more success with ParallelMap over using Table in the last line above, but what is the proper function to use here: Scan, ParallelTable, ParallelMap, etc...?

Any and all help would be appreciated!

Answer 1

If you're using URLSaveAsynchronous then you're automatically downloading in parallel and don't need to mess around with Parallel and other explicitly parallel constructs.

I wrote a bit of code to handle scraping images from the web some time ago. First we get our list of URLs:

urls = "http://www.explainxkcd.com" <> # & /@ 
  StringCases[
   Import["http://www.explainxkcd.com/wiki/index.php?title=1190:_Time", "Text"], 
   "/wiki/images/" ~~ HexadecimalCharacter ~~ "/" ~~ 
    HexadecimalCharacter .. ~~ "/time" ~~ DigitCharacter ... ~~ ".png"];

(As you can probably tell, we're going to be downloading this xkcd in it's entirety.)

First I'm going to define our state variables: remaining, which is a queue of URLs to download, and progressQueue, an Association which maps files to their download status.

remaining = urls;
progressQueue = Association[];

We can catch events passed to URLSaveAsynchronous using a function, which I'm calling urlFunction (imaginative, isn't it?). I slightly modify the example given in the documentation to record "progress" events into our Association:

urlFunction[task_, 
  "progress", {dlnow_, dltotal_, _, _}] := (progressQueue[task] = 
   dlnow/dltotal)

(If you're familiar with Javascript, then you'll probably recognize this as a callback function. It works pretty much exactly the same way it does in Javascript.)

When we receive a "data" event, we set the completion status of the object to 1 (100% completed), then call our download initiator function startDownload:

urlFunction[task_, "data", _] := (progressQueue[task] = 1;
  startDownload[])

If we get an "error", we Print out the URL to inform ourselves of the failure. (You could make the function retry upon failure, but that's extremely bad behavior for any networked program, especially so for a web scraper). Finally we call the function again with a "data" event as if success occurred, so that we continue to download.

urlFunction[task_, "error", 
  e_] := (Print["Error downloading ", First[task], ": ", e]; 
  urlFunction[task, "data", {}])

Finally, we define the dowload initiator function, startDowload. If there are no URLs remaining, we return. Otherwise, we peel one URL (u) off of remaining and download it, extracting the filename and setting the download path of the file. We pass our urlFunction so that events from the download are captured.

startDownload[] := (If[Length[remaining] == 0, Return[]]; 
  Module[{u = First[remaining]},
   remaining = Rest[remaining];
   URLSaveAsynchronous[u, 
    "C:\\Users\\Robert\\Pictures\\time\\" <> 
     Last@StringCases[u, "time" ~~ DigitCharacter ... ~~ ".png"], 
    urlFunction, "Progress" -> True]])

Finally, we call our function 16 times to start 16 parallel downloads. We also Total all of the progress values in progressQueue inside a Dynamic to, what else, dynamically check the total download progress.

Do[startDownload[], {16}];
Dynamic[ProgressIndicator[Total[Values[progressQueue]]/Length[urls]]]

This downloads all 3,099 images (28 MB) in about thirty seconds on my computer.

With Great Power Comes Great Responsibility

Think before you download. If you use this code irresponsibly then you're essentially DDoSing the target site. Services like Google and Twitter will likely throttle you if you make excessive requests; if this happens, reduce the number of parallel downloads.