1
$\begingroup$

Is this the best way to batch save a list of image urls in parallel?

save[url_, path_:"~/Downloads/"] := Module[{imgName},
        imgName = FileBaseName[url]<>"."<>FileExtension[url];
        URLSaveAsynchronous[url, FileNameJoin[{path, imgName}], Beep];
    ]

urls = Cases[Import["http://www.ebay.com/sch/i.html?_from=R40&_trksid=p2050601.m570.l1313.TR11.TRC1.A0.H0.Xguitar.TRS0&_nkw=guitar&ghostText=&_sacat=0","ImageLinks"], x_ /; StringMatchQ[x,___~~".jpg"]]

Table[save[u], {u,urls}]

I would like to address the following issues:

  1. When I run this on only 6k urls, the kernel crashes immediately and I don't know why. Update: when I switched Identity to Beep in the above function this solved the immediate crash, I'd rather not have the Beep though...
  2. I need to monitor the progress of all downloads. I tried to print a cell that monitors Length[AsynchronousTasks[]] with a ProgressIndicator but since my function is inside a package, the $CellContext is causing the Dynamic box expressions to fail.
  3. Many times some of the Asynchronous tasks silently fail, which isn't optimal. I need a way to resume and/or retry the downloads that fail and see why.
  4. I've had more success with ParallelMap over using Table in the last line above, but what is the proper function to use here: Scan, ParallelTable, ParallelMap, etc...?

Any and all help would be appreciated!

$\endgroup$
  • $\begingroup$ Should I even be using URLSaveAsyncronous as opposed to Import if so/not why? $\endgroup$ – M.R. Mar 17 '15 at 0:14
5
$\begingroup$

If you're using URLSaveAsynchronous then you're automatically downloading in parallel and don't need to mess around with Parallel and other explicitly parallel constructs.

I wrote a bit of code to handle scraping images from the web some time ago. First we get our list of URLs:

urls = "http://www.explainxkcd.com" <> # & /@ 
  StringCases[
   Import["http://www.explainxkcd.com/wiki/index.php?title=1190:_Time", "Text"], 
   "/wiki/images/" ~~ HexadecimalCharacter ~~ "/" ~~ 
    HexadecimalCharacter .. ~~ "/time" ~~ DigitCharacter ... ~~ ".png"];

(As you can probably tell, we're going to be downloading this xkcd in it's entirety.)

First I'm going to define our state variables: remaining, which is a queue of URLs to download, and progressQueue, an Association which maps files to their download status.

remaining = urls;
progressQueue = Association[];

We can catch events passed to URLSaveAsynchronous using a function, which I'm calling urlFunction (imaginative, isn't it?). I slightly modify the example given in the documentation to record "progress" events into our Association:

urlFunction[task_, 
  "progress", {dlnow_, dltotal_, _, _}] := (progressQueue[task] = 
   dlnow/dltotal)

(If you're familiar with Javascript, then you'll probably recognize this as a callback function. It works pretty much exactly the same way it does in Javascript.)

When we receive a "data" event, we set the completion status of the object to 1 (100% completed), then call our download initiator function startDownload:

urlFunction[task_, "data", _] := (progressQueue[task] = 1;
  startDownload[])

If we get an "error", we Print out the URL to inform ourselves of the failure. (You could make the function retry upon failure, but that's extremely bad behavior for any networked program, especially so for a web scraper). Finally we call the function again with a "data" event as if success occurred, so that we continue to download.

urlFunction[task_, "error", 
  e_] := (Print["Error downloading ", First[task], ": ", e]; 
  urlFunction[task, "data", {}])

Finally, we define the dowload initiator function, startDowload. If there are no URLs remaining, we return. Otherwise, we peel one URL (u) off of remaining and download it, extracting the filename and setting the download path of the file. We pass our urlFunction so that events from the download are captured.

startDownload[] := (If[Length[remaining] == 0, Return[]]; 
  Module[{u = First[remaining]},
   remaining = Rest[remaining];
   URLSaveAsynchronous[u, 
    "C:\\Users\\Robert\\Pictures\\time\\" <> 
     Last@StringCases[u, "time" ~~ DigitCharacter ... ~~ ".png"], 
    urlFunction, "Progress" -> True]])

Finally, we call our function 16 times to start 16 parallel downloads. We also Total all of the progress values in progressQueue inside a Dynamic to, what else, dynamically check the total download progress.

Do[startDownload[], {16}];
Dynamic[ProgressIndicator[Total[Values[progressQueue]]/Length[urls]]]

This downloads all 3,099 images (28 MB) in about thirty seconds on my computer.

With Great Power Comes Great Responsibility

Think before you download. If you use this code irresponsibly then you're essentially DDoSing the target site. Services like Google and Twitter will likely throttle you if you make excessive requests; if this happens, reduce the number of parallel downloads.

$\endgroup$
  • $\begingroup$ Since you're using ~ and / in your path, I assume you're on a Linux system (yay!). You can also use a short bash one-liner to achieve parallel downloads: <list.txt xargs -n 1 -P 8 wget -q. $\endgroup$ – 2012rcampion Mar 17 '15 at 3:56
  • $\begingroup$ Would it be possible for you to update your wonderful answer to support v11.2 with URLDownloadSubmit (which is supposed to replace URLSaveAsynchronous)? @2012rcampion $\endgroup$ – M.R. Oct 17 '17 at 3:23
  • $\begingroup$ Ping - did you see my comment? If you can't that's ok but would be very helpful! $\endgroup$ – M.R. Mar 7 at 22:02
  • $\begingroup$ @M.R. I'm on 11.1 currently, sorry. $\endgroup$ – 2012rcampion Mar 7 at 22:05
  • $\begingroup$ @2012campion No worries then. I wonder if this technique could work to replace ParallelMap with async tasks? Definitely easier to monitor than a multi-kernel ParallelXXX, but could it be made as fast? $\endgroup$ – M.R. Mar 7 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.