4
$\begingroup$

I have a few thousand image urls that I want to download asynchronously, how can I do that while monitoring progress? I'm really asking for 5 things:

  • Show dynamic progress in bar
  • Downloads must be asynchronous
  • Avoid any filename collisions
  • Save with correct file extensions (even if not present in url)
  • Show list of failed download tasks and why

Here's an example to get started:

Monitor[
    URLDownload[
         WebImageSearch["dog", "ImageHyperlinks", MaxItems -> 10], 
         "~/Downloads/"
    ]
] 

Updated response to comment

I don't believe URLDownloadSubmit takes a directory, this is the behavior I see (no progress indication):

dogs = WebImageSearch["dog", "ImageHyperlinks", MaxItems -> 10]
URLDownloadSubmit[dogs, "~/Downloads", 
 HandlerFunctions -> <|"TaskProgress" -> Print, 
   "TaskComplete" -> Print|>, 
 HandlerFunctionsKeys -> {"FractionComplete", "ByteCountDownloaded"}]

enter image description here

And the filenames are wrong:

enter image description here

Related but not duplicate:

$\endgroup$
4
  • 2
    $\begingroup$ You can use HandlerFunctions like so: URLDownloadSubmit[dogs, "~/Desktop/test", HandlerFunctions -> <|"TaskProgress" -> Print, "TaskComplete" -> Print|>, HandlerFunctionsKeys -> {"FractionComplete", "ByteCountDownloaded"}] to print the FractionComplete and ByteCountDownloaded. It's probably a documentation bug that there's no good example of how to do it, but it is mentioned in the usage of URLDownload and URLDownloadSubmit. You can use ProgressIndicator[Dynamic[frac]] and frac = #FractionDownloaded instead of Print. $\endgroup$
    – Carl Lange
    Aug 12, 2020 at 21:26
  • $\begingroup$ @CarlLange thanks, see my update $\endgroup$
    – M.R.
    Aug 12, 2020 at 21:38
  • 1
    $\begingroup$ I think the message is because "~/Desktop/test" is a file and you are trying to download multiple files to it. Try just "~/Desktop". $\endgroup$
    – Carl Lange
    Aug 12, 2020 at 22:11
  • $\begingroup$ @CarlLange Sorry for the typo, I fixed that but it doesn't look like it's doing anything still. If you could your snippet with a progress bar I'll accept! $\endgroup$
    – M.R.
    Aug 12, 2020 at 23:06

2 Answers 2

4
+75
$\begingroup$

What I would do is run URLDownloadSubmit once on each URL that you want to download. That will make all those files start to download asynchronously in the background as a result.

When downloading that many images, it's not worth it to keep track of how many bytes have been downloaded of each image. It's better to just count the number of files that have been downloaded so far. That should be granular enough.

Start by getting the URLs:

urls = WebImageSearch["dog", "ImageHyperlinks", MaxItems -> 10];

Create a progress indicator:

i = 0;
ProgressIndicator[Dynamic[i], {0, Length[urls]}]

Let outputDir be your output directory. This will start downloading the files:

submitURL[url_] := URLDownloadSubmit[
  url,
  FileNameJoin[{outputDir, Last@Information[url, "Path"]}],
  HandlerFunctions -> <|
    "TaskFinished" -> (i++ &)
    |>]

submitURL /@ urls;

I used FileNameJoin[{outputDir, Last@Information[url, "Path"]}] to keep the original names as requested. However, note that the names may collide with each other. The advantage with UUIDs, as in your screenshot, is that they are always unique.

$\endgroup$
13
  • $\begingroup$ When there are missing urls, the progress indicator will never be 100%. How can you add an error handler to count and show the few bad ones? $\endgroup$
    – M.R.
    Aug 24, 2020 at 22:25
  • $\begingroup$ If you can add error handler I will accept $\endgroup$
    – M.R.
    Aug 25, 2020 at 1:29
  • $\begingroup$ Actually, that collision fix isn't perfect as FileExtension returns "" on some urls, is there a better way to inject the uuid into the filename? $\endgroup$
    – M.R.
    Aug 25, 2020 at 5:55
  • $\begingroup$ It’s nice that this is abortable but how to handle failures in a nice way? $\endgroup$
    – M.R.
    Aug 25, 2020 at 7:23
  • $\begingroup$ @M.R. It depends. Perhaps it would work to just make a list of all the files in that have been downloaded and compare it with the URLs that you tried to download. That way you can make a list of all the files that you failed to download. It would be nice to have some examples of filenames where FileExtension fails, perhaps these are especially tricky cases that need special treatmeant not matter how it is done. $\endgroup$
    – C. E.
    Aug 25, 2020 at 8:03
3
$\begingroup$

Does something like this work for you?

With[
    {
    links = WebImageSearch["dog", "ImageHyperlinks", MaxItems->10],
    dir = CreateDirectory[]
    },

    Monitor[
        Table[URLDownload[links[[i]], dir], {i, Length[links]}],
        ProgressIndicator[i / Length[links]]
    ]
]

Your original question wasn't clear to me that the downloads were to be asynchronous. Perhaps you can use LocalSubmit for this purpose, e.g.:

With[
    {
    links=WebImageSearch["dog","ImageHyperlinks",MaxItems->10],
    dir=CreateDirectory[]
    },

    p=0;
    LocalSubmit[
        Table[
            URLDownload[links[[i]],dir];
            Print[i],
            {i, Length[links]}
        ],
        HandlerFunctions-><|"PrintOutputGenerated" -> Function[p = First @ #PrintOutput]|>,
        HandlerFunctionsKeys->{"PrintOutput"}
    ];
    Dynamic @ ProgressIndicator[p/Length[links]]
]

Response:

Thanks for updating, here are 100 image urls to test on:

urls = CloudGet["https://www.wolframcloud.com/obj/f691b40d-b12a-41b5-9017-9130bc797fd0"]; 

It seems like it is still downloading synchronously because it takes ~40 seconds, but @C.E.'s only takes ~10 seconds (and speed is the most important factor to me). On the plus side, it seems all 100 images download correctly and the filenames avoid collisions, but the With block won't Abort.

$\endgroup$
7
  • $\begingroup$ Well not really, because it's not async, and so will be waaay to slow for 10k images. $\endgroup$
    – M.R.
    Aug 17, 2020 at 22:06
  • $\begingroup$ Do you see what I mean? $\endgroup$
    – M.R.
    Aug 19, 2020 at 3:36
  • $\begingroup$ You solution is too slow because it is sychronous, so do see a way around this? $\endgroup$
    – M.R.
    Aug 22, 2020 at 2:49
  • $\begingroup$ @C.E. Your comment doesn't make sense. LocalSubmit uses one kernel for its code, not one kernel per file. $\endgroup$
    – Carl Woll
    Aug 22, 2020 at 15:43
  • 1
    $\begingroup$ @CarlWoll No, that was what I first thought that LocalSubmit did. That would have given terrible performance. I assure you that URLDownloadSubmit does not use multiple kernels. $\endgroup$
    – C. E.
    Aug 25, 2020 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.