3
$\begingroup$

I'm trying to download a few thousand images (using an authenticated get request). This explanation of parallel downloading was a great answer to the problem of handling massive parallel asynchronous downloads. But that solution no longer works for me, and now in v11.2 there are completely new URL-related functions...

The function URLDownloadSubmit is supposed to replace URLFetchAsynchronous, but I'm finding that it's not asynchronous and very slow. Here's what my code looks like now using URLDownloadSubmit, and it's clearly downloading things sequentially:

enter image description here

Has anyone come across this in v11.2 and know how to address it?

Additional Details:

  • Even if I DistributeDefinitions and then ParallelMap my submitting function, nothing happens:

    enter image description here)

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ This is gonna be some throttle at the asynchronous task level. If you look at the DownValues I think URLFetchAsynchronous has an implementation at the CURL level, hence the difference. It looks buggy to me. Ask support. $\endgroup$
    – b3m2a1
    Commented Oct 16, 2017 at 20:51
  • $\begingroup$ @b3m2a1 seems like they keep making new versions of url related functions, but they keep getting worse! $\endgroup$
    – M.R.
    Commented Oct 17, 2017 at 3:17
  • $\begingroup$ Just from the DownValues it looks like they're trying to move away from using an external library for all of this stuff, which makes sense, but the transition is gonna be a bit bumpy. Until PacletFind["CURLLink"] is empty (which probably won't be until V15 or something), the old stuff will work. $\endgroup$
    – b3m2a1
    Commented Oct 17, 2017 at 3:18

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.