8
$\begingroup$

The URL related functions have confused me yet again (I have v11.1)!

What is the Mathematica equivalent for the following generic curl command?

curl -u user:pass http://someurl.file --output filename

I tried what @Kuba suggested, but its options are colored like errors and it didn't work:

enter image description here

Separate Issues:

  • URLDownload doesn't respect the Interactive option either:

enter image description here

Follow-ups:

  • How can I do this for a long list of large images? e.g. is URLDownload multithreaded?
  • How can I do these url downloads in a pauseable or resumable fashion?
$\endgroup$
  • 2
    $\begingroup$ What is your Mathematica version? $\endgroup$ – Szabolcs Apr 21 '17 at 7:06
  • $\begingroup$ Take a look at documentation of URLSave or URLDownload for newer versions. $\endgroup$ – Kuba Apr 21 '17 at 7:17
  • $\begingroup$ I have V11.1 @Szabolcs, I didn't see any mention about user/passwords in those docs $\endgroup$ – user5601 Apr 21 '17 at 7:40
  • 4
    $\begingroup$ @user5601 it says URLDownload takes same options as URLRead so I guess URLDownload["http://someurl.file", "filename", "Username" -> "user", "Password" -> "pass"], does it work for you? $\endgroup$ – Kuba Apr 21 '17 at 7:41
  • $\begingroup$ About Interactive issue: 143423. p.s. have you tried with URLSave? The first message doesn't seem to be related to those options btw. $\endgroup$ – Kuba Apr 21 '17 at 16:35
10
$\begingroup$

Assuming you're using Basic Authentication (vs. say, NTML Auth), the following code should work:

basicAuth[u_, p_] := "Basic " <> ExportString[u <> ":" <> p, "Base64"]
Import@URLRead@
  HTTPRequest[<|"Method" -> "GET", "Scheme" -> "https", 
    "Domain" -> "server.foo.com", "Port" -> 443,
    "Path" -> "/path/to.jpg",
    "Headers" -> {"Authorization" -> basicAuth[username, password]}|>]

You'll want to change "Scheme" and "Port" for the HTTP instead of HTTPS.

Once you confirm that works, then you can switch to URLDownload:

URLDownload@
      HTTPRequest[<|"Method" -> "GET", "Scheme" -> "https", 
        "Domain" -> "server.foo.com", "Port" -> 443,
        "Path" -> "/path/to.jpg",
        "Headers" -> {"Authorization" -> basicAuth[username, password]}|>]

I haven't tried it, but you should be able to create a bunch of HTTPRequest[] objects and pass them to URLDownload[{req1,req2,...}].

Finally, I was surprised to find that ExportString["...","Base64"] adds a "\n" at the end of the resulting string. I thought I'd need to strip it out but I haven't seen it make a difference. YMMV.

Hope this helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.