10
$\begingroup$

The URL related functions have confused me yet again (I have v11.1)!

What is the Mathematica equivalent for the following generic curl command?

curl -u user:pass http://someurl.file --output filename

I tried what @Kuba suggested, but its options are colored like errors and it didn't work:

enter image description here

Separate Issues:

  • URLDownload doesn't respect the Interactive option either:

enter image description here

Follow-ups:

  • How can I do this for a long list of large images? e.g. is URLDownload multithreaded?
  • How can I do these url downloads in a pauseable or resumable fashion?
$\endgroup$
8
  • 2
    $\begingroup$ What is your Mathematica version? $\endgroup$
    – Szabolcs
    Apr 21, 2017 at 7:06
  • $\begingroup$ Take a look at documentation of URLSave or URLDownload for newer versions. $\endgroup$
    – Kuba
    Apr 21, 2017 at 7:17
  • $\begingroup$ I have V11.1 @Szabolcs, I didn't see any mention about user/passwords in those docs $\endgroup$
    – user5601
    Apr 21, 2017 at 7:40
  • 4
    $\begingroup$ @user5601 it says URLDownload takes same options as URLRead so I guess URLDownload["http://someurl.file", "filename", "Username" -> "user", "Password" -> "pass"], does it work for you? $\endgroup$
    – Kuba
    Apr 21, 2017 at 7:41
  • $\begingroup$ About Interactive issue: 143423. p.s. have you tried with URLSave? The first message doesn't seem to be related to those options btw. $\endgroup$
    – Kuba
    Apr 21, 2017 at 16:35

1 Answer 1

11
$\begingroup$

Assuming you're using Basic Authentication (vs. say, NTML Auth), the following code should work:

basicAuth[u_, p_] := "Basic " <> ExportString[u <> ":" <> p, "Base64"]
Import@URLRead@
  HTTPRequest[<|"Method" -> "GET", "Scheme" -> "https", 
    "Domain" -> "server.foo.com", "Port" -> 443,
    "Path" -> "/path/to.jpg",
    "Headers" -> {"Authorization" -> basicAuth[username, password]}|>]

You'll want to change "Scheme" and "Port" for the HTTP instead of HTTPS.

Once you confirm that works, then you can switch to URLDownload:

URLDownload@
      HTTPRequest[<|"Method" -> "GET", "Scheme" -> "https", 
        "Domain" -> "server.foo.com", "Port" -> 443,
        "Path" -> "/path/to.jpg",
        "Headers" -> {"Authorization" -> basicAuth[username, password]}|>]

I haven't tried it, but you should be able to create a bunch of HTTPRequest[] objects and pass them to URLDownload[{req1,req2,...}].

Finally, I was surprised to find that ExportString["...","Base64"] adds a "\n" at the end of the resulting string. I thought I'd need to strip it out but I haven't seen it make a difference. YMMV.

Hope this helps!

$\endgroup$
2
  • $\begingroup$ Can you update this for v12.3? $\endgroup$
    – user5601
    Sep 29, 2021 at 22:58
  • $\begingroup$ @user5601 Just tried it in 12.3.1 and it worked fine. Was able to pull down the html from a server with basic auth enabled. $\endgroup$
    – Paul Clip
    Sep 30, 2021 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.