What is the equivalent of curl?

Question

The URL related functions have confused me yet again (I have v11.1)!

What is the Mathematica equivalent for the following generic curl command?

curl -u user:pass http://someurl.file --output filename

I tried what @Kuba suggested, but its options are colored like errors and it didn't work:

Separate Issues:

• URLDownload doesn't respect the Interactive option either:

Follow-ups:

• How can I do this for a long list of large images? e.g. is URLDownload multithreaded?
• How can I do these url downloads in a pauseable or resumable fashion?

Answer 1

Assuming you're using Basic Authentication (vs. say, NTML Auth), the following code should work:

basicAuth[u_, p_] := "Basic " <> ExportString[u <> ":" <> p, "Base64"]
Import@URLRead@
  HTTPRequest[<|"Method" -> "GET", "Scheme" -> "https", 
    "Domain" -> "server.foo.com", "Port" -> 443,
    "Path" -> "/path/to.jpg",
    "Headers" -> {"Authorization" -> basicAuth[username, password]}|>]

You'll want to change "Scheme" and "Port" for the HTTP instead of HTTPS.

Once you confirm that works, then you can switch to URLDownload:

URLDownload@
      HTTPRequest[<|"Method" -> "GET", "Scheme" -> "https", 
        "Domain" -> "server.foo.com", "Port" -> 443,
        "Path" -> "/path/to.jpg",
        "Headers" -> {"Authorization" -> basicAuth[username, password]}|>]

I haven't tried it, but you should be able to create a bunch of HTTPRequest[] objects and pass them to URLDownload[{req1,req2,...}].

Finally, I was surprised to find that ExportString["...","Base64"] adds a "\n" at the end of the resulting string. I thought I'd need to strip it out but I haven't seen it make a difference. YMMV.

Hope this helps!