Curl analog with URLRead/Execute?

Question

I have a working curl command:

curl https://myapp.herokuapp.com/doit -F filedata=@path/to/img.jpg

How do I write this in the modern way with URLRead or URLExecute?

I've seen answers on how to do this with URLFetch, but it is deprecated.

Answer 1

To keep this example self-contained, we will use the HTTP introspection service at https://httpbin.org. This is a very handy site for verifying and debugging HTTP transactions.

First we will create a small JPG file:

Export["zot.jpg", Image[{{1}}]]

Then we will create a multi-part HTTP request that will post that file:

$req = HTTPRequest[
  "https://httpbin.org/anything"
  , <| "Method" -> "POST"
     , "Body" -> {"filedata" -> File["d:\\documents\\zot.jpg"]}
    |>
  ];

We can submit the request using URLRead and then inspect the body of the response. The introspection service returns a JSON description of the original request:

$resp = URLRead[$req];

$resp["Body"]

(*
{
  "args": {}, 
  "data": "", 
  "files": {
    "filedata": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEASABIAAD/4QCCRXhpZgAASUkqAAgAAAADADEBAgA0AAAAMgAAADIBAgAUAAAAZgAAACqIBAABAAAA+v///wAAAABDcmVhdGVkIHdpdGggdGhlIFdvbGZyYW0gTGFuZ3VhZ2UgOiB3d3cud29sZnJhbS5jb20AMjAyMDowNzowMyAxMDo0NDowOQD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/wAALCAABAAEBAREA/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/9oACAEBAAA/APf6/9k="
  }, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "deflate, gzip", 
    "Content-Length": "653", 
    "Content-Type": "multipart/form-data; boundary=------------------------f14f77e69b501379", 
    "Host": "httpbin.org", 
    "User-Agent": "Wolfram HTTPClient 12.1", 
    "X-Amzn-Trace-Id": "Root=1-5eff61aa-a60799b6493fef5b401d85c4"
  }, 
  "json": null, 
  "origin": "52.0.14.116", 
  "url": "https://httpbin.org/anything"
}
*)

Alternatively, we could use URLExecute which automatically imports the JSON response body as nested lists of rules:

URLExecute[$req]

(*
{ "args" -> {}
, "data" -> ""
, "files" -> {"filedata" -> "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEASABIAAD/4QCCRXhpZgAASUkqAAgAAAADADEBAgA0AAAAMgAAADIBAgAUAAAAZgAAACqIBAABAAAA+v///wAAAABDcmVhdGVkIHdpdGggdGhlIFdvbGZyYW0gTGFuZ3VhZ2UgOiB3d3cud29sZnJhbS5jb20AMjAyMDowNzowMyAxMDo0NDowOQD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/wAALCAABAAEBAREA/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/9oACAEBAAA/APf6/9k="}
, "form" -> {}
, "headers" ->
  { "Accept" -> "*/*"
  , "Accept-Encoding" -> "deflate, gzip"
  , "Content-Length" -> "653"
  , "Content-Type" -> "multipart/form-data; boundary=------------------------61723d168bdcb569"
  , "Host" -> "httpbin.org"
  , "User-Agent" -> "Wolfram HTTPClient 12.1"
  , "X-Amzn-Trace-Id" -> "Root=1-5eff6281-e0a4c5e08b883fc0f0437148"
  }
, "json" -> Null
, "origin" -> "52.0.14.116"
, "url" -> "https://httpbin.org/anything"
}
*)

The result returned by the introspection service makes it easy for us to verify that these results are equivalent to using curl:

curl https://httpbin.org/anything -F filedata=@d:\documents\zot.jpg

{
  "args": {},
  "data": "",
  "files": {
    "filedata": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEASABIAAD/4QCCRXhpZgAASUkqAAgAAAADADEBAgA0AAAAMgAAADIBAgAUAAAAZgAAACqIBAABAAAA+v///wAAAABDcmVhdGVkIHdpdGggdGhlIFdvbGZyYW0gTGFuZ3VhZ2UgOiB3d3cud29sZnJhbS5jb20AMjAyMDowNzowMyAxMDo0NDowOQD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/wAALCAABAAEBAREA/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/9oACAEBAAA/APf6/9k="
  },
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "653",
    "Content-Type": "multipart/form-data; boundary=------------------------16397e5167d6e462",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.55.1",
    "X-Amzn-Trace-Id": "Root=1-5eff6094-0034631c571243182edd4b3c"
  },
  "json": null,
  "method": "POST",
  "origin": "52.0.14.116",
  "url": "https://httpbin.org/anything"
}

Answer 2

curl is different from the built-in Curl.

curl is a command-line tool and library for transferring data with URLs.

This aks for a Mathematica built-in or a call to the OS command line.

For a built-in URLDownload may be the corresponding command. But curl is used to exploration purposes too, so URLRead takes this part. For the evaluation of the response URLResponse is offered by Mathematica.

The most modern pair of built-ins in Mathematica is URLRead[HTTPRequest[\[Ellipsis]]].

curl is Open Source and evergreen as long as the ISOOSI Standard will be applied and URIs will be in use. So $ScriptCommandLine is the built-in of recommendation

ReadList["!bash -c 'curl --help'"]  (*linux,MacOSX*)

(* {Usage : curl[options ...] < url && url > -socket - unix + --abstract && -socket - unix + --abstract < path && path > abstract Connect domain socket Unix via, -svc + --alt < file name && file name > alt Enable - cache file svc this with, any authentication method Pick (--anyauth), $Failed} *)

On Windows:

RunProcess[{"cmd", "/c", "\"curl /?\""}]

There are some messages that can be ignored.

The output goes into the shell until a file is specified. This can be read by Import. Path and name management are separate. Error management in the case is necessary.

On Windows it might be generalized to

RunProcess[$SystemShell, All, "curl --help
 echo example line 2
 exit
 "]

To do some further processing in one step.

For simple error communication see the answer of @WReach.

Form one of the websites for curl, https://curl.haxx.se/:

DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, HTTP/3, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, NTLM, Negotiate and Kerberos), file transfer resume, proxy tunneling and more.

Comparison to URLRead[URLRequest[..]] to curl is filling more than a book.

There might be a website were the Mathematica solution is inappropriate. Think of Linus OSes that may be outdated or even optimized for curl.

An import trend is to require licenses and keys is another topic. So differences are to be expected with Cookies. Since curl is free, the addition to Mathematica is a snap. Other way round is more expensive.