Using URLFetch/URLExecute to post to an internal server

Question

I've been looking up how to do this for a while and I haven't seen any thorough explanations, just hints. Is there like a secret repository of clearly defined Mathematica code somewhere that I cant find? Anyways, I have an element in JSon and want to add it to a internal url specific to that element and for which I have write permissions. For example: https://www.internalSite.com/master/siteID/elementID

What I've been seeing consistently is something like

URLFetch["url", "Method"->"Post", "RawJSON"]

Which is strange because the reference guide says URLFetch only takes in two arguments: the url and "elements." It does mention methods (but not how to use them), but only the "Get" method.

So my question is basically, URLFetch was replaced w/ URLExecute, so how can I use it to "Put" Json on the website? What arguments do I need to specify, how does defining a method work, etc?

i.e. When I go to the above url, I want to see the JSon I put there.

EDIT: I just remembered I'll actually have to create the url as well as add the element to it. We have a server "internalSite.com/master/siteID" for example that contains links to all the JSon elements. I need to create another linked url "https://www.internalSite.com/master/siteID/elementID" for example and add the raw JSon to said url.

Thank you

Answer 1

Lets begin by installing an "internal server". CouchDB is a database that uses the HTTP protocol (no DatabaseLink needed here). It works using REST like many other web services out there. If you are on a Mac or PC it is just a matter of running the installer with default options. After this we can confirm that the server is working:

In[1]:= Import["http://127.0.0.1:5984"]
Out[1]= {"couchdb":"Welcome","version":"2.0.0","vendor":{"name":"The Apache Software Foundation"}}

This server always replies in JSON format so we can fetch and parse in one step:

In[2]:= Import["http://127.0.0.1:5984","RawJSON"]
Out[2]= <|couchdb->Welcome,version->2.0.0,vendor-><|name->The Apache Software Foundation|>|>

I use Import[] for fetching because it is the easiest way to generate and parse simple GET requests. But you can also use other commands like URLRead or URLFetch.

Let's create a database. In CouchDB tutorial, they do it like this:

curl -X PUT http://127.0.0.1:5984/baseball

from Mathematica, one way of doing this is:

In[3]:= URLExecute[HTTPRequest["http://127.0.0.1:5984/mydatabase", <|"Method" -> "PUT"|>]]
Out[3]= {"ok" -> True}

By default, CouchDB allows all operations (admin party) but only from the same computer where it is installed. If you add a user/password then you need to provide your credentials or a cookie.

Now lets get a list of the databases in order to verify that "mydatabase" is there:

In[4]:= Import["http://127.0.0.1:5984/_all_dbs","RawJSON"]
Out[4]= {_global_changes, _metadata, _replicator, _users, mydatabase}

It is there. Now let's put in there do some documents. We can do this with PUT requests or with POST requests. With PUT, we have to specify the document ID in the URL:

In[5]:= URLExecute[HTTPRequest[
     "http://127.0.0.1:5984/mydatabase/1",
     <|"Method"->"PUT",
       "ContentType" -> "application/json", 
       "Body"-> ExportString[<|"random"->RandomInteger[10,10]|>,"RawJSON"]|>]]

Out[5]= {"rev" -> "1-683f50d046d9d04090215fac932eb395", "id" -> "1", "ok" -> True}

Now lets read this back:

In[6]:= Import["http://127.0.0.1:5984/mydatabase/1","RawJSON"]["random"]
Out[6]= {0, 7, 7, 10, 2, 10, 8, 8, 9, 8}

Lets now create a document with a POST request:

In[7]:= newWithPOST =  URLExecute[ HTTPRequest[
    "http://127.0.0.1:5984/mydatabase/",
    <|"Method" -> "POST", 
      "ContentType" -> "application/json", 
      "Body" -> ExportString[<|"random" -> RandomInteger[10, 10]|>,"RawJSON"]|>],"RawJSON"]

Out[7]= {"rev" -> "1-7c2799fa64b5dd6d9d66be7af3f8de15",  "id" -> "0467a21cd602ed6827dc8fad3e0014a2", "ok" -> True}

And now we read it back:

In[8]:= Import[URLBuild[{"http://127.0.0.1:5984/mydatabase/",newWithPOST["id"]}],"RawJSON"]["random"]
Out[8]= {3,8,4,5,0,8,5,10,5,1}

We can also use other commands like URLFetch but it doesn't seem natural to "insert" using a "fetch".

Answer 2

I didn't quite catch the last part about building the URL, but maybe you can use URLBuild[]. About sending requests, however, I think I know the answer:

I had to deal with something similar recently, where I wanted to simulate the requests made by an external API.

To test your requests, you can use Hookbin.com. I've created an endpoint which I can access using:

https://hookb.in/v953z87G

In Mathematica, I've learned to use HTTPRequest[] like this:

url="https://hookb.in/v953z87G";
body=ExportString[Import["/path/to/file.json","RawJSON"],"RawJSON"];

req = HTTPRequest[
    url,
    <|"Method" -> "POST","Body" -> body,
        "ContentType" -> "application/json"|>];

URLExecute[req]

which if done correctly, will return:

{"success" -> True}

by changing the last line, you can control the structure of the results:

URLExecute[req, {}, "RawJSON"]

which will return:

<|"success" -> True|>

Now, refreshing the hookbin page, shows you the result. You should see the JSON content in "Body" section.