Creating a custom web API ServiceConnection

Question

Can I build a custom API service connection that links into the standard framework?

I know there are paclets that seem to implement this. How are they structured and how can I write my own?

Answer 1

Template Notebook

I sunk some time into making a good template notebook for integrating with all this. It lives here:

https://github.com/b3m2a1/mathematica-BTools/blob/master/Resources/Templates/ServiceConnectionTemplate.nb

Original:

If you just want the code I used to generate the paclet it lives here:

PacletInstall["CustomServiceConnection",
 "Site"->"http://www.wolframcloud.com/objects/b3m2a1.paclets/PacletServer"
 ]

If you just want the StackExchange connection (it now also supports authorization and the basic answers, users, questions, and comments interface) you can get it here:

PacletInstall["ServiceConnection_StackExchange",
 "Site"->"http://www.wolframcloud.com/objects/b3m2a1.paclets/PacletServer"
 ]

Note that this paclet isn't a great example of what to do as it's clumsy, being generated from code, not typed by hand.

---

Explanation

I'll provide an example of building a connection for the StackExchange API.

Paclet Setup

First we provide a name for our paclet. This looks like ServiceConnection_<service name>. For us this is ServiceConnection_StackExchange.

The paclet has three core implementation files, as can be seen in, for example, the ServiceConnection_OpenLibrary paclet.

These are:

load.m
<service_name>.m
<service_name>Load.m
<service_name>Functions.m

The last is not, strictly speaking, necessary, but it's a convenience to have.

The best thing to do is identify the implemented service whose API authentication behavior is most similar to yours and copy then modify that.

For us this will be the OpenLibrary connection as I will use the unauthenticated parts of the StackExchange API.

If you have an auth key that should be copied, an example of that is the ChemSpider connection.

If you have an OAuth2 setup that implements the redirect_uri interface, the Facebook connection might be the way to go (I'm not certain one can use the WolframConnector system that Facebook uses, but maybe. If not it is possible to set something up using the ChannelFramework.)

load.m

This file is very simple and merely provides the interface to loading your service.

< service_name >Load.m

This just implements setting up the auth / fetch system used. For unauthenticated APIs this is the KeyClient system. For OAuth clients this is the OAuth system. Just copy the file from an appropriate existing connection and modify the names.

< service_name >Functions.m

These are just functions you can use when implementing your service. Put processing functions here.

< service name >.m

This has three distinct chunks, the service info, the raw import functions, and the post-processing functions.

Service Info

This simply provides the basic info the ServiceConnection system uses. In the most basic case one just has something that looks like this:

<service_name>data[]:={
 "ServiceName"->"<service_name>",
 "ClientInfo":>(authentication_function),
 "Gets" -> get_call_names,
 "Posts" -> post_call_names,
 "RawGets"-> get_raw_data_call_names,
 "RawPosts"-> get_raw_data_call_names
 } 

For the case of the (quick example version of the) StackExchange API we have:

stackexchangedata[] = {
  "ServiceName" -> "StackExchange",
  "URLFetchFun" :>
   Function[
    With[{f = URLSave[#1, CreateFile[], ##2]},
     ((DeleteFile[f]; #1) &)[Import[f, {"GZIP", "Text"}]]
     ]
    ],
  "ClientInfo" :> {},
  "Gets" -> {"Search"},
  "RawGets" -> {"RawSearch"}
  }

Note that "URLFetchFun" which just describes how to import the call. It uses the URLFetch structure so one will most likely need to use URLFetch or URLSave in the call somewhere.

This data can be much more complex when OAuth is concerned.

If one wants an icon they simply set it via:

<service_name>data["icon"]:=icon

I set the standard one for the StackExchange API

Raw Import Functions

In general a ServiceObject method is defined as a "raw" call that imports the result string and a processing function that takes the raw data and processes it.

The raw import functions are the "RawGets" and "RawPosts" from the info.

Each call should be defined like:

<service_name>data[call_name]:=
 {
  "URL"->url_base,
  "HTTPSMethod"->get_or_post,
  "PathParameters"->path_parameter_names,
  "Parameters"->query_parameter_names,
  "RequiredParameters"->required_path_or_query_parameter_names,
  }

Note the "URL" can be a function in which case it is passed the values of the "PathParameters" and should return the proper URL.

The "HTTPSMethod" is merely the "Method" in HTTPRequest.

I only provided one method for the StackExchange connection, which is search. Here is the raw version:

stackexchangedata["RawSearch"] := {
  "URL" -> "https://api.stackexchange.com/2.2/search/advanced",
  "HTTPSMethod" -> "GET",
  "Parameters" -> {
    "q", "accepted", "answers", "body", "closed", "migrated", 
    "nottagged", "title", "user", "url", "views", "wiki", "sort", 
    "order", "page", "pagesize", "fromdate", "todate", "min", "max", 
    "site"
    },
  "RequiredParameters" -> {"site"}
  }

Methods

Then the calls proper are defined as:

<service_name>cookeddata[call_name,id_,args___]:=
 Block[{params = Association[args], raw},
  raw =
   processFunction[
    <client_type>`rawkeydata[id, base_call, Normal@params]];
  Dataset[raw]
  ]

Where the <client_type> is either the KeyClient or the OAuth client and the base_call is the "raw" call that the function depends on. This is how we will define our "Search" call:

stackexchangecookeddata["Search",id_,args___]:=
 Block[{params = Association[args], raw},
  raw =
   With[{a = ImportString[KeyClient`rawkeydata[id, base_call, 
    Normal@params], "RawJSON"]}, 
  If[KeyMemberQ[a, "quota_max"], $stackexchangecallmax = 
   a["quota_max"]]; 
  If[KeyMemberQ[a, "quota_calls"], $stackexchangecalls = 
   a["quota_calls"]]; If[KeyMemberQ[a, "items"], a["items"], a]
  ];
  Dataset[raw]
  ]

sendmessage

I don't know what this function does (haven't seen it have a meaningful implementation outside of the Twilio API)

For now stick with:

<service_name>sendmessage[___]:=$Failed

Returned Functions

The last thing in our <service_name>.m file should then be these three functions:

{<service_name>data, <service_name>cookeddata, <service_name>sendmessage}

These implement the service connection interface.

---

Testing

With that in place we pack and save our paclet and we can then test how it works:

In[1]:= $se = ServiceConnect@"StackExchange"

Out[1]= ServiceObject["StackExchange", 
 "ID" -> "connection-be87e3a01215e882e70342ae55ea8522"]

And we can see our icon got included:

And our basic search works:

In[9]:= $se["Search", {
    "site" -> "mathematica.stackexchange.com",
    "q" -> "curated data paclets"
    }
   ] // Normal // First

Out[9]= <|"tags" -> {"curated-data", "paclets"}, 
 "owner" -> <|"reputation" -> 7252, "user_id" -> 38205, 
   "user_type" -> "registered", "accept_rate" -> 41, 
   "profile_image" -> 
    "https://www.gravatar.com/avatar/12a17db78babaa782233d5c4128c0025?\
s=128&d=identicon&r=PG&f=1", "display_name" -> "MB1965", 
   "link" -> 
    "https://mathematica.stackexchange.com/users/38205/mb1965"|>, 
 "is_answered" -> True, "view_count" -> 18, "answer_count" -> 1, 
 "score" -> 3, "last_activity_date" -> 1495744664, 
 "creation_date" -> 1495744664, "question_id" -> 146894, 
 "link" -> 
  "https://mathematica.stackexchange.com/questions/146894/custom-\
curated-data-function-for-multiple-datasets", 
 "title" -> "Custom curated data function for multiple datasets"|>

Hopefully this is enough to get ones own paclets built.