How can I customize my service connection functions?

Question

I've got a service connection function where I want to be able to pass the body arguments as a multipart JSON argument, unfortunately the best the standard service connect framework will let me do is pass a "MultipartData" argument. Can I hack this in some way?

Answer 1

So after digging through the service connection implementation ( Lookup[PacletInformation/@PacletFind["*OAuth*"],"Location"] will find it for you) we find that if we reassign ServiceConnections`Private`urlfetchFun[id] we can use whatever import function we want. Of course, we won't have the id until the ServiceObject is constructed, but as long as we restrict ourselves to working with our cookeddata functions (see this) we should be fine. In each function we'll insert:

ServiceConnections`Private`urlfetchFun[id]=
 $serviceconnectionurlfetchfun;

before the primary call, where $serviceconnectionurlfetchfun is the function we use to import.

That way we can go past a raw URLFetch.

The next thing we need to worry about is collecting our "MulitpartData" parameters from the "BodyData". Here let's make a function that transfers the parameters, call it $serviceconnectionreformatmultipartbodydata.

Right before we call <Client_Name>`raw<client_name>data in our cookeddata function we'll apply this to the base_call, id, and params and it will spit out new params. So our cookeddata function now looks something like:

<service_name>cookeddata[call_name,id_,args___]:=
 Block[{params = Association[args], raw},
  ServiceConnections`Private`urlfetchFun[id]=
   $serviceconnectionurlfetchfun;
  raw =
   processFunction[
    <Client_Name>`raw<client_name>data[
     id, 
     base_call, 
     Normal@
      $serviceconnectionreformatmultipartbodydata[
        base_call,
        id,
        params
        ]
     ]];
  Dataset[raw]
  ]

We call it like this so we can take advantage of ServiceConnections`Private`getQueryData[id,baseCall] from which we can extract which params are taken to be part of the "BodyData".

The issue, however, is that our MultipartData will be stripped away before the call to $serviceconnectionurlfetchfun occurs, but we can get this back by putting a patching parameter ($serviceconnectionpatchedparameters) for things we want to make it into the final call in $serviceconnectionurlfetchfun and the Block there.

So we have, at the end,

<service_name>cookeddata[call_name,id_,args___]:=
 Block[{
  params = Association[args], 
  $serviceconnectionpatchedparameters,
  raw
  },
  ServiceConnections`Private`urlfetchFun[id]=
   $serviceconnectionurlfetchfun;
  raw =
   processFunction[
    <Client_Name>`raw<client_name>data[
     id, 
     base_call, 
     Normal@
      $serviceconnectionreformatmultipartbodydata[
        base_call,
        id,
        params
        ]
     ]];
  Dataset[raw]
  ]

And then we have the $serviceconnectionreformatmultipartbodydata push its results to $serviceconnectionpatchedparameters, which is used by $serviceconnectionurlfetchfun to push stuff into the final call that would generally be stripped out.

Here's a brief example of this in action.

I implemented this in my google drive service connection to take care of uploads with metadata.

We start by connecting:

$gd = ServiceConnect["GoogleDrive"(*,"New"*)]

And then google drive uploads can take meta-data as well as content data, in which case we use multipart encoding where the metadata is passed as JSON and the content data comes next as raw data.

But I wanted to be able to pass the metadata not as a multipart element, but just as standard arguments to the call, e.g.:

$gd["UploadFile",
 "name" -> "test.txt",
 "BodyContent" -> "test string",
 "MIMEType" -> "text/plain",
 "uploadType" -> "multipart"
 ]

This isn't the simplest thing to do though, as we have to deal with path and query parameters too. But that's where .The "BodyContent" and "MIMEType" are fields that define the file content and then anything else in ServiceConnections`Private`getQueryData[id,baseCall][[5]] (the body data parameters) gets reformatted into metadata.

In the end for this call my patched in "MultipartData" looks like:

{
 {"metadata","application/json"}->
  ToCharacterCode@ExportString[{"name"->"test.txt"},"JSON"],
 {"contentdata",<"MIMEType">}->
  ToCharacterCode@<"BodyContent">
 }

Note that for things like File objects we'll want to pass ReadString[<"BodyContent">] instead of the raw content.

But with this in place and the "Parameters" stripped out it all flows fine:

In[23]:= $gd["UploadFile",
 "name" -> "test.txt",
 "BodyContent" -> "test string",
 "MIMEType" -> "text/plain",
 "uploadType" -> "multipart"
 ]//Normal

Out[23]= <|"kind" -> "drive#file", "id" -> "0B3O7g3kj2eCYMzNpWUJmOF9RS28", 
 "name" -> "test.txt", "mimeType" -> "text/plain"|>

And then we can delete it with:

$gd["DeleteFile",
 "fileId" -> %23["id"]
 ]

I took this type of structure (a parameter reformatting "Pre" function, a custom "Fetch" function -- plus an "Import" function to wrap around the call and a "Post" function to apply to the dataset) and automated it, so that all I have to supply to my templater is:

"UploadFile" -> {
  "Method" -> "POST",
  "Call" -> "RawUploadFile",
  "Post" -> Dataset@*imp@*Normal,
  "Import" -> googledrivepatchedurlfetchblock,
  "Pre" -> googledrivepatchmultipartparams
  }

And the googledrivepatchmultipartparams patches in the multipart params and googledrivepatchedurlfetchblock actually forces URLFetch to use the patched in params.

This type of format is a good way to make connecting to services (like Google Drive) that implement more complicated call structures easier.