5
$\begingroup$

I noticed a strange behavior regarding the APIFunctionwhen used with URLDispatcher. It does not allow me to access the body of a HTTP request payload. Similar question was asked here. However the answer does not help.

Let me provide a simple example. We prepare a test body as a "JSON" string and define the APIFunction which returns the body of the HTTP request payload after importing it as a "JSON" string.

api$Test = 
  APIFunction[{}, 
   Block[{s},
             s = HTTPRequestData[{"Method", "Headers", "Body"}];
             ImportString[s[[3]],"JSON"]
         ]&
       ];

Create a test body. Now testing the sanity of the APIFunction is straightforward.

body = ExportString[
{"User-1"->{"Name"->"Rob","Age"->28},"User-2"->{"Name"->"Bob","Age" ->32}},"JSON"];
GenerateHTTPResponse[
  api$Test, {"Method" -> "POST",
  "Headers" -> {"accept" -> "*/*","content-type" ->"application/json"},
  "Body" -> body}]["Body"]

{"User-1" -> {"Name" -> "Rob", "Age" -> 28}, "User-2" -> {"Name" -> "Bob", "Age" -> 32}}

Now let's try exposing the above APIFunction with URLDispatcherand access it locally using the builtin toy web server.

$Rules1 = URLDispatcher[{"/Test" -> api$Test}];
ws=HTTPHandling`StartWebServer[$Rules1,"Port"-> 7028]

With the web server running green, we can make an identical HTTP request that we used with GenerateHTTPResponseand our function api$Test. The APIFunction now used behind the URLDispatcher should return the same result as above.

exp = HTTPRequest[
   "http://localhost:7028/Test",<|"Method" -> "POST", 
    "Headers" -> {"accept" -> "*/*","content-type" -> "application/json"},
    "Body" -> body|>];
URLExecute[exp]

However one will get an ImportString::string error as the APIFunction can not access the body (which is None at this point not a "JOSN" string) thus it can not import the body as json using ImportString.

Am I doing something wrong or it is a bug of the toy web server or even the URLDispatcher? One can see that the "Header" is accessible.

{Accept->{/},Content-Length->{130},User-Agent->{Wolfram HTTPClient 11.1},Content-Type->{application/json}}

The specified "Method" is also accessible by the APIFunction but just like a magic, only the "Body" becomes None. Above was tested in mma version 11.1.1.0.

Update 1

I tried tracing the error

Trace[URLExecute[exp], TraceBackward -> True]

and found the following.

ImportString::string: First argument None is not a string.
ByteArray::lend: The argument at position 1 in ByteArray[#ContentData] should be a vector of unsigned byte values or a Base64-encoded string.
HTTPResponse::encfailed: Failed to encode HTTPResponse body
HTTPResponse::encfailed: Failed to encode HTTPResponse body
HTTPResponse::encfailed: Failed to encode HTTPResponse body

Strangely the error ByteArray::lend appears only during the first time evaluation. Passing the body as Base64-encoded string should not help and it does not either. The URLUtilities package might have some problem with reading the HTTP response. Experts can shade some lights!

Update 2

As suggested in the answer from @b3m2a1 checking out the Main.m shows that the "AsyncHandler" of the WebServer is working with a data where the body part of the HTTP request seems to be already missing. In the definition of the (ws:WebServer[_?AssociationQ])["AsyncHandler"] we can see the local variable json should contain the response.

json =  Association[ImportString[FromCharacterCode[Normal[msg["Data"]]], "JSON"]]

However it seems the message coming from the MQTT subscriber is having corrupt "Data" and failing to capture the body part of the HTTP POST request.

$\endgroup$
3
+50
$\begingroup$

So after some digging and breaking "MQTTLink`" for while I found that "MQTTLink`" is passing garbage out to its handler functions.

The heart of the problem is what's passed to that "AsyncHandler" you found. MQTT must work via some message / topic framework, as we find this in the source, when configuring the server:

MQTTLink`TopicSubscribe[client,
 "GenerateHTTPResponse",
 "MessageReceivedFunction":>ws["AsyncHandler"]
 ];

where ws is a WebServer. And then we can change that subscription function like so:

resetSubscription[ws_HTTPHandling`WebServer,
   pattern : _String : "GenerateHTTPResponse",
   unsubscribe : True | False : False,
   ops__?OptionQ] :=
  Block[{
    $attempts = If[IntegerQ@$attempts, $attempts, 1],
    $failed = False,
    $unsub = False
    },
   Quiet[
    If[unsubscribe,
     Check[
      MQTTLink`TopicUnsubscribe[ws["Meta"]["Connection"], 
       "GenerateHTTPResponse"];
      Pause[.1],
      $failed = True,
      MQTTLink`TopicUnsubscribe::noSubscribe
      ]
     ];
    Check[
     MQTTLink`TopicSubscribe[
      ws["Meta"]["Connection"],
      "GenerateHTTPResponse",
      ops
      ],
     $failed = True;
     $unsub = True,
     MQTTLink`TopicSubscribe::alreadySubscribed
     ],
    {
     MQTTLink`TopicUnsubscribe::noSubscribe,
     MQTTLink`TopicSubscribe::alreadySubscribed
     }
    ];
   If[$failed && ($attempts++ < 5),
    resetSubscription[ws, pattern, $unsub, ops],
    If[$attempts >= 5,
     $Failed
     ]
    ]
   ];
resetSubscription[ws_HTTPHandling`WebServer,
   pattern : _String : "GenerateHTTPResponse",
   unsubscribe : True | False : False,
   f_Function
   ] :=
  resetSubscription[ws,
   pattern,
   unsubscribe,
   "MessageReceivedFunction" -> f
   ];

We can then see what it's spitting out:

resetSubscription[ws, Function[$messageData = {##}; Abort[]]];
HTTPRequest[
  ws["AbsoluteURL"] <> "Test",
  <|
   "Method" -> "POST",
   "Headers" -> {"accept" -> "*/*", 
     "content-type" -> "application/json"},
   "Body" -> body
   |>
  ] // URLRead;
$messageData[[3, "Data"]] // Normal // 
  FromCharacterCode // JSONTools`FromJSON

{"Headers" -> {"Accept" -> {"*/*"}, "Content-Length" -> {"130"}, 
   "User-Agent" -> {"Wolfram HTTPClient 11.1"}, 
   "Content-Type" -> {"application/json"}}, "Scheme" -> "http", 
 "Domain" -> "", "MultipartElements" -> {}, "Parameters" -> {}, 
 "PathString" -> "/Test", 
 "Connection" -> "0c9deaa6-ce10-439a-8544-bd80ad5566cf", 
 "QueryString" -> "", "Port" -> "7000", "Method" -> "POST"}

No "Body" as you can see. This is something that comes straight out of the server, I believe, so it's not like we can tweak anything before the data comes back out to toplevel. But everything else is right so we can define a function to stick the body back in:

reinsertBody[data_, body_] :=
  ByteArray@
   ToCharacterCode@
    JSONTools`ToJSON@
     Append[
      data // Normal // FromCharacterCode // JSONTools`FromJSON,
      "Body" -> body
      ];
reinsertBody[data_] :=
  reinsertBody[data, $requestBody];
reinsertMessageBody[msgData_] :=
 ReplacePart[msgData,
  "Data" -> reinsertBody[msgData["Data"]]
  ]

And then we can add this to our server and get the desired result:

With[{ws = ws},
  resetSubscription[ws,
   Function[
    ws["AsyncHandler"][
     #,
     #2,
     reinsertMessageBody[#3]
     ]
    ]
   ]
  ];
HTTPRequest[
  ws["AbsoluteURL"] <> "Test",
  <|
   "Method" -> "POST",
   "Headers" -> {"accept" -> "*/*", 
     "content-type" -> "application/json"},
   "Body" -> ($requestBody = body)
   |>
  ] // URLExecute

{"User-1" -> {"Name" -> "Rob", "Age" -> 28}, 
 "User-2" -> {"Name" -> "Bob", "Age" -> 32}}

(Note: this can be finnicky and may require a Quit to get everything working nicely again)

Obviously this is a kinda junky solution, as we always have to use the pattern $requestBody = body, but for casual applications this should work just fine.


A note on MQTT link

There is a system exposed to start new brokers, see this in the source: MQTTLink`StartBroker[1883]. And it would be interesting to try to get one of these up and running.

We can also easily start a broker on a new port, e.g., MQTTLink`StartBroker[1884] but this isn't terribly interesting. More interesting is that we can pass MQTT options vis the "ConfigOptions" to StartBroker where the options come out of this

This theoretically lets us leverage more of the power of MQTT in Mathematica. Whether this is useful for anyone is something I do not know.


Original

It's clearly an issue with HTTPHandling`WebServer as this works:

bleh =
  CloudDeploy[
    URLDispatcher[{"/Test" -> api$Test}],
    "bleeblop",
    Permissions -> "Public"
    ][[1]];
HTTPRequest[
  URLBuild@{bleh, "Test"},
  <|
   "Method" -> "POST",
   "Headers" -> {"accept" -> "*/*", 
     "content-type" -> "application/json"},
   "Body" -> body|>
  ] // URLExecute

{"User-1" -> {"Name" -> "Rob", "Age" -> 28}, 
 "User-2" -> {"Name" -> "Bob", "Age" -> 32}}

But this doesn't:

serv =
  HTTPHandling`StartWebServer[URLDispatcher[{"/Test" -> api$Test}]];

HTTPRequest[
  "http://localhost:7000/Test",
  <|
   "Method" -> "POST",
   "Headers" -> {"accept" -> "*/*", 
     "content-type" -> "application/json"},
   "Body" -> body|>
  ] // URLExecute

ImportString::string: First argument None is not a string.

ImportString[None, "JSON"]

You can try to contact WRI about it or see if you can debug yourself from:

FileNameJoin@{
   PacletFind["HTTPHandling"][[1]]["Location"],
   "Main.m"
   } // SystemOpen

It seems to use MQTT on the backend which seems, from a cursory glance, to be pretty widely used (and thus likely the issue arises not with it, but with the Mathematica side of things).

$\endgroup$
  • $\begingroup$ Thanks for the pointer. Experimenting with the "Main.m" file suggests that the AsyncHandler of the WebServer function is working with a response data where the body part of the HTTP request is already missing. I guess the MQTT subscriber is messing up with the response it gets from MQTT. But no idea where to look for that part. $\endgroup$ – PlatoManiac Aug 21 '17 at 9:12
  • $\begingroup$ Thx for the help!! Now above allows POST to the endpoint from within MMA as we specifically need to set that $requestBody = body in the HTTPSRequest body. Do you see any possibility to extend the solution to allow POST from say a browser. When I try via a local browser to POST on http://localhost:7028/Test,with some JSON body, as expected it does not work. Do you see any scope to fix this so that my casual browser app can POST on the end point. If we can make the browser work, any external application (e.g. CURL,HTTP Client, or getpostman.com) will work too. Keeping my fingers crossed :) $\endgroup$ – PlatoManiac Aug 24 '17 at 15:13
  • $\begingroup$ @PlatoManiac Unfortunately I'm afraid I think that's probably buried in MQTTLink so I doubt I can help you there. Try to contact WRI or the creators and ask. $\endgroup$ – b3m2a1 Aug 27 '17 at 17:46
  • $\begingroup$ No worries thx anyways. Did not get any response at the worlfram community either. I chose to go the hard way ;) Using some web microframework to route and expose the mma api functions via KernelLink. In that way no nee to dive into MQTT code and wait for WRI. Better should have gone with this route to start with :) May be in 11.2 it will be fixed but I do not think so. $\endgroup$ – PlatoManiac Aug 28 '17 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.