APIFunction can not access HTTP request body when used with URLDispatcher

Question

I noticed a strange behavior regarding the APIFunctionwhen used with URLDispatcher. It does not allow me to access the body of a HTTP request payload. Similar question was asked here. However the answer does not help.

Let me provide a simple example. We prepare a test body as a "JSON" string and define the APIFunction which returns the body of the HTTP request payload after importing it as a "JSON" string.

api$Test = 
  APIFunction[{}, 
   Block[{s},
             s = HTTPRequestData[{"Method", "Headers", "Body"}];
             ImportString[s[[3]],"JSON"]
         ]&
       ];

Create a test body. Now testing the sanity of the APIFunction is straightforward.

body = ExportString[
{"User-1"->{"Name"->"Rob","Age"->28},"User-2"->{"Name"->"Bob","Age" ->32}},"JSON"];
GenerateHTTPResponse[
  api$Test, {"Method" -> "POST",
  "Headers" -> {"accept" -> "*/*","content-type" ->"application/json"},
  "Body" -> body}]["Body"]

{"User-1" -> {"Name" -> "Rob", "Age" -> 28}, "User-2" -> {"Name" -> "Bob", "Age" -> 32}}

Now let's try exposing the above APIFunction with URLDispatcherand access it locally using the builtin toy web server.

$Rules1 = URLDispatcher[{"/Test" -> api$Test}];
ws=HTTPHandling`StartWebServer[$Rules1,"Port"-> 7028]

With the web server running green, we can make an identical HTTP request that we used with GenerateHTTPResponseand our function api$Test. The APIFunction now used behind the URLDispatcher should return the same result as above.

exp = HTTPRequest[
   "http://localhost:7028/Test",<|"Method" -> "POST", 
    "Headers" -> {"accept" -> "*/*","content-type" -> "application/json"},
    "Body" -> body|>];
URLExecute[exp]

However one will get an ImportString::string error as the APIFunction can not access the body (which is None at this point not a "JOSN" string) thus it can not import the body as json using ImportString.

Am I doing something wrong or it is a bug of the toy web server or even the URLDispatcher? One can see that the "Header" is accessible.

{Accept->{/},Content-Length->{130},User-Agent->{Wolfram HTTPClient 11.1},Content-Type->{application/json}}

The specified "Method" is also accessible by the APIFunction but just like a magic, only the "Body" becomes None. Above was tested in mma version 11.1.1.0.

Update 1

I tried tracing the error

Trace[URLExecute[exp], TraceBackward -> True]

and found the following.

ImportString::string: First argument None is not a string.
ByteArray::lend: The argument at position 1 in ByteArray[#ContentData] should be a vector of unsigned byte values or a Base64-encoded string.
HTTPResponse::encfailed: Failed to encode HTTPResponse body
HTTPResponse::encfailed: Failed to encode HTTPResponse body
HTTPResponse::encfailed: Failed to encode HTTPResponse body

Strangely the error ByteArray::lend appears only during the first time evaluation. Passing the body as Base64-encoded string should not help and it does not either. The URLUtilities package might have some problem with reading the HTTP response. Experts can shade some lights!

Update 2

As suggested in the answer from @b3m2a1 checking out the Main.m shows that the "AsyncHandler" of the WebServer is working with a data where the body part of the HTTP request seems to be already missing. In the definition of the (ws:WebServer[_?AssociationQ])["AsyncHandler"] we can see the local variable json should contain the response.

json =  Association[ImportString[FromCharacterCode[Normal[msg["Data"]]], "JSON"]]

However it seems the message coming from the MQTT subscriber is having corrupt "Data" and failing to capture the body part of the HTTP POST request.

Answer 1

So after some digging and breaking "MQTTLink`" for while I found that "MQTTLink`" is passing garbage out to its handler functions.

The heart of the problem is what's passed to that "AsyncHandler" you found. MQTT must work via some message / topic framework, as we find this in the source, when configuring the server:

MQTTLink`TopicSubscribe[client,
 "GenerateHTTPResponse",
 "MessageReceivedFunction":>ws["AsyncHandler"]
 ];

where ws is a WebServer. And then we can change that subscription function like so:

resetSubscription[ws_HTTPHandling`WebServer,
   pattern : _String : "GenerateHTTPResponse",
   unsubscribe : True | False : False,
   ops__?OptionQ] :=
  Block[{
    $attempts = If[IntegerQ@$attempts, $attempts, 1],
    $failed = False,
    $unsub = False
    },
   Quiet[
    If[unsubscribe,
     Check[
      MQTTLink`TopicUnsubscribe[ws["Meta"]["Connection"], 
       "GenerateHTTPResponse"];
      Pause[.1],
      $failed = True,
      MQTTLink`TopicUnsubscribe::noSubscribe
      ]
     ];
    Check[
     MQTTLink`TopicSubscribe[
      ws["Meta"]["Connection"],
      "GenerateHTTPResponse",
      ops
      ],
     $failed = True;
     $unsub = True,
     MQTTLink`TopicSubscribe::alreadySubscribed
     ],
    {
     MQTTLink`TopicUnsubscribe::noSubscribe,
     MQTTLink`TopicSubscribe::alreadySubscribed
     }
    ];
   If[$failed && ($attempts++ < 5),
    resetSubscription[ws, pattern, $unsub, ops],
    If[$attempts >= 5,
     $Failed
     ]
    ]
   ];
resetSubscription[ws_HTTPHandling`WebServer,
   pattern : _String : "GenerateHTTPResponse",
   unsubscribe : True | False : False,
   f_Function
   ] :=
  resetSubscription[ws,
   pattern,
   unsubscribe,
   "MessageReceivedFunction" -> f
   ];

We can then see what it's spitting out:

resetSubscription[ws, Function[$messageData = {##}; Abort[]]];
HTTPRequest[
  ws["AbsoluteURL"] <> "Test",
  <|
   "Method" -> "POST",
   "Headers" -> {"accept" -> "*/*", 
     "content-type" -> "application/json"},
   "Body" -> body
   |>
  ] // URLRead;
$messageData[[3, "Data"]] // Normal // 
  FromCharacterCode // JSONTools`FromJSON

{"Headers" -> {"Accept" -> {"*/*"}, "Content-Length" -> {"130"}, 
   "User-Agent" -> {"Wolfram HTTPClient 11.1"}, 
   "Content-Type" -> {"application/json"}}, "Scheme" -> "http", 
 "Domain" -> "", "MultipartElements" -> {}, "Parameters" -> {}, 
 "PathString" -> "/Test", 
 "Connection" -> "0c9deaa6-ce10-439a-8544-bd80ad5566cf", 
 "QueryString" -> "", "Port" -> "7000", "Method" -> "POST"}

No "Body" as you can see. This is something that comes straight out of the server, I believe, so it's not like we can tweak anything before the data comes back out to toplevel. But everything else is right so we can define a function to stick the body back in:

reinsertBody[data_, body_] :=
  ByteArray@
   ToCharacterCode@
    JSONTools`ToJSON@
     Append[
      data // Normal // FromCharacterCode // JSONTools`FromJSON,
      "Body" -> body
      ];
reinsertBody[data_] :=
  reinsertBody[data, $requestBody];
reinsertMessageBody[msgData_] :=
 ReplacePart[msgData,
  "Data" -> reinsertBody[msgData["Data"]]
  ]

And then we can add this to our server and get the desired result:

With[{ws = ws},
  resetSubscription[ws,
   Function[
    ws["AsyncHandler"][
     #,
     #2,
     reinsertMessageBody[#3]
     ]
    ]
   ]
  ];
HTTPRequest[
  ws["AbsoluteURL"] <> "Test",
  <|
   "Method" -> "POST",
   "Headers" -> {"accept" -> "*/*", 
     "content-type" -> "application/json"},
   "Body" -> ($requestBody = body)
   |>
  ] // URLExecute

{"User-1" -> {"Name" -> "Rob", "Age" -> 28}, 
 "User-2" -> {"Name" -> "Bob", "Age" -> 32}}

(Note: this can be finnicky and may require a Quit to get everything working nicely again)

Obviously this is a kinda junky solution, as we always have to use the pattern $requestBody = body, but for casual applications this should work just fine.

---

A note on MQTT link

There is a system exposed to start new brokers, see this in the source: MQTTLink`StartBroker[1883]. And it would be interesting to try to get one of these up and running.

We can also easily start a broker on a new port, e.g., MQTTLink`StartBroker[1884] but this isn't terribly interesting. More interesting is that we can pass MQTT options vis the "ConfigOptions" to StartBroker where the options come out of this

This theoretically lets us leverage more of the power of MQTT in Mathematica. Whether this is useful for anyone is something I do not know.

---

Original

It's clearly an issue with HTTPHandling`WebServer as this works:

bleh =
  CloudDeploy[
    URLDispatcher[{"/Test" -> api$Test}],
    "bleeblop",
    Permissions -> "Public"
    ][[1]];
HTTPRequest[
  URLBuild@{bleh, "Test"},
  <|
   "Method" -> "POST",
   "Headers" -> {"accept" -> "*/*", 
     "content-type" -> "application/json"},
   "Body" -> body|>
  ] // URLExecute

{"User-1" -> {"Name" -> "Rob", "Age" -> 28}, 
 "User-2" -> {"Name" -> "Bob", "Age" -> 32}}

But this doesn't:

serv =
  HTTPHandling`StartWebServer[URLDispatcher[{"/Test" -> api$Test}]];

HTTPRequest[
  "http://localhost:7000/Test",
  <|
   "Method" -> "POST",
   "Headers" -> {"accept" -> "*/*", 
     "content-type" -> "application/json"},
   "Body" -> body|>
  ] // URLExecute

ImportString::string: First argument None is not a string.

ImportString[None, "JSON"]

You can try to contact WRI about it or see if you can debug yourself from:

FileNameJoin@{
   PacletFind["HTTPHandling"][[1]]["Location"],
   "Main.m"
   } // SystemOpen

It seems to use MQTT on the backend which seems, from a cursory glance, to be pretty widely used (and thus likely the issue arises not with it, but with the Mathematica side of things).