11
$\begingroup$

Working in the creations of APIs with Mathematica 12.2, I got this problem with diacritics when sending JSON data.

Here is a toy code using httpbin.

HTTPRequest["https://httpbin.org/anything",
      <|
      Method->"POST"
      ,"Body" -> (<|"textWithDiacritics"-> "àéíóú"|>//ExportString[#, "RawJSON"]&)
        |>
     ]//URLRead[#, "BodyBytes"]&//FromCharacterCode//ImportString[#, "RawJSON"]&

enter image description here

I found this related post, It seams to be a bug in ExportString for RawJSON.

Any suggestions on how to get around this case?

Cross post in Wolfram Community

$\endgroup$
9
$\begingroup$

ExportString JSON produces UTF8 encoded json string, if put as HTTPRequest body it will undergo another encoding unless you prevent it with an option:

HTTPRequest[..., CharacterEncoding -> None]

It is analogous to the double decoding issue which you address with bodybytes//FromCharacterCode//ImportString.


Exchanging JSON via http requests

This is very common these days so for the record:

Building requests (for URLRead input):

HTTPRequest[
  url_
, <|
    "ContentType" -> "application/json"
  , "Body"        -> ExportString[ jsonCompliant_String, "RawJSON"]
  , "Method"      -> "POST"
  , "Headers"     -> {"Accept"->"application/json"} (* less important *)
  |>
, CharacterEncoding -> None
]

Receiving requests ( for APIFunction )

I will skip the part of handling different payloads or error handling, just assuming the incomming request is on generated as above will need:

APIFunction[
  {}
, Function[whatever (*we don't care about query params*)
  , HTTPRequestData["BodyBytes"] // FromCharacterCode //
      ImportString[#, "RawJSON"]& //
      doSomethingWithAssoAndGenerateResponse
  ]
] 

Building responses (* in API/FormFunctions *)

For APIFunction etc there is a nice wrapper ExportForm which works with GenerateHTTPResponse which is called on the result of APIFunction's function.

So a quick way:

  APIFunction[{}, ExportForm[association, "RawJSON" ]& ]

And a longer way:

HTTPResponse[
    ExportString[association_ , "RawJSON" ]
  , <| "ContentType"->"application/json" |>
  , CharacterEncoding -> None
  ]

I wish there was ExportForm support for building HTTPRequests, I even asked WRI, the status is 'maybe'.

Parsing responses (* what URLRead needs to do *)

Worth to mention that URLExecute does it well wrt encoding but I always need more controll on status codes etc so I need to use URLRead. So as you mentioned:

URLRead[...]["BodyBytes"]& // FromCharacterCode//ImportString[#, "RawJSON"]&

I would not try to decode "Body" because it is very unclear what the body is according to developers, the behavior of "Body" worsened in 12.2 it became even less predictable. So URLExecute or "BodyBytes" are the way to go.

More reading:

HTTPRequest syntax problem

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

$\endgroup$
2
  • $\begingroup$ This is cool too. Now I don't know which one to accept! :) $\endgroup$
    – Murta
    Mar 11 at 19:36
  • $\begingroup$ @Murta thanks for the accept. I added a guideline section so you won't be sorry :) $\endgroup$
    – Kuba
    Mar 12 at 5:58
7
$\begingroup$

I've had luck using the functions WriteRawJSONString and ReadRawJSONString from Developer context. Check out: WriteRawJSONString vs ExportString?

HTTPRequest[
    "https://httpbin.org/anything", <|Method -> "POST", 
     "Body" -> (<|"textWithDiacritics" -> "àéíóú"|> // 
        Developer`WriteRawJSONString)|>] // 
   URLRead[#, "BodyBytes"] & // 
  FromCharacterCode // Developer`ReadRawJSONString

(* <|"args" -> <||>, "data" -> "{
    \"textWithDiacritics\":\"àéíóú\"
}", "files" -> <||>, "form" -> <||>, 
 "headers" -> <|"Accept" -> "*/*", 
   "Accept-Encoding" -> "deflate, gzip", "Content-Length" -> "38", 
   "Content-Type" -> "text/plain;charset=utf-8", 
   "Host" -> "httpbin.org", "User-Agent" -> "Wolfram HTTPClient 12.2",
    "X-Amzn-Trace-Id" -> "Root=1-604a65d2-560dfd4155a830d409b373c9"|>,
  "json" -> <|"textWithDiacritics" -> "àéíóú"|>, "method" -> "POST", 
 "origin" -> "165.254.183.48", 
 "url" -> "https://httpbin.org/anything"|> *)
$\endgroup$
1
  • $\begingroup$ This is great, tks Chuy. $\endgroup$
    – Murta
    Mar 11 at 19:35
4
$\begingroup$

For what it's worth, the issue is in how ExportString is handling the diacritics. We can see that it botches them, but the function being used under the hood doesn't

(<|"textWithDiacritics" -> "àéíóú"|> // ExportString[#, "RawJSON"] &)

"{
    \"textWithDiacritics\":\"àéíÃ\.b3ú\"
}"

<|"textWithDiacritics" -> "àéíóú"|> // Developer`WriteRawJSONString

"{
    \"textWithDiacritics\":\"àéíóú\"
}"

That issue can be traced back to the way ExportString works (I think): it opens an OutputStream, uses a standard exporter to write there, then reads back into Mathematica as a "String", because it's fast.

We can see that issue clearly if we mimic that approach:

meh = OpenWrite[];
Developer`WriteRawJSONStream[meh, <|"textWithDiacritics" -> "àéíóú"|>];
Import[Close[meh], "String"]

"{
    \"textWithDiacritics\":\"àéíÃ\.b3ú\"
}"

meh = OpenWrite[];
Developer`WriteRawJSONStream[meh, <|"textWithDiacritics" -> "àéíóú"|>];
Import[Close[meh], "Text"]

"{
    \"textWithDiacritics\":\"àéíóú\"
}"

So what's the solution? Well, I think you can just leave the body as an expression and Mathematica will write it out as a raw JSON string for you, i.e. this works for me

HTTPRequest[
    "https://httpbin.org/anything",
    <|Method -> "POST", 
     "Body" -> <|"textWithDiacritics" -> "àéíóú"|>
     |>
    ] // 
   URLRead[#, "BodyBytes"] & // FromCharacterCode // 
 ImportString[#, "RawJSON"]["form"] &

<|"textWithDiacritics" -> "àéíóú"|>

But in any case, any of the existing solutions provide real answers, this was mostly just to point how what's going wrong.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.