Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

Question

Bug introduced in 11.0 or earlier and persists through 12.0

(confirmed by WRI Support but read summary below)

---

Answers summary

I think we need a quick summary here to justify keeping bug.

Original problem is not a bug. Here is why:

URLRead[#, "Body"]&:        <raw http response> -> json_String
   (*decoding, driven by content-type/charset header*)

ImportString[#,"RawJSON"]&: json_String -> wlJSON_(Association|List)
   (* decoding driven by assumption that JSONString was UTF8 encoded*)

So the problem occurs because of double decoding. Workaround? Use BodyBytes (see below) or URLExecute. It should be easier to understand this and documentation does not help so much. Even WRI Support was confused.

What it also implies is that if you are preparing json with ExportString for HTTPRequest / HTTPResponse then you need to include CharacterEncoding->None for them to avoid double encoded message.

There is an issue though, which I'm tempted to call a documentation bug. Handling content-type, charset and content-encoding by family of URL* functions is not documented and changes with versions (improves) but it is not clear what to expect from them. It is not documented what a http response body is and what to expect from it with respect to mentioned headers.

There is a bug, JSON by its standard is UTF8 encoded and charset should not play any role, it can be there or not. content-type:application/json header should be enough to expect http response body to be decoded. It is not when there is no charset spec. Then the problem does not occur and the confusion is doubled (was for me when I tried to prepare a minimal example with different source of http responses)

The bug can be confirmed with:

urls = CloudDeploy @ Delayed[
  HTTPResponse[
    ExportString[{"\[Dash]"}, "RawJSON", "Compact" -> True]
  , <|"ContentType" -> "application/json" <> #|>
  , CharacterEncoding -> None
  ]
] & /@ {"", "; charset=utf-8"};

bodies = URLRead[#, "Body"] & /@ urls
ImportString[#, "RawJSON"] & /@ bodies

{{"\[Dash]"}, $Failed}

This is NOT about $Failed it is about two different results caused by different charset spec. Which should not matter for JSON.

---

Original question

Background

V11.1.1 I'm using URLRead to fetch some data,

can't show everything but headers contain:

"content-type->application/json; charset=utf-8"

and body (returned from URLRead) contains something you can create by:

body = "[\"" <> FromCharacterCode[8211] <> "\"]";

---

Problem

Since header is correct I'd expect the body to be ready for ImportString, but it is not:

ImportString[body, "JSON"]

(*$Failed*)

ImportString[body, "RawJSON"]

(*$Failed and 
  General::jsonoutofrangeunicode :  Out of range unicode code point encountered.
*)

What works though is:

ImportString[ ToString[body, OutputForm, CharacterEncoding -> "UTF8"], "RawJSON"]

{"\[Dash]"}

Who is to blame, me, mathematica or the server for malformed response? Or maybe no one but then my method seems ugly for something that should be a standard procedure.

I tried to dig in encoding, responses, headers etc but I got lost in what should happen when.

Would appreciate clarification.

---

Update

I tried to mimic a round trip:

jsonBytes = ByteArray[Join[
    ToCharacterCode["[\"", "UTF-8"],
    ToCharacterCode[FromCharacterCode[8211], "UTF-8"],
    ToCharacterCode["\"]", "UTF-8"]
    ]];

co = CloudDeploy @ Delayed[
    HTTPResponse[
     jsonBytes,
     <|"ContentType" -> "application/json", 
      "CharacterEncoding" -> "UTF8"|>
     ]
    ];

URLRead[co, {"Headers", "Body"}]

<|"Headers" -> { ...
      ,  "content-type" -> "application/json"
      , "vary" -> "Accept-Encoding"
      ,  "transfer-encoding" -> "chunked"}
    , "Body" -> "[\"â\"]"
 |>

Notice the body!, it looks 'ok' now:

URLRead[co, {"Headers", "Body"}]["Body"] //  ImportString[#, "RawJSON"] &

{"\[Dash]"}

But the encoding information is missing in headers. I just said "CharacterEncoding" -> "UTF8", didn't I?

If I force the encoding in content type field:

... <|"ContentType" -> "application/json; charset=utf-8"|> ...

then it is preserved

URLRead[co, {"Headers", "Body"}]

but body is incorrect:

<|"Headers" -> {..., 
  "content-type" -> "application/json;charset=utf-8", 
  "vary" -> "Accept-Encoding", 
  "transfer-encoding" -> "chunked"}
,"Body" -> "[\"\[Dash]\"]"
|>

And import string fails.

Summing up:

• ignored character encoding, should not happen in my opinion

• misinterpreted encoding when encoding is provided correctly

• my interpretation is that URLRead and friends handle communcation from MMA to WPC if you don't care so much but it looks like some things are assumed instead of read from e.g. headers so communication with external services is flawed.

What is the story? I don't have time for that...

---

Related:

• Importing malformed XML: Import[... "XMLObject"] vs ImportString[..., "XML"]
• https://mathematica.stackexchange.com/a/145771/5478
• URL functions overload

Answer 1

I'm Riccardo, current developer of URLRead in WL and I have some experience working with encoding in WL.

I would like to inform you that this is not a bug.

In modern versions of mathematica we have ByteArray, and this is a representation of bytes. But for decades strings have been both bytes and "unicode" at the same time.

The problem here is that all Import functions are expecting bytes as input and all Export functions are producing bytes as Output.

Let's take your example, <|"a" -> "\[Dash]"|>, and let's produce JSON out of it by using ExportString.

In[9]:= ExportString[<|"a" -> "\[Dash]"|>, "RawJSON", "Compact" -> True]
Out[9]= "{\"a\":\"â\"}"

What you get out is a string, but the string has been encoded in UTF-8 and now it's "unreadable". the output of ExportString is always a string that contains bytes in the range {0, 255}.

if you try to do the opposite operation, ImportString, you are getting back an association with encoded string:

In[12]:= ImportString[ExportString[<|"a" -> "\[Dash]"|>, "RawJSON", "Compact" -> True], "RawJSON"]
Out[12]= <|"a" -> "\[Dash]"|>

Trying to call ImportString over something that was decoded won't work. Infact, this is the "bug" you are experiencing:

ImportString["{\"a\":\"\[Dash]\"}", "RawJSON"]
During evaluation of In[14]:= General::jsonoutofrangeunicode: Out of range unicode code point encountered.
During evaluation of In[14]:= Import::jsoninvalidtoken: Invalid token found.
During evaluation of In[14]:= Import::jsonhintposition: An error occurred at line 1:8
Out[14]= $Failed

you are trying to import from a string that was already decoded, unfortunately there is NO WAY to distinguish between a string that has been encoded vs a string that is bytes, which is why the import is failing.

now, let's speak of URLRead. URLRead[..., "Body"] is returning the decoded body of the response, which is what you expect this method to do.

In[17]:= co = CloudDeploy@
   Delayed[HTTPResponse[
     ByteArray[
      Join[ToCharacterCode["[\"", "UTF-8"], 
       ToCharacterCode[FromCharacterCode[8211], "UTF-8"], 
       ToCharacterCode["\"]", "UTF-8"]]], <|
      "ContentType" -> "application/json; charset=utf-8"|>]];    

In[18]:= URLRead[co, "Body"] 
Out[18]= "[\"\[Dash]\"]"

now as I was explaining, the problem here is that you are importing using a decoded string. so calling ImportString over the Body won't work, it will fail because Body is not a string rappresenting bytes. what you should do is to use bytes, not a decoded string:

In[20]:= ImportString[FromCharacterCode[URLRead[co, "BodyBytes"]], "JSON"]
Out[20]= {"\[Dash]"}

in 11.2 there are functions to import / export using byte array instead of strings, so until then you need to use BodyBytes, starting from 11.2 you can pass to Import a bytearray.

Another small note: charset=...; is something you should specify only for text/* contenttypes, everything else (application, image) are not text formats and they do not accept a charset.

JSON only accepted encoded is UTF-8.

https://www.rfc-editor.org/rfc/rfc7159

Note: No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients.

I hope this is helpful to you. Let me know if you have any other questions.

Riccardo Di Virgilio

Answer 2

Tracing the evaluation (Mathematica 11.1.1) shows that the string is passed to Developer`ReadRawJSONStream which actually produces the messages:

Developer`ReadRawJSONStream[StringToStream@string, "IssueMessagesAs" -> Import]

General::jsonoutofrangeunicode: Out of range unicode code point encountered.

Import::jsoninvalidtoken: Invalid token found.

Import::jsonhintposition: An error occurred at line 1:4

$Failed

Since the first message says that "out of range unicode code point encountered", we can try to find out which Unicode code points are allowed:

Cases[Table[{n, Quiet@Developer`ReadRawJSONStream[
      StringToStream["[\"" <> FromCharacterCode[n] <> "\"]"], 
      "IssueMessagesAs" -> Import]}, {n, 0, 50000}], {n_, Except[$Failed]} :> n] // MinMax

{1, 254}

Obviously, even ASCII table isn't fully supported! No one Unicode symbol is allowed in the input stream.

But considering the workaround you already found we can conclude that it is an encoding issue, for some reason we must use only single-byte encoding:

ToString[body, OutputForm, CharacterEncoding -> "UTF8"] // FullForm

"[\"\[AHat]\.80\.93\"]"

We can achieve the same simpler and (probably) faster via ToCharacterCode/FromCharacterCode:

ImportString[FromCharacterCode@ToCharacterCode[body, "UTF-8"], "RawJSON"]

{"–"}

I still strongly suspect that current behavior is a bug and recommend reporting it to the tech support.

It is also worth to note that in version 8.0.4 ImportString[body, "JSON"] works, but in version 11.1.1 it fails, what supports that we have a bug in the current importer.

Answer 3

Alexey Popkov's answer is correct. So as to who is to blame, it's Mathematica. The standard requires applications to accept all Unicode characters in strings.

Quote from RFC 7159:

Section 7:

All Unicode characters may be placed within the quotation marks, except for the characters that must be escaped: quotation mark, reverse solidus, and the control characters (U+0000 through U+001F).

Section 9:

A JSON parser MUST accept all texts that conform to the JSON grammar.