# Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

Bug introduced in 11.0 or earlier and persists through 11.3

(confirmed by WRI Support but read summary below)

I think we need a quick summary here to justify keeping .

Original problem is not a bug. Here is why:

URLRead[#, "Body"]&:        <raw http response> -> json_String

ImportString[#,"RawJSON"]&: json_String -> wlJSON_(Association|List)
(* decoding driven by assumption that JSONString was UTF8 encoded*)


So the problem occurs because of double decoding. Workaround? Use BodyBytes (see below) or URLExecute. It should be easier to understand this and documentation does not help so much. Even WRI Support was confused.

What it also implies is that if you are preparing json with ExportString for HTTPRequest / HTTPResponse then you need to include CharacterEncoding->None for them to avoid double encoded message.

There is an issue though, which I'm tempted to call a documentation bug. Handling content-type, charset and content-encoding by family of URL* functions is not documented and changes with versions (improves) but it is not clear what to expect from them. It is not documented what a http response body is and what to expect from it with respect to mentioned headers.

There is a bug, JSON by its standard is UTF8 encoded and charset should not play any role, it can be there or not. content-type:application/json header should be enough to expect http response body to be decoded. It is not when there is no charset spec. Then the problem does not occur and the confusion is doubled (was for me when I tried to prepare a minimal example with different source of http responses)

The bug can be confirmed with:

urls = CloudDeploy @ Delayed[
HTTPResponse[
ExportString[{"\[Dash]"}, "RawJSON", "Compact" -> True]
, <|"ContentType" -> "application/json" <> #|>
, CharacterEncoding -> None
]
] & /@ {"", "; charset=utf-8"};

bodies = URLRead[#, "Body"] & /@ urls
ImportString[#, "RawJSON"] & /@ bodies

{{"\[Dash]"}, $Failed}  This is NOT about $Failed it is about two different results caused by different charset spec. Which should not matter for JSON.

# Original question

## Background

V11.1.1 I'm using URLRead to fetch some data,

can't show everything but headers contain:

"content-type->application/json; charset=utf-8"


and body (returned from URLRead) contains something you can create by:

body = "[\"" <> FromCharacterCode[8211] <> "\"]";


## Problem

Since header is correct I'd expect the body to be ready for ImportString, but it is not:

ImportString[body, "JSON"]

(*$Failed*) ImportString[body, "RawJSON"] (*$Failed and
General::jsonoutofrangeunicode :  Out of range unicode code point encountered.
*)


What works though is:

ImportString[ ToString[body, OutputForm, CharacterEncoding -> "UTF8"], "RawJSON"]

{"\[Dash]"}


Who is to blame, me, mathematica or the server for malformed response? Or maybe no one but then my method seems ugly for something that should be a standard procedure.

I tried to dig in encoding, responses, headers etc but I got lost in what should happen when.

Would appreciate clarification.

## Update

I tried to mimic a round trip:

jsonBytes = ByteArray[Join[
ToCharacterCode["[\"", "UTF-8"],
ToCharacterCode[FromCharacterCode[8211], "UTF-8"],
ToCharacterCode["\"]", "UTF-8"]
]];

co = CloudDeploy @ Delayed[
HTTPResponse[
jsonBytes,
<|"ContentType" -> "application/json",
"CharacterEncoding" -> "UTF8"|>
]
];


<|"Headers" -> { ...
,  "content-type" -> "application/json"
, "vary" -> "Accept-Encoding"
,  "transfer-encoding" -> "chunked"}
, "Body" -> "[\"â\"]"
|>


Notice the body!, it looks 'ok' now:

URLRead[co, {"Headers", "Body"}]["Body"] //  ImportString[#, "RawJSON"] &

{"\[Dash]"}


But the encoding information is missing in headers. I just said "CharacterEncoding" -> "UTF8", didn't I?

If I force the encoding in content type field:

... <|"ContentType" -> "application/json; charset=utf-8"|> ...


then it is preserved

URLRead[co, {"Headers", "Body"}]


but body is incorrect:

<|"Headers" -> {...,
"content-type" -> "application/json;charset=utf-8",
"vary" -> "Accept-Encoding",
"transfer-encoding" -> "chunked"}
,"Body" -> "[\"\[Dash]\"]"
|>


And import string fails.

Summing up:

• ignored character encoding, should not happen in my opinion

• misinterpreted encoding when encoding is provided correctly

• my interpretation is that URLRead and friends handle communcation from MMA to WPC if you don't care so much but it looks like some things are assumed instead of read from e.g. headers so communication with external services is flawed.

What is the story? I don't have time for that...

Related:

• The problem is strongly related to the question: what is the expected output from ToCharacterCode["\[Dash]", "UTF-8"]? Currently it is {226, 128, 147} and it isn't quite clear, why. There seems to be a confusion even on the developer's side: looks like URLRead interprets "UTF-8" as "Unicode" while JSON importers works like ToCharacterCode. In any case, we have a bug. – Alexey Popkov Aug 22 '17 at 9:44
• @AlexeyPopkov {226,128,147} seems correct: fileformat.info/info/unicode/char/2013/index.htm – Kuba Aug 22 '17 at 9:50
• Thanks. But I still think that current behavior of DeveloperReadRawJSONStream is inconsistent: it should accept Unicode characters in the string. And I would expect URLRead to import UTF8-encoded string "–" as "\[Dash]", not as "\[AHat]\.80\.93". – Alexey Popkov Aug 22 '17 at 10:00
• @AlexeyPopkov I didn't mean to defend anything :) only to point the source of UTF8 bytes. – Kuba Aug 22 '17 at 10:03

I'm Riccardo, current developer of URLRead in WL and I have some experience working with encoding in WL.

I would like to inform you that this is not a bug.

In modern versions of mathematica we have ByteArray, and this is a representation of bytes. But for decades strings have been both bytes and "unicode" at the same time.

The problem here is that all Import functions are expecting bytes as input and all Export functions are producing bytes as Output.

Let's take your example, <|"a" -> "\[Dash]"|>, and let's produce JSON out of it by using ExportString.

In[9]:= ExportString[<|"a" -> "\[Dash]"|>, "RawJSON", "Compact" -> True]
Out[9]= "{\"a\":\"â\"}"


What you get out is a string, but the string has been encoded in UTF-8 and now it's "unreadable". the output of ExportString is always a string that contains bytes in the range {0, 255}.

if you try to do the opposite operation, ImportString, you are getting back an association with encoded string:

In[12]:= ImportString[ExportString[<|"a" -> "\[Dash]"|>, "RawJSON", "Compact" -> True], "RawJSON"]
Out[12]= <|"a" -> "\[Dash]"|>


Trying to call ImportString over something that was decoded won't work. Infact, this is the "bug" you are experiencing:

ImportString["{\"a\":\"\[Dash]\"}", "RawJSON"]
During evaluation of In[14]:= General::jsonoutofrangeunicode: Out of range unicode code point encountered.
During evaluation of In[14]:= Import::jsoninvalidtoken: Invalid token found.
During evaluation of In[14]:= Import::jsonhintposition: An error occurred at line 1:8
Out[14]= $Failed  you are trying to import from a string that was already decoded, unfortunately there is NO WAY to distinguish between a string that has been encoded vs a string that is bytes, which is why the import is failing. now, let's speak of URLRead. URLRead[..., "Body"] is returning the decoded body of the response, which is what you expect this method to do. In[17]:= co = CloudDeploy@ Delayed[HTTPResponse[ ByteArray[ Join[ToCharacterCode["[\"", "UTF-8"], ToCharacterCode[FromCharacterCode[8211], "UTF-8"], ToCharacterCode["\"]", "UTF-8"]]], <| "ContentType" -> "application/json; charset=utf-8"|>]]; In[18]:= URLRead[co, "Body"] Out[18]= "[\"\[Dash]\"]"  now as I was explaining, the problem here is that you are importing using a decoded string. so calling ImportString over the Body won't work, it will fail because Body is not a string rappresenting bytes. what you should do is to use bytes, not a decoded string: In[20]:= ImportString[FromCharacterCode[URLRead[co, "BodyBytes"]], "JSON"] Out[20]= {"\[Dash]"}  in 11.2 there are functions to import / export using byte array instead of strings, so until then you need to use BodyBytes, starting from 11.2 you can pass to Import a bytearray. Another small note: charset=...; is something you should specify only for text/* contenttypes, everything else (application, image) are not text formats and they do not accept a charset. JSON only accepted encoded is UTF-8. https://tools.ietf.org/html/rfc7159 Note: No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients. I hope this is helpful to you. Let me know if you have any other questions. Riccardo Di Virgilio • @Riccardo Thank you very much for the detailed response. Please clarify: was your statement "all Import functions are expecting bytes as input and all Export functions are producing bytes as Output" meant to be about JSON only? Or was it about every Import/Export format? For example, ExportString["\[Dash]", "String"] outputs "\[Dash]" what contradicts this statement. Or is it just "String" format that is so special? – Alexey Popkov Aug 25 '17 at 16:37 • I mean every single Import / Export function (PNG, JPEG, ecc). String is a particular case: String is WL string, which is producing an output using WL encoding (example: \[Dash]), which is different from Text, which produces an arbitrary encoding. – Riccardo Di Virgilio Aug 26 '17 at 12:35 • – Kuba Aug 28 '17 at 7:46 • @AlexeyPopkov and Riccardo, what is your opinion on chat.stackexchange.com/transcript/message/43639148#43639148 – Kuba Mar 27 '18 at 7:40 • I'm glad to announce that with Mathematica 11.3 you can use ImportByteArray to properly handle encoding. sample usage is ImportByteArray[ URLRead[ "raw.githubusercontent.com/riccardodivirgilio/rfuncs/master/…", "BodyByteArray" ], "RawJSON" ] – Riccardo Di Virgilio May 2 '18 at 12:53 Tracing the evaluation (Mathematica 11.1.1) shows that the string is passed to DeveloperReadRawJSONStream which actually produces the messages: DeveloperReadRawJSONStream[StringToStream@string, "IssueMessagesAs" -> Import]  General::jsonoutofrangeunicode: Out of range unicode code point encountered. Import::jsoninvalidtoken: Invalid token found. Import::jsonhintposition: An error occurred at line 1:4 $Failed


Since the first message says that "out of range unicode code point encountered", we can try to find out which Unicode code points are allowed:

Cases[Table[{n, Quiet@DeveloperReadRawJSONStream[
StringToStream["[\"" <> FromCharacterCode[n] <> "\"]"],
"IssueMessagesAs" -> Import]}, {n, 0, 50000}], {n_, Except[\$Failed]} :> n] // MinMax

{1, 254}


Obviously, even ASCII table isn't fully supported! No one Unicode symbol is allowed in the input stream.

But considering the workaround you already found we can conclude that it is an encoding issue, for some reason we must use only single-byte encoding:

ToString[body, OutputForm, CharacterEncoding -> "UTF8"] // FullForm

"[\"\[AHat]\.80\.93\"]"


We can achieve the same simpler and (probably) faster via ToCharacterCode/FromCharacterCode:

ImportString[FromCharacterCode@ToCharacterCode[body, "UTF-8"], "RawJSON"]

{"–"}


I still strongly suspect that current behavior is a bug and recommend reporting it to the tech support.

It is also worth to note that in version 8.0.4 ImportString[body, "JSON"] works, but in version 11.1.1 it fails, what supports that we have a bug in the current importer.

Alexey Popkov's answer is correct. So as to who is to blame, it's Mathematica. The standard requires applications to accept all Unicode characters in strings.

Quote from RFC 7159:

Section 7:

All Unicode characters may be placed within the quotation marks, except for the characters that must be escaped: quotation mark, reverse solidus, and the control characters (U+0000 through U+001F).

Section 9:

A JSON parser MUST accept all texts that conform to the JSON grammar.