Bug introduced in 11.0 or earlier and persists through 12.0
(confirmed by WRI Support but read summary below)
Answers summary
I think we need a quick summary here to justify keeping bug.
Original problem is not a bug. Here is why:
URLRead[#, "Body"]&: <raw http response> -> json_String
(*decoding, driven by content-type/charset header*)
ImportString[#,"RawJSON"]&: json_String -> wlJSON_(Association|List)
(* decoding driven by assumption that JSONString was UTF8 encoded*)
So the problem occurs because of double decoding. Workaround? Use BodyBytes
(see below) or URLExecute
. It should be easier to understand this and documentation does not help so much. Even WRI Support was confused.
What it also implies is that if you are preparing json with ExportString
for HTTPRequest / HTTPResponse
then you need to include CharacterEncoding->None
for them to avoid double encoded message.
There is an issue though, which I'm tempted to call a documentation bug. Handling content-type, charset and content-encoding by family of URL*
functions is not documented and changes with versions (improves) but it is not clear what to expect from them. It is not documented what a http response body is and what to expect from it with respect to mentioned headers.
There is a bug, JSON by its standard is UTF8 encoded and charset should not play any role, it can be there or not. content-type:application/json header should be enough to expect http response body to be decoded. It is not when there is no charset spec. Then the problem does not occur and the confusion is doubled (was for me when I tried to prepare a minimal example with different source of http responses)
The bug can be confirmed with:
urls = CloudDeploy @ Delayed[
HTTPResponse[
ExportString[{"\[Dash]"}, "RawJSON", "Compact" -> True]
, <|"ContentType" -> "application/json" <> #|>
, CharacterEncoding -> None
]
] & /@ {"", "; charset=utf-8"};
bodies = URLRead[#, "Body"] & /@ urls
ImportString[#, "RawJSON"] & /@ bodies
{{"\[Dash]"}, $Failed}
This is NOT about $Failed
it is about two different results caused by different charset
spec. Which should not matter for JSON
.
Original question
Background
V11.1.1 I'm using URLRead
to fetch some data,
can't show everything but headers contain:
"content-type->application/json; charset=utf-8"
and body (returned from URLRead) contains something you can create by:
body = "[\"" <> FromCharacterCode[8211] <> "\"]";
Problem
Since header is correct I'd expect the body
to be ready for ImportString
, but it is not:
ImportString[body, "JSON"]
(*$Failed*)
ImportString[body, "RawJSON"]
(*$Failed and
General::jsonoutofrangeunicode : Out of range unicode code point encountered.
*)
What works though is:
ImportString[ ToString[body, OutputForm, CharacterEncoding -> "UTF8"], "RawJSON"]
{"\[Dash]"}
Who is to blame, me, mathematica or the server for malformed response? Or maybe no one but then my method seems ugly for something that should be a standard procedure.
I tried to dig in encoding, responses, headers etc but I got lost in what should happen when.
Would appreciate clarification.
Update
I tried to mimic a round trip:
jsonBytes = ByteArray[Join[
ToCharacterCode["[\"", "UTF-8"],
ToCharacterCode[FromCharacterCode[8211], "UTF-8"],
ToCharacterCode["\"]", "UTF-8"]
]];
co = CloudDeploy @ Delayed[
HTTPResponse[
jsonBytes,
<|"ContentType" -> "application/json",
"CharacterEncoding" -> "UTF8"|>
]
];
URLRead[co, {"Headers", "Body"}]
<|"Headers" -> { ... , "content-type" -> "application/json" , "vary" -> "Accept-Encoding" , "transfer-encoding" -> "chunked"} , "Body" -> "[\"â\"]" |>
Notice the body!, it looks 'ok' now:
URLRead[co, {"Headers", "Body"}]["Body"] // ImportString[#, "RawJSON"] &
{"\[Dash]"}
But the encoding information is missing in headers. I just said "CharacterEncoding" -> "UTF8"
, didn't I?
If I force the encoding in content type field:
... <|"ContentType" -> "application/json; charset=utf-8"|> ...
then it is preserved
URLRead[co, {"Headers", "Body"}]
but body is incorrect:
<|"Headers" -> {...,
"content-type" -> "application/json;charset=utf-8",
"vary" -> "Accept-Encoding",
"transfer-encoding" -> "chunked"}
,"Body" -> "[\"\[Dash]\"]"
|>
And import string fails.
Summing up:
ignored character encoding, should not happen in my opinion
misinterpreted encoding when encoding is provided correctly
my interpretation is that
URLRead
and friends handle communcation from MMA to WPC if you don't care so much but it looks like some things are assumed instead of read from e.g. headers so communication with external services is flawed.
What is the story? I don't have time for that...
Related:
ToCharacterCode["\[Dash]", "UTF-8"]
? Currently it is{226, 128, 147}
and it isn't quite clear, why. There seems to be a confusion even on the developer's side: looks likeURLRead
interprets "UTF-8" as "Unicode" while JSON importers works likeToCharacterCode
. In any case, we have a bug. $\endgroup${226,128,147}
seems correct: fileformat.info/info/unicode/char/2013/index.htm $\endgroup$Developer`ReadRawJSONStream
is inconsistent: it should accept Unicode characters in the string. And I would expectURLRead
to import UTF8-encoded string"–"
as"\[Dash]"
, not as"\[AHat]\.80\.93"
. $\endgroup$