0
$\begingroup$

Update

The short answer is that when getting results back from an API into an HTTPResponse object the "Body" property is a post-decoded form of the result. ImportString with "JSON" expects the encoded form of the result. Therefore you should use the "BodyBytes" property with FromCharacterCode.

With res a HTTPResponse object with the result, then

ImportString[FromCharacterCode@res["BodyBytes"], "JSON"]

brings in the result without issue.

URLExecute can also be used to directly return the imported JSON but you don't get a nice HTTPResponse object to check the status of your result.

With req a HTTPRequest object, then

URLExecute[req, {}, "JSON"]

brings in the result without issue.


OP

I have many large JSON results from a web API that contain Unicode characters. ImportString with "JSON" seems unable to import these. As a minimal example try,

ImportString["{\"id\":1,\"text\":\"ñía\"}", "JSON"]

Import::utf8expcontinuation: Input is not a valid UTF8 byte sequence. 237 is expected to be a continuation byte.

Import::jsonhintposition: An error occurred at line 1:19

I have tried adding option CharacterEncoding -> "UTF8" and get the same result.

Finally I tried

# -> ImportString["{\"id\":1,\"text\":\"ñía\"}", "JSON", 
    CharacterEncoding -> #] & /@ $CharacterEncodings

Every encoding failed.

How do I import Unicode JSON? I have many such results to import.

I am using Win10 Mma 11.3

$\endgroup$
  • 1
    $\begingroup$ Round tripping to and from ByteArray works on your small example for some encodings: Cases[Quiet[# -> ImportString[ByteArrayToString[StringToByteArray["{\"id\":1,\"text\":\"ñía\"}"], #], "JSON"]] & /@ $CharacterEncodings, _[_, _List]] $\endgroup$ – Chip Hurst Mar 21 '19 at 23:26
  • $\begingroup$ @ChipHurst A few of them do work with the round trip. Any ideas why hoops must be jumped through to get what should be a basic import. Bug, perhaps? $\endgroup$ – Edmund Mar 22 '19 at 0:04
  • 1
    $\begingroup$ It feels like a bug, but I don’t have enough background to know for sure. $\endgroup$ – Chip Hurst Mar 22 '19 at 0:05
  • 3
    $\begingroup$ Where does this string come from? If from URLRead Body then it is not the way to go and you have to import bytes > FromCharacterCode > ImportString. For explanation see the first section in mathematica.stackexchange.com/q/154245/5478 $\endgroup$ – Kuba Mar 22 '19 at 0:21
  • 1
    $\begingroup$ @Edmund yes if URLExecute can be used then go with it. Sometimes one needs to use URLRead when a more fine grained control is needed. $\endgroup$ – Kuba Mar 22 '19 at 13:43