9
$\begingroup$

I'm not sure if it's a bug with Mathematica or if I'm missing something, but when trying to ExportString a UTF-8 string with diacritics as JSON, it's messing up the character encoding and producing invalid JSON. Here's an example:

ExportString[{"1" -> "Conexión"}, "JSON"]

It doesn't matter if I add CharacterEncoding -> "Unicode" or whatever, the output always ends up as

{
    "1": "ConexiÃ\.b3n"
}

Has anybody encountered this problem before, or know how to solve it? I'm running Mathematica 10.0.2, in case that matters.

$\endgroup$
  • 1
    $\begingroup$ The problem is still here in 10.1. $\endgroup$ – C. E. May 9 '15 at 11:46
  • $\begingroup$ I think you may have found a bug in ExportString (see my extended comment below as well). Would you contact Wolfram support to see what they think? $\endgroup$ – MarcoB May 11 '15 at 5:20
  • 1
    $\begingroup$ @MarcoB I've already filed a bug report, I'll post any useful replies here. $\endgroup$ – R. Kazeno May 11 '15 at 5:27
  • 1
    $\begingroup$ ExportString doesn't accept CharacterEncoding like that works well in Export, not only the JSON, also HTML,I've tried @ windows 8.1 and Mma 10.1 $\endgroup$ – HyperGroups May 11 '15 at 6:32
  • $\begingroup$ In 10.2 only, RawJSON works with associations: ExportString[<|"1" -> "Conexión"|>, "RawJSON"] $\endgroup$ – Szabolcs Sep 4 '15 at 15:52
6
$\begingroup$

Actually this problem is still in Mma 10.2

here is a solution

ExportString["Conexión", "JSON"]

(*
    "ConexiÃ\.b3n"
*)

ExportString["Conexión", "JSON", CharacterEncoding -> "UTF8"]

(*
    "ConexiÃ\.b3n"
*)

FromCharacterCode[

 ToCharacterCode[ExportString["Conexión", "JSON"]], "UTF8"]

(*
    "Conexión"
*)

ExportString["Conexión", "RawJSON"]

(*
    "Conexión"
*)

ExportString["Conexión", "HTMLFragment"]

Here Actually I want to add a Chinese Word but because this problem...

(*
    Conexi&oacute;n
*)

A problem is sometimes we should find two matched encodings

decode[str_, encode_, decode_] := 

 FromCharacterCode[ToCharacterCode[str, encode], decode]

Here is a collection of some problem about encodings.

$\endgroup$
  • $\begingroup$ I'm accepting your answer since FromCharacterCode[ ToCharacterCode[ExportString[{"1" -> "Conexión"}, "JSON"]], "UTF8"] is the best workaround I've found. $\endgroup$ – R. Kazeno Sep 6 '15 at 3:48
  • $\begingroup$ If you state the problem as "how do I decode the UTF-8 encoding of a JSON string", then this is the right solution. But ExportString[expr, "JSON"] by itself is doing what it's supposed to, including the encoding into UTF-8. $\endgroup$ – Joel Klein Oct 1 '15 at 19:02
12
$\begingroup$

I don't believe that this is a bug. Here's why:

What exactly happens? Export and ExportString will always encode "JSON" as UTF-8. This may be intentional, as UTF-8 is the default encoding for JSON files. From the documentation:

Strings in the Wolfram Language are represented in JSON as UTF-8 strings, escaped as required by the JSON standard.

Strings in Mathematica have sort of a dual function. They can be used to store:

  • text, which might contain various unicode characters such as "γő"
  • binary data (including the null character), if they only have character codes 0-255.

The Import/Export formats to use for these two different purposes are "Text" and "String", respectively.

The result of ExportString is supposed to be binary data that can be literally written into a file (without any transformations) and produce a correct output.

How to fix it? The question is why you want to ExportString JSON?

Is it to stitch it together with other strings, and then finally export to a file? Then make sure all other strings are also encoded into UTF-8 before stitching, and Export as "String".

Ot is it to display to the user? Then decode the UTF-8 by re-importing (ImportString) this string representing binary data as "Text":

raw = ExportString["Conexión", "JSON"];

ImportString[raw, "Text"]
(* "\"Conexión\"" *)

An inconsistency The "RawJSON" import/export format behaves differently:

ExportString["ő", "RawJSON"]
(* "\"ő\"" *)

This inconsistency is a bit disturbing because it means that we cannot treat the output of JSON and RawJSON the same way. Exporting this result as "String" would give a text file containing the ASCII characters "o''" and not "ő".

$\endgroup$
  • $\begingroup$ Wolfram Support confirmed by email it is a bug, and the problem here is that the string I'm trying to encode is supposed to be UTF-8 already. Also, trying to encode my original rule as RawJSON throws an "Expression Rule cannot be exported as RawJSON" error, so that's not the solution either. $\endgroup$ – R. Kazeno Sep 6 '15 at 3:45
  • 2
    $\begingroup$ @R.Kazeno The string you are exporting has non-ASCII characters. How Mathematica stores these internally should not ever be of concern or even visible to the user. (I believe Mathematica uses UCS-2 internally, or at least it used to in the past.) Thus saying that it is UTF-8 already doesn't make sense. The encoding only comes into play once you export it to something. I do believe that the support representative who confirmed it as a bug is mistaken, but it is a good thing that they were alerted to this discrepancy between JSON and RawJSON anyway. $\endgroup$ – Szabolcs Sep 6 '15 at 10:46
  • $\begingroup$ @R.Kazeno Please check the documentation on RawJSON. It expected associations, not lists of rules. $\endgroup$ – Szabolcs Sep 6 '15 at 10:46
  • $\begingroup$ I never considered that Mathematica could be using a different encoding internally, thanks for the insight. Also you're right on the RawJSON export, ExportString[<|"1" -> "Conexión"|>, "RawJSON"] does work without issues. I'm not accepting your answer just based on the technicality that the question was about a list of rules, not an association; however it is very useful, so thank you very much. $\endgroup$ – R. Kazeno Sep 7 '15 at 5:49
  • 2
    $\begingroup$ The workarounds are solving a slightly different problem, which is how to decode the UTF-8 they get from producing JSON output. In a similar vein you might as well ask "why is the output from ExportString[expr, "PNG"] not showing me a picture?" The answer is because the string you get back actually stores the binary for the PNG, which requires decoding from something that understand the PNG serialization format to render the image. $\endgroup$ – Joel Klein Oct 1 '15 at 19:04
2
$\begingroup$

This is unfortunately not an answer, but an extended comment that outgrew the comment form. In short, this might be a bug in ExportString.

I am using Mathematica v. 10.1.0 on Windows 7 - 64 bit. Here's what I tried so far.


I tried to export your JSON content to a file, using the same format you proposed. This works just fine.

Export["fromExport.json", {"1" -> "Conexión"}, "JSON"]
Import["fromExport.json", "JSON"]

(* Out: {"1" -> "Conexión"} *)

The JSON file generated above also contains the same information:

{
    "1": "Conexión"
}

I tried other strings, and other diacritics: no problems here.


Although I have typically found Export and ExportString to behave very similarly, in this case ExportString fails as you showed on my system as well.

It doesn't seem to be a front-end problem either (i.e. the output is correct, but the front end messes up when formatting it for display). I tried to write the generated string out to a file, and it's still messed up:

stringout = ExportString[{"1" -> "Conexión"}, "JSON"];

filestream = OpenWrite["fromWriteString.json"];
WriteString[filestream, stringout];
Close[filestream];

Import["fromWriteString.json", "JSON"]

(* Out: Import::fmterr: Cannot import data as JSON format. *)

As you said, the JSON string generated is invalid. For completeness, on my system the fromWriteString.json file contains:

{
    "1": "ConexiA~\263n"
}

Hope this helps!

$\endgroup$
  • $\begingroup$ After you Export to JSON, try doing BinaryReadList on the file and comparing the bytes you see with ToCharacterCode on the string coming back from ExportString. Except for a possible difference in newlines, they should be identical. The reason Import of the Exported JSON string "works" is that it is converting back from UTF-8. @Szabolcs articulate answer explains what is happening and why it is correct behavior. In this case, what you get from ExportString is essentially a string representing a byte sequence, so what you see looks "messed up", as it shows the bare UTF-8 representation. $\endgroup$ – Joel Klein Oct 1 '15 at 18:39
  • $\begingroup$ You could also do a FilePrint on the Exported file; it will show identical results to ExportString. $\endgroup$ – Joel Klein Oct 1 '15 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.