Export dataset to JSON including Unicode Characters

Question

I'm experiencing a weird issue. Consider a dataset defined like this:

dataset = Dataset[{<|"a" -> 1, "b" -> 2|>, <|"a" -> 3, "b" -> 4|>}];

Now I would expect Export["test.json", dataset] to create a JSON file with these contents:

[{"a": 1, "b": 2}, {"a": 3, "b": 4}].

Instead, the command returns

Export::badval: The element Data contains invalid values. >>.

A workaround is to run

Export["test.json", Normal[Normal[dataset]]] 

but that seems merely a workaround. Is there anyone here who knows a neat solution?

Answer 1

I ran into the same issue. Dataset and nested Associations have a structure ideally suited for JSON export, but the JSON exporter only supports lists of rules. Any other type of expression triggers the error you see.

Export

The workaround is to recursively convert all nested associations to lists of rules.

normalAsc[expr_] := expr //. a_Association :> Normal[a]

normalAsc@Normal[dataset]

(* ==> {{"a" -> 1, "b" -> 2}, {"a" -> 3, "b" -> 4}} *)

This can now be exported to JSON.

ExportString[normalAsc@Normal[dataset], "JSON"]
(* "[
    {
        \"a\": 1,
        \"b\": 2
    },
    {
        \"a\": 3,
        \"b\": 4
    }
]" *)

Import

After importing JSON, we also need a way to convert it back to a nested association, then a Dataset. This can be done as follows:

Dataset@Replace[json, {r__Rule} :> Association[r], {0, Infinity}]

The key here is to use Replace, which scans the expression from the inside to outside. ReplaceAll would go from the outside to the inside, and once a list of rules has been changed to an Association, it won't be possible to replace inside it any more.

Update: Looks like there's a simpler way, ToAssociation (undocumented).

---

A while ago I sent a suggestion to WRI support about allowing nested associations and possibly Dataset to be exported to JSON. The more people suggest a feature, the more likely that it'll get implemented. You can consider suggesting it too.

Answer 2

Using Mathematica v10.3

dataset = Dataset[{<|"a" -> 1, "b" -> 2|>, <|"a" -> 3, "b" -> 4|>}];

Export["test.json", dataset]

works without any errors and results in a file with the following, expected structure:

[
    {"a": 1, "b": 2},
    {"a": 3, "b": 4}
]

To get the Dataset back from this file one can use

dataset = Dataset@Import["test.json", "RawJSON"]

Answer 3

In Mathematica 10.3 the answer above is not working with Unicode characters

dataset=Dataset[{<|"a"->1,"b"->2,"\[CapitalLambda]"->3|>,<|"a"->3,"b"->4,"\[CapitalLambda]"->5|>}]

Export["test.json", dataset]

File content is

[
     {"a": 1, "b": 2, "\[CapitalLambda]": 3},
     {"a": 3, "b": 4, "\[CapitalLambda]": 5}
]

But if you try now

dataset = Dataset@Import["test.json", "RawJSON"]

Import::jsoninvescchar: Unexpected escaped character '[' >>
Import::jsonkvsep: "Missing key value separator ':'.
Import::jsonhintposandchar: An error occurred near character 'C', at line 2:23 >>
Dataset`ExtractRawData::dataextr: Data extraction failed.
Dataset[$Failed]

The trick is to export first in "RawJSON" then import the "RawJSON" but Export does not seem to have any option for Unicode. That is why I used ExportString

In:  jstr=ExportString[Normal@dataset, "RawJSON", "Compact"->True]
Out: [{"a":1,"b":2,"\[CapitalLambda]":3},{"a":3,"b":4,"\[CapitalLambda]":5}]

Export["test.txt",jstr]

Now you can get the dataset with

dataset=Dataset@Import["test.txt","RawJSON"]