# Export dataset to JSON including Unicode Characters

I'm experiencing a weird issue. Consider a dataset defined like this:

dataset = Dataset[{<|"a" -> 1, "b" -> 2|>, <|"a" -> 3, "b" -> 4|>}];


Now I would expect Export["test.json", dataset] to create a JSON file with these contents:

[{"a": 1, "b": 2}, {"a": 3, "b": 4}].


Export::badval: The element Data contains invalid values. >>.

A workaround is to run

Export["test.json", Normal[Normal[dataset]]]


but that seems merely a workaround. Is there anyone here who knows a neat solution?

-
The JSON doc page says: Export["file.json", expr] exports a list of rules to JSON. Clearly, a Dataset is no such thing. Your "workaround" does the required conversion to this list of rules in a straightforward manner. I wouldn't call it a workaround, as there is no bug to work around. – Sjoerd C. de Vries Aug 13 '14 at 13:27
Export, which is probably a mess of special cases internally, doesn't seem to have been given the full update treatment in V10.0.0. It dosesn't handle the new structured data objects gracefully. We are probably stuck with this for now. Hopefully, it will be fixed in V10.0.1, coming real soon now (this month, according to WRI tech support). – m_goldberg Aug 13 '14 at 13:27
ExportString ? – Sjoerd C. de Vries Aug 13 '14 at 13:32
This won't be fixed for 10.0.1, but it's a high-priority issue, and we'll try tackle it for 10.0.2. – Taliesin Beynon Aug 13 '14 at 17:47
I too would like to see easy Mathematica Export of Dataset objects to formats such as XML, JSON, CSV and others. It will need to address the various kinds of data that can be wrapped in a Dataset. Yes, experts can use Normal and knowledge of Mathematica to fiddle and get a reasonable Export, but the "Automation Principal" suggests this should not be necessary. I appreciate that it's a high-priority issue. – Seth Chandler Aug 18 '14 at 14:16

I ran into the same issue. Dataset and nested Associations have a structure ideally suited for JSON export, but the JSON exporter only supports lists of rules. Any other type of expression triggers the error you see.

### Export

The workaround is to recursively convert all nested associations to lists of rules.

normalAsc[expr_] := expr //. a_Association :> Normal[a]

normalAsc@Normal[dataset]

(* ==> {{"a" -> 1, "b" -> 2}, {"a" -> 3, "b" -> 4}} *)


This can now be exported to JSON.

ExportString[normalAsc@Normal[dataset], "JSON"]
(* "[
{
\"a\": 1,
\"b\": 2
},
{
\"a\": 3,
\"b\": 4
}
]" *)


### Import

After importing JSON, we also need a way to convert it back to a nested association, then a Dataset. This can be done as follows:

Dataset@Replace[json, {r__Rule} :> Association[r], {0, Infinity}]


The key here is to use Replace, which scans the expression from the inside to outside. ReplaceAll would go from the outside to the inside, and once a list of rules has been changed to an Association, it won't be possible to replace inside it any more.

Update: Looks like there's a simpler way, ToAssociation (undocumented).

A while ago I sent a suggestion to WRI support about allowing nested associations and possibly Dataset to be exported to JSON. The more people suggest a feature, the more likely that it'll get implemented. You can consider suggesting it too.

-
Thanks, Szabolcs. This is at least a proper workaround. How should we go about making a feature request to WRI? – Thijs Aug 13 '14 at 14:52
@Thijs I think either here or usual email to support at wolfram.com. I wouldn't expect much unless several people ask for the same though. (At least two of us for now.) – Szabolcs Aug 13 '14 at 15:09
btw, the undocumented ToAssociation can also be used to convert imported json to associations. – R. M. Aug 21 '14 at 14:18

Using Mathematica v10.3

dataset = Dataset[{<|"a" -> 1, "b" -> 2|>, <|"a" -> 3, "b" -> 4|>}];

Export["test.json", dataset]


works without any errors and results in a file with the following, expected structure:

[
{"a": 1, "b": 2},
{"a": 3, "b": 4}
]


To get the Dataset back from this file one can use

dataset = Dataset@Import["test.json", "RawJSON"]

-
It is not working with Unicode, see my answer below – Athanassios Jan 30 at 8:07

In Mathematica 10.3 the answer above is not working with Unicode characters

dataset=Dataset[{<|"a"->1,"b"->2,"\[CapitalLambda]"->3|>,<|"a"->3,"b"->4,"\[CapitalLambda]"->5|>}]

Export["test.json", dataset]


File content is

[
{"a": 1, "b": 2, "\[CapitalLambda]": 3},
{"a": 3, "b": 4, "\[CapitalLambda]": 5}
]


But if you try now

dataset = Dataset@Import["test.json", "RawJSON"]

Import::jsoninvescchar: Unexpected escaped character '[' >>
Import::jsonkvsep: "Missing key value separator ':'.
Import::jsonhintposandchar: An error occurred near character 'C', at line 2:23 >>
DatasetExtractRawData::dataextr: Data extraction failed.
Dataset[\$Failed]


The trick is to export first in "RawJSON" then import the "RawJSON" but Export does not seem to have any option for Unicode. That is why I used ExportString

In:  jstr=ExportString[Normal@dataset, "RawJSON", "Compact"->True]
Out: [{"a":1,"b":2,"\[CapitalLambda]":3},{"a":3,"b":4,"\[CapitalLambda]":5}]

Export["test.txt",jstr]


Now you can get the dataset with

dataset=Dataset@Import["test.txt","RawJSON"]
`
-
I think this post is relevant on the discussion here – Athanassios Jan 29 at 22:18
Please notice that [CapitalLamba] character is Greek lambda (Λ), Unicode: 039B, and the whole point of any similar test like the one above is that after importing a dataset with Unicode strings you should get a Notebook output formatted with your Language encoding, NOT with full names, e.g. [CapitalLambda] – Athanassios Jan 30 at 7:11