Exporting held expressions through JSON

Question

Trying to round-trip expressions through JSON, I'm getting unexpected errors for held expressions, and would be grateful for advice or clues. Consider, first, something that works well

Export[Environment["USERPROFILE"] <> "\\AppData\\Local\\test.json", {1, 2, 3},"JSON"]

and read it back in

Import[Environment["USERPROFILE"] <> "\\AppData\\Local\\test.json","JSON"]

producing the expected {1, 2, 3}. However, when I try a held expression, such as (and now you can see why I might want to do this):

Export[
  Environment["USERPROFILE"] <> "\\AppData\\Local\\test.json",
  HoldComplete[myList = {1, 2, 3}],
  "JSON"]

and we get

Export::badval: The element Data contains invalid values. >>

I haven't been able to find anything useful on this error message. I suspect it's something to do with HoldComplete and its friends not being real expressions, but rather some kinds of special syntax in the front end or the kernel, but it's a bit surprising since one of the oft-repeated slogans in Mathematica is everything is an expression.

Btw, lest we think that the assignment to myList is the problem, the following fails with the same message:

Export[
  Environment["USERPROFILE"] <> "\\AppData\\Local\\test.json",
  HoldComplete[{1, 2, 3}],
  "JSON"]

Answer 1

The short answer is that, as @FJRA noted in the comment, only certain types are supported. Which types? Enter the long answer.

Why the converter behaves as it does

Long answer: JSON supports only certain types, and their nested combinations, as defined e.g. here. Mathematica converter maps JSON objects to lists of rules, arrays to lists, strings to strings, plus has some special cases for True, False and Null. Once Mathematica JSON converter sees a general expression, it does not know what to do with it.

The problem with the "obvious" solution to convert to string and store as a string is that there will be no automatic way (without imposing some additional conventions) to tell which strings are really strings and which are stringified Mathematica expressions. So, IMO, the converter is doing the right thing.

Digging deeper

You can actually quite easily trace the execution the the functions of interest. If we use my debug function (from here), as

debug@Export[
   Environment["USERPROFILE"] <> "\\AppData\\Local\\test.json", 
   HoldComplete[{1, 2, 3}], "JSON"]

It will quickly tell us to look at the function System`Convert`JSONDump`iexportJSON, which in turn points to System`Convert`JSONDump`toString. Inspecting the DownValues of the latter, you will see the procedure I described above.

Making the JSON import - export more liberal (for illustration purposes only !)

If you really want to make the JSON import - export more liberal, so that, upon seeing an unrecognized general expression, it somehow converts it to string for export, and back to expression during import, here is one way:

ClearAll[withLiberalJsonTostring];
SetAttributes[withLiberalJsonTostring, HoldAll];
withLiberalJsonTostring[code_] :=
   Block[{dv = DownValues[System`Convert`JSONDump`toString], 
         System`Convert`JSONDump`toString},
      DownValues[System`Convert`JSONDump`toString] = Most[dv];
      System`Convert`JSONDump`toString[expr_, _Integer] :=
       StringJoin[
          "StringifiedOpen",
          StringReplace[ToString[FullForm@expr],
            {"[" :> "EscapeOpen", "]" :> "EscapeClose", "," :> "EscapeComma"}],
          "StringifyClose"
       ];
      code];

and the import counterpart:

ClearAll[withLiberalJsonImport];
SetAttributes[withLiberalJsonImport, HoldAll];
withLiberalJsonImport[code_] :=
   With[{result = code},
      result /. 
        s_String :> 
          StringReplace[
             s, 
            {"EscapeOpen" :> "[", "EscapeClose" :> "]", "EscapeComma" :> ","}
          ] /. 
        s_String /; StringMatchQ[s, "StringifiedOpen" ~~ __ ~~ "StringifyClose"] :>
            ToExpression@ StringReplace[s, "StringifiedOpen" | "StringifyClose" :> ""]
   ]; 

Note that the escaping strings are arbitrary, and this will break if these particular strings are also used in different capacities in the JSON expression.

With this, we can do:

withLiberalJsonTostring[
  Export[Environment["USERPROFILE"] <> "\\AppData\\Local\\test.json", 
  {1, 2, HoldComplete[{1, 2, 3}]}, 
  JSON"]]

and

withLiberalJsonImport@
   Import[Environment["USERPROFILE"] <> "\\AppData\\Local\\test.json", "JSON"]

(* 
  ==> {1, 2, HoldComplete[{1, 2, 3}]}
*)

Note that I don't really recommend this method as robust, just posted this code for an illustration, and to aid the understanding of the matter. This is not robust on many grounds, incuding dependence on implementation details, the escaping procedure being arbitrary and not really robust, etc.

A robust solution would be to write an alternative converter (importer / exporter) to JSON, which would do the thing you want, and use that intead. Also, please have a look at the solution by @celtschk, which is a lot cleaner and simpler.

EDIT

As @celtschk pointed out in the comments, escaping is not really necessary if we add extra string quatation marks. The mechanism to distinguish strings from stringified expressions (to be converted back to expressions during import) is still needed however.

Answer 2

The reason is that the JSON format is quite limited. It doesn't support arbitrary expressions. You can e.g. see that by trying the folllowing:

Export["test.json",someSymbol,"JSON"]

You'll get the same error.

If your goal is just to pass the expressions around (i.e. on the other side is another Mathematica instance to interpret them), the simplest solution is to package the expression in a string before sending:

Export["test.json",ToString[HoldComplete[myList={1,2,3}],InputForm],"JSON"]

and translating that string back into an expression on receiving:

ToExpression@Import["test.json","JSON"]
(*
==> HoldComplete[myList={1,2,3}]
*)

Note that I used InputForm in order to enable passing expressions which are normally represented specially. For example, Graph[{1->2,2->3}] could not be passed through otherwise.

If you want to distinguish between Mathematica expressions and other stuff, you can do the following:

Export["test.json",{"MathematicaExpression"->ToString[HoldComplete[myList={1,2,3}],InputForm]},"JSON"]

This will produce the following JSON file:

{"MathematicaExpression" : "HoldComplete[myList = {1, 2, 3}]"}

You can then read it in with

ToExpression@("MathematicaExpression"/.Import["test.json","JSON"])

You can of course first check that the Import resulted in something matching { "MathematicaExpression -> x_String }. The following will give the Mathematica expression if a Mathematica expression was explicitly passed using that mechanism, and whatever Import resulted in otherwise:

Import["test.json","JSON"]/.{{"MathematicaExpression"->x_String}:>ToExpression[x]}