17
$\begingroup$

I reported this to WRI support. Tracking number is CASE:3368636

Official answer is:

Thank you for contacting Wolfram Technical Support. This is a known issue and our developers are working on a resolution. Thank you once again for taking the time and bringing this issue to our attention and helping us improve Mathematica.

On my Linux (Ubuntu 14.04 LTS 64bit) with Mathematica version 10.1 the following was unexpected to me:

dataset = Dataset[{<|"a" -> 1|>}];
Export["main.wdx", dataset]

Mathematica graphics

The documentation of the WDX format states that

  • Used by the Wolfram Language for storing and exchanging expressions and data.
  • Stores arbitrary Wolfram Language expressions in a serialized, platform-independent form.

This sounds to me like it should work. Can someone try this on a different system and/or share some insight?

Improve this question
$\endgroup$
10
  • $\begingroup$ This is going to be different for 10.2. Regardless, WDX isn't really the best serialization format to use. To serialize a Dataset, I would use MX instead. $\endgroup$ – Leonid Shifrin Jun 23 '15 at 4:44
  • 1
    $\begingroup$ @LeonidShifrin Long story short: I had a big Excel table (not mine) and turned it into a DataSet and I wanted to store it so that I don't have to do this conversion again. I distinctly remember reading somewhere that MX is portable now, but I wasn't sure and I was too lazy to search; the doc of MX explicitly says it's non-portable so I tried WDX. Is the documentation wrong? $\endgroup$ – halirutan♦ Jun 23 '15 at 8:00
  • $\begingroup$ @LeonidShifrin To be clear, I only need to be portable across operating systems, not over different Mathematica versions! $\endgroup$ – halirutan♦ Jun 23 '15 at 8:03
  • 2
    $\begingroup$ @AlexeyPopkov It is obsolete. Since version 10, MX files became de facto cross-platform (although not cross-architecture, so 32-bit and 64-bit mx files are not compatible with each other). $\endgroup$ – Leonid Shifrin Jun 25 '15 at 14:54
  • 1
    $\begingroup$ @AlexeyPopkov I don't know enough about WDX to give a fair comparison, but the main disadvantage is speed (bot sure about cross-architecture, but, frankly, at this point, who cares? 32-bit processors are becoming rare in general-purpose computers). MX is a very fast binary format. WDX is empirically quite slow to Import from / Export to it. Also, while this is a subjective opinion, but right now it looks to me like WDX has no future (not much if any internal developments are using it), while MX seems to have a bright future. $\endgroup$ – Leonid Shifrin Jun 25 '15 at 15:39
17
$\begingroup$

Preface:

For everyone: be aware that this behaviour is very likely to change soon (the name of the hook variable flag, etc). In fact, I did change this for 10.2, where Export would fall back to the standard Export in case when the specialised hook fails. Leonid Shifrin

I get the same behaviour using Mathematica version 10.1 on Windows 7/64-bit.

Dataset has an up-value for altering the behaviour of Export. The printed message PATCH! looks very much like it comes from some left-over debugging code in that definition.

The up-value definition implements custom serialization of dataset objects for the formats listed in the error message, namely these formats:

Dataset;
Dataset`Integration`PackagePrivate`$DatasetExportFormats

(* {"CSV", "TSV", "JSON", "MX", "M"} *)

The absence of WDX from this list is probably an oversight although it is possible that it was omitted on purpose due to some implementation limitation (so beware). The principal dataset-exporting function can be inspected thus:

Dataset;
Needs["GeneralUtilities`"]
PrintDefinitions[Dataset`Integration`PackagePrivate`DatasetString]

As a work-around, we can apparently perform a successful WDX export like this:

Block[{Dataset`$DatasetImportHookEnabled = False}
, dataset = Dataset[{<|"a"->1|>}]
; Export["main.wdx", dataset]
]

This follows the implementation of the dataset-specific MX export. I imagine that WDX will be added to the list of special formats at some point.

All of this information is unofficial and current as of version 10.1.0.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Every time I think I'm getting pretty good at MMA, an answer like this shows up. Impressive knowledge of the system and how to traverse it. (+1) $\endgroup$ – kale Jun 23 '15 at 2:26
  • $\begingroup$ @WReach Thanks for the digging. I already reported it to WRI. $\endgroup$ – halirutan♦ Jun 23 '15 at 2:38
  • 5
    $\begingroup$ +1, nice analysis. For everyone: be aware that this behavior is very likely to change soon (the name of the hook variable flag, etc). In fact, I did change this for 10.2, where Export would fall back to the standard Export in case when the specialized hook fails. $\endgroup$ – Leonid Shifrin Jun 23 '15 at 4:42
  • 2
    $\begingroup$ @halirutan This has been fixed in the sources, although I didn't test specifically WDX. $\endgroup$ – Leonid Shifrin Jun 23 '15 at 4:55
  • $\begingroup$ @Leonid bugs tag? $\endgroup$ – Mr.Wizard Jun 23 '15 at 9:34
10
$\begingroup$

WDX is not a good format. We are likely to deprecate it or entirely replace it with a different implementation that is not backward-compatible (which is obviously problematic). Dataset will never directly support it.

There are candidates for a possible native format for dataset, like Cap'n Proto, HDF5, and a couple others. Or XML. No, just kidding :-)

Improve this answer
$\endgroup$
3
  • 4
    $\begingroup$ If HDF5 is the winner, would this mean that HDF5 support in general will be extended? It would be nice if so. HDF5 is very fast and the format seems to be very general, but at the moment its support in Mathematica is only really good enough for storing numerical arrays. $\endgroup$ – Oleksandr R. Jul 2 '15 at 23:19
  • $\begingroup$ @OleksandrR if HDF5 wins, of course we will! maybe at first there will be an internal API that is not documented. but this is speculation until we decide on a native format for Datasets. $\endgroup$ – Taliesin Beynon Jul 3 '15 at 7:24
  • $\begingroup$ Can you comment on this more? What's the likelihood WDX gets removed completely vs merely not used going forward? I think it's a useful format for things like this and, of course, it underlies things like ElementData. $\endgroup$ – b3m2a1 Jun 16 '17 at 3:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.