25
$\begingroup$

Mathematica has a cross platform data exchange format, WDX. Unfortunately importing or exporting large data from/to WDX is very slow. Using MX files is very fast, but they are not compatible across different computer architectures (32 or 64 bit).

Sometimes it is suggested to Compress the data and write out or read it in manually.

Question: How can we extend Import and Export to allow convenient and fast importing/exporting of arbitrary Mathematica expressions using Compress?

The aim is to define a fast-to-load format, and make importing from it as easy as Import["data.mmaz"] or Import["http://server.com/myfile.mmaz"], by integrating it into the Import/Export framework. Ideally the format should be recognized based on a file extension.

If there is a better solution than using Compress, I'd like to hear it!

Improve this question
$\endgroup$
8
  • $\begingroup$ I'm planning to do this today. I'll probably post an answer in a day or so if I manage. But I'd like to see some better solutions first :-) $\endgroup$
    – Szabolcs
    Feb 3, 2012 at 10:47
  • $\begingroup$ Not an answer for you but I moved all my data from WDX to a MySQL database a couple of years ago. One of the best things I did. $\endgroup$
    – Mike Honeychurch
    Feb 3, 2012 at 12:03
  • $\begingroup$ @Mike I didn't forget about your suggestion, but this question is mainly about how to develop an Import/Export converter. I thought it would be good to have a practical tutorial on this here at Mma.SE $\endgroup$
    – Szabolcs
    Feb 3, 2012 at 12:07
  • $\begingroup$ @Szabolcs, further down you mention something about tutorials, have looked at these tutorial/DevelopingAnExportConverter, tutorial/DevelopingAnImportConverter? $\endgroup$
    – user21
    Feb 3, 2012 at 13:29
  • $\begingroup$ @ruebenko Yes, I am reading those now. This was more of an experiment about this: meta.mathematica.stackexchange.com/q/194/12 But now I think I asked the wrong question. $\endgroup$
    – Szabolcs
    Feb 3, 2012 at 13:35

3 Answers 3

Reset to default
17
$\begingroup$

In this case, developing the converters is dead-easy (which is not a good thing IMO, since it means that we really don't utilize the power of Import/Export framework, but rather are adding syntactic sugar):

CompressedFormat`CompressedFormatImport[filename_String, options___] :=
    {"Data" -> Uncompress@Import[filename, "String"]};

CompressedFormat`CompressedFormatExport[filename_String, data_, opts___] :=
    Export[filename, Compress@data, "String"];

ImportExport`RegisterImport[
   "CompressedFormat",
   CompressedFormat`CompressedFormatImport
]

ImportExport`RegisterExport[
   "CompressedFormat", 
   CompressedFormat`CompressedFormatExport 
]

Example:

file = $TemporaryPrefix <> "test";
Export[file, Range[1000000], "CompressedFormat"];
Import[file, {"CompressedFormat", "Data"}] // Length

(* 
  ==>  1000000
*)

That said, I think using Import - Export framework makes much more sense for specific formats where you can specify distinct elements and the framework makes it convenient to create importers for those elements (possibly avoiding full imports when unnecessary). So, for a meaningful exposition of the importer-writing procedure using Import/Export framework, some e.g. particular graphics of numerical format would be a better choice IMO, because your stated goal is too general for that.

For that matter, I think that my large data framework (perhaps when extended and generalized) will make for a much better case for Import/Export framework use, as well as cover your use case and many more, because it:

  • Does use Compress under the cover
  • Uses lazy loading, which opens many possibilities to define certain elements for Import/Export, which are loaded individually / efficiently
  • Does not have a limitation that the file must fit in memory
  • Can be very fast for large files
  • In practice, we use large files much more frequently than carry them around from platform to platform. My framework can switch from extremely fast .mx files to Compress-ed non-.mx files very easily, and the details can be completely hidden from the user, who will just use Import in all cases, and have great performance.

In other words, I feel that the direction I outlined there, does contain your suggestion as a special case, and is much more fruitful both for further development of the large-data framework / file format, and for the utilization of the power of the Import/Export framework (and, sure enough, this is the direction I will be extending the large-data framework in the future).

Improve this answer
$\endgroup$
9
  • $\begingroup$ @Szabolcs I did benchmark that before posting - Import was 3 times faster. As I said, you are mixing two questions: essential question on loading data fast and cross-platform, with a question of using Import - Export, which in this case is IMO more like syntactic sugar (because your format does not have any complex structure). I was mostly answering the Import/Export part, and did not intend to fully answer the loading / saving part (for reasons I outlined). Will look into automatic file extension resolution. Feel free to edit and add benchmarks and whatever else you'd like. $\endgroup$
    – Leonid Shifrin
    Feb 3, 2012 at 13:44
  • $\begingroup$ Yes, you are right, the question was not well formulated. Now it feels like I should not have asked, epsecially since the tutorials in the documentation are good enough. $\endgroup$
    – Szabolcs
    Feb 3, 2012 at 13:49
  • $\begingroup$ @Szabolcs I think the part regarding the fast cross-platform load/save is a good and valid question - I just think that we'd be better off addressing the more general one by extending the framework I posted. As to the Import/Export, some simple but composite format like some custom tabular data would probably be better, but then there is an example like that in the docs, as you said. $\endgroup$
    – Leonid Shifrin
    Feb 3, 2012 at 14:05
  • $\begingroup$ @Szabolcs I actually think this was a useful experiment, and now I think that the answer to that question of Mr.Wizard is: ask a question in two cases - either when you do know the answer really well, but think that the question and answers would benefit the community, or when you don't, and need it for yourself. Both cases are fine because the asker has put in a lot of thought in either case. Mixing them is not since then this may not be the case. I will post a version of this on meta as another answer, to add to yours. $\endgroup$
    – Leonid Shifrin
    Feb 3, 2012 at 14:10
  • $\begingroup$ I'll clean up the mess later, perhaps European-tonight ... got to finish some work today $\endgroup$
    – Szabolcs
    Feb 3, 2012 at 14:35
12
$\begingroup$

One simple way to store data in compressed form could use the following:

ExportCompressed[filename_,data_]:=
    Export[filename,"Uncompress@"<>"\""<>Compress[data]<>"\"","String"]

This simply compresses and prepends the Uncompress statement to the resulting string. You can now simply use Get[] to import your data.

I use this to store compressed graphics expressions. Compressing can take a long time (I´d like to see that sped up big time, because several minutes for a few MB of graphics expression is way too long), but mostly you get very good compression.

On the other hand, import of these expressions is really fast. This seems kind of related to the WDX performance.

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ Hi Yves, welcome to the party! $\endgroup$
    – user21
    Feb 3, 2012 at 12:26
  • 3
    $\begingroup$ Finally, a question that was not way over my head and/or answered immediately - just what I was looking for to get my foot into the door. $\endgroup$
    – Yves Klett
    Feb 3, 2012 at 12:50
  • $\begingroup$ Hi Yves, welcome to Mathematica.SE! Could you complement this with some code that integrates this into the Import and Export functions? $\endgroup$
    – Szabolcs
    Feb 3, 2012 at 12:56
  • 1
    $\begingroup$ @Szabolcs: If you Import[] such a file, you get a string. You can then use ToExpression to evaluate this and get back your uncompressed. Using Get[] does the evaluation on the spot. I am not sure how to properly splice this transparently into Import and Export. $\endgroup$
    – Yves Klett
    Feb 3, 2012 at 13:18
  • $\begingroup$ @Yves I know that, but directly integrating into the Import/Export framework has many advantages: I don't need to change my code when I change the file format: just pass it different file names. Also, web import will work out of the box. $\endgroup$
    – Szabolcs
    Feb 3, 2012 at 13:21
1
$\begingroup$

A bit late, but the extension can be registered using RegisterFormat from the Wolfram Function Repository

https://resources.wolframcloud.com/FunctionRepository/resources/RegisterFormat

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.