Against my better judgement, and in the interest of speedy I/O, I've decided to use .mx files for a project - limitations and all. I'm dealing with a few gigabyte-sized datasets that I need in core for reasonably efficient computations and, despite other options, the ol .mx remains my best option (with some hand-rolled HDF5 coming in a close second).

I try to be reasonably clever and name the file with the name of the symbol or context I am exporting. Despite my cleverness I occasionally dump things into the Global' context and then misname the file.

I've perused the always insightful work of @MrWizard on the subject, especially here but I'm still adrift in a sea of Definition when it comes to this particular problem.

For a while, I figured that looking what was defined in Global' before and after, then comparing, would be of assistance, but, if the symbol is already defined (and subsequently gets clobbered) it means I'm out of luck there.

I feel like I'm missing something pretty obvious here - or not.

  • 1
    $\begingroup$ You could load your mx file in another (temporary) context, like described e.g.here, and then analyze names created in that context (and then perhaps remove them all). B.t.w., for the datasets of the size you mention, you could try using undocumented Streaming functionality, which is also based on mx files, but where lots of things has been already automated (I hope that Streaming will soon become officially part of Mathematica). $\endgroup$ Feb 10, 2017 at 16:01
  • $\begingroup$ Quick comment here, if I look at a hexdump of the file I can see the symbol names of known .mx dumps... it just doesn't seem to be at a 'nice' location in the file, like a header or something. It's always about ½ way through the file somewhere. $\endgroup$
    – flip
    Feb 10, 2017 at 16:02
  • 3
    $\begingroup$ @flip If you use MX, try not to use DumpSave. DumpSave (and Save) will save definitions, and that can be pretty inconvenient, as you discovered. Use Export instead. Thus, instead of a=Range[100]; DumpSave["foo.mx", a], then Get["foo.mx"] and thinking, "What was the name of that symbol? a? b? Somthing else?", just use Export["foo.mx", a] and a = Import["foo.mx"]. This won't save the symbol name and definition, it just exports the expression assigned to the symbol. Then you can re-assign to whatever symbol you like upon loading the file. $\endgroup$
    – Szabolcs
    Feb 10, 2017 at 16:08
  • 2
    $\begingroup$ @Szabolcs Actually, one thing to keep in mind regarding using Import / Export to store the data is that it is several times slower than using something like getData = Function[file, Block[{data}, Get[file];data]] and putData = Function[{file, dt}, Block[{data = dt}, DumpSave[file, data]]], where data is some symbol (typically living in a package's private part, so that Block isn't too dangerous to use). $\endgroup$ Feb 10, 2017 at 16:14
  • 3
    $\begingroup$ Possible duplicates: (2900), (25027) $\endgroup$
    – Mr.Wizard
    Feb 11, 2017 at 0:40


Browse other questions tagged or ask your own question.