I have a massive, (mostly) static dataset as an Association that I want to be able to access parts of without importing the whole thing as that's slow and eats up my memory. Is there a convenient way to do this?


1 Answer 1


This is exactly what the "WDX" format appears to have been built for. Even if WDX's days are numbered it's still a good tool for this job, as Mathematica supports it natively and it's, overall, reasonably efficient.

We'll take advantage of the "DataTable" format for WDX as detailed here and I'll just provide some read sugar for it.

First we generate our "big" dataset:

hugeDS =
   RandomReal[1, 1000] &,
   intToAlpha /@ Range[1000]
hugeDS // ByteCount

(* 9461936 *)

Then we export it as described in the previous link:

  $wdxFile =
      "Keys" -> Keys@hugeDS,
      "Properties" -> {"Data"},
      "Data" ->
  ] // First

(* 8.85185 *)

This export time is why it pays to have a largely static dataset, as WDX is not built to be edited conveniently.

Then we build a WDX importer which will allow us to just access certain properties of it:

wdxImporter[wdx_] :=
 Module[{wdxRead, wdxIndex},
  ClearAttributes[{wdxRead, wdxIndex}, Temporary];
  wdxIndex =
   AssociationThread @@
      {"WDX", "DataTable"}
     {"Keys", "Index"}
  wdxRead[Keys] :=
  wdxRead[Values] :=
  wdxRead[All] :=
   AssociationThread[wdxRead[Keys], wdxRead[Values]];
  wdxRead[bit_?(KeyMemberQ[wdxIndex, #] &), 
    keepOpen : True | False : False] :=

   With[{strm = OpenRead[wdx, BinaryFormat -> True]},
    SetStreamPosition[strm, wdxIndex[bit]];
       If[Not@keepOpen, Close[strm]]; #
       ) &@System`Convert`WDXDump`readWDXObject@strm
  wdxRead[bits : {__?(KeyMemberQ[wdxIndex, #] &)}, 
    keepOpen : True | False : False] :=

   With[{strm = OpenRead[wdx, BinaryFormat -> True]},
       If[Not@keepOpen, Close[strm]]; #
       ) &@
      SetStreamPosition[strm, wdxIndex[b]];
      {b, bits}

The trick here is that the WDX "Index" describes the StreamPosition where the element starts and the next byte describes what type it is and then the next bytes are the data. But System`Convert`WDXDump`readWDXObject handles the actual import. We just need to set the position.

This factory returns an import function which we can use to access elements from the stream:

read = wdxImporter[$wdxFile]
(* wdxRead$352020 *)

read["AAA"] // Length
(* 1000 *)

And notice that we actually gained some memory efficiency by using WDX:

$wdxFile // FileByteCount
(* 9022326 *)

hugeDS // ByteCount
(* 9461936 *)

And if you have lots of duplicate elements you'll gain even more, as WDX is configured to remove duplicates by default.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.