Extracting data from Data paclet

Question

In the $UserBasePacletsDirectory there are lots of paclets that have only Data as an element. These are loaded via the DataPaclets` functionality using the standard form:

DataPaclets`ImportData[
 pacletName<>"_"<>dataPart<>".wdx",
 DataPaclets`GetDataPacletResource[pacletName<>"_"<>dataPart, 
  dataSubpart<>".wdx"]
 ]

E.g.:

DataPaclets`ImportData[
 "ChemicalData_Index",
 DataPaclets`GetDataPacletResource[
  "ChemicalData_Index",
  "Properties.wdx"
  ]
 ]

This only works though when the .wdx file has the standard "Expression" part, e.g.:

DataPaclets`ImportData[
 "ChemicalData_Index",
 DataPaclets`GetDataPacletResource[
  "ChemicalData_Index",
  "Index.wdx"
  ]
 ]

this fails.

On the other hand for these indices we can pull stuff like:

DataPaclets`ImportData[
 "ChemicalData_Index",
 DataPaclets`GetDataPacletResource[
  "ChemicalData_Index",
  "Index.wdx"
  ],
 {"DataIndex", "Sources", "Data"}
 ]

Which will give us a long set of data specifying in what part we can find the actual data. For example we'll get something like "Part01" -> { ... } which seems to mean that the data for keys listed in that list can be found via:

DataPaclets`ImportData[
 "ChemicalData_Part01",
 DataPaclets`GetDataPacletResource[
  "ChemicalData_Part01",
  "Part01.wdx"
  ],
 {"DataTable", "Index"} (* Note that now we find the DataTable element *)
 ]

The problem is that the return from this is just a long list of really large ints and I have no idea what they mean and how to convert them to proper data.

I want to make my own Data paclet, but unfortunately I need to know how to export my data to make it work.

Does anyone know how to convert this? I really have no idea what it's supposed to be as the numbers range from around 440k to roughly 6 mil

Answer 1

Data Import

After doing some digging into the WDX export function (System`Convert`WDXDump`ExportWDX), for a "DataTable" one finds that the "Index" property is just the stream positions of a list of elements with duplicate elements removed (done via the "DetectDuplicates" option). Hence the observation that the "Index" is primarily increasing with some discrete jumps.

The objects themselves have varying byte representations coming from System`Convert`WDXDump`writeWDXObject and can be read with System`Convert`WDXDump`readWDXObject.

This means we can write something import the actual WDX data by something like this:

wdx =
  DataPaclets`GetDataPacletResource[
   "ChemicalData_Part01", "Part01.wdx"
   ];
idx =
  DataPaclets`ImportData[
   "ChemicalData_Part01",
   wdx,
   {"DataTable", "Index"}
   ];
read[piece_, n : _Integer : 1] :=

 With[{strm = OpenRead[wdx, BinaryFormat -> True]},
  SetStreamPosition[strm, idx[[piece]]];
  (Close[strm]; #) &@
   Table[System`Convert`WDXDump`readWDXObject@strm, n]
  ]

And if we want the 15-25th expressions in the data underlying (in this case the first chunk of the data underlyingChemicalData) we can get it:

In[439]:= read[15, 10] // NewlineateInput

Out[439]= {
 "B2Be",
 "6.1",
 {1566},
 {},
 {{"Be", 29.419}, {"B", 70.58}},
 {{"Be", 1}, {"B", 2}},
 "EU235-443-0",
 {-1, -1, 2},
 "beryllium boride",
 "Gmelin171277"
 }

This then corresponds to the some stuff in the "Keys" and "Properties" portions of the data.

Seeing as my question was really about the export form that problem will be left to the reader.

Data Export

In terms of export System`Convert`WDXDump`ExportWDX tells us how to set that up. All we need is to specify the "Keys", "Properties" and "Data" parameters. Any missing parameters will default to an empty list. The code suggests that an "Attributes" property should be applicable too as a list of options with string keys. Any way this gets the job done for me:

Export[
 "file.wdx",
 {
  "Keys" ->
    Keys@entityStore["Entities"],
  "Properties" ->
    Thread@List@Keys@entityStore["Properties"],
  "Data" ->
    Normal@Map[Values, Values@entityStore["Entities"]]
  },
 "DataTable"
 ]