Custom curated data function for multiple datasets

Question

I have a bunch of curated data that I can't practically distribute as a single curated function, but I don't really want to pass it around as a bunch of paclets as it's really all just one big dataset.

How can I get around this?

Answer 1

This is divided into two parts, laying out the wrapper paclet we're gonna use and an example using NIST data at the end.

The paclet with all of the necessary code is at the end along with a link to a cloud notebook that just details everything I did (it won't actually work if you try to evaluate in it -- I hope ).

Wrapper Paclet

We'll essentially build a curated data function that serves up curated data functions and their components.

The basic component our our curated data wrapper will be an association that maps string names to curated data functions.

Let's call that $CuratedDataFunctions. We'll store this in $CuratedData["DataFunctions"] where $CuratedData is the wrapper function we're building.

We'll also create a tracking association to make sure we don't check our load conditions too many times:

If[! AssociationQ@$DataFunctionsLoaded,
  $DataFunctionsLoaded = <||>
  ];

Then we simply build our $CuratedData as a router to our various data functions that will install them if not installed and load them if not loaded, then call the specified function:

$CuratedData[data_?(KeyMemberQ[$DataFunctions, #] &), 
   "DataFunction"] :=
  (
   If[! TrueQ@
      Lookup[$DataFunctionsLoaded, $DataFunctions[data], False],
    With[{functionName = SymbolName[$DataFunctions[data]]},
     If[Length@PacletManager`PacletFind[functionName] === 0,
      PacletManager`PacletInstall[functionName];
      ];
     If[Length@DownValues@Evaluate@$DataFunctions[data] === 0,
      Get[functionName <> "`"]
      ];
     $DataFunctionsLoaded[$DataFunctions[data]] = True
     ]
    ];
   $DataFunctions[data]
   );
$CuratedData[data_?(KeyMemberQ[$DataFunctions, #] &), 
   args___] :=
  $CuratedData[data, "DataFunction"][args];
$CuratedData[] :=
  Values@$DataFunctions;

where $DataFunctions is a symbol storing the values of $CuratedDataFunctions, just to make the DownValues of $CuratedData nicer looking.

Then we'll upload all of our paclets.

Example

I wanted something big with lots of chunks so I'm uploading the data in the JSON files here: https://catalog.data.gov/dataset/nist-computational-chemistry-comparison-and-benchmark-database-srd-101

Pretty much everything I do here now is the same as what is done here except I add in the wrapper paclet

EntityStore

First I built an EntityStore with all 18 JSON tables (see this for details)

Google Drive Paclets

I then exported that entity store to paclets and also added in the wrapper paclet NISTData` which wraps around all the paclets generated from the EntityStore data:

$dataPaclets =
  CuratedDataExport[
   "NISTData",
   nistDataEntityStore
   ];

Upload the paclets to Google Drive:

$pacletLinks =
  PacletAPIUpload[#] & /@

   Cases[$dataPaclets, _String?FileExistsQ, \[Infinity]];

Wolfram Cloud / PacletSite

$pacletUploads =
  PacletUpload[
   $pacletLinks,
   "UploadSiteFile" -> True
   ];

In[28]:= $pacletSite =
 ReplacePart[#,
     "Path" -> Drop[#Path, -1]
     ] &@
   URLParse[
    $pacletUploads["PacletSiteFile"] // First
    ] // URLBuild

Out[28]= "http://www.wolframcloud.com/objects/user-e4d1d43a-267f-4924-\
934a-2ba2321519a9/paclets/NISTData"

Install and Test

Then we install the site and the NISTData` paclet and then that should route to everything else we need:

PacletSiteAdd[
  $pacletSite,
  "NISTData Paclet"
  ];

PacletInstall[
  "NISTData",
  "Site" -> $pacletSite
  ];

Then we'll quit and test this:

In[1]:= << NISTData`

In[2]:= NISTData["BondList", "22"]["Association"]

Out[2]= <|EntityProperty["NISTDataBondList", "Label"] -> "22", 
 EntityProperty["NISTDataBondList", "AtomNumber"] -> {"1", "2", "3", 
   "4", "5", "6", "7", "8"}, 
 EntityProperty["NISTDataBondList", "Charge"] -> {"0", "0", "0", "0", 
   "0", "0", "0", "0"}, 
 EntityProperty["NISTDataBondList", "CASNumber"] -> "22", 
 EntityProperty["NISTDataBondList", "Atom1"] -> {"2", "1", "2", "2", 
   "2", "5", "6", "6"}, 
 EntityProperty["NISTDataBondList", 
   "Atom2"] -> {Missing["NotAvailable"], "3", Missing["NotAvailable"],
    Missing["NotAvailable"], "6", "7", Missing["NotAvailable"], 
   Missing["NotAvailable"]}, 
 EntityProperty["NISTDataBondList", 
   "Atom3"] -> {Missing["NotAvailable"], "4", Missing["NotAvailable"],
    Missing["NotAvailable"], Missing["NotAvailable"], "8", 
   Missing["NotAvailable"], Missing["NotAvailable"]}, 
 EntityProperty["NISTDataBondList", "Atom4"] -> 
  Missing["NotAvailable"], 
 EntityProperty["NISTDataBondList", "Atom5"] -> 
  Missing["NotAvailable"], 
 EntityProperty["NISTDataBondList", "Atom6"] -> 
  Missing["NotAvailable"]|>

And the "BondList" paclet has been installed:

In[5]:= Length@PacletFind["NISTDataBondListData"] > 0

Out[5]= True

But the "References" paclet has not:

In[7]:= Length@PacletFind["NISTDataReferencesData"] > 0

Out[7]= False

But it'll install when needed:

In[15]:= First[NISTData["References"]]["Association"]

Out[15]= <|EntityProperty["NISTDataReferences", "Label"] -> "1", 
 EntityProperty["NISTDataReferences", "Reference"] -> 
  "LR Maxwell, SB Hendricks, VM Mosley \"Electron Diffraction by \
Gases\" J. Chem. Phys. 3, 699, 1935", 
 EntityProperty["NISTDataReferences", "Journal"] -> 
  "J. Chem. Phys., 1935, 3, 699", 
 EntityProperty["NISTDataReferences", "SQUIB"] -> "1935Max/Hen:699", 
 EntityProperty["NISTDataReferences", "DOI"] -> "10.1063/1.1749580", 
 EntityProperty["NISTDataReferences", "Category"] -> 
  Missing["NotAvailable"]|>

And we'll just check that:

In[17]:= Length@PacletFind["NISTDataReferencesData"] > 0

Out[17]= True

This strategy makes it easier to pack more data into fewer symbols and paclets to install.

You can get the paclet that I use to build all of this here:

"http://www.wolframcloud.com/objects/user-e4d1d43a-267f-4924-934a-\
2ba2321519a9/paclets/CuratedDataExporter/Paclets/CuratedDataExporter-\
1.0.0.paclet" // PacletInstall

And here's the link to the notebook containing all of the code