Hosting a curated data function / server in Google Drive

Question

I want to make a curated data function, but I have no good place to store the paclets.

Per the pricing page I only get:

And that's really not enough to host all of my data plus give me room to do stuff.

How can I get around this? Could I, for instance, host my paclet in Google Drive?

Answer 1

The trick here is that we'll host our paclets in Google Drive, but deploy a paclet server in the Wolfram Cloud that will route to the download links.

In doing so we get the full 15GB of Google Drive space, but don't have to find a kludge (okay maybe this is a bit of a kludge) for the fact that Google Drive hosts its files as https://drive.google.com/.../fileID.

Just for the sake of argument I'll use this dataset https://datasource.kapsarc.org/explore/dataset/us-vehicle-fuel-economy-data-1984-2017/export/ as it was the first one I found (after some clicking around) after Googling "large datasets"

We'll do this in ~3 steps. First we'll clean the data into an EntityStore, then we'll deploy the .paclet files to Google Drive or the server of our choice, and finally we'll create a server in the Wolfram Cloud that serves the download links for these paclets.

I share my code for all this at the end.

EntityStore

I'll prep this as an EntityStore because that's the basic format a curated data function uses

I downloaded the CSV, but it was in a semi-colon separated format, so I converted that to TSV:

$raw = "~/Desktop/us-vehicle-fuel-economy-data-1984-2017.csv";
$cleaner =
  Export[
   "~/Desktop/us-vehicle-fuel-economy-data-1984-2017_CLEANER.tsv",
   StringReplace[
    Import[$raw, "Text"],
    ";" -> "\t"
    ],
   "Text"
   ];
$imp = Import[$cleaner];

Then we'll clean up the keys so they're camel case and thread these over the properties:

$keys =
  Capitalize /@
   Nest[
    StringReplace[#, {
       "_" ~~ d_ :>
        Capitalize@d,
       "Dscr" -> "Description",
       "atv" -> "ATV",
       "co2" -> "CO2",
       "Gpm" -> "GPM",
       "mpg" -> "MPG",
       "mfr" -> "MFR",
       "ghg" -> "GHG",
       "evM" -> "EVM",
       "Id" -> "ID",
       "id" -> "ID"
       }] &,
    $imp[[1]],
    2
    ];
$props =
  AssociationMap[
   <|
     "Label" ->
      deCamelCase@#
     |> &,
   $keys
   ];

Then thread construct the entity store:

$ents =
  AssociationThread[$keys, #] & /@ Rest@$imp;

$dataset =
  "FuelEconomyVehicle" ->
   <|
    "Entities" ->
     Association[
      #Make <> ToString@#Model <> ToString@#Year -> # & /@ $ents
      ],
    "Properties" ->
     $props
    |>;

Upload Paclets

With this in place I have some code that will automatically construct a set of paclets for this based on the set-up I laid out in Can I build my own curated data function?.

$dataPaclets =
  CuratedDataExport[
   $es
   ];

This just builds the requisite .paclet files. If you do your chunking right you're likely to have numerous. You want big enough chunks that you're not constantly serving new data, but not so big it takes a which to send. It seems WRI aims for about 5 MB. I over-chunked and ended up with 57 paclets in total.

Upload to Google Drive

Then we can do one of two things. Write a Google Drive API connection or add the paclets to drive by hand and copy the links. The latter is undoubtedly the quicker, cleaner way. I went the former, then hooked it into my set of paclet tools.

Note that theoretically this should be easy in Mathematica with Dropbox as there is the Dropbox service connection. That's probably the best route, but it's not the one I took:

$pacletLinks =
  PacletAPIUpload[#] & /@

   Flatten@Values@$dataPaclets["FuelEconomyVehicle"];

This just uploads the .paclet files, makes them publicly accessible, and returns the download links. Here's a sample output:

PacletManager`Paclet[
 "Name" -> "FuelEconomyVehicleData", "Version" -> "1.0.0", 
  "Extensions" -> {{
    "Kernel", Root -> ".", Context -> "FuelEconomyVehicleData`"}, {
    "AutoCompletionData", 
     "Root" -> "AutoCompletionData"}}] -> \
"https://drive.google.com/uc?id=0BwAKizeHn8kqekJIN0xBZkc2VXc&export=\
download"

I collect the Paclet expressions so I can easily build an automatic PacletSite file.

Create Server in Wolfram Cloud

Then all we do is create a paclet server in the Wolfram cloud where each of those links is transformed into:

CloudDeploy@HTTPRedirect[link]

And it's given the name <paclet_name>-<paclet_version>.paclet so that the paclet manager thinks it's a real paclet.

Note that any collection of links will do, as long as they download a paclet file (what they download does not have to have the .paclet extension as the PacletManager will download it as <paclet_name>-<paclet_version>.paclet)

We also need to upload the appropriate PacletSite.mz file, which will just store all those Paclet expressions from before. I have code to do this automatically:

$pacletUploads =
  PacletUpload[
   $pacletLinks,
   "UploadSiteFile" -> True
   ];

The output is just a bunch of an association where each redirect is a CloudObject

It also uploads that PacletSite.mz file.

Install and Test

Then all we need is to install the appropriate paclet site:

In[60]:= $pacletSite =
 ReplacePart[#,
     "Path" -> Drop[#Path, -1]
     ] &@
   URLParse[
    $pacletUploads["PacletSiteFile"] // First
    ] // URLBuild

Out[60]= "http://www.wolframcloud.com/objects/user-e4d1d43a-267f-4924-\
934a-2ba2321519a9/paclets/FuelEconomyVehicleData"

PacletSiteAdd[
  $pacletSite,
  "Fuel economy paclet"
  ];

And we can install and use our paclet:

In[61]:= << FuelEconomyVehicleData`

In[62]:= FuelEconomyVehicleData["BMWi82014"]

Out[62]= Entity["FuelEconomyVehicle", "BMWi82014"]

In[63]:= FuelEconomyVehicleData["BMWi82014", "CO2"]

Out[63]= 198

I have no idea what this data does, but it's big and I don't have to host it all locally, so our job here is done.

Code

If you want my cloud notebook here it is: https://www.wolframcloud.com/objects/user-affd7b1c-ecb6-4ccc-8cc4-4d107e2bf04a/curated_data_example

If you want the paclet that implements all of this here it is (I'm not sharing my Google APIs client ID and client secret in there, but if you want to use the API, let me know, and I'll show you how to set it up for yourself)

"http://www.wolframcloud.com/objects/user-e4d1d43a-267f-4924-934a-2ba2321519a9/paclets/CuratedDataExporter/Paclets/CuratedDataExporter-1.0.0.paclet" // PacletInstall