# Hosting a curated data function / server in Google Drive

I want to make a curated data function, but I have no good place to store the paclets.

Per the pricing page I only get:

And that's really not enough to host all of my data plus give me room to do stuff.

How can I get around this? Could I, for instance, host my paclet in Google Drive?

The trick here is that we'll host our paclets in Google Drive, but deploy a paclet server in the Wolfram Cloud that will route to the download links.

In doing so we get the full 15GB of Google Drive space, but don't have to find a kludge (okay maybe this is a bit of a kludge) for the fact that Google Drive hosts its files as https://drive.google.com/.../fileID.

Just for the sake of argument I'll use this dataset https://datasource.kapsarc.org/explore/dataset/us-vehicle-fuel-economy-data-1984-2017/export/ as it was the first one I found (after some clicking around) after Googling "large datasets"

We'll do this in ~3 steps. First we'll clean the data into an EntityStore, then we'll deploy the .paclet files to Google Drive or the server of our choice, and finally we'll create a server in the Wolfram Cloud that serves the download links for these paclets.

I share my code for all this at the end.

### EntityStore

I'll prep this as an EntityStore because that's the basic format a curated data function uses

I downloaded the CSV, but it was in a semi-colon separated format, so I converted that to TSV:

$raw = "~/Desktop/us-vehicle-fuel-economy-data-1984-2017.csv";$cleaner =
Export[
"~/Desktop/us-vehicle-fuel-economy-data-1984-2017_CLEANER.tsv",
StringReplace[
Import[$raw, "Text"], ";" -> "\t" ], "Text" ];$imp = Import[$cleaner];  Then we'll clean up the keys so they're camel case and thread these over the properties: $keys =
Capitalize /@
Nest[
StringReplace[#, {
"_" ~~ d_ :>
Capitalize@d,
"Dscr" -> "Description",
"atv" -> "ATV",
"co2" -> "CO2",
"Gpm" -> "GPM",
"mpg" -> "MPG",
"mfr" -> "MFR",
"ghg" -> "GHG",
"evM" -> "EVM",
"Id" -> "ID",
"id" -> "ID"
}] &,
$imp[[1]], 2 ];$props =
AssociationMap[
<|
"Label" ->
deCamelCase@#
|> &,
$keys ];  Then thread construct the entity store: $ents =
AssociationThread[$keys, #] & /@ Rest@$imp;

$dataset = "FuelEconomyVehicle" -> <| "Entities" -> Association[ #Make <> ToString@#Model <> ToString@#Year -> # & /@$ents
],
"Properties" ->
$props |>;  ### Upload Paclets With this in place I have some code that will automatically construct a set of paclets for this based on the set-up I laid out in Can I build my own curated data function?. $dataPaclets =
CuratedDataExport[

$pacletLinks, "UploadSiteFile" -> True ];  The output is just a bunch of an association where each redirect is a CloudObject It also uploads that PacletSite.mz file. # Install and Test Then all we need is to install the appropriate paclet site: In[60]:=$pacletSite =
ReplacePart[#,
"Path" -> Drop[#Path, -1]
] &@
URLParse[
$pacletUploads["PacletSiteFile"] // First ] // URLBuild Out[60]= "http://www.wolframcloud.com/objects/user-e4d1d43a-267f-4924-\ 934a-2ba2321519a9/paclets/FuelEconomyVehicleData" PacletSiteAdd[$pacletSite,
"Fuel economy paclet"
];


And we can install and use our paclet:

In[61]:= << FuelEconomyVehicleData

In[62]:= FuelEconomyVehicleData["BMWi82014"]

Out[62]= Entity["FuelEconomyVehicle", "BMWi82014"]

In[63]:= FuelEconomyVehicleData["BMWi82014", "CO2"]

Out[63]= 198


I have no idea what this data does, but it's big and I don't have to host it all locally, so our job here is done.

### Code

If you want my cloud notebook here it is: https://www.wolframcloud.com/objects/user-affd7b1c-ecb6-4ccc-8cc4-4d107e2bf04a/curated_data_example

If you want the paclet that implements all of this here it is (I'm not sharing my Google APIs client ID and client secret in there, but if you want to use the API, let me know, and I'll show you how to set it up for yourself)

"http://www.wolframcloud.com/objects/user-e4d1d43a-267f-4924-934a-2ba2321519a9/paclets/CuratedDataExporter/Paclets/CuratedDataExporter-1.0.0.paclet" // PacletInstall
`
• Why would you want your dataset in the cloud, except as backup? For access in milliseconds, instead of seconds, consider a large external hard disk for data storage, or even better, a NAS with RAID for error checking and prevention. Here is my FreeNAS setup: SuperMicro X9SCL-F-O mobo, Xeon 1220-V2 CPU, 16 GB Kingston KVR16E11 memory, 3 x Hitachi HDS721010CLA330 1TB, Patriot Memory 16 GB USB 3 stick for O/S. W/O the three hard disks, the FreeNAS cost about \$200 to set up. Enterprise grade hard disks cost more, but worth it. Access times run up to 990 Mbps. – CElliott Jun 1 '17 at 12:00
• @CElliott there are 2 primary reasons. The first is keeping cruft off your machine. Mathematica's ElementedData datasets and stuff are really big so it's nice to keep them off your machine. WRI clearly uses this DataPaclets framework to send their data out as chunks. The other is distribution. Doing it like this means others can easily use your data by calling PacletInstall once. – b3m2a1 Jun 1 '17 at 14:00