11
$\begingroup$

I want to make a curated data function, but I have no good place to store the paclets.

Per the pricing page I only get:

And that's really not enough to host all of my data plus give me room to do stuff.

How can I get around this? Could I, for instance, host my paclet in Google Drive?

$\endgroup$
10
$\begingroup$

The trick here is that we'll host our paclets in Google Drive, but deploy a paclet server in the Wolfram Cloud that will route to the download links.

In doing so we get the full 15GB of Google Drive space, but don't have to find a kludge (okay maybe this is a bit of a kludge) for the fact that Google Drive hosts its files as https://drive.google.com/.../fileID.

Just for the sake of argument I'll use this dataset https://datasource.kapsarc.org/explore/dataset/us-vehicle-fuel-economy-data-1984-2017/export/ as it was the first one I found (after some clicking around) after Googling "large datasets"

We'll do this in ~3 steps. First we'll clean the data into an EntityStore, then we'll deploy the .paclet files to Google Drive or the server of our choice, and finally we'll create a server in the Wolfram Cloud that serves the download links for these paclets.

I share my code for all this at the end.

EntityStore

I'll prep this as an EntityStore because that's the basic format a curated data function uses

I downloaded the CSV, but it was in a semi-colon separated format, so I converted that to TSV:

$raw = "~/Desktop/us-vehicle-fuel-economy-data-1984-2017.csv";
$cleaner =
  Export[
   "~/Desktop/us-vehicle-fuel-economy-data-1984-2017_CLEANER.tsv",
   StringReplace[
    Import[$raw, "Text"],
    ";" -> "\t"
    ],
   "Text"
   ];
$imp = Import[$cleaner];

Then we'll clean up the keys so they're camel case and thread these over the properties:

$keys =
  Capitalize /@
   Nest[
    StringReplace[#, {
       "_" ~~ d_ :>
        Capitalize@d,
       "Dscr" -> "Description",
       "atv" -> "ATV",
       "co2" -> "CO2",
       "Gpm" -> "GPM",
       "mpg" -> "MPG",
       "mfr" -> "MFR",
       "ghg" -> "GHG",
       "evM" -> "EVM",
       "Id" -> "ID",
       "id" -> "ID"
       }] &,
    $imp[[1]],
    2
    ];
$props =
  AssociationMap[
   <|
     "Label" ->
      deCamelCase@#
     |> &,
   $keys
   ];

Then thread construct the entity store:

$ents =
  AssociationThread[$keys, #] & /@ Rest@$imp;

$dataset =
  "FuelEconomyVehicle" ->
   <|
    "Entities" ->
     Association[
      #Make <> ToString@#Model <> ToString@#Year -> # & /@ $ents
      ],
    "Properties" ->
     $props
    |>;

Upload Paclets

With this in place I have some code that will automatically construct a set of paclets for this based on the set-up I laid out in Can I build my own curated data function?.

$dataPaclets =
  CuratedDataExport[
   $es
   ];

This just builds the requisite .paclet files. If you do your chunking right you're likely to have numerous. You want big enough chunks that you're not constantly serving new data, but not so big it takes a which to send. It seems WRI aims for about 5 MB. I over-chunked and ended up with 57 paclets in total.

Upload to Google Drive

Then we can do one of two things. Write a Google Drive API connection or add the paclets to drive by hand and copy the links. The latter is undoubtedly the quicker, cleaner way. I went the former, then hooked it into my set of paclet tools.

Note that theoretically this should be easy in Mathematica with Dropbox as there is the Dropbox service connection. That's probably the best route, but it's not the one I took:

$pacletLinks =
  PacletAPIUpload[#] & /@

   Flatten@Values@$dataPaclets["FuelEconomyVehicle"];

This just uploads the .paclet files, makes them publicly accessible, and returns the download links. Here's a sample output:

PacletManager`Paclet[
 "Name" -> "FuelEconomyVehicleData", "Version" -> "1.0.0", 
  "Extensions" -> {{
    "Kernel", Root -> ".", Context -> "FuelEconomyVehicleData`"}, {
    "AutoCompletionData", 
     "Root" -> "AutoCompletionData"}}] -> \
"https://drive.google.com/uc?id=0BwAKizeHn8kqekJIN0xBZkc2VXc&export=\
download"

I collect the Paclet expressions so I can easily build an automatic PacletSite file.

Create Server in Wolfram Cloud

Then all we do is create a paclet server in the Wolfram cloud where each of those links is transformed into:

CloudDeploy@HTTPRedirect[link]

And it's given the name <paclet_name>-<paclet_version>.paclet so that the paclet manager thinks it's a real paclet.

Note that any collection of links will do, as long as they download a paclet file (what they download does not have to have the .paclet extension as the PacletManager will download it as <paclet_name>-<paclet_version>.paclet)

We also need to upload the appropriate PacletSite.mz file, which will just store all those Paclet expressions from before. I have code to do this automatically:

$pacletUploads =
  PacletUpload[
   $pacletLinks,
   "UploadSiteFile" -> True
   ];

The output is just a bunch of an association where each redirect is a CloudObject

It also uploads that PacletSite.mz file.

Install and Test

Then all we need is to install the appropriate paclet site:

In[60]:= $pacletSite =
 ReplacePart[#,
     "Path" -> Drop[#Path, -1]
     ] &@
   URLParse[
    $pacletUploads["PacletSiteFile"] // First
    ] // URLBuild

Out[60]= "http://www.wolframcloud.com/objects/user-e4d1d43a-267f-4924-\
934a-2ba2321519a9/paclets/FuelEconomyVehicleData"

PacletSiteAdd[
  $pacletSite,
  "Fuel economy paclet"
  ];

And we can install and use our paclet:

In[61]:= << FuelEconomyVehicleData`

In[62]:= FuelEconomyVehicleData["BMWi82014"]

Out[62]= Entity["FuelEconomyVehicle", "BMWi82014"]

In[63]:= FuelEconomyVehicleData["BMWi82014", "CO2"]

Out[63]= 198

I have no idea what this data does, but it's big and I don't have to host it all locally, so our job here is done.

Code

If you want my cloud notebook here it is: https://www.wolframcloud.com/objects/user-affd7b1c-ecb6-4ccc-8cc4-4d107e2bf04a/curated_data_example

If you want the paclet that implements all of this here it is (I'm not sharing my Google APIs client ID and client secret in there, but if you want to use the API, let me know, and I'll show you how to set it up for yourself)

"http://www.wolframcloud.com/objects/user-e4d1d43a-267f-4924-934a-2ba2321519a9/paclets/CuratedDataExporter/Paclets/CuratedDataExporter-1.0.0.paclet" // PacletInstall
$\endgroup$
  • $\begingroup$ Why would you want your dataset in the cloud, except as backup? For access in milliseconds, instead of seconds, consider a large external hard disk for data storage, or even better, a NAS with RAID for error checking and prevention. Here is my FreeNAS setup: SuperMicro X9SCL-F-O mobo, Xeon 1220-V2 CPU, 16 GB Kingston KVR16E11 memory, 3 x Hitachi HDS721010CLA330 1TB, Patriot Memory 16 GB USB 3 stick for O/S. W/O the three hard disks, the FreeNAS cost about $200 to set up. Enterprise grade hard disks cost more, but worth it. Access times run up to 990 Mbps. $\endgroup$ – CElliott Jun 1 '17 at 12:00
  • $\begingroup$ @CElliott there are 2 primary reasons. The first is keeping cruft off your machine. Mathematica's ElementedData datasets and stuff are really big so it's nice to keep them off your machine. WRI clearly uses this DataPaclets framework to send their data out as chunks. The other is distribution. Doing it like this means others can easily use your data by calling PacletInstall once. $\endgroup$ – b3m2a1 Jun 1 '17 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.