8
$\begingroup$

Seems like there are a lot of useful functions in this undocumented package AWSLink:

enter image description here

Any developers on here that can demo it a bit?

$\endgroup$
  • 2
    $\begingroup$ They seem to all have usage messages. Use ?AWSLink`* and keep clicking on the symbol names. $\endgroup$ – Szabolcs Mar 15 '18 at 17:09
  • 1
    $\begingroup$ @Szabolcs yes, but a tutorial would be nice! $\endgroup$ – user5601 Mar 15 '18 at 17:46
  • 4
    $\begingroup$ That was a hint that if you're interested in the topic, you can start experimenting with guidance from the usage messages. Once you start to get comfortable with the package, you can share a few examples with the rest of us. $\endgroup$ – Szabolcs Mar 15 '18 at 19:27
9
$\begingroup$

I haven't used these submarined functions for EC2, but I have found the S3 functionalities in this package to be extremely useful. Since there is currently no documentation in v11.3, I thought at the very least I could share a brief guide to the most important operations.

Setup AWS CLI

AWSLink uses the AWS command line tool under the hood, so you need install the cli and configure it. Just follow the linked tutorial, which helps you set up your ~/.aws/credentials file and aws-mfs to authenticate (if you use multi-factor auth).

Once you are able to use aws on the command line like this:

aws s3 ls s3://your_bucket/path_to_your_data  # to list bucket contents

you will be good to go.

Package Features

There are three main classes of functions in the package: general functionality (symbols like AWS*), operators for Elastic Compute Cloud (EC2*), and those relating to Simple Storage Service (S3*).

enter image description here

Connecting

To get started, use AWSClientConnect to create a connection object for one of the two supported services you want to use (lowercase "ec2" or "s3"):

enter image description here

Create buckets

Given the client object, you can create a bucket:

S3CreateBucket[c, "datascience"]

List all buckets

You can list all top-level buckets:

S3DescribeBuckets[c]

enter image description here

Inspect buckets

Say you have a specific top-level bucket named s3://datascience, you can retrieve a nested Association describing its contents

ds = S3Bucket[c, "datascience"]
S3DescribeBucketContent @ ds

enter image description here

or get a summary of bucket properties {"CreationDate", "BucketName", "Owner", "Region"} with Dataset[ds].

Access sub-directories

Some important AWSLink symbols return inert objects that meant to be used as arguments to other functions that page through or act on them lazily. One such function is AWSCollectionReadList, which we use to inspect folders within a bucket.

For example, to see the files inside s3://datascience/features/example1/

objs = AWSCollectionReadList[
  S3BucketObjects[ds, Prefix -> "features/example1/"]]

enter image description here

which returns a list of S3Objects. In this example, there were four files in that folder. If I didn't use the StartAfter option we would have gotten five things in the list. AWSCollectionReadList always returns the bucket folder (which is really just a key itself) as the first S3Object in the list.

Download from a bucket

After using AWSCollectionReadList you get a list of S3Object's. To pull those objects down use S3DownloadAsynchronous, which works like so:

enter image description here

For example,

S3DownloadAsynchronous[{objs[[1]] -> File["~/local/path/to/downloaded.tsv.gz"]}]

Uploate to a bucket

To upload a single file to a top-level bucket use S3UploadAsynchronous, this returns an S3TaskObject:

enter image description here

Upload to a specific folder

To upload files to various paths in a specific bucket use S3UploadAsynchronous. For example, you can upload a single file

S3UploadAsynchronous[ds, {File["~/local/path/to/your_file.tsv.gz"]} -> 
{"features/example2/your_file.tsv.gz.h5"}]

or upload a folder of thousands of images:

fns = FileNames["*.jpg", "~/images_to_upload"]
uploads =  File /@ fns -> (("features/example2/" <> FileNameTake[#, -1]) & /@ fns)
Dynamic[AsynchronousTasks[], UpdateInterval -> 3] (* to monitor progress *)
st = S3UploadAsynchronous[ds, uploads]

Note that conveniently, any intermediate directories that don't currently exist will be created for you, but you need to list out each file explicitly, it doesn't do recursive upload on a directory (like cp -R).

Ok, hope this is useful to someone!

$\endgroup$
  • 1
    $\begingroup$ Thank you this is amazing! $\endgroup$ – user5601 Jun 6 '18 at 18:25
  • 2
    $\begingroup$ If the are any other things you’d like to see explained just add a comment $\endgroup$ – M.R. Jun 12 '18 at 21:18
  • 1
    $\begingroup$ @M.R. How to pass --profile in the AWSClientConnect? I have configured the AWS CLI for different profiles (docs.aws.amazon.com/cli/latest/userguide/…). For example this aws s3 ls --profile test works on my terminal and lists the buckets. $\endgroup$ – PlatoManiac Jun 17 '18 at 15:57
  • 1
    $\begingroup$ Heads up for version 12. S3 functions have been moved to ExternalStorage` , which is another unpublished library. There are new function names too so older code will need to be updated. Source: Wolfram support $\endgroup$ – Eric William Smith Feb 26 at 15:53
  • $\begingroup$ @EricWilliamSmith Thanks for the note, btw, can you answer Platomaniac's question above? $\endgroup$ – M.R. Feb 26 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.