Seems like there are a lot of useful functions in this undocumented package AWSLink:

Any developers on here that can demo it a bit?

• They seem to all have usage messages. Use ?AWSLink* and keep clicking on the symbol names. – Szabolcs Mar 15 '18 at 17:09
• @Szabolcs yes, but a tutorial would be nice! – user5601 Mar 15 '18 at 17:46
• That was a hint that if you're interested in the topic, you can start experimenting with guidance from the usage messages. Once you start to get comfortable with the package, you can share a few examples with the rest of us. – Szabolcs Mar 15 '18 at 19:27
• @ericwilliamsmith @murta Would you be able to add a new answer to show how the s3 functionalities in ExternalStorage work in 12.1? – M.R. Apr 12 at 23:08

I haven't used these submarined functions for EC2, but I have found the S3 functionalities in this package to be extremely useful. Since there is currently no documentation in v11.3, I thought at the very least I could share a brief guide to the most important operations.

Setup AWS CLI

AWSLink uses the AWS command line tool under the hood, so you need install the cli and configure it. Just follow the linked tutorial, which helps you set up your ~/.aws/credentials file and aws-mfs to authenticate (if you use multi-factor auth).

Once you are able to use aws on the command line like this:

aws s3 ls s3://your_bucket/path_to_your_data  # to list bucket contents

you will be good to go.

Package Features

There are three main classes of functions in the package: general functionality (symbols like AWS*), operators for Elastic Compute Cloud (EC2*), and those relating to Simple Storage Service (S3*).

Connecting

To get started, use AWSClientConnect to create a connection object for one of the two supported services you want to use (lowercase "ec2" or "s3"):

Create buckets

Given the client object, you can create a bucket:

S3CreateBucket[c, "datascience"]

List all buckets

You can list all top-level buckets:

S3DescribeBuckets[c]

Inspect buckets

Say you have a specific top-level bucket named s3://datascience, you can retrieve a nested Association describing its contents

ds = S3Bucket[c, "datascience"]
S3DescribeBucketContent @ ds

or get a summary of bucket properties {"CreationDate", "BucketName", "Owner", "Region"} with Dataset[ds].

Access sub-directories

Some important AWSLink symbols return inert objects that meant to be used as arguments to other functions that page through or act on them lazily. One such function is AWSCollectionReadList, which we use to inspect folders within a bucket.

For example, to see the files inside s3://datascience/features/example1/

S3BucketObjects[ds, Prefix -> "features/example1/"]]

which returns a list of S3Objects. In this example, there were four files in that folder. If I didn't use the StartAfter option we would have gotten five things in the list. AWSCollectionReadList always returns the bucket folder (which is really just a key itself) as the first S3Object in the list.

After using AWSCollectionReadList you get a list of S3Object's. To pull those objects down use S3DownloadAsynchronous, which works like so:

For example,

Uploate to a bucket

To upload files to various paths in a specific bucket use S3UploadAsynchronous. For example, you can upload a single file

{"features/example2/your_file.tsv.gz.h5"}]

or upload a folder of thousands of images: