When I read Wolfram's post about the new features in MMA 12.2 I was intrigued by RemoteBatchSubmit. He wrote:
I have only ever used MMA from my laptop, and never used any other cloud computing or high-performance computing service. I've always thought of these as a service for really heavy computations fancy scientists (or companies) do.
Now, looking into the documentation, the user guide to set up AWS (Amazon Web Services), confused users in Wolfram community, and the hundreds of product versions/tiers/pricings for AWS computing services, not to mention that apparently one also needs something called "Wolfram Service Credits" to do this!??...
I wonder: could anybody here who has experience on the matter and a good understanding of the nuances, explain in simple terms (Edits: I've been adding a couple of additional questions to this list, plus some more "extra" thoughts/doubts at the end.):
What is needed to send a computation to RemoteBatchSubmit? (Besides my code, internet, and the basic free AWS account) I.e.: Do I need some paid AWS tier or the "free tier" services (I think is 750 computing hours per month in something called EC2) is "enough" for some decent computing speed-up? I know this depends obviously on the specific task, but think of some program that takes overnight in a "regular" laptop: would it be worth to send this to AWS batch? Or is it more cost-hassle effective only with some seriously crunchy computations (CERN-level folks? haha).
What are Wolfram Service Credits and how do they work in this context? I.e. Would I need to know beforehand how many credits to buy for a specific computation? In the possible issues section of the documentation, it seems that one possible issue is that such credits can run-off. How am I supposed to know how many to buy beforehand? How are these credits related to the credits or payments to AWS services?
How I am supposed to know, if I've never done something like this, how many "nodes", "cores", "region", whether to request vCPUs, GPUs, and other seemingly important details, when sending my batch job?
If you could write some example(s), step by step, but in plain language, that would be great.
Once you submit a job to the cloud, I imagine you can turn your computer off or whatever. How do you retrieve the results later? Say, I quit Mathematica: from a new (blank) notebook, can I retrieve the results from a computation I sent from another notebook a previous day, or do I need the notebook where I sent the job from? Can you get a notification when it is done or you have to check the status over time?
From a single interactive session in a notebook, can I launch many different jobs? If so, do I need to make a new environment for each one, or just create independent jobs using the same environment? (e.g., using a single
envas the environment, can I then create
job1 = RemoteBatchSubmit[env, code1],
job2 = RemoteBatchSubmit[env, code2], etc?
Are the variables, data, functions, and other definitions of the local notebook sent/carried onto the job submission, or do I have to define everything inside the job
For context, I make a bunch of computations on a decent desktop, but some of these take overnight; some even multiple days. The computations themselves are nothing fancy: some computation on large networks. The "output" is a .mx file of about 100MB, so nothing that requires too much storage-memory either.
Wolfram's blog seemed to suggest it is quite simple to just send these jobs for batch processing, is it really that easy? Consider folks like me, who ignore many of the technical terms of cloud/high-performance computing. The other questions I've seen are more specific than what I'm looking for; I'm looking for an intuitive, introductory, yet complete explanation on how to get started. Let's unleash the power of MMA's high-performance computing to the masses! XD
P.S. Edit 1: If you do give some example or share your experience with some project(s) you've done, it would be great if you could share some estimate on how much ($) did you spend, whether with AWS or Wolfram Credits (I still don't know whether I'd need both types of "credits").
P.S. Edit 2: I know that people somehow have set up Wolfram Engine running in Jupyter, to then use AWS for cluster/batch computing. But here, I'm specifically asking how to use
RemoteBatchSubmit from your run-of-the-mill Wolfram notebook.
P.S. Edit 3: Interested to know whether any type of code would benefit from sending it to the cloud. For instance: is it a big
Table-like worth sending to the cloud, or only things that can be run in parallel (e.g.
ParallelTable) would be faster/more convenient. Related to this: when sending batch jobs to the cloud, should I code things differently than what I do in my laptop? (if not using GPU/parallelization). Finally: how do we install packages in the cloud service? I'm using IGraphM a lot in my computations, and as I understand AWS will run on the Wolfram Engine; how do I get to install/use a specific package there? Do I need to install a package every time I make an environment/new-session?
P.S. Edit 4: I was consistently getting the error mentioned here. It seems one has to "Switching locations on AWS, then creating a new BatchComputeEnvironment, solved the issue".