20
$\begingroup$

I have spent a lot of time lately training neural networks. However, my poor laptop has only an AMD GPU, so I am stuck training these networks on the CPU. That means I get to train networks over multiple days only to learn that I made a mistake in my loss function.

I don't have enough money to build a new computer or buy an eGPU, so it seems that my best solution is to train these networks on a cloud provider instead.

There are a lot of options nowadays for these cloud GPUS - AWS, Paperspace, Floydhub, etc.

The good news is that MMA neural networks use MXNET under the hood, and you are able to export any network as MXNET files - Export["mynet.json", network] appears to work quite well (though I haven't tried to install MXNET to see if they'll actually work correctly).

Has anybody trained these MMA-defined MXNET neural networks on these cloud providers? If so, what are your recommendations? I am specifically looking for:

  • ease of use - ideally, export->upload->train->download->use in MMA, with limited hassle - I feel that the data is going to be an annoying step here. I'm currently training on 4000 image-mask pairs, a few hundred MB worth of data. Getting some python written to actually train the network is also not going to be very fun.
  • pricing - it seems to be that FloydHub is the best option in terms of price?
  • does anyone know if the Wolfram Cloud is going to support GPU training anytime soon and I can save myself this hassle?
Improve this question
$\endgroup$
6
  • $\begingroup$ Why not just install WL on an AWS/Paperspace/etc machine and use the NN framework to train directly? (what does exporting to MXNet get you?) $\endgroup$
    – Sebastian
    Commented Sep 3, 2018 at 14:38
  • $\begingroup$ I don't think my license allows me to do that, but maybe I'm wrong, I haven't read it in detail. But this would definitely be the easiest solution otherwise... $\endgroup$
    – Carl Lange
    Commented Sep 3, 2018 at 18:30
  • 1
    $\begingroup$ @CarlLange Please let us know if you succeeded $\endgroup$
    – gogoolplex
    Commented Sep 19, 2018 at 12:57
  • $\begingroup$ @gogoolplex I haven't tried yet to be honest. It seems that using a remote kernel on an AWS server is the easiest option but I haven't had the money to spend to do that. I would dearly love to be able to do Device->"Cloud" and not have to deal with any of this, but then transferring datasets etc would be hassle. $\endgroup$
    – Carl Lange
    Commented Sep 19, 2018 at 14:19
  • 1
    $\begingroup$ @CarlLange Ok. I know that Google gives you like 300USD for free in the first year and I guess AWS has a similar offer. So lets hope that 12 brings an easy solution to use whatever cloud you want easily. $\endgroup$
    – gogoolplex
    Commented Sep 19, 2018 at 17:19

2 Answers 2

Reset to default
11
$\begingroup$

No need to export to MXNet, put your neural net on a wl script instead and run it on AWS. Install CDFPlayer for linux on a Deep-Learning Ubuntu AMI on a EC2 GPU-compute instance, then download and install manually the drivers and run your training script with TargetDevice->"GPU"

Remember to screen your session to avoid interruption. Check GPU usage with nvidia-smi -l 1

Improve this answer
$\endgroup$
5
  • $\begingroup$ That's really helpful. I didn't realise CDFPlayer would be usable for training my network. I'll try this out over the next few weeks and approve your answer then; in theory looks good! $\endgroup$
    – Carl Lange
    Commented Dec 10, 2018 at 15:07
  • $\begingroup$ @CarlLange Yes, the Mathkernel that gets installed with the CDF seems able to do the same things the Mathematica one does; it might be the very same. $\endgroup$
    – Fortsaint
    Commented Dec 10, 2018 at 17:40
  • $\begingroup$ I never actually got around to testing this, but the new Wolfram Engine seems suitable for this as well. $\endgroup$
    – Carl Lange
    Commented May 20, 2019 at 17:21
  • $\begingroup$ @CarlLange Yes I confirm. What I haven't tested yet is the possibility to have a working Wolfram Engine installation on different system simultaneously, something you could definitively have with the previous CDFPlayer kernel. $\endgroup$
    – Fortsaint
    Commented May 21, 2019 at 11:17
  • $\begingroup$ (No disrespect intended with the un-accept - I did so because there is now official support within the language to do this. Your answer is still totally valid and useful, but now especially so for those running versions below 12.2. Thanks for the original answer!) $\endgroup$
    – Carl Lange
    Commented Dec 16, 2020 at 19:49
10
$\begingroup$

With version 12.2, this functionality now exists using RemoteBatchSubmit and related functionality.

From the release notes:

env = RemoteBatchSubmissionEnvironment[
  "AWSBatch", <|"JobQueue" -> 
    "arn:aws:batch:us-east-1:123456789012:job-queue/MyQueue", 
   "JobDefinition" -> 
    "arn:aws:batch:us-east-1:123456789012:job-definition/MyDefinition:\
1", "IOBucket" -> "my-job-bucket"|>]

RemoteBatchSubmit[env, 
 NetTrain[NetModel["LeNet"], "MNIST", TargetDevice -> "GPU"],
 RemoteProviderSettings -> <|"GPUCount" -> 1|>]

This will run NetTrain on AWS with a GPU.

Improve this answer
$\endgroup$
7
  • $\begingroup$ But you lose all the nice monitoring of NetTrain $\endgroup$
    – M.R.
    Commented Dec 17, 2020 at 6:16
  • $\begingroup$ Ah, that may be true. I suppose you can still misuse TrainingProgressReporting to email you though - or output to a Databin that you can then watch live. $\endgroup$
    – Carl Lange
    Commented Dec 17, 2020 at 9:58
  • 2
    $\begingroup$ I think it's noteworthy that in addition to AWS costs, this also incurs costs as Wolfram Service Credits (wolfram.com/service-credits). If I understood it correctly, two service credits are consumed by a hour of usage of one kernel, minimum charge being one credit. This is in the ballpark of one cent per hour, which should be rather minor expense in comparison to corresponding AWS costs. $\endgroup$
    – kirma
    Commented Dec 19, 2020 at 12:49
  • 1
    $\begingroup$ Yes, very relevant, that's a great insight. AWS GPU instance costs are about 2 orders of magnitude higher than the service credit cost. $\endgroup$
    – Carl Lange
    Commented Dec 19, 2020 at 14:36
  • 4
    $\begingroup$ (From the RemoteBatchSubmit developer, who doesn't have enough rep to comment): It's true that you can't have the same dynamic graphics monitoring that you get from NetTrain in a frontend session, but if you're using AWS Batch you can get live logs with the "JobLog" property of RemoteBatchJobObject. These logs will show the same monitoring info you get from NetTrain in a standalone kernel. The 12.2 blog post has a (suboptimally cropped) example of this: writings.stephenwolfram.com/2020/12/… $\endgroup$
    – ClydeTheGhost
    Commented Dec 28, 2020 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.