17
$\begingroup$

I have spent a lot of time lately training neural networks. However, my poor laptop has only an AMD GPU, so I am stuck training these networks on the CPU. That means I get to train networks over multiple days only to learn that I made a mistake in my loss function.

I don't have enough money to build a new computer or buy an eGPU, so it seems that my best solution is to train these networks on a cloud provider instead.

There are a lot of options nowadays for these cloud GPUS - AWS, Paperspace, Floydhub, etc.

The good news is that MMA neural networks use MXNET under the hood, and you are able to export any network as MXNET files - Export["mynet.json", network] appears to work quite well (though I haven't tried to install MXNET to see if they'll actually work correctly).

Has anybody trained these MMA-defined MXNET neural networks on these cloud providers? If so, what are your recommendations? I am specifically looking for:

  • ease of use - ideally, export->upload->train->download->use in MMA, with limited hassle - I feel that the data is going to be an annoying step here. I'm currently training on 4000 image-mask pairs, a few hundred MB worth of data. Getting some python written to actually train the network is also not going to be very fun.
  • pricing - it seems to be that FloydHub is the best option in terms of price?
  • does anyone know if the Wolfram Cloud is going to support GPU training anytime soon and I can save myself this hassle?
$\endgroup$
6
  • $\begingroup$ Why not just install WL on an AWS/Paperspace/etc machine and use the NN framework to train directly? (what does exporting to MXNet get you?) $\endgroup$ – Sebastian Sep 3 '18 at 14:38
  • $\begingroup$ I don't think my license allows me to do that, but maybe I'm wrong, I haven't read it in detail. But this would definitely be the easiest solution otherwise... $\endgroup$ – Carl Lange Sep 3 '18 at 18:30
  • 1
    $\begingroup$ @CarlLange Please let us know if you succeeded $\endgroup$ – gogoolplex Sep 19 '18 at 12:57
  • $\begingroup$ @gogoolplex I haven't tried yet to be honest. It seems that using a remote kernel on an AWS server is the easiest option but I haven't had the money to spend to do that. I would dearly love to be able to do Device->"Cloud" and not have to deal with any of this, but then transferring datasets etc would be hassle. $\endgroup$ – Carl Lange Sep 19 '18 at 14:19
  • 1
    $\begingroup$ @CarlLange Ok. I know that Google gives you like 300USD for free in the first year and I guess AWS has a similar offer. So lets hope that 12 brings an easy solution to use whatever cloud you want easily. $\endgroup$ – gogoolplex Sep 19 '18 at 17:19
9
$\begingroup$

No need to export to MXNet, put your neural net on a wl script instead and run it on AWS. Install CDFPlayer for linux on a Deep-Learning Ubuntu AMI on a EC2 GPU-compute instance, then download and install manually the drivers and run your training script with TargetDevice->"GPU"

Remember to screen your session to avoid interruption. Check GPU usage with nvidia-smi -l 1

$\endgroup$
5
  • $\begingroup$ That's really helpful. I didn't realise CDFPlayer would be usable for training my network. I'll try this out over the next few weeks and approve your answer then; in theory looks good! $\endgroup$ – Carl Lange Dec 10 '18 at 15:07
  • $\begingroup$ @CarlLange Yes, the Mathkernel that gets installed with the CDF seems able to do the same things the Mathematica one does; it might be the very same. $\endgroup$ – Fortsaint Dec 10 '18 at 17:40
  • $\begingroup$ I never actually got around to testing this, but the new Wolfram Engine seems suitable for this as well. $\endgroup$ – Carl Lange May 20 '19 at 17:21
  • $\begingroup$ @CarlLange Yes I confirm. What I haven't tested yet is the possibility to have a working Wolfram Engine installation on different system simultaneously, something you could definitively have with the previous CDFPlayer kernel. $\endgroup$ – Fortsaint May 21 '19 at 11:17
  • $\begingroup$ (No disrespect intended with the un-accept - I did so because there is now official support within the language to do this. Your answer is still totally valid and useful, but now especially so for those running versions below 12.2. Thanks for the original answer!) $\endgroup$ – Carl Lange Dec 16 '20 at 19:49
8
$\begingroup$

With version 12.2, this functionality now exists using RemoteBatchSubmit and related functionality.

From the release notes:

env = RemoteBatchSubmissionEnvironment[
  "AWSBatch", <|"JobQueue" -> 
    "arn:aws:batch:us-east-1:123456789012:job-queue/MyQueue", 
   "JobDefinition" -> 
    "arn:aws:batch:us-east-1:123456789012:job-definition/MyDefinition:\
1", "IOBucket" -> "my-job-bucket"|>]

RemoteBatchSubmit[env, 
 NetTrain[NetModel["LeNet"], "MNIST", TargetDevice -> "GPU"],
 RemoteProviderSettings -> <|"GPUCount" -> 1|>]

This will run NetTrain on AWS with a GPU.

$\endgroup$
7
  • $\begingroup$ But you lose all the nice monitoring of NetTrain $\endgroup$ – M.R. Dec 17 '20 at 6:16
  • $\begingroup$ Ah, that may be true. I suppose you can still misuse TrainingProgressReporting to email you though - or output to a Databin that you can then watch live. $\endgroup$ – Carl Lange Dec 17 '20 at 9:58
  • 2
    $\begingroup$ I think it's noteworthy that in addition to AWS costs, this also incurs costs as Wolfram Service Credits (wolfram.com/service-credits). If I understood it correctly, two service credits are consumed by a hour of usage of one kernel, minimum charge being one credit. This is in the ballpark of one cent per hour, which should be rather minor expense in comparison to corresponding AWS costs. $\endgroup$ – kirma Dec 19 '20 at 12:49
  • 1
    $\begingroup$ Yes, very relevant, that's a great insight. AWS GPU instance costs are about 2 orders of magnitude higher than the service credit cost. $\endgroup$ – Carl Lange Dec 19 '20 at 14:36
  • 4
    $\begingroup$ (From the RemoteBatchSubmit developer, who doesn't have enough rep to comment): It's true that you can't have the same dynamic graphics monitoring that you get from NetTrain in a frontend session, but if you're using AWS Batch you can get live logs with the "JobLog" property of RemoteBatchJobObject. These logs will show the same monitoring info you get from NetTrain in a standalone kernel. The 12.2 blog post has a (suboptimally cropped) example of this: writings.stephenwolfram.com/2020/12/… $\endgroup$ – ClydeTheGhost Dec 28 '20 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.