2
$\begingroup$

Since 12.3, the RemoteBatchSubmit to EC2 does not work anymore.

First, it features the same bug as Version 12.3.1 Problems with NetTrain, where the MXNetResources paclet is 12.4 instead of 12.3.

However, by manually installing:

PacletInstall["MXNetResources"->"12.3.406", ForceVersionInstall->True];

It still doesn't work and fails inside the library.

Anyone had any success? Is there any way to force version 12.2 instead of 12.3 on EC2 RemoteBatchSubmit?

$\endgroup$
3
  • $\begingroup$ Welcome to Mathematica.SE! I hope you will become a regular contributor. To get started, 1) take the introductory tour now, 2) when you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge, 3) remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign, and 4) give help too, by answering questions in your areas of expertise. $\endgroup$
    – bbgodfrey
    Jul 13, 2021 at 21:33
  • 1
    $\begingroup$ This issue is being investigated. For the time being, you can force jobs to run under version 12.2 by adjusting the "Job container image" parameter when creating/modifying a submission environment in CloudFormation. Change "wolframengine:cuda" to "wolframengine:12.2.0-cuda". $\endgroup$ Jul 14, 2021 at 0:05
  • $\begingroup$ Good suggestion! it worked, up to a point. Changing to 12.2.0-cuda works, and with paclet MXNetResources (12.2.404), NetTrain can begin... until the wolfram engine is killed because of a licence error. I can't guess what is wrong, so hopefully cuda 12.3 will be functional soon. $\endgroup$ Jul 14, 2021 at 17:42

1 Answer 1

3
$\begingroup$

The main issue here should now be resolved with a newly released version of the Wolfram Engine Docker image containing version 12.3.1 of the Wolfram Engine and version 12.3.406 of MXNetResources. If you've worked around the issue by changing the image used in your compute environment stack as suggested here, you can now reset it to the default wolframresearch/wolframengine:cuda.

I did discover an issue with the behavior of AWS Batch that compounds this issue, and may have been responsible for the library error you encountered after manually installing version 12.3.406 of MXNetResources. When a RemoteBatchSubmit submission environment CloudFormation stack is created from the template, the compute environment resource in the stack is locked at the version of the ECS AMI that was current at the time of stack creation. For example, I was testing the new image with a stack created several months ago, which was stuck on an old version of the AMI with outdated GPU drivers, causing a library error likely similar to the one you saw. (This general issue about outdated AMIs is discussed on the AWS forums here. The minimum driver requirements for NetTrain are documented here.)

I am aware of two remedies for this issue:

  1. Create an entirely new submission environment stack per the "Set Up the AWS Batch Computation Provider" workflow. This will include a freshly created compute environment with the latest AMI.
  2. Update your existing submission environment CloudFormation stack in such a way as to trigger recreation of the compute environment resource. An easy way to do this is to tweak the "Available instance types" stack parameter - I found that changing the default c5,m5,r5,p3 to c5,r5,p3 worked. After the stack updates you can update it again to change the parameter back to its original value (which will recreate the compute environment yet again).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.