Why I can only train some NetModel with GPU

Question

I can train NetModel's models with CPU. I'm on Windows 10 with Mathematica 11.2 GPU is GTX1080 with 8G memory.

My datasets is 1000 pictures with 224*224.

I can train some NetModel like Inception V1 Trained on ImageNet Competition Data with TargetDevice->"GPU"

But I cannot train some NetModel like ResNet-152 Trained on ImageNet Competition Data with TargetDevice->"GPU"

NetTrain::interr: An internal error occurred. Please contact Wolfram Research.

Since I can train some model with GPU, I think my graphics card driver is OK. Since I can train them with CPU, I think Mathematica implement their layers.

I can use ImageRestyle with TargetDevice->"GPU", if I encounter with the problem An internal error occurred in training NetModel, then ImageRestyle also gives the same error information unless I Quit or Restart Mathematica.

---

I can train with GPU:

"Inception V1 Trained on ImageNet Competition Data"

"Wolfram ImageIdentify Net for WL 11.1"

"Yahoo Open NSFW Model V1"

"SqueezeNet V1.1 Trained on ImageNet Competition Data" crashes my kernel like Quit.

---

Well, some net can be trained with 20/30 pictures, but some can be trained by 1000 pictures....I'm not sure whether this is normal?

Even I use File object, the Out of Core training, it also gives the error information, and has limit number of training set.

Answer 1

The cause is the GPU running out of memory, and Mathematica not correctly reporting the failure (ResNet 152 is very memory hungry!).

This can be seen by running Internal`$LastInternalFailure, which will include a line "cudaMalloc failed: out of memory". I have reported this incorrect reporting as a bug.

Try a smaller BatchSize in NetTrain to reduce your memory usage in order to train a large net like this.