4
$\begingroup$

I am training a convolutional Network with a dataset consisting of Images of the same size on my GPU (200x100 pixels, GTX 1080Ti). The images are previously resized with ImageResize[] as their size varies. The training runs perfectly fine.

However, when I try to increase resolution by resizing all the images to 250x150 (for example) NetTrain[] quits with the following error

NetTrain: An internal error occurred. Please contact Wolfram Research.

I guess this has something to do with my GPU as the training proceeds normally on my CPU. Is there a way to debug this / find the limitation of my GPU? The error message is not helping at all.

Best, Max

$\endgroup$
  • 1
    $\begingroup$ I had a similar problem and solved it by reducing the batch size. But better debugging options would be good. $\endgroup$ – Niki Estner Sep 13 '17 at 16:03
  • $\begingroup$ I also had a similar problem in this answer, but my image was much larger (2000 by 1600). $\endgroup$ – xslittlegrass Sep 13 '17 at 17:04
  • 1
    $\begingroup$ The way I deal with this is by using the freeware monitoring program GPUz to check my graphics card memory usage. Start from a small batch size, and initiate NetTrain. See how much graphics memory is used, then, if you didn't crash, increase the batch size to something that you think won't quite fill the memory, and repeat. $\endgroup$ – Yss Sep 14 '17 at 13:35
  • $\begingroup$ Apologies, the incorrect reporting of cuda memory allocation failures is a bug (same as mathematica.stackexchange.com/questions/155924/…). This has been reported, and should be fixed in the next version. $\endgroup$ – Sebastian Sep 18 '17 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.