I have a model with NetTrain that works fine. However, as I increase the amount of training data, the number of inputs per second to my GPU drops very, very substantially. In fact, when I include all of the data, almost no time is spend on the GPU and it takes a very long time to do anything. Is there a reason for this? For example, I have almost 2000 examples in my training dataset. If I use 100 of them, I get an input of 30 per second. However, if I use 1000 of them, Mathematica reports an input of 0 per second. Over time it does train, but hardly any time is spend on the GPU.

Currently, I am using the default batch size.

Is there a way to change this? Why is this happening?

  • $\begingroup$ I am not an expert for NetTrain but transferring data to and from the GPU is very, very costly... $\endgroup$ – Henrik Schumacher Feb 25 '18 at 20:44
  • $\begingroup$ Is it possible that the examples are cached to disk? Have you tried using the same batch size? $\endgroup$ – Niki Estner Feb 26 '18 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.