# Training speed of NetTrain on GPU

I have a model with NetTrain that works fine. However, as I increase the amount of training data, the number of inputs per second to my GPU drops very, very substantially. In fact, when I include all of the data, almost no time is spend on the GPU and it takes a very long time to do anything. Is there a reason for this? For example, I have almost 2000 examples in my training dataset. If I use 100 of them, I get an input of 30 per second. However, if I use 1000 of them, Mathematica reports an input of 0 per second. Over time it does train, but hardly any time is spend on the GPU.

Currently, I am using the default batch size.

Is there a way to change this? Why is this happening?

• I am not an expert for NetTrain but transferring data to and from the GPU is very, very costly... – Henrik Schumacher Feb 25 '18 at 20:44
• Is it possible that the examples are cached to disk? Have you tried using the same batch size? – Niki Estner Feb 26 '18 at 7:09