I have a model with NetTrain that works fine. However, as I increase the amount of training data, the number of inputs per second to my GPU drops very, very substantially. In fact, when I include all of the data, almost no time is spend on the GPU and it takes a very long time to do anything. Is there a reason for this? For example, I have almost 2000 examples in my training dataset. If I use 100 of them, I get an input of 30 per second. However, if I use 1000 of them, Mathematica reports an input of 0 per second. Over time it does train, but hardly any time is spend on the GPU.
Currently, I am using the default batch size.
Is there a way to change this? Why is this happening?