8
$\begingroup$

I am working on something in which I need to train a neural network many times with slightly different training data. I noticed that after many training runs the input rate had dropped significantly.

Below is a simple illustrative example showing the phenomenon.

inputSW = RandomReal[{-1, 1}, {10000, 5}];
target = RandomInteger[{1, 4}, 10000];
trainingData = MapThread[Normalize[#1] -> #2 &, {inputSW, target}];
rates = {};
Do[temp = 
   NetTrain[NetChain[{16, Ramp, 12, Ramp, 4, SoftmaxLayer[]}, 
   "Output" -> NetDecoder[{"Class", Range[4]}]], trainingData, 
   "ResultsObject", MaxTrainingRounds -> 4000, BatchSize -> 10000, 
   TargetDevice -> {"GPU", 2}, TrainingProgressReporting -> "Panel"];
   rate = temp["MeanInputsPerSecond"]; PrintTemporary[rate]; 
   rates = Append[rates, rate], 600]

ListPlot@rates

GPU #2 is a GTX 1050 Ti and the list plot looks like this:

Inputs per second

Performing more training rounds with:

rates = {};
Do[temp = 
  NetTrain[NetChain[{16, Ramp, 12, Ramp, 4, SoftmaxLayer[]}, 
  "Output" -> NetDecoder[{"Class", Range[4]}]], trainingData, 
  "ResultsObject", MaxTrainingRounds -> 4000, BatchSize -> 10000, 
  TargetDevice -> {"GPU", 2}, TrainingProgressReporting -> "Panel"];
  rate = temp["MeanInputsPerSecond"]; PrintTemporary[rate]; 
  rates = Append[rates, rate], 600]

continues the trend:

Inputs per second

Restarting the kernel and running more iterations with:

inputSW = RandomReal[{-1, 1}, {10000, 5}];
target = RandomInteger[{1, 4}, 10000];
trainingData = MapThread[Normalize[#1] -> #2 &, {inputSW, target}];
rates = {};
Do[temp = 
   NetTrain[NetChain[{16, Ramp, 12, Ramp, 4, SoftmaxLayer[]}, 
   "Output" -> NetDecoder[{"Class", Range[4]}]], trainingData, 
   "ResultsObject", MaxTrainingRounds -> 4000, BatchSize -> 10000, 
   TargetDevice -> {"GPU", 2}, TrainingProgressReporting -> "Panel"];
   rate = temp["MeanInputsPerSecond"]; PrintTemporary[rate]; 
   rates = Append[rates, rate], 600]

ListPlot@rates

resets the input rate:

Inputs per second

Without restarting the kernel I train more networks on GPU #1 (Quadro M4000) with

rates = {};
Do[temp = 
  NetTrain[NetChain[{16, Ramp, 12, Ramp, 4, SoftmaxLayer[]}, 
  "Output" -> NetDecoder[{"Class", Range[4]}]], trainingData, 
  "ResultsObject", MaxTrainingRounds -> 4000, BatchSize -> 10000, 
  TargetDevice -> {"GPU", 2}, TrainingProgressReporting -> "Panel"];
  rate = temp["MeanInputsPerSecond"]; PrintTemporary[rate]; 
  rates = Append[rates, rate], 600]

and get

Inputs per second

Restarting the kernel and training again on GPU #1 gives

Inputs per second

This shows that switching the GPU used to train the network "resets" the input rate. Continuing to train on this GPU without restarting the kernel shows the input rate continue to drop

Inputs per second

Can others reproduce this? Does anyone have any ideas what might be happening (and how to solve it)?

I am using Mathematica 11.3 on Windows 10.

Update:

Same trend training on a CPU:

Inputs per second

Improve this question
$\endgroup$
3
  • $\begingroup$ I can reproduce this. GTX1050Ti, Mathematica 11.3, Windows 10. The initial decrease can be explained as turning off GPU Boost. But it's hard to explain further decrease. $\endgroup$
    – Alexey Golyshev
    Commented Feb 1, 2019 at 15:39
  • $\begingroup$ Per my update, you see the same trend training with a CPU. Also, I've monitored the GPU during these tests and there is nothing (clock speeds, temperature, etc.) that seems to explain these behavior. $\endgroup$
    – EricMock
    Commented Feb 1, 2019 at 18:43
  • 3
    $\begingroup$ Wolfram has confirmed this is a bug and hope to have it fixed in the next release. $\endgroup$
    – EricMock
    Commented Feb 5, 2019 at 0:59

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.