NetTrain seems to leak memory.


We define a simple NetChain:

net = NetInitialize@NetChain[{5, LogisticSigmoid, 1}, "Input" -> 3];

We define the input data:

hues = RandomReal[1, {1000000, 1}];
rgbs = List @@ ColorConvert[Hue[#[[1]]], "RGB"] & /@ hues;
rules = Inner[Rule, Hold @@@ rgbs, Hold @@@ hues, List] /. 
   Hold -> List;

The memory used, according to Mathematica:


(* 373849120 *)

My task manager shows 412.0 MB used.

Now we train the net:

net = NetTrain[net, rules, BatchSize -> 512, MaxTrainingRounds -> 5]

And we check the memory usage:


(* 444727112 *) (* ~70MB increase *)

My task manager shows 497.1 MB (~85MB increase).

So far, it seems okay. Mathematica had to load NeuralNetworks things, so it seems natural that the memory usage increased.

Now we train the net more (it doesn't matter if you generate new rules):

net = NetTrain[net, rules, BatchSize -> 512, MaxTrainingRounds -> 5]

The memory usage doesn't really change:


(* 444963728 *)

But in the task manager, Mathematica is now using 517.1 MB (20MB increase).


What is causing the increase in memory allocation of Mathematica?

  • 2
    $\begingroup$ When I train a large neural net, the memory leak becomes more severe; Mathematica takes over 8 GB of RAM while it thinks it is using only about 200 MB. $\endgroup$ Commented Aug 29, 2016 at 4:09
  • $\begingroup$ What's your platform? Windows? $\endgroup$ Commented Aug 29, 2016 at 6:55
  • $\begingroup$ @TaliesinBeynon "11.0.0 for Microsoft Windows (64-bit) (July 28, 2016)" $\endgroup$ Commented Aug 29, 2016 at 13:39

1 Answer 1


For CPU usage, the answer is simple: the learning backend (customized version of MXNet) uses the MKL library on CPUs, which is parallelized and utilizes all available cores. That's a good thing!

The memory usage, please try the following: before you run anything, set $HistoryLength = 0. Does that improve matters? Also, after training, see if running NeuralNetworks`ClearCache[] changes what Task Manager shows.

Last thing that would help us track down what's going on here, can you run the following code (on a fresh kernel) and paste its output a) before training b) after the first NetTrain c) after the second NetTrain?

  MXNetLink`GetManagedLibraryKeys /* Length, 
  {"MXExecutor", "NDArray", "MXSymbol", "MXOptimizer"}]
  • $\begingroup$ My $HistoryLength is already 0. NeuralNetworks`ClearCache[] doesn't seem to do anything. For your code, the output before training is <|"MXExecutor" -> 1, "NDArray" -> 1, "MXSymbol" -> 1, "MXOptimizer" -> 1|>. The outputs after the first training and the second training are <|"MXExecutor" -> 0, "NDArray" -> 0, "MXSymbol" -> 0, "MXOptimizer" -> 0|>. $\endgroup$ Commented Aug 29, 2016 at 13:44
  • $\begingroup$ @JHM I'm inclined to think that this is a false positive. For example, memory allocated and deallocated during training (though there shouldn't be much) might cause the heap to become fragmented, and the OS is unable to reclaim it from the application. What actual pain is it causing you? $\endgroup$ Commented Sep 1, 2016 at 22:28
  • 1
    $\begingroup$ I have to export the net and restart the kernel more than 80 times if I want to train my net with all data available. $\endgroup$ Commented Sep 1, 2016 at 23:05
  • $\begingroup$ @JHM go in chatroom? $\endgroup$ Commented Sep 1, 2016 at 23:18
  • $\begingroup$ Sure. How do I join? $\endgroup$ Commented Sep 1, 2016 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.