7
$\begingroup$

Issue

NetTrain seems to leak memory.

Example

We define a simple NetChain:

net = NetInitialize@NetChain[{5, LogisticSigmoid, 1}, "Input" -> 3];

We define the input data:

hues = RandomReal[1, {1000000, 1}];
rgbs = List @@ ColorConvert[Hue[#[[1]]], "RGB"] & /@ hues;
rules = Inner[Rule, Hold @@@ rgbs, Hold @@@ hues, List] /. 
   Hold -> List;

The memory used, according to Mathematica:

MemoryInUse[]

(* 373849120 *)

My task manager shows 412.0 MB used.


Now we train the net:

net = NetTrain[net, rules, BatchSize -> 512, MaxTrainingRounds -> 5]

And we check the memory usage:

MemoryInUse[]

(* 444727112 *) (* ~70MB increase *)

My task manager shows 497.1 MB (~85MB increase).

So far, it seems okay. Mathematica had to load NeuralNetworks things, so it seems natural that the memory usage increased.


Now we train the net more (it doesn't matter if you generate new rules):

net = NetTrain[net, rules, BatchSize -> 512, MaxTrainingRounds -> 5]

The memory usage doesn't really change:

MemoryInUse[]

(* 444963728 *)

But in the task manager, Mathematica is now using 517.1 MB (20MB increase).

Question

What is causing the increase in memory allocation of Mathematica?

$\endgroup$
  • 2
    $\begingroup$ When I train a large neural net, the memory leak becomes more severe; Mathematica takes over 8 GB of RAM while it thinks it is using only about 200 MB. $\endgroup$ – JungHwan Min Aug 29 '16 at 4:09
  • $\begingroup$ What's your platform? Windows? $\endgroup$ – Taliesin Beynon Aug 29 '16 at 6:55
  • $\begingroup$ @TaliesinBeynon "11.0.0 for Microsoft Windows (64-bit) (July 28, 2016)" $\endgroup$ – JungHwan Min Aug 29 '16 at 13:39
3
$\begingroup$

For CPU usage, the answer is simple: the learning backend (customized version of MXNet) uses the MKL library on CPUs, which is parallelized and utilizes all available cores. That's a good thing!

The memory usage, please try the following: before you run anything, set $HistoryLength = 0. Does that improve matters? Also, after training, see if running NeuralNetworks`ClearCache[] changes what Task Manager shows.

Last thing that would help us track down what's going on here, can you run the following code (on a fresh kernel) and paste its output a) before training b) after the first NetTrain c) after the second NetTrain?

AssociationMap[
  MXNetLink`GetManagedLibraryKeys /* Length, 
  {"MXExecutor", "NDArray", "MXSymbol", "MXOptimizer"}]
$\endgroup$
  • $\begingroup$ My $HistoryLength is already 0. NeuralNetworks`ClearCache[] doesn't seem to do anything. For your code, the output before training is <|"MXExecutor" -> 1, "NDArray" -> 1, "MXSymbol" -> 1, "MXOptimizer" -> 1|>. The outputs after the first training and the second training are <|"MXExecutor" -> 0, "NDArray" -> 0, "MXSymbol" -> 0, "MXOptimizer" -> 0|>. $\endgroup$ – JungHwan Min Aug 29 '16 at 13:44
  • $\begingroup$ @JHM I'm inclined to think that this is a false positive. For example, memory allocated and deallocated during training (though there shouldn't be much) might cause the heap to become fragmented, and the OS is unable to reclaim it from the application. What actual pain is it causing you? $\endgroup$ – Taliesin Beynon Sep 1 '16 at 22:28
  • 1
    $\begingroup$ I have to export the net and restart the kernel more than 80 times if I want to train my net with all data available. $\endgroup$ – JungHwan Min Sep 1 '16 at 23:05
  • $\begingroup$ @JHM go in chatroom? $\endgroup$ – Taliesin Beynon Sep 1 '16 at 23:18
  • $\begingroup$ Sure. How do I join? $\endgroup$ – JungHwan Min Sep 1 '16 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.