1
$\begingroup$

I have a list of 200 neural nets nets0. I have to measure the output of several layers of each of them on a list of 1000 associations trainSmallAssocList. Using the following code

$HistoryLength = 0;
Report[n_] := 
 Block[{start = SessionTime[], xs, nets0, res0 = results[n]},
  nets0 = nets[res0["CheckpointingFiles"]];
  xs = Table[
   Print["n = ", n, " i = ", i, " ", (SessionTime[] - start)/60., " min ", MemoryAvailable[]/(1024 1024 1024.), " Gb"];
   start = SessionTime[];
   NetMeasurements[nets0[[i]], #, {NetPort[{"lenet", 2, "Output"}], NetPort[{"lenet", "x Output"}]}] & /@ trainSmallAssocList, {i, 200}];
  DumpSave["data" <> ToString[n], xs];
  ]

I get for Report[2]

n = 2 i = 1 1.10612 min 8.66111 Gb
n = 2 i = 2 0.124991 min 8.64914 Gb
n = 2 i = 3 0.15243 min 8.64968 Gb
n = 2 i = 4 0.17889 min 8.65084 Gb
n = 2 i = 5 0.204045 min 8.65091 Gb
...

One can see that the time of each iteration increases gradually. After some time it gets too long to wait. However, the nets at different i differ very slightly and the measurement should take the same time. If I abort the computation and relaunch it I get

n = 2 i = 1 2.37871 min 8.33028 Gb
n = 2 i = 2 1.12449 min 8.30013 Gb
n = 2 i = 3 1.12593 min 8.30732 Gb
n = 2 i = 4 1.15771 min 8.31129 Gb
n = 2 i = 5 1.19696 min 8.21004 Gb
...

One can see that the iteration time increases again but it is longer from the beginning although it calculates exactly the same.

I started measuring the available memory after the computation stopped with the message

General::nomem: The current computation was aborted because there was insufficient memory available to complete the computation.
SystemException["MemoryAllocationFailure"]

However there is enough memory. Windows confirms it. I thought that the problem was in manipulating the large list xs in memory as it got larger. Therefore I started to write it to disk. I changed Table[...] to

Do[
   xs = NetMeasurements[nets0[[i]], #, {NetPort[{"lenet", 2, "Output"}], NetPort[{"lenet", "x Output"}]}] & /@ trainSmallAssocList;
   PutAppend[xs, file]; 
   Print["n = ", n, " i = ", i, " ", (SessionTime[] - start)/60., " min ", MemoryAvailable[]/(1024 1024 1024.), " Gb"]; 
  start = SessionTime[];, {i, 200}];

The iteration time increased gradually again. Using Map or Outer did not help either. The following code with Outer takes too long so that I abort:

sets1 = Function[{x, y}, NetMeasurements[x,  y, {NetPort[{"lenet", 2, "Output"}], NetPort[{"lenet", "x Output"}]}]]
e=Outer[sets1[##]&,nets0,trainSmallAssocList];

Using Map via

e2[net_, asslist_] := Block[{xs = NetMeasurements[net, #, {NetPort[{"lenet", 2, "Output"}], NetPort[{"lenet", "x Output"}]}] & /@ asslist},
xs]
e = e2[#, trainSmallAssocList] & /@ nets0[[;;40]];

works in small batches, 40 e.g. However the effect of slowing down persists in the following way. The time necessary for the first 40 nets is about 2.5 - 3 times less than the time for the next 40, which is about 1.6 times less than the time for the next 40, etc. At some point I relaunch the kernel and get the small time again. According to Windows, Mathematica never takes more than 2 Gb of memory, usually between 1 and 2. It is 12.3, Win10, 16Gb.

I do not understand if it is a neural network related issue or I am writing a memory leaking code or something else. Could you please tell me what causes slowing down and how to avoid it?

$\endgroup$
1
  • $\begingroup$ I don't see any obvious problems with your code. If you could put the example somewhere on Github I could try to reproduce it. Internals of NeuralNets use lots of caching so maybe something goes wrong there. What is the version of Mathematica btw ($Version)? $\endgroup$ Nov 3 '21 at 6:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.