# Repeated calculation with neural network measurement slows down

I have a list of 200 neural nets nets0. I have to measure the output of several layers of each of them on a list of 1000 associations trainSmallAssocList. Using the following code

$HistoryLength = 0; Report[n_] := Block[{start = SessionTime[], xs, nets0, res0 = results[n]}, nets0 = nets[res0["CheckpointingFiles"]]; xs = Table[ Print["n = ", n, " i = ", i, " ", (SessionTime[] - start)/60., " min ", MemoryAvailable[]/(1024 1024 1024.), " Gb"]; start = SessionTime[]; NetMeasurements[nets0[[i]], #, {NetPort[{"lenet", 2, "Output"}], NetPort[{"lenet", "x Output"}]}] & /@ trainSmallAssocList, {i, 200}]; DumpSave["data" <> ToString[n], xs]; ]  I get for Report[2] n = 2 i = 1 1.10612 min 8.66111 Gb n = 2 i = 2 0.124991 min 8.64914 Gb n = 2 i = 3 0.15243 min 8.64968 Gb n = 2 i = 4 0.17889 min 8.65084 Gb n = 2 i = 5 0.204045 min 8.65091 Gb ...  One can see that the time of each iteration increases gradually. After some time it gets too long to wait. However, the nets at different i differ very slightly and the measurement should take the same time. If I abort the computation and relaunch it I get n = 2 i = 1 2.37871 min 8.33028 Gb n = 2 i = 2 1.12449 min 8.30013 Gb n = 2 i = 3 1.12593 min 8.30732 Gb n = 2 i = 4 1.15771 min 8.31129 Gb n = 2 i = 5 1.19696 min 8.21004 Gb ...  One can see that the iteration time increases again but it is longer from the beginning although it calculates exactly the same. I started measuring the available memory after the computation stopped with the message General::nomem: The current computation was aborted because there was insufficient memory available to complete the computation. SystemException["MemoryAllocationFailure"]  However there is enough memory. Windows confirms it. I thought that the problem was in manipulating the large list xs in memory as it got larger. Therefore I started to write it to disk. I changed Table[...] to Do[ xs = NetMeasurements[nets0[[i]], #, {NetPort[{"lenet", 2, "Output"}], NetPort[{"lenet", "x Output"}]}] & /@ trainSmallAssocList; PutAppend[xs, file]; Print["n = ", n, " i = ", i, " ", (SessionTime[] - start)/60., " min ", MemoryAvailable[]/(1024 1024 1024.), " Gb"]; start = SessionTime[];, {i, 200}];  The iteration time increased gradually again. Using Map or Outer did not help either. The following code with Outer takes too long so that I abort: sets1 = Function[{x, y}, NetMeasurements[x, y, {NetPort[{"lenet", 2, "Output"}], NetPort[{"lenet", "x Output"}]}]] e=Outer[sets1[##]&,nets0,trainSmallAssocList];  Using Map via e2[net_, asslist_] := Block[{xs = NetMeasurements[net, #, {NetPort[{"lenet", 2, "Output"}], NetPort[{"lenet", "x Output"}]}] & /@ asslist}, xs] e = e2[#, trainSmallAssocList] & /@ nets0[[;;40]];  works in small batches, 40 e.g. However the effect of slowing down persists in the following way. The time necessary for the first 40 nets is about 2.5 - 3 times less than the time for the next 40, which is about 1.6 times less than the time for the next 40, etc. At some point I relaunch the kernel and get the small time again. According to Windows, Mathematica never takes more than 2 Gb of memory, usually between 1 and 2. It is 12.3, Win10, 16Gb. I do not understand if it is a neural network related issue or I am writing a memory leaking code or something else. Could you please tell me what causes slowing down and how to avoid it? • I don't see any obvious problems with your code. If you could put the example somewhere on Github I could try to reproduce it. Internals of NeuralNets use lots of caching so maybe something goes wrong there. What is the version of Mathematica btw ($Version)? Nov 3 '21 at 6:08