Why does `MemoryInUse` keep increasing?

Question

Consider MemoryInUse. After kernel initiation, I have noticed that its value keeps increasing, even when not evaluating any cell, as seen

ml = Table[MemoryInUse[], {t, 0, 10000}];
ListPlot[ml, PlotRange -> {Automatic, {Mean@ml - 100000, Mean@ml + 100000}}, Frame -> True]

Any idea why this happens? Where could I read more about memory management within Mathematica notebooks? Any way to reset it?

Edit: Also, I have noticed that by re-evaluating a cell with the code MemoryInUse[], the value increases every time I evaluate it.

Answer 1

Explanation

At least on my system, the slope Mean@Differences@ml is equal to $16$ bytes, which is equal to ByteCount[MemoryInUse[]].

It looks like the MemoryInUse increases exactly by the ByteCount amount it takes to store the output of MemoryInUse, as the Table executes and stores each value.

I don't find that surprising at all, as you fill the table with data that is stored in memory, that increases the memory used.

Solution

To understand this better, unsurprisingly, the best option would be to read the documentation about Memory Management. It's unclear what you mean with "Reset" the memory in use, but if yu mean minimizing the amount of MemoryInUse[] could be a question on its own.

If you want to avoid that increase during the table execution, you will need to either pre-allocate the data in memory or not use memory at all, by writing on disk.

Pre-allocation

You can pre-allocate the Table and update its content like this

ml = ConstantArray[0,10000];
Do[ ml[[t]]=MemoryInUse[], {t, 10000}];

Write to disk

Alternatively, you could store the values on disk instead of memory.

Block[
    {
        fname = FileNameJoin[{$TemporaryDirectory, "testfile"}],
        s
    },
    s = OpenWrite[fname];
    Do[ Write[s, MemoryInUse[] ], 1000];
    Close[fname];
    ListPlot[ ReadList[fname] ]
]