Possible Memory Leak Issues in NestWhile

Question

Consider the following code snippet:

NestWhile[{#[[1]]+1,[email protected];ConstantArray[0,{100, 100}]}&,
  {1}, (#[[1]] <= 10000) &, 2]

In theory, when evaluating, NestWhile shall keep track of the last two results generated, thus consuming approximately 160kB of memory.

However, in reality, memory consumption will continue to grow at a pace of approximately 80MB/s before NestWhile finishes calculation. Furthermore, regardless of the fourth parameter (as long as it is not 1), the speed of memory consumption is the same. These two phenomena indicate that Mathematica evaluates the expression and stored all results in memory until all computations are finished then picks out last two elements.

This behavior is rather disturbing for me: why save the previous evaluation results in memory if they are used in nowhere?

---

This behavior exists in v12.0 and v11.2

---

Update

Received reply from technical support which said:

This behavior of storing all intermediate steps of a calculation is intended. Clearing intermediate information that has been stored can be done with the command ClearSystemCache[]

However, after altering the code to:

NestWhile[(ClearSystemCache[]; {#[[1]] + 1, [email protected]; ConstantArray[0, {100, 100}]})&,
  {1}, (#[[1]] <= 10000) &, 2]

still cannot stop fast increasing memory consumption during evaluation.

Furthermore, a new observation, when evaluating this piece of code, Mathematica will eventually consume 99% of memory, but will not try to use virutal memory. So theoretically this behavior will not influence the performance of Mathematica, however, if I open another program, Mathematica will encounter memory related issues sometimes and crash. But by all means, I think consuming all memory is not a good choice.

Also, if Mathematica actually saves all these intermediate steps for acceleration of evaluation, then why Mathematica do not store them when the fourth argument is 1? I'm still a bit confused with this reply...

---

Update 2

Received another reply from technical support after explaining that clearing cache will not help and this behavior did no good to any evaluation while causing memory explosion.

I have sent a suggestion report to the appropriate people in our development team so changes to how NestWhile handles memory can be considered for future versions of Mathematica.

Hope this bug can be fixed in the next version of Mathematica...

Now, before this problem is solved by Wolfram officially, let this question be:

"How to implement a efficient myNestWhile which works exactly as NestWhile?"

Answer 1

This works without hogging memory:

nestWhile[f_, expr_, test_, m_Integer:1, max_:∞, n_Integer:0] /; m>=1&&n>=0 := 
  Module[{c = m - 1, r = NestList[f, expr, m - 1]},
    While[test @@ r && c++ < max, r = Append[Rest@r, f@Last@r]];
    Nest[f, Last@r, n]]

In contrast to the real NestWhile, the above does not implement the following:

• NestWhile[f,expr,test,All] supplies all results so far as arguments for test at each step.
• NestWhile[f,expr,test,{mmin,m}] does not start applying test until at least mmin results have been generated. At each step it then supplies as arguments to test as many recent results as possible, up to a maximum of m.
• NestWhile[f,expr,test,m,max,-n] returns the result found when f had been applied n fewer times.

Suggestions and improvements highly welcome!