Measuring memory use

Here is some simple code:

Quit[]

B[d_] := Sum[Sqrt[1 + k^2] // N[#, d] &, {k, 0, 2 d}]
LaunchKernels[];
k = \$KernelCount


outputting 6 for k. Next, I have

ClearSystemCache[];
d = 10^4;
mem1 = MemoryInUse[];
B[ d];
mem2 = MemoryInUse[];
(mem2 - mem1)/d // N


outputing

3903.61

Replacing ClearSystemCache[; in the last 6 lines of code by ClearSystemCache[] // ParallelEvaluate; gives a much smaller number, 29.0552, instead of 3903.61. This is completely unexpected to me. Doesn't ClearSystemCache[] // ParallelEvaluate; clear the memory more thoroughly than ClearSystemCache[]; ?

Repeating again with ClearSystemCache[] // ParallelEvaluate; gives a yet smaller number, 11.8872, in place of 29.0552.

Now I try to measure memory use per kernel:

ClearSystemCache[] // ParallelEvaluate;
d = 10^4;
mem1 = MemoryInUse[] // ParallelEvaluate;
B[d];
mem2 = MemoryInUse[] // ParallelEvaluate;
(mem2 - mem1)/(k d) // N // Total


and get 0 (!!).

Next, I replace Sum[] in the definition of B[] by ParallelSum[]:

Quit[]

BPar[d_] := ParallelSum[Sqrt[1 + j^2] // N[#, d] &, {j, 0, 2 d}]


Using BPar[] in place of B[] yields 8.0696 and 1.2688 in place of the mentioned numbers 3903.61 and 29.0552, respectively.

If, in addition, I also replace the two instances of MemoryInUse[] by MemoryInUse[] // ParallelEvaluate, to measure memory use per kernel:

ClearSystemCache[] // ParallelEvaluate;
d = 10^4;
mem1 = MemoryInUse[] // ParallelEvaluate;
BPar[d];
mem2 = MemoryInUse[] // ParallelEvaluate;
(mem2 - mem1)/(k d) // N // Total


then I get 1.21805; this does not seem to decrease with repetitions.

When BPar[d_] is replaced by

BParCoarse[d_] :=
ParallelSum[Sqrt[1 + j^2] // N[#, d] &, {j, 0, 2 d},
Method -> "CoarsestGrained"]


then I get 1.2952 and 2022.59 in place of the mentioned numbers 8.0696 and 1.21805.

So, there seems to be no agreement whatsoever between any of the output numbers, which vary very widely. Nothing of this makes any sense to me. I would very much appreciate your help.

Replacing ClearSystemCache[; in the last 6 lines of code by ClearSystemCache[] // ParallelEvaluate; gives a much smaller number, 29.0552, instead of 3903.61. This is completely unexpected to me.

Paralellization is achieved by running several Mathematica processes (called subkernels) which do not share memory. MemoryInUse[] reports the memory usage of the kernel on which it is evaluated (i.e. the main kernel). When you replace ClearSystemCache[] by ParallelEvaluate[ClearSystemCache[]], you are no longer clearing the cache on the main kernel, yet you are only measuring memory use on the main kernel.

Doesn't ClearSystemCache[] // ParallelEvaluate; clear the memory more thoroughly than ClearSystemCache[]; ?

It clears the system cache on the parallel kernels only, not on the main kernel.

Also, it is somewhat misleading to say that ClearSystemCache[] clears the memory. It simply clears certain caches (i.e. saved results that can be re-used). There are many other things that can take up memory, e.g. any variables you may have assigned values to.

Using BPar[] in place of B[] yields 8.0696 and 1.2688 in place of the mentioned numbers 3903.61 and 29.0552, respectively.

That's because now you are doing the computation on the subkernels and measuring memory use on the main kernel.

... then I get 1.21805; this does not seem to decrease with repetitions.

If I remove the Total, I get {711.794, 713.942, 707.588, 706.157} with every repetition.

• Thank you very much for your answer. I had a (mistaken?) impression that the main kernel takes the same, or a very similar, part in parallel calculations as the other kernels. Otherwise, it would seem to be a waste of resources. In fact, the computer I am using now has 6 cores, and it seems that 6 kernels (rather than 5) work in parallel, unless restrictions on that are placed. So, there is something I don't quite understand here. – Iosif Pinelis Mar 3 '17 at 2:50
• Thanks to your answer, I am now getting steady numbers in Mathematica's memory use reports, which agree with one another. However, these numbers seem way too big. I have detailed this concern in another question, at mathematica.stackexchange.com/questions/139098/… . Could you please help me with that as well? Thank you again! – Iosif Pinelis Mar 3 '17 at 2:56