Measuring memory use

Question

Here is some simple code:

Quit[]

B[d_] := Sum[Sqrt[1 + k^2] // N[#, d] &, {k, 0, 2 d}]
LaunchKernels[];
k = $KernelCount

outputting 6 for k. Next, I have

ClearSystemCache[];
d = 10^4;
mem1 = MemoryInUse[];
B[ d];
mem2 = MemoryInUse[];
(mem2 - mem1)/d // N

outputing

3903.61

Replacing ClearSystemCache[; in the last 6 lines of code by ClearSystemCache[] // ParallelEvaluate; gives a much smaller number, 29.0552, instead of 3903.61. This is completely unexpected to me. Doesn't ClearSystemCache[] // ParallelEvaluate; clear the memory more thoroughly than ClearSystemCache[]; ?

Repeating again with ClearSystemCache[] // ParallelEvaluate; gives a yet smaller number, 11.8872, in place of 29.0552.

Now I try to measure memory use per kernel:

ClearSystemCache[] // ParallelEvaluate;
d = 10^4;
mem1 = MemoryInUse[] // ParallelEvaluate;
B[d];
mem2 = MemoryInUse[] // ParallelEvaluate;
(mem2 - mem1)/(k d) // N // Total

and get 0 (!!).

Next, I replace Sum[] in the definition of B[] by ParallelSum[]:

Quit[]

BPar[d_] := ParallelSum[Sqrt[1 + j^2] // N[#, d] &, {j, 0, 2 d}]

Using BPar[] in place of B[] yields 8.0696 and 1.2688 in place of the mentioned numbers 3903.61 and 29.0552, respectively.

If, in addition, I also replace the two instances of MemoryInUse[] by MemoryInUse[] // ParallelEvaluate, to measure memory use per kernel:

ClearSystemCache[] // ParallelEvaluate;
d = 10^4;
mem1 = MemoryInUse[] // ParallelEvaluate;
BPar[d];
mem2 = MemoryInUse[] // ParallelEvaluate;
(mem2 - mem1)/(k d) // N // Total

then I get 1.21805; this does not seem to decrease with repetitions.

When BPar[d_] is replaced by

BParCoarse[d_] := 
 ParallelSum[Sqrt[1 + j^2] // N[#, d] &, {j, 0, 2 d}, 
  Method -> "CoarsestGrained"]

then I get 1.2952 and 2022.59 in place of the mentioned numbers 8.0696 and 1.21805.

So, there seems to be no agreement whatsoever between any of the output numbers, which vary very widely. Nothing of this makes any sense to me. I would very much appreciate your help.

Answer 1

Replacing ClearSystemCache[; in the last 6 lines of code by ClearSystemCache[] // ParallelEvaluate; gives a much smaller number, 29.0552, instead of 3903.61. This is completely unexpected to me.

Paralellization is achieved by running several Mathematica processes (called subkernels) which do not share memory. MemoryInUse[] reports the memory usage of the kernel on which it is evaluated (i.e. the main kernel). When you replace ClearSystemCache[] by ParallelEvaluate[ClearSystemCache[]], you are no longer clearing the cache on the main kernel, yet you are only measuring memory use on the main kernel.

Doesn't ClearSystemCache[] // ParallelEvaluate; clear the memory more thoroughly than ClearSystemCache[]; ?

It clears the system cache on the parallel kernels only, not on the main kernel.

Also, it is somewhat misleading to say that ClearSystemCache[] clears the memory. It simply clears certain caches (i.e. saved results that can be re-used). There are many other things that can take up memory, e.g. any variables you may have assigned values to.

Using BPar[] in place of B[] yields 8.0696 and 1.2688 in place of the mentioned numbers 3903.61 and 29.0552, respectively.

That's because now you are doing the computation on the subkernels and measuring memory use on the main kernel.

... then I get 1.21805; this does not seem to decrease with repetitions.

If I remove the Total, I get {711.794, 713.942, 707.588, 706.157} with every repetition.