Mathematica takes way too much memory?

Question

Here is some simple code:

ClearSystemCache[];
d = 10^4;
mem1 = MemoryInUse[];
Sum[Sqrt[1 + j^2] // N[#, d] &, {j, 1, d}];
mem2 = MemoryInUse[];
(mem2 - mem1)/d // N

which outputs

3764.59

I wonder why Mathematica takes so much memory for this calculation. Trying to nudge it in the desired direction, I replaced the line Sum[Sqrt[1 + j^2] // N[#, d] &, {j, 1, d}]; by

s = 0; Do[s = s + Sqrt[1 + j^2] // N[#, d] &, {j, 1, d}] 

but got an even bigger output,

3971.35

As I see it, to compute the sum using the Do[] command, one only needs to keep in memory at most d digits of the current value of s, a few digits for the values of d=10^4 and j=1, ..., d, and however many digits needed to compute the current value of

Sqrt[1 + j^2] // N[#, d] &, which seems to be no more than just 1.2d digits on an average (somewhat surprisingly to me); indeed,

ClearSystemCache[];
d = 10^4;
mem1 = MemoryInUse[];
Sqrt[1 + 1^2] // N[#, d] &;
Sqrt[1 + d^2] // N[#, d] &;
mem2 = MemoryInUse[];
(mem2 - mem1)/d // N

outputs

2.3144

half of which is less than 1.2.

So, it seems that the total memory needed for this calculation should be, at most, about d+1.2d=2.2d digits, whereas Mathematica seems to say it takes almost 4000d bytes of memory.

Where is my mistake? More importantly, is it possible to get the desired behavior from Mathematica (preferably without losing the speed), and if so, how? Thank you for your help.

Answer 1

Mathematica caches all sorts of expressions that come up on the course of evaluation. Some but not all of these are cleared by ClearSystemCache.

By your use of the expression Sqrt[1 + j^2] // N[#, d] & you are performing exact symbolic computations and only then converting these to numeric equivalents. Mathematica will cache values in this process.

Put another ClearSystemCache[]; in your code before you check mem2 and you will see a very different number:

ClearSystemCache[];
d = 10^4;
mem1 = MemoryInUse[];
Sum[Sqrt[1 + j^2] // N[#, d] &, {j, 1, d}];
ClearSystemCache[];
mem2 = MemoryInUse[];
(mem2 - mem1)/d // N

9.9984

More suitable however is to avoid exact symbolic calculation in the first place.

Here I replace 1 with an arbitrary precision one with d digits of Precision:

$HistoryLength = 0;
d = 10^4
one = SetPrecision[1, d];
mem1 = MemoryInUse[]
out = Sum[Sqrt[one + j^2], {j, 1, d}];
mem2 = MemoryInUse[]
(mem2 - mem1)/d // N

10000

26294360

26298432

0.4072

This is sufficient to compute your result:

Precision[out]

10007.

In the code above simply allowing the result of d = 10^4 to print by omitting ; uses some memory, resulting in the final delta being smaller. (Since mem1 = MemoryInUse[] comes after this.) I suspect that this is because certain mechanisms are not fully primed until an expression is printed.

I also added $HistoryLength = 0.