Making sure that Timing[] isn't thrown off by caching

Question

In playing with and comparing the various answers to my question here, I tried applying Timing[] to see how fast things ran, but it seemed that some of the answers (but not all of them) were a lot faster when run for a second time. How can I make sure that there isn't some kind of caching of intermediate results throwing off the Timing[]?

Answer 1

Completely restarting the kernel will of course work. If we don't restart the kernel, then we need to clear all caches.

The caches used for symbolic and some numeric calculations can be cleared using ClearSystemCache[]. The documentation page of this function says:

ClearSystemCache can be useful in generating worst-case timing results independent of previous computations.

I do not know if there are any other caches as well, not affected by this.

Answer 2

When experimenting with Mathematica's caching, sometimes it can be clearer to look at memory usage rather than timing. When looking at individual functions, Mathematica is able to measure memory much more accurately than timing. The are a number of reasons why timing accuracy is off, such as the system clock resolution and fluctuations in the system use while testing. For more of an example check out the answer to this question.

When using ClearSystemCache, it has to be continually called in benchmarking studies. Lets look at the following example:

---

One call

Quit[]; (* Has to be in a cell by itself or the rest will not be evaluated. *)

m = MemoryInUse[];
N[Pi, 10^5];
MemoryInUse[] - m

Results:

42368

---

Iterating over the call 10 times. Not clearing the cache. FYI I am using map to collect the values for me.

Quit[];

Map[(m = MemoryInUse[]; N[Pi, 10^5]; MemoryInUse[] - m) &, Range[10]]
Map[(AbsoluteTiming[N[Pi, 10^5]][[1]]) &, Range[10]]

Results:

{42368, 0, 0, 0, 0, 0, 0, 0, 0, 0}
{0.068365, 0.000013, 5.*10^-6, 4.*10^-6, 5.*10^-6, 5.*10^-6, 4.*10^-6,
  5.*10^-6, 4.*10^-6, 5.*10^-6}

Clearly we are seeing caching going on.

---

Iterating over the call 10 times. Clearing the cache and m value.

Quit[];

ClearSystemCache[]; Map[(ClearSystemCache[]; Clear[m]; 
   m = MemoryInUse[]; N[Pi, 10^5]; MemoryInUse[] - m) &, Range[10]]
ClearSystemCache[]; Map[(ClearSystemCache[]; 
   AbsoluteTiming[N[Pi, 10^5]][[1]]) &, Range[10]]

Results:

{42368, 42304, 42304, 42304, 42304, 42304, 42304, 42304, 42304, 42304}
{0.063965, 0.061679, 0.072600, 0.095275, 0.075488, 0.068122, 0.061311,
  0.066843, 0.076715, 0.071130}

---

Alternatively using SetSystemOptions instead of ClearSystemCache.

Quit[];

SetSystemOptions[{"CacheOptions" -> {"CacheKeyMaxBytes" -> 0, 
    "CacheResultMaxBytes" -> 0, "Constants" -> False, 
    "Numeric" -> False, "Symbolic" -> False}}]
Map[(Clear[m]; m = MemoryInUse[]; N[Pi, 10^5]; MemoryInUse[] - m) &, 
 Range[10]]
Map[(AbsoluteTiming[N[Pi, 10^5]][[1]]) &, Range[10]]

Results:

{42072, 42072, 42072, 42072, 42072, 42072, 42072, 42072, 42072, 42072}
{0.059663, 0.054014, 0.053179, 0.053858, 0.053549, 0.053493, 0.053547,
  0.053773, 0.054064, 0.053623}

I have not been able to explain the difference in memory used, 42368 vs 42304 vs 42072. Maybe Map is caching something? Note, that clearing / turning off caching may not effect all functions. Take Factorial (!) There must be some other caching going on. Neither ClearSystemCache nor SetSystemOptions has an effect. Both:

Quit[];

ClearSystemCache[]; Map[(ClearSystemCache[]; Clear[m]; 
   m = MemoryInUse[]; 5!; MemoryInUse[] - m) &, Range[10]]

And:

Quit[];

SetSystemOptions[{"CacheOptions" -> {"CacheKeyMaxBytes" -> 0, 
    "CacheResultMaxBytes" -> 0, "Constants" -> False, 
    "Numeric" -> False, "Symbolic" -> False}}]

Result in:

{608, 24, 24, 24, 24, 24, 24, 24, 24, 24}

Answer 3

It's just a shot in the wild, but I would guess that you can switch off the caching with appropriate values to the corresponding system options:

SystemOptions["CacheOptions"]

(*
==> {"CacheOptions" -> {"CacheKeyMaxBytes" -> 1000000, 
   "CacheResultMaxBytes" -> 1000000, "Constants" -> True, 
   "Numeric" -> True, "Symbolic" -> True}}
*)

It's hard to say whether playing with these will in fact give more reliable results, I could well imagine that some stuff just gets very slow. But it's probably worth a try.

A completeley different approach that I have used before is to ensure that there is nothing that can be cached, which often but not always is possible. One way is to achieve this is to use some randomness in the code you want to test, see e.g. my answer to this question.