# Making sure that Timing[] isn't thrown off by caching

In playing with and comparing the various answers to my question here, I tried applying Timing[] to see how fast things ran, but it seemed that some of the answers (but not all of them) were a lot faster when run for a second time. How can I make sure that there isn't some kind of caching of intermediate results throwing off the Timing[]?

-
Not your question, but perhaps it is better to use AbsoluteTiming for "benchmarking", Timing does strange stuff. For example it has problems with multiple CPUs. –  user21 Feb 1 '12 at 17:58
@ruebenko: That sounds like a good idea—it's often very useful to answer the question you think I should have been asking (or try to address the underlying or larger issue), rather than just focusing on what I actually asked. –  Isaac Feb 1 '12 at 20:17

Completely restarting the kernel will of course work. If we don't restart the kernel, then we need to clear all caches.

The caches used for symbolic and some numeric calculations can be cleared using ClearSystemCache[]. The documentation page of this function says:

ClearSystemCache can be useful in generating worst-case timing results independent of previous computations.

I do not know if there are any other caches as well, not affected by this.

-
It appears that Factorial (!) is not effected by ClearSystemCache. For example: ClearSystemCache[]; Map[(ClearSystemCache[]; Clear[m]; m = MemoryInUse[]; N[Pi, 10^5]; MemoryInUse[] - m) &, Range[10]] returns {42304, 42304, 42304, 42304, 42304, 42304, 42304, 42304, 42304, 42304} Where: ClearSystemCache[]; Map[(ClearSystemCache[]; Clear[m]; m = MemoryInUse[]; 5!; MemoryInUse[] - m) &, Range[10]] returns {608, 24, 24, 24, 24, 24, 24, 24, 24, 24} The 608 value only happens if the kernel has been quit. –  mmorris Mar 12 '12 at 22:52

It's just a shot in the wild, but I would guess that you can switch off the caching with appropriate values to the corresponding system options:

SystemOptions["CacheOptions"]

(*
==> {"CacheOptions" -> {"CacheKeyMaxBytes" -> 1000000,
"CacheResultMaxBytes" -> 1000000, "Constants" -> True,
"Numeric" -> True, "Symbolic" -> True}}
*)


It's hard to say whether playing with these will in fact give more reliable results, I could well imagine that some stuff just gets very slow. But it's probably worth a try.

A completeley different approach that I have used before is to ensure that there is nothing that can be cached, which often but not always is possible. One way is to achieve this is to use some randomness in the code you want to test, see e.g. my answer to this question.

-
If I were to change these, will they be reset to the defaults when I quit and relaunch Mathematica? –  Isaac Feb 1 '12 at 20:25
I don't think so, but I think you can reset them by setting them to Automatic. Nevertheless you problably want to store the defaults before starting to play with these. I should probably also cite the documentation of SetSystemOptions: System options specify parameters relevant to the internal operation of Mathematica. They should not normally be modified except under specific direction. If they are modified, Mathematica may not operate correctly. –  Albert Retey Feb 1 '12 at 20:33
Oh, and of course I'm by no means entitled to give you those specific directions :-) –  Albert Retey Feb 1 '12 at 20:47
@Isaac Yes if you call SetSystemOptions["CacheOptions" -> "CacheKeyMaxBytes" -> 0]; Once you quit the kernel it will reset to 1000000. –  mmorris Mar 12 '12 at 22:58
It is more reliable to just interleave ClearSystemCache[] between calls to whatever you are timing. Changing these system options is quite safe. The problem is that some functions, e.g. Together and Integrate, use intermediate caching to speed themselves (not just later calls to same function). So you could get degraded performance that would not reflect usual behavior if caching is disabled for intermediate steps. –  Daniel Lichtblau Mar 13 '12 at 15:41

When experimenting with Mathematica's caching, sometimes it can be clearer to look at memory usage rather than timing. When looking at individual functions, Mathematica is able to measure memory much more accurately than timing. The are a number of reasons why timing accuracy is off, such as the system clock resolution and fluctuations in the system use while testing. For more of an example check out the answer to this question.

When using ClearSystemCache, it has to be continually called in benchmarking studies. Lets look at the following example:

One call

Quit[]; (* Has to be in a cell by itself or the rest will not be evaluated. *)

m = MemoryInUse[];
N[Pi, 10^5];
MemoryInUse[] - m


Results:

42368


Iterating over the call 10 times. Not clearing the cache. FYI I am using map to collect the values for me.

Quit[];

Map[(m = MemoryInUse[]; N[Pi, 10^5]; MemoryInUse[] - m) &, Range[10]]
Map[(AbsoluteTiming[N[Pi, 10^5]][[1]]) &, Range[10]]


Results:

{42368, 0, 0, 0, 0, 0, 0, 0, 0, 0}
{0.068365, 0.000013, 5.*10^-6, 4.*10^-6, 5.*10^-6, 5.*10^-6, 4.*10^-6,
5.*10^-6, 4.*10^-6, 5.*10^-6}


Clearly we are seeing caching going on.

Iterating over the call 10 times. Clearing the cache and m value.

Quit[];

ClearSystemCache[]; Map[(ClearSystemCache[]; Clear[m];
m = MemoryInUse[]; N[Pi, 10^5]; MemoryInUse[] - m) &, Range[10]]
ClearSystemCache[]; Map[(ClearSystemCache[];
AbsoluteTiming[N[Pi, 10^5]][[1]]) &, Range[10]]


Results:

{42368, 42304, 42304, 42304, 42304, 42304, 42304, 42304, 42304, 42304}
{0.063965, 0.061679, 0.072600, 0.095275, 0.075488, 0.068122, 0.061311,
0.066843, 0.076715, 0.071130}


Alternatively using SetSystemOptions instead of ClearSystemCache.

Quit[];

SetSystemOptions[{"CacheOptions" -> {"CacheKeyMaxBytes" -> 0,
"CacheResultMaxBytes" -> 0, "Constants" -> False,
"Numeric" -> False, "Symbolic" -> False}}]
Map[(Clear[m]; m = MemoryInUse[]; N[Pi, 10^5]; MemoryInUse[] - m) &,
Range[10]]
Map[(AbsoluteTiming[N[Pi, 10^5]][[1]]) &, Range[10]]


Results:

{42072, 42072, 42072, 42072, 42072, 42072, 42072, 42072, 42072, 42072}
{0.059663, 0.054014, 0.053179, 0.053858, 0.053549, 0.053493, 0.053547,
0.053773, 0.054064, 0.053623}


I have not been able to explain the difference in memory used, 42368 vs 42304 vs 42072. Maybe Map is caching something? Note, that clearing / turning off caching may not effect all functions. Take Factorial (!) There must be some other caching going on. Neither ClearSystemCache nor SetSystemOptions has an effect. Both:

Quit[];

ClearSystemCache[]; Map[(ClearSystemCache[]; Clear[m];
m = MemoryInUse[]; 5!; MemoryInUse[] - m) &, Range[10]]


And:

Quit[];

SetSystemOptions[{"CacheOptions" -> {"CacheKeyMaxBytes" -> 0,
"CacheResultMaxBytes" -> 0, "Constants" -> False,
"Numeric" -> False, "Symbolic" -> False}}]


Result in:

{608, 24, 24, 24, 24, 24, 24, 24, 24, 24}

-
"FYI I am using map to collect the values for me." -- why use Map that way when you could use Array or Table? (+1 on your answer, by the way) –  Mr.Wizard Mar 13 '12 at 7:42
@Mr.Wizard No particular reason, other than I came across Map and have not much exposure to Array or Table. I have found my work in Mathematica a bit ... evolutionary. ;) I'll take a look. Thanks –  mmorris Mar 13 '12 at 13:57
I see. I didn't mean to criticize, though I do think those are more standard in this application. Many people here have their own methods and tricks and I have learned a lot from them. –  Mr.Wizard Mar 13 '12 at 14:01
@Mr.Wizard I did not take it as criticism. I appreciate the advice. –  mmorris Mar 13 '12 at 14:43
An example of how my work has evolved: Summing the items of the list = {1, 2, 3, 4}. Background I was looking for a one argument function like Min, Max, Mean, ... and did not have much initial success with help. I started with: sum = 0; i = 0; While[i++ < Length[list], sum += Part[list, i]]; sum which evolved to: Fold[Plus, 0, list] followed by: Sum[i, {i, list}] and finally: Total[list]. I used "finally" with hesitation, as there is always a chance that there is something more efficient. ;) –  mmorris Mar 13 '12 at 15:30