# Benchmarking: unexplained switching between high and low performance

tl;dr I am trying to accurately benchmark some vectorized operations, and compare them between systems. But benchmarking is hard to do well, and I am getting inconsistent results: performance is switching, apparently randomly, between "slow" and "fast". Why?Link to Woflram Community version.

Here is some code that benchmarks adding two packed arrays of size n, where n is just above a million. The timing is measured 5 times, to ensure consistency, then n is increased a bit, then the summation is timed again, etc. The whole benchmark is repeated twice.

Table[
n = 1000000 + k;
SeedRandom[120];
a = RandomReal[1, n];
b = RandomReal[1, n];
{k, Table[First@RepeatedTiming[a + b;], {5}]},
{2}, {k, 20000, 200000, 20000}
]


The results are below. In each row, the first number is the array size, the rest are the 5 timings.

{{ {20000, {0.000799, 0.000801, 0.000797, 0.000804, 0.000800}},
{40000, {0.00224, 0.00225, 0.00223, 0.00224, 0.00223}},
{60000, {0.00226, 0.00226, 0.00227, 0.00226, 0.00226}},
{80000, {0.00229, 0.00229, 0.00229, 0.00229, 0.00229}},
{100000, {0.00087, 0.000868, 0.000874, 0.000873, 0.00089}},
{120000, {0.00235, 0.00236, 0.00235, 0.00236, 0.00235}},
{140000, {0.00240, 0.00240, 0.00240, 0.00239, 0.00240}},
{160000, {0.00245, 0.00246, 0.00245, 0.00246, 0.00245}},
{180000, {0.00097, 0.000964, 0.000965, 0.000961, 0.000963}},
{200000, {0.00255, 0.00258, 0.00254, 0.00256, 0.00254}}},

{{20000, {0.00224, 0.00224, 0.00224, 0.00220, 0.00221}},
{40000, {0.00224, 0.00224, 0.00223, 0.00224, 0.00223}},
{60000, {0.00227, 0.00227, 0.00227, 0.00226, 0.00227}},
{80000, {0.00234, 0.00235, 0.00233, 0.00230, 0.00230}},
{100000, {0.00233, 0.00232, 0.00232, 0.00233, 0.00233}},
{120000, {0.00234, 0.00238, 0.00235, 0.00239, 0.00237}},
{140000, {0.00238, 0.00238, 0.00238, 0.00238, 0.00238}},
{160000, {0.00247, 0.00245, 0.00245, 0.00246, 0.00245}},
{180000, {0.000965, 0.000961, 0.000962, 0.000967, 0.000968}},
{200000, {0.00254, 0.00259, 0.00255, 0.00254, 0.00254}}}}


Things to notice:

• The 5 timings for the same array are always consistent.
• The timings are generally proportional to the array size.
• However, I see some "fast" (about 0.0008 s) and some "slow" (about 0.002 s) timings.
• Between the two runs, it is not always the same array size that is fast. Look at 20,000, 80,000 and 180,000 in the first run and 180,000 in the second run. These change randomly between runs.
• "Slow" and "fast" differ by a very significant factor of about 2.5-2.6.

Why do I see this switching between fast and slow timing? What is causing it? It prevents me from getting consistent benchmark results.

The measurements were done with Mathematica 11.1.0 on a 2014 MacBook Pro (Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz, 4 cores) connected to AC power. Turbo Boost is disabled using this tool.

I closed other programs as much as possible (but there are always many background tasks on a modern OS).

You might think that such short timings are not relevant in real-world applications. But remember that RepeatedTiming repeats the operation enough times to run for at least a second. I still get some fast timings if I increase this to as many as 15 seconds, or if I run the test many times consecutively.

I compared RepeatedTiming with AbsoluteTiming@Do[..., {bigNumber}], and there is no difference (other than the occasional fast-slow switching).

I noticed that longer arrays are less likely to produce fast timings than shorter ones, but I am not sure. I also noticed that running a long one tends to cause the subsequent short one to be slow again. Due to the fickle nature of the results, it is hard to be sure about these things.

On first sight, this may not look like a Mathematica question. But benchmarking is hard, and many things can go wrong. If I post it on another site, people may rightfully suspect that it is something specific to Mathematica that is causing it.

I believe that vector arithmetic is parallelized in Mathematica. There is an interesting talk here by Mark Sofroniou from WTC2016, which also discusses how the stragety used to distribute parlallel threads between cores can have a significant impact on performance.

UPDATE

Here's modified code by Sander Huisman on Wolfram Community. This makes it easier to reproduce the slow-fast pattern. It also shows another effect: when evaluating the same thing multiple times, it gets slower with each evaluation.

timings = Table[n = 1000000 + 8000 j;
SeedRandom[120];
a = RandomReal[1, n];
b = RandomReal[1, n];
First@AbsoluteTiming[Do[a + b;, {500}]], {10}, {j, 20}]
ListPlot[Flatten[timings], PlotRange -> {0, All}]


With Turbo Boost turn on on my machine:

With Turbo Boost off:

Running this takes a long time because I wanted to have timings that are over a second, for reliability. To just try this out, reduce the {500} to {50} in the Do loop.

UPDATE 2:

I did a comparison with MATLAB 2017a. I believe that both Mathematica and MATLAB use the MKL, and this should perform similarly on vector arithmetic.

a=rand(1,1000000);
b=rand(1,1000000);

tic; for i=1:1000
c=a+b;
end;toc


The timing (divided by 1000 to account for the repetitions) is 0.00065 s. This happens to be the same as the "fast" timing I get from Mathematica. But in MATLAB, I get the "fast" timing consistently.

This suggests that what I see is "a bug" in Mathematica, or rather that there is a potential for a speedup by a factor of 2.5 or more on my hardware and OS.

• Yes I meant what you thought. Looking a bit more I am also now of the opinion that it is not a granularity issue. It might or might not be an issue in RepeatedTiming internals. I'm leaning toward it being determined by packed array byte alignment, but that is still just a guess. – Daniel Lichtblau Mar 30 '17 at 16:09
• What OS are you using? Have you tired switching off hyper-threading in the BIOS? – user21 Mar 31 '17 at 8:05
• It would be interesting to see if this is Mac only issue, as there the MKL is linked in a different manner than other OSes. Has anyone with a non Mac reproduced this? – user21 Mar 31 '17 at 11:37
• @Szabolcs I would be curious if there is any improvement if the timing is measured with InternalArithmeticTiming[Plus, a, b]. – ilian Mar 31 '17 at 16:25
• It's a low-level method to (almost) directly time the actual operation, so this likely means the arithmetic is performing as expected while the fluctuations have more to do with things like going through the evaluator, repeatedly allocating memory for the result and/or caching. Why Mac would be different from other operating systems is still a bit of an open question. – ilian Apr 11 '17 at 17:37

You did supply any details on your platform. Mine is Windows 10, Prof, Anniversary Update, 64-bit Intel Xeon CPU E3-1220 v3 @ 3.10GHz, 4 Core(s), no Hyperthreading 32 GB ECC RAM memory Mathematica 11.0

I don't know how to insert code on StackExchange and I cannot find an explanation anywhere on how to do it, so I must describe what I did.

Flatten[%, 1]
`