Why the performance of list of Timings depend on the order?

It seems that the result of list of Timings and AbsoluteTimings depends on the order, while RepeatedTiming is free from this problem. How to understand this behaviour of timing functions?

The related codes are as follows. (The original aim is to compare the performance of similar functions on different types of data.)

Tested function

types = {"integer", "letter", "symbolLetter", "word", "symbolWord"}
functions = {complementVerbatim, Complement}
argnumber = 2;
(*argnumber specifies the number of arguments*)

complementVerbatim[list1_List, list2_List] :=
DeleteCases[list1, Alternatives @@ Verbatim /@ list2];


Sample space

Choose portion of the samples as test data and save them into downvalues of data.

size = 2 10^4;
portion = 0.6;

sampleSpace["integer"] = RandomInteger[2 size, size];
sampleSpace["letter"] = RandomChoice[Alphabet[], size];
sampleSpace["word"] = RandomWord[size];
sampleSpace["symbolLetter"] = ToExpression /@ sampleSpace["letter"];
sampleSpace["symbolWord"] = ToExpression /@ sampleSpace["word"];

Table[data[type, argnumber] =
Table[RandomSample[sampleSpace[type], portion size // Floor], {i, argnumber}],
{type, types}];


Timing

Now evaluate the functions on data with different order and timing functions.

Table[timing[Sequence @@ exception] := "/", {exception, exceptions}];

timing[function_, type_, argnumber, timingFunction_] :=
First@timingFunction[function @@ data[type, argnumber];]

timing[abs, fun] =
Outer[timing[#1, #2, argnumber, AbsoluteTiming] &, functions, types];
timing[abs, type] =
Outer[timing[#2, #1, argnumber, AbsoluteTiming] &, types, functions] // Transpose;

timing[re, fun] =
Outer[timing[#1, #2, argnumber, RepeatedTiming] &, functions, types];
timing[re, type] =
Outer[timing[#2, #1, argnumber, RepeatedTiming] &, types, functions] // Transpose;

timing[fun] =
Outer[timing[#1, #2, argnumber, Timing] &, functions, types];
timing[type] =
Outer[timing[#2, #1, argnumber, Timing] &, types, functions] // Transpose;

timingShow[x__] := SparseArraySparseBlockMatrix[{
{1, 1} -> {{""}},
{1, 2} -> {types},
{2, 1} -> List /@ functions,
{2, 2} -> timing[x]
}] // TableForm


Result

Interestingly the three timing functions give different results.

The results of RepeatedTiming don't depend on the order:

while the results of Timings and AbsoluteTimings do depend, and they are different from single runs like

timing[Complement, "integer", argnumber, Timing]
(*0.011819*)
(*comparing with 0.004266 in the table.*)


hence are untrustable.

• There is a reason why timeAvg[] was invented and then RepeatedTiming was added to Mma. Timing is said to be unreliable in a parallelized/hyperthreaded environment because not everything is counted the same on every system. AbsoluteTiming measures elapsed time according to the "wall clock" and depends on how busy your computer is. Jun 26 at 16:13
• @MichaelE2 I'm new here. For timeAvg[] do you mean this answer? mathematica.stackexchange.com/a/22427/86893 Jun 27 at 10:34
• Yep, and it appears in earlier posts, too. The earliest variant I know is this (or better ref). Jun 27 at 13:35

• "Timing is accurate only down to a granularity of at least $TimeUnit seconds." See here. • "AbsoluteTiming is always accurate down to a granularity of $TimeUnit seconds, but on many systems is much more accurate." See here.
I just evaluated $TimeUnit on my machine and got $$\frac{1}{100}$$, hence I should not expect timings of less than a few tens of milliseconds to be accurate when obtained using Timing or AbsoluteTiming. Therefore use RepeatedTiming if you are anywhere close to $TimeUnit.
• Thanks for pointing me the $TimeUnit issue! I haven't notice that before. Jun 29 at 21:49 • I set the size to 10^7, and indeed the timings above $TimeUnit don't depend on the order of the list anymore. But the timings below \$TimeUnit are probably not random, since I have run the code many times, and the results are stable. Jun 29 at 21:55
• Sure. If for your original size you expect there is a systematic difference between say timing[fun] versus timing[type], and you wanted to really know, then you could run each many (say 100) times, store all the results, then draw a Histogram` to compare. Or make a proper statistical analysis to see if the difference is statistically significant. Jun 30 at 4:34