0
$\begingroup$

It seems that the result of list of Timings and AbsoluteTimings depends on the order, while RepeatedTiming is free from this problem. How to understand this behaviour of timing functions?

The related codes are as follows. (The original aim is to compare the performance of similar functions on different types of data.)

Tested function

types = {"integer", "letter", "symbolLetter", "word", "symbolWord"}
functions = {complementVerbatim, Complement}
argnumber = 2;
(*`argnumber` specifies the number of arguments*)

complementVerbatim[list1_List, list2_List] := 
  DeleteCases[list1, Alternatives @@ Verbatim /@ list2];

Sample space

Choose portion of the samples as test data and save them into downvalues of data.

size = 2 10^4;
portion = 0.6;

sampleSpace["integer"] = RandomInteger[2 size, size];
sampleSpace["letter"] = RandomChoice[Alphabet[], size];
sampleSpace["word"] = RandomWord[size];
sampleSpace["symbolLetter"] = ToExpression /@ sampleSpace["letter"];
sampleSpace["symbolWord"] = ToExpression /@ sampleSpace["word"];

Table[data[type, argnumber] = 
   Table[RandomSample[sampleSpace[type], portion size // Floor], {i, argnumber}],
  {type, types}];

Timing

Now evaluate the functions on data with different order and timing functions.

Table[timing[Sequence @@ exception] := "/", {exception, exceptions}];

timing[function_, type_, argnumber, timingFunction_] :=
First@timingFunction[function @@ data[type, argnumber];]

timing[abs, fun] = 
  Outer[timing[#1, #2, argnumber, AbsoluteTiming] &, functions, types];
timing[abs, type] = 
  Outer[timing[#2, #1, argnumber, AbsoluteTiming] &, types, functions] // Transpose;

timing[re, fun] = 
  Outer[timing[#1, #2, argnumber, RepeatedTiming] &, functions, types];
timing[re, type] = 
  Outer[timing[#2, #1, argnumber, RepeatedTiming] &, types, functions] // Transpose;

timing[fun] = 
  Outer[timing[#1, #2, argnumber, Timing] &, functions, types];
timing[type] = 
  Outer[timing[#2, #1, argnumber, Timing] &, types, functions] // Transpose;

timingShow[x__] := SparseArray`SparseBlockMatrix[{
    {1, 1} -> {{""}},
    {1, 2} -> {types},
    {2, 1} -> List /@ functions,
    {2, 2} -> timing[x]
}] // TableForm

Result

Interestingly the three timing functions give different results.

The results of RepeatedTiming don't depend on the order:

while the results of Timings and AbsoluteTimings do depend, and they are different from single runs like

timing[Complement, "integer", argnumber, Timing]
(*0.011819*)
(*comparing with 0.004266 in the table.*)

hence are untrustable.

$\endgroup$
3
  • 1
    $\begingroup$ There is a reason why timeAvg[] was invented and then RepeatedTiming was added to Mma. Timing is said to be unreliable in a parallelized/hyperthreaded environment because not everything is counted the same on every system. AbsoluteTiming measures elapsed time according to the "wall clock" and depends on how busy your computer is. $\endgroup$
    – Michael E2
    Jun 26 at 16:13
  • $\begingroup$ @MichaelE2 I'm new here. For timeAvg[] do you mean this answer? mathematica.stackexchange.com/a/22427/86893 $\endgroup$
    – lilyric
    Jun 27 at 10:34
  • 1
    $\begingroup$ Yep, and it appears in earlier posts, too. The earliest variant I know is this (or better ref). $\endgroup$
    – Michael E2
    Jun 27 at 13:35

1 Answer 1

2
$\begingroup$

I think your table does not show a systematic bias, just random variation. If you run again, then you will get different results. From the documentation:

  • "Timing is accurate only down to a granularity of at least $TimeUnit seconds." See here.

  • "AbsoluteTiming is always accurate down to a granularity of $TimeUnit seconds, but on many systems is much more accurate." See here.

I just evaluated $TimeUnit on my machine and got $\frac{1}{100}$, hence I should not expect timings of less than a few tens of milliseconds to be accurate when obtained using Timing or AbsoluteTiming. Therefore use RepeatedTiming if you are anywhere close to $TimeUnit.

$\endgroup$
3
  • $\begingroup$ Thanks for pointing me the $TimeUnit issue! I haven't notice that before. $\endgroup$
    – lilyric
    Jun 29 at 21:49
  • $\begingroup$ I set the size to 10^7, and indeed the timings above $TimeUnit don't depend on the order of the list anymore. But the timings below $TimeUnit are probably not random, since I have run the code many times, and the results are stable. $\endgroup$
    – lilyric
    Jun 29 at 21:55
  • $\begingroup$ Sure. If for your original size you expect there is a systematic difference between say timing[fun] versus timing[type], and you wanted to really know, then you could run each many (say 100) times, store all the results, then draw a Histogram to compare. Or make a proper statistical analysis to see if the difference is statistically significant. $\endgroup$
    – user293787
    Jun 30 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.