14
$\begingroup$

How can I ingest Mathematica / WL benchmark results posted at Mathematica Stack Exchange (MSE), like the ones posted here?

Assumptions:

  • There are multiple answers with benchmarks posted

  • The benchmark results have a certain clear "list of rules" form

See:

Needs["Benchmarking`"]
Benchmark[]

(* {"MachineName" -> "macbook-pro", "System" -> "Mac OS X x86 (64-bit)", "BenchmarkName" -> "WolframMark", "FullVersionNumber" -> "12.1.1", "Date" -> "November 26, 2020",
 "BenchmarkResult" -> 3.028, "TotalTime" -> 4.571, "Results" -> {{"Data Fitting", 0.291}, {"Digits of Pi", 0.249}, {"Discrete Fourier Transform", 0.363}, {"Eigenvalues of a Matrix", 0.297},
   {"Elementary Functions", 0.375}, {"Gamma Function", 0.356}, {"Large Integer Multiplication", 0.323}, {"Matrix Arithmetic", 0.227}, {"Matrix Multiplication", 0.218}, {"Matrix Transpose", 0.404},
   {"Numerical Integration", 0.463}, {"Polynomial Expansion", 0.068}, {"Random Number Sort", 0.449}, {"Singular Value Decomposition", 0.221}, {"Solving a Linear System", 0.267}}} *)
$\endgroup$
  • $\begingroup$ Please, if you upvoted (or inclined to do so) this question then (consider and) post benchmarks in the linked MSE discussion. And if you do post your benchmark results, please post WL code output from Benchmark[] not just screenshots. $\endgroup$ – Anton Antonov Nov 26 '20 at 19:13
  • $\begingroup$ What is "WL"? In some Mathematica format, file extension .wl? Or simply Mathematica code? Wolfram Language Packages? $\endgroup$ – Peter Mortensen Nov 27 '20 at 8:24
  • $\begingroup$ @PeterMortensen 1. "What is "WL"?" -- WL stands for "Wolfram Language". I used "Mathematica" and "WL" as synonyms. 2. "Or simply Mathematica code?" -- Yes, simply Mathematica code. (The output code from Benchmark[].) $\endgroup$ – Anton Antonov Nov 27 '20 at 11:23
  • $\begingroup$ I challenge whoever downvoted to explain the reasons for the downvote. :) $\endgroup$ – Anton Antonov Nov 28 '20 at 23:29
15
$\begingroup$

Ingestion

Get all answers from MSE discussion in an XML object:

xmlObject = 
  Import["https://mathematica.stackexchange.com/questions/234881/benchmarking-with-mathematica-v-12-for-up-to-date-comparision-of-mathematica-acr/235384#235384", "XMLObject"];

Get code XML elements:

lsRes1 = Cases[xmlObject, XMLElement["code", {}, code_] :> code, \[Infinity]];

(Note of that at this point code strings are obtained.)

Remove Benchmark computation commands and convert the code strings into WL code:

lsRes2 = Map[(ToExpression[Map[If[StringQ[#] && SyntaxQ[#], #, Nothing] &, #]] &) @
    StringReplace[#, {"LaunchKernels[]" -> "", "BenchmarkReport[]" -> "", "Benchmark[]" -> ""}] &, lsRes1]

(*21*)

Select code (blocks) that are lists of rules:

lsRes3 = Select[Flatten /@ lsRes2, MatchQ[#, {_Rule ..}] && Length[#] > 1 &];
Length[lsRes3]

(*21*)

For each code block flatten the metadata and benchmark results into associations:

lsRes4 = Map[Association@ Cases[Flatten[# /.  HoldPattern["Results" -> r_] :> (Rule @@@ r)], _Rule] &, lsRes3];
Length[lsRes4]

(*21*)

Make sure all associations have the same keys:

lsRes5 = 
 Block[{lsAllKeys = Union[Flatten[Keys /@ lsRes4]]}, 
  Map[Join[AssociationThread[lsAllKeys, "NA"], #] &, lsRes4]
 ];
Length[lsRes5]

(*21*)

(Otherwise, the obtained dataset might be hard to use in some computations.)

Make a comparison dataset:

dsBenchmarks = Dataset[lsRes5];
dsBenchmarks = dsBenchmarks[SortBy[#BenchmarkResult &]]

enter image description here

Basic analysis

Summarize the dataset:

ResourceFunction["RecordsSummary"][dsBenchmarks]

enter image description here

(Bar) plot results:

Multicolumn@
 KeyValueMap[
  If[VectorQ[DeleteCases[#2, _Missing | "NA"], NumericQ], 
    BarChart[#2, PlotLabel -> #1, PlotTheme -> "Detailed"], 
    Nothing
  ] &, 
  Normal@Transpose@dsBenchmarks
]

enter image description here

Breakdown statistics

Partition benchmark results per system (or other criteria):

aRes = GroupBy[Normal@dsBenchmarks, #System &, Dataset];
aRes = Map[#[All, Select[#, NumericQ] &] &, aRes];

Separate the overall benchmark results from the benchmark component results:

aRes2 = Map[#[All, KeyTake[#, {"BenchmarkResult", "TotalTime"}] &] &, aRes];
aRes3 = Map[#[All, KeyDrop[#, {"BenchmarkResult", "TotalTime"}] &] &, aRes];

Show the overall benchmark results:

aRes2

enter image description here

Show distribution statistical plots per system:

aResPlots3 = 
  BoxWhiskerChart[Transpose[Normal[#[Values]]], 
     PlotLabel -> Row[{"Number of benchmarks:", Spacer[2], Length[#]}], 
     ChartStyle -> 56, ImageSize -> Medium, PlotTheme -> "Detailed", 
     ChartLegends -> Keys[Normal@#[[1]]]] & /@ aRes3;
Grid[List @@@ Normal[aResPlots3], Dividers -> All, FrameStyle -> GrayLevel[0.7]]

enter image description here

(Note, that we have to keep an eye on the number of available benchmarks per system. Those numbers are too small here...)

Older plots

enter image description here

$\endgroup$
  • 1
    $\begingroup$ I’ve also come to find that the benchmark rarely uses 100% of my systems cpu to do calculations...i’ll get spikes up to 65% and a score that varies +- 0.3 $\endgroup$ – morbo Nov 28 '20 at 10:47
  • 1
    $\begingroup$ @morbo Hmm... interesting observation. I guess you can post question(s) at MSE about modifying the “standard” benchmark in a way that makes it more computationally challenging and/or how to get/include the CPU utilization. (Of course, certain algorithms just cannot be executed that effectively on certain CPU-OS configurations.) $\endgroup$ – Anton Antonov Nov 28 '20 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.