16
$\begingroup$

How can I ingest Mathematica / WL benchmark results posted at Mathematica Stack Exchange (MSE), like the ones posted here?

Assumptions:

  • There are multiple answers with benchmarks posted

  • The benchmark results have a certain clear "list of rules" form

See:

Needs["Benchmarking`"]
Benchmark[]

(* {"MachineName" -> "macbook-pro", "System" -> "Mac OS X x86 (64-bit)", "BenchmarkName" -> "WolframMark", "FullVersionNumber" -> "12.1.1", "Date" -> "November 26, 2020",
 "BenchmarkResult" -> 3.028, "TotalTime" -> 4.571, "Results" -> {{"Data Fitting", 0.291}, {"Digits of Pi", 0.249}, {"Discrete Fourier Transform", 0.363}, {"Eigenvalues of a Matrix", 0.297},
   {"Elementary Functions", 0.375}, {"Gamma Function", 0.356}, {"Large Integer Multiplication", 0.323}, {"Matrix Arithmetic", 0.227}, {"Matrix Multiplication", 0.218}, {"Matrix Transpose", 0.404},
   {"Numerical Integration", 0.463}, {"Polynomial Expansion", 0.068}, {"Random Number Sort", 0.449}, {"Singular Value Decomposition", 0.221}, {"Solving a Linear System", 0.267}}} *)
$\endgroup$
5
  • $\begingroup$ Please, if you upvoted (or inclined to do so) this question then (consider and) post benchmarks in the linked MSE discussion. And if you do post your benchmark results, please post WL code output from Benchmark[] not just screenshots. $\endgroup$ Nov 26, 2020 at 19:13
  • $\begingroup$ What is "WL"? In some Mathematica format, file extension .wl? Or simply Mathematica code? Wolfram Language Packages? $\endgroup$ Nov 27, 2020 at 8:24
  • $\begingroup$ @PeterMortensen 1. "What is "WL"?" -- WL stands for "Wolfram Language". I used "Mathematica" and "WL" as synonyms. 2. "Or simply Mathematica code?" -- Yes, simply Mathematica code. (The output code from Benchmark[].) $\endgroup$ Nov 27, 2020 at 11:23
  • $\begingroup$ I challenge whoever downvoted to explain the reasons for the downvote. :) $\endgroup$ Nov 28, 2020 at 23:29
  • $\begingroup$ Related community post: "Ingest and analyze benchmark results posted at MathematicaStackExchange". (You can get the notebook from there.) $\endgroup$ Nov 1, 2021 at 15:21

1 Answer 1

16
$\begingroup$

Ingestion

Get all answers from MSE discussion in an XML object:

xmlObject = 
  Import["https://mathematica.stackexchange.com/questions/234881/benchmarking-with-mathematica-v-12-for-up-to-date-comparision-of-mathematica-acr/235384#235384", "XMLObject"];

Get code XML elements:

lsRes1 = Cases[xmlObject, XMLElement["code", {}, code_] :> code, \[Infinity]];

(Note of that at this point code strings are obtained.)

Remove Benchmark computation commands and convert the code strings into WL code:

lsRes2 = Map[(ToExpression[Map[If[StringQ[#] && SyntaxQ[#], #, Nothing] &, #]] &) @
    StringReplace[#, {"LaunchKernels[]" -> "", "BenchmarkReport[]" -> "", "Benchmark[]" -> ""}] &, lsRes1]

Select code (blocks) that are lists of rules:

lsRes3 = Select[Flatten /@ lsRes2, MatchQ[#, {_Rule ..}] && Length[#] > 1 &];
Length[lsRes3]

(*31*)

For each code block flatten the metadata and benchmark results into associations:

lsRes4 = Map[Association@ Cases[Flatten[# /.  HoldPattern["Results" -> r_] :> (Rule @@@ r)], _Rule] &, lsRes3];
Length[lsRes4]

(*31*)

Make sure all associations have the same keys:

lsRes5 = 
 Block[{lsAllKeys = Union[Flatten[Keys /@ lsRes4]]}, 
  Map[Join[AssociationThread[lsAllKeys, "NA"], #] &, lsRes4]
 ];
Length[lsRes5]

(*31*)

(Otherwise, the obtained dataset might be hard to use in some computations.)

Make a comparison dataset:

dsBenchmarks = Dataset[lsRes5];
dsBenchmarks = dsBenchmarks[SortBy[#BenchmarkResult &]]

enter image description here

Basic analysis

Summarize the dataset:

ResourceFunction["RecordsSummary"][dsBenchmarks]

enter image description here

(Bar) plot results:

Multicolumn@
 KeyValueMap[
  If[VectorQ[DeleteCases[#2, _Missing | "NA"], NumericQ], 
    BarChart[#2, PlotLabel -> #1, PlotTheme -> "Detailed"], 
    Nothing
  ] &, 
  Normal@Transpose@dsBenchmarks
]

enter image description here

Breakdown statistics

Partition benchmark results per system (or other criteria):

aRes = GroupBy[Normal@dsBenchmarks, #System &, Dataset];
aRes = Map[#[All, Select[#, NumericQ] &] &, aRes];

Separate the overall benchmark results from the benchmark component results:

aRes2 = Map[#[All, KeyTake[#, {"BenchmarkResult", "TotalTime"}] &] &, aRes];
aRes3 = Map[#[All, KeyDrop[#, {"BenchmarkResult", "TotalTime"}] &] &, aRes];

Show the overall benchmark results:

aRes2

enter image description here

Show distribution statistical plots per system:

aResPlots3 = 
  BoxWhiskerChart[Transpose[Normal[#[Values]]], 
     PlotLabel -> Row[{"Number of benchmarks:", Spacer[2], Length[#]}], 
     ChartStyle -> 56, ImageSize -> Medium, PlotTheme -> "Detailed", 
     ChartLegends -> Keys[Normal@#[[1]]]] & /@ aRes3;
Grid[List @@@ Normal[aResPlots3], Dividers -> All, FrameStyle -> GrayLevel[0.7]]

enter image description here

(Note, that we have to keep an eye on the number of available benchmarks per system. Those numbers are too small here...)

Older plots

enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ I’ve also come to find that the benchmark rarely uses 100% of my systems cpu to do calculations...i’ll get spikes up to 65% and a score that varies +- 0.3 $\endgroup$ Nov 28, 2020 at 10:47
  • 1
    $\begingroup$ @morbo Hmm... interesting observation. I guess you can post question(s) at MSE about modifying the “standard” benchmark in a way that makes it more computationally challenging and/or how to get/include the CPU utilization. (Of course, certain algorithms just cannot be executed that effectively on certain CPU-OS configurations.) $\endgroup$ Nov 28, 2020 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.