# BoxWhiskerChart too slow on big datasets, how to speed it up?

I have very big dataset, with over 14 000 000 records. And maybe more! Each record consists of Data ( Real ) and Time ( integer ).

I group records by time interval, resulting in 300 groups. In each group there are 70 000 records.

I need to draw a BoxWhiskerChart.

BUT, The system has frozen for 15 minutes. And then work is very slow. I think the reason this froze, is that I give it groups that are very large. But what can I do?

-
Welcome to SE Mathematica, Andrey. Could you specify where your code freezes? Could you load the data and Mathematica only fails plotting the chart, or you couldn't even import it successfully? Some information on how to load big data is here. – István Zachar Apr 18 '12 at 10:05
Andrey, can I also suggest that you register? Any future posts will then be linked properly to each other. – Verbeia Apr 18 '12 at 11:13

Even on my rather old laptop, calculating the various statistics used for plotting the whisker of data sets of that size takes less than a minute so I suspect the problem is the number of groups rather than the number of records in each group. For each box in your plot, Mathematica generates a Tooltip containing a DynamicsBox. With 300 boxes, this will slow down the front end quite a lot.

It's a bit more involved than using BoxWhiskerChart out of the box, but as an alternative you could build a chart by hand using graphics primitives, for example

plot[data_] := DynamicModule[{n, min, max, q25, med, q75, fun},
n = Length[data];
{min, q25, med, q75, max} =
Transpose[
Through[{Min, Quantile[#, 1/4] &, Median, Quantile[#, 3/4] &, Max}[#]] & /@ data];
fun[dat_] := Transpose[{Range[n], dat}];
Dynamic[Tooltip[Graphics[{
{LightGray, Polygon[Join[fun[min], Reverse@fun[max]]]},
{LightBlue, Polygon[Join[fun[q25], Reverse@fun[q75]]]},
{Thick, Black, Line[fun[med]]},
{Gray, Line[{{#, min[[#]]}, {#, max[[#]]}}]},
{Thick, Blue, Line[{{#, q25[[#]]}, {#, q75[[#]]}}]}},
Axes -> True, AspectRatio -> 1/GoldenRatio],
Grid[{{"Index", #}, {"Max", max[[#]]}, {"75%",
q75[[#]]}, {"Median", med[[#]]}, {"25%", q25[[#]]},
{"Min", min[[#]]}}, Frame -> All]] &@
Clip[Round[MousePosition["Graphics"] /. None -> {1}][[1]], {1, n}]]]

data = Table[RandomVariate[NormalDistribution[i, i^2], 1000], {i, .1, 1, 1/300}];

plot[data]


Here, I've chosen to plot the boxes and whiskers as polygons since with 300 boxes side-by-side it's hard to distinguish the individual boxes anyway.

-
Using PerformanceGoal -> "Speed" as an option to BoxWhiskerChart will stop the tooltips. – wxffles Apr 18 '12 at 21:18