4
$\begingroup$

I am trying to visualize a list of about 1 million {x, y} tuples, and for this I made

Histogram3D[data, {5},
 PlotRange -> {{0, 100}, {0, -100}, Automatic},
 AxesLabel -> {"x", "y", "Count"},
 LabelingFunction -> Tooltip
]

Histogram3D

This is good enough to see a general trend, but now I'd like to pick a bin (say, $x \in [-5,0]$ or $y \in [50,55]$) and take a cross section from there to better understand how changing that one variable affects the distribution of the other.

What is a good way of doing this interactively? I tried controlling the plot range with Manipulate[hist, {xrange, 0, 100, 5}] but it's extremely slow and always aborts. I assume it's recalculating the bin counts for the whole data set every time the value changes. It would also mean I have to edit the code if I want to see it from a bin on the $y$ axis which is not what I want.

Something like making all but the relevant bars transparent, or showing multiple discrete 2D histograms on a Manipulate window would be fine.

$\endgroup$

1 Answer 1

6
$\begingroup$

You should give up on displaying histograms as counts and use the "PDF" option as you're wanting to estimate a probability distribution. Also, you should give up on histograms as there is an automatic way to obtain a "smooth histogram" which is generally what one expects the "true" distribution to be (i.e., the underlying probability density function is many times expected to be relatively smooth).

Generate some data and produce a smooth estimate of the underlying distribution. (I've only used 10,000 observations to speed things up.)

dist1 = CopulaDistribution[{"FGM", 0.2}, {BetaDistribution[1, 3], BetaDistribution[3, 1]}];
dist2 = CopulaDistribution[{"FGM", 0.2}, {BetaDistribution[6, 6], BetaDistribution[6, 6]}];
n = 10000;
SeedRandom[12345];
data = Join[RandomVariate[dist1, n/2], RandomVariate[dist2, n/2]];
data[[All, 1]] = -100 data[[All, 1]];
data[[All, 2]] = 100 data[[All, 2]];

SmoothHistogram3D[data, {Automatic, {"Bounded", {{-100, 0}, {0, 100}}, "Gaussian"}}, "PDF",
 AxesLabel -> {"x", "y", "Probability density"},
 PlotRange -> {{-100, 0}, {0, 100}, {0, Automatic}}, 
 RotationAction -> "Clip",
 SphericalRegion -> True, ImageSize -> Large, 
 ImagePadding -> {{100, 100}, {10, 10}}]

Smooth 3D density plot

You can produce estimates of the probability density conditional on a range of y values in the following manner:

Manipulate[
 x = Select[data, y - binWidth/2 <= #[[2]] < y + binWidth/2 &][[All, 1]];
 SmoothHistogram[x, {Automatic, {"Bounded", {-100, 0}, "Gaussian"}}, "PDF",
  PlotRange -> {{-100, 0}, {0, maxVerticalAxis}}, PlotPoints -> 200, 
  MaxRecursion -> 3, Frame -> True, FrameLabel -> {"x", "Probability density"}],
 {{y, 50}, 1, 99, Appearance -> "Labeled"},
 {{binWidth, 5}, 2, 20, Appearance -> "Labeled"},
 {{maxVerticalAxis, 0.06}, 0.01, 0.20, Appearance -> "Labeled"}]

Manipulate function of density slices

Addition:

If you need speed for a demo, then constructing the conditional pdf's ahead of time will fix that. Here's a limited example:

p = ConstantArray[0, 99];
Do[x = Select[data, y - 5/2 <= #[[2]] < y + 5/2 &][[All, 1]];
 p[[y]] = SmoothHistogram[x, {Automatic, {"Bounded", {-100, 0}, "Gaussian"}},
   "PDF", PlotRange -> {{-100, 0}, {0, 0.06}}, PlotPoints -> 200,
   Frame -> True, FrameLabel -> {"x", "Probability density"},
   PlotLabel -> ToString[NumberForm[Max[0, y - 2.5], 3]] <> " <= y <= " <> 
     ToString[NumberForm[Min[100, y + 2.5], 3]]],
 {y, 1, 99}]

Manipulate[p[[i]],
 {{i, 1, "y"}, 1, 99, 1}, TrackedSymbols :> {i}, SaveDefinitions -> True]

Manipulate of conditional pdf

If you save the Manipulate[... line to a separate file, then the SaveDefinitions -> True option will store the individual images so you don't have to recreate those each time you open that file.

$\endgroup$
7
  • $\begingroup$ Great answer, but hey, while you're at it why not give the Manipulate a 2nd view - sections of the x axis as well? $\endgroup$
    – Jagra
    Jul 27 at 21:30
  • 1
    $\begingroup$ Nice answer @JimB (+1)! It pains me when people use histograms in Mathematica or any other serious tools that offer much better PDF estimators. In my books, if somebody is using a histogram on a continuous variable, most likely, they are doing the wrong thing. $\endgroup$
    – rhermans
    Jul 28 at 10:22
  • $\begingroup$ Very nice, thank you! To be fair I went for a discrete histogram and not for a KDE because I thought it would be much faster for something with Dynamic $\endgroup$
    – user589632
    Jul 28 at 14:55
  • $\begingroup$ "thought" ? Did you try it? The most time consuming part with a million observations is the selection of a subset of the data (2.2 seconds), followed by 0.24 seconds for the smooth histogram and 0.07 seconds for the discrete histogram. The discrete histogram is 3 to 4 times faster than the smooth histogram but does that really make a difference? $\endgroup$
    – JimB
    Jul 28 at 15:23
  • $\begingroup$ If speed (because you probably want this as a nice demonstration to others) is what is important, then you can compute the smooth histograms ahead of time and make it lightning fast. $\endgroup$
    – JimB
    Jul 28 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.