3
$\begingroup$

Suppose that data is a large data sample, and that Histogram[data] produces a very smooth-looking distribution.

Since data eats up a lot of RAM, I'd like to summarize it with some analytic model. (For many operations I want to do, particularly while prototyping visualizations, etc., it would be fine to work with an analytic approximation to the actual data.)

Mathematica has functions like HistogramDistribution, EmpiricalDistribution, and SmoothKernelDistribution, which provide functional replacements for the original data, but AFAICT, the resulting objects still encapsulate all the original datapoints, and therefore their use would entail no reduction in the amount of memory required.

Does Mathematica have a ready-made way to extract an analytic model from the data sample?

$\endgroup$
  • 2
    $\begingroup$ As @JJM has shown in his answer, there exists functions to do so. However, boiling down a large data set to a few parameters and a functional form does not allow one to infer that the dataset is the equivalent of many random and independent samples from such a probability distribution along with any other properties of such sampling. If your samples are indeed from independent random samples, then everything's fine. But is there serial correlation over time? Is the variability constant over time? The point is the usefulness also depends on how the data came into being. $\endgroup$ – JimB May 19 '16 at 19:26
2
$\begingroup$

If you know the model form of the distribution you can use

EstimatedDistribution[data, GammaDistribution[\[Alpha], \[Beta]]]

or any other parametric distribution. These have a small memory footprint.

For a non-parametric approach you can use the bin specifications of HistogramDistribution to reduce the memory footprint of the resulting DataDistribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.