3
$\begingroup$

BOUNTY GOAL

To get bounty I am asking to build a function that serves as an analog of FindDistribution. You can also simply re-implement FindDistribution if you know how it works. Your solution can work in same or different ways, have same or different syntax and options, but it needs to give a similar output: best fitted distributions to a data, desirably ranked by fitness. Minimal working function demoed on ay data example is acceptable, even in a quite raw state.

ORIGINAL POST

Friends, especially those with math and stats background, perhaps you could enlighten. We face the following general problem quite often. Perhaps as a community we should find solution working especially well inside Wolfram ecosystem. Moreover - I think some of you might have such strategies - would be nice to share with community.

FindDistribution is an excellent automation for search of analytic statistical models fitting data. But it is sophisticated large machine learning implementation and we cannot control the algorithm there unless we explicitly rewrite or reinvent it. I wonder if we can build a simple but exhaustive algorithm taking in account all available in Wolfram Language statistical distributions (see related) that acts similar to FindDistribution.

For the data consider any that is listed in the APPLICATIONS section of docs on FindDistribution. Or any data you like.

I suggest the following very simple idea for a start:

  • Take all set of analytical distributions in Wolfram Language (suitable to your data - e.g. all continuous or all discrete) -- also related

  • Use EstimatedDistribution or FindDistributionParameters to fit all of them

  • Run some tests - like PearsonChiSquareTest or DistributionFitTest or anything similar and find top models according to some criteria

  • If even top models are still far from a good fit use things like MixtureDistribution and TransformedDistribution etc. to derive better models. See Derived Statistical Distributions

  • We could also think of additional criteria that chooses models of smaller complexity - in terms of e.g. number of fitted parameters.

This is very blueprint-ish :-) I lack the deeper vision that takes in combination both: stats knowledge and Wolfram ecosystem structure. I hope some of you got the insight.

Feel free to demonstrate any strategy on any simple data I described above. Thsi might give some scope:

Improve this question
$\endgroup$
5
  • $\begingroup$ As a statistician I hope that a "solution" is never found as a one-size-fits-all solution doesn't exist. While I agree that FindDistribution is excellent in terms of automation, it doesn't challenge the user with respect to how the data was collected or to the objective of obtaining the distribution. That information is not an intrinsic property of the data values. Such a question might be better asked at CrossValidated (stats.stackexchange.com). $\endgroup$
    – JimB
    Commented Apr 24, 2023 at 0:08
  • $\begingroup$ @JimB Agree with all, but CrossValidated. I changed the title. I meant specifically relying on Wolfram set of statistical distributions and various stats tests and other tools. $\endgroup$
    – Vitaliy Kaurov
    Commented Apr 24, 2023 at 4:58
  • 3
    $\begingroup$ One more attempt at convincing you about CrossValidated: Determining a "reasonable approach" to what you want is (generally) independent of the software used. Then with a reasonable approach, implementing with Mathematica can happen. (And to repeat my standard sermon: Taking into consideration the objective and subject is not found in a set of numbers.) $\endgroup$
    – JimB
    Commented Apr 24, 2023 at 5:35
  • 2
    $\begingroup$ CrossValidated has a bunch of mathematicians, they require precisely specified problems. This SE has physicists, physicists can answer questions mathematicians can't :) $\endgroup$
    – Yaroslav Bulatov
    Commented May 1, 2023 at 18:20
  • 2
    $\begingroup$ @YaroslavBulatov You are so wrong so many ways. Many (if not most) of the experts on CrossValidated are statisticians. And I hope that not all physicists think that this issue does not have a long statistical history with a great deal of literature and needs to start from scratch because of ignorance of the subject. Hence my suggestion to start with CrossValidated and then back here for implementation. (Although that doesn't fix my concern about thinking that all one needs is a set of data. One needs how the data was collected and the objective (which certainly comes from physicists). $\endgroup$
    – JimB
    Commented May 1, 2023 at 20:23

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.