6
$\begingroup$

I have a data set of the form $d=\{(y_1,x_1),(y_2,x_2)...(y_n,x_n)\}$ for a large $n$. A non-parametric plot of this data (a scatter plot where all observations are sorted along $x$ and joined with a line in the $y-x$ plane) looks like this: enter image description here

I want to know how to construct (or define) an empirical distribution, for which the CDF corresponds to this plot. I know that the function EmpiricalDistribution could create a distribution function from $d$. The question, however, is how to generate in Mathematica a function from an empirical CDF.

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ Yes, it is related to Mathematica. Mathematica can clearly form an EmpiricalDistribution from event data. The question is whether there is a way to create an EmpiricalDistribution (or possibly its smoother cousins) from an empirical CDF. Legitimate question that we should be able to answer. $\endgroup$
    – Seth Chandler
    Commented Mar 25, 2013 at 20:38

2 Answers 2

Reset to default
7
$\begingroup$

If you can use Mathematica 9, WeightedData does the trick:

(* "Measured" cumulative probability data { {cum_prob, position}, ... } *)
cum = {CDF[NormalDistribution[], #], #} & /@ 
          RandomVariate[UniformDistribution[{-3, 3}], 100];
cumSorted = SortBy[cum, #[[2]]&] // Transpose;

(* Weighted data. Weights correspond to "histogram step sizes." *)
weighted = WeightedData[cumSorted[[2]], 
              Differences[Prepend[cumSorted[[1]], 0]]];

(* Various distributions and probability functions can take weighted data *)
empirical = EmpiricalDistribution[weighted];
smooth = SmoothKernelDistribution[weighted];

(* CDFs. empirical and smooth are first-class citizens of the distribution land! *)
DiscretePlot[{CDF[empirical, x], CDF[smooth, x]}, {x, -3, 3, .01}]

<code>DiscretePlot</code> of weighted data CDF

EDIT: EmpiricalDistribution does also support (presumably already before version 9) following weighted data: EmpiricalDistribution[weights -> values].

Disclaimer: don't base your nuclear reactor safety on this. I'm not a mathematician.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ You can also directly estimate parametric distributions this way using EstimatedDistribution on your WeightedData. For example EstimatedDistribution[weighted, NormalDistribution[a,b]]. $\endgroup$
    – Andy Ross
    Commented Mar 26, 2013 at 2:32
  • $\begingroup$ @Andy Ross If you want to estimate the distribution using a specific parametric distribution, your example works. (And surely, data seems to fit normal distribution well.) Exact semantics of the original question are a bit hard to follow, I believe the original question was related specifically to empirical distribution and other DataDistributions. $\endgroup$
    – kirma
    Commented Mar 26, 2013 at 6:06
  • $\begingroup$ @kirma Thanks. Very useful (works alright with Mathematica 8 as you suggest). Yes, I am dealing with empirical data, so I don't know exactly the data generating process and would hence prefer not to impose any parametric assumptions (and some of my other plots seem to fit less neatly normal distributions). $\endgroup$
    – OO_SE
    Commented Mar 26, 2013 at 11:43
2
$\begingroup$

If your answer does not have to be perfect you could write the following, where the data is in the form {{x1,y1},{x2,y2} ... {xn,yn}} (note I have switched the order of your data so that it is a format that Mathematica uses more frequently). 

sampleData={{-0.4, 0}, {0.2, 0.3}, {0.6, 0.7}, {2, 1}};
dist = EmpiricalDistribution[
Differences[Last /@ sampleData] -> Mean /@ Partition[First /@ sampleData, 2, 1]]

I was hoping to be able to use EventData but I can't get EmpiricalDistribution to work with weights and EventData.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I accepted kirma's answer because it came first and gives an exact solution. I however upvoted your answer because it is a useful approximation (and works directly with Mathematica 8). Thanks! $\endgroup$
    – OO_SE
    Commented Mar 26, 2013 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.