Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a data set of the form $d=\{(y_1,x_1),(y_2,x_2)...(y_n,x_n)\}$ for a large $n$. A non-parametric plot of this data (a scatter plot where all observations are sorted along $x$ and joined with a line in the $y-x$ plane) looks like this: enter image description here

I want to know how to construct (or define) an empirical distribution, for which the CDF corresponds to this plot. I know that the function EmpiricalDistribution could create a distribution function from $d$. The question, however, is how to generate in Mathematica a function from an empirical CDF.

share|improve this question
3  
Yes, it is related to Mathematica. Mathematica can clearly form an EmpiricalDistribution from event data. The question is whether there is a way to create an EmpiricalDistribution (or possibly its smoother cousins) from an empirical CDF. Legitimate question that we should be able to answer. –  Seth Chandler Mar 25 '13 at 20:38

2 Answers 2

up vote 3 down vote accepted

If you can use Mathematica 9, WeightedData does the trick:

(* "Measured" cumulative probability data { {cum_prob, position}, ... } *)
cum = {CDF[NormalDistribution[], #], #} & /@ 
          RandomVariate[UniformDistribution[{-3, 3}], 100];
cumSorted = SortBy[cum, #[[2]]&] // Transpose;

(* Weighted data. Weights correspond to "histogram step sizes." *)
weighted = WeightedData[cumSorted[[2]], 
              Differences[Prepend[cumSorted[[1]], 0]]];

(* Various distributions and probability functions can take weighted data *)
empirical = EmpiricalDistribution[weighted];
smooth = SmoothKernelDistribution[weighted];

(* CDFs. empirical and smooth are first-class citizens of the distribution land! *)
DiscretePlot[{CDF[empirical, x], CDF[smooth, x]}, {x, -3, 3, .01}]

<code>DiscretePlot</code> of weighted data CDF

EDIT: EmpiricalDistribution does also support (presumably already before version 9) following weighted data: EmpiricalDistribution[weights -> values].

Disclaimer: don't base your nuclear reactor safety on this. I'm not a mathematician.

share|improve this answer
1  
You can also directly estimate parametric distributions this way using EstimatedDistribution on your WeightedData. For example EstimatedDistribution[weighted, NormalDistribution[a,b]]. –  Andy Ross Mar 26 '13 at 2:32
    
@Andy Ross If you want to estimate the distribution using a specific parametric distribution, your example works. (And surely, data seems to fit normal distribution well.) Exact semantics of the original question are a bit hard to follow, I believe the original question was related specifically to empirical distribution and other DataDistributions. –  kirma Mar 26 '13 at 6:06
    
@kirma Thanks. Very useful (works alright with Mathematica 8 as you suggest). Yes, I am dealing with empirical data, so I don't know exactly the data generating process and would hence prefer not to impose any parametric assumptions (and some of my other plots seem to fit less neatly normal distributions). –  EOO Mar 26 '13 at 11:43

If your answer does not have to be perfect you could write the following, where the data is in the form {{x1,y1},{x2,y2} ... {xn,yn}} (note I have switched the order of your data so that it is a format that Mathematica uses more frequently).

sampleData={{-0.4, 0}, {0.2, 0.3}, {0.6, 0.7}, {2, 1}};
dist = EmpiricalDistribution[
Differences[Last /@ sampleData] -> Mean /@ Partition[First /@ sampleData, 2, 1]]

I was hoping to be able to use EventData but I can't get EmpiricalDistribution to work with weights and EventData.

share|improve this answer
    
I accepted kirma's answer because it came first and gives an exact solution. I however upvoted your answer because it is a useful approximation (and works directly with Mathematica 8). Thanks! –  EOO Mar 26 '13 at 11:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.