0
$\begingroup$

Is there an option to map the minimum value of a list to 0 using CDF of EmpiricalDistribution?

EDIT

I'm looking for the inverse of the Quantile function, which by definition equals 0 at the min value, and 1 at the max value. In fac, using CDF of EmpiricalDistribution is motivated by Sjoerd's answer to my question on that topic

END EDIT

Given a list of random reals in [1,2]:

lst = Table[RandomReal[{1, 2}], 10]

{1.77852, 1.06913, 1.59777, 1.95967, 1.64152, 1.77021, 1.16423, 1.66417, 1.59076, 1.08503}

Note that the Max value of the list maps to 1, but the Min maps to a very small but positive value, so I suspect there's some logic that involves less than versus less than or equal thats to blame.

lst // AssociationMap[CDF[EmpiricalDistribution[lst], #] &]

<|1.77852 -> 0.9, 1.06913 -> 0.1, 1.59777 -> 0.5, 1.95967 -> 1, 1.64152 -> 0.6, 1.77021 -> 0.8, 1.16423 -> 0.3, 1.66417 -> 0.7, 1.59076 -> 0.4, 1.08503 -> 0.2|>

Same goes for random integers:

lst = Table[RandomInteger[{1, 100}], 10]

{3, 22, 40, 90, 97, 98, 72, 37, 53, 30}

Gives again a CDF whose max value is 1 but with positive min value:

lst // AssociationMap[CDF[EmpiricalDistribution[lst], #] &]

<|3->1/10,22->1/5,40->1/2,90->4/5,97->9/10,98->1,72->7/10,37->2/5,53->3/5,30->3/10|>

$\endgroup$
  • $\begingroup$ The usual definition of a CDF is $Pr(X\leq x)$ but is what you want $Pr(X \lt x)$? $\endgroup$ – JimB Apr 25 at 20:16
2
$\begingroup$

If you want $Pr(X < x)$ rather than $Pr(X \leq x)$ (the usual definition of a CDF), then the following function might work for you:

lst = Table[RandomReal[{1, 2}], 10];
dist = EmpiricalDistribution[lst];
cdfModified[x_] := Probability[X < x, X \[Distributed] dist];

cdfModified[Min[lst]]
(* 0 *)

But that means you'll get

cdfModified[Max[lst]]
(* 0.9 *)

Stating what you need in mathematical/statistical terms such as $Pr(X<x)$ would clarify things. Any proposed manipulation of the data in isolation of a model or definition of the objective can likely lead to undesirable consequences.

For example, just dropping the smallest value will achieve one part of the stated objective: the value of the empirical CDF will be zero at that minimum value. However, the value will now also be zero for any value less than the 2nd smallest value.

Edit:

From your comment what you want to do doesn't seem unclear but rather imaginative in terms of properties of probability and statistics. (Or, it might very well be that I'm not so imaginative.)

One could simply replace the smallest value with the smallest value incremented by a very small amount. (Again, I cannot imagine a situation where that would make sense. So I offer this just to see if I'm understanding what you want to do.)

(* Generate some data and sort it from low to high *)
lst = Table[RandomReal[{1, 2}], 10]
lst = Sort[lst]

(* Save the minimum value *)
minlst = lst[[1]]

(* Add a small amount to the minimum value based on the smallest differences
among all of the values *)
lst[[1]] = lst[[1]] + Min[Differences[lst]]/100

(* Construct the empirical distribution *)
dist = EmpiricalDistribution[lst];

(* Check the value of the empirical CDF with the modified value and the minimum value *)
CDF[dist, minlst]
(* 0 *)
CDF[dist, lst[[1]]]
(* 0.1 *)

Another alternative:

lst = Table[RandomReal[{1, 2}], 10];
minlst = Min[lst];
dist = EmpiricalDistribution[lst];
cdfModified[z_, zmin_] := Piecewise[{{CDF[dist, z], z > zmin}}, 0]

cdfModified[minlst, minlst]
(* 0 *)
cdfModified[minlst + 0.001, minlst]
(* 0.1 *)
| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks, doesn't help as it creates the mirror problem. If it's not obvious from my q, I'm looking to normalize discrete data list to [0,1] $\endgroup$ – alancalvitti Apr 26 at 0:34
  • $\begingroup$ Why am I being so obstinate about stating that you shouldn't be modifying the common definition of an empirical distribution function? First, all definitions are arbitrary. But if one deviates from the common definition/interpretation of a statistical procedure, then one can be mislead. I'm all for modifying definitions but only when there's at least a proposed justification. What I see here is what appears to be arbitrary manipulation of the numbers. Now this forum isn't necessarily the right place for the justification. But it would get you better answers if you could expound on that. $\endgroup$ – JimB Apr 26 at 16:30
  • $\begingroup$ I'm looking for the inverse of the Quantile function, which by definition equals 0 at the min value, and 1 at the max value. In fact I got the CDF of EmpiricalDistribution from Sjoerd's answer to my question on that topic $\endgroup$ – alancalvitti Apr 27 at 14:02
  • $\begingroup$ It would be great if you could add that to your question. $\endgroup$ – JimB Apr 27 at 14:06
  • $\begingroup$ I edited my question, thanks. $\endgroup$ – alancalvitti Apr 27 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.