# Discretizing data using quantiles in Mathematica

I am trying to discretize a long array of real numbers into a uniformly distributed integer array. To do this I am identifying the quantile where each real value sits and writing it out to the output array as shown in the code below.

But I am not sure this is the most efficient / optimised way to do this. Is there any more efficient code to discretize arrays?

Qsections = Range[1/#, 1, 1/#] &;
FindQuantile3[data0_, qvalue0_, qts0_] :=
Module[{data = data0, qvalue = qvalue0, qts = qts0},
quantilesT = Quantile[data, Qsections[qts]];
Position[quantilesT, Nearest[quantilesT, qvalue0][[1]]][[1, 1]]
];
FindQuantile3[RandomReal[10,1000],5,7]


out: 3 (* meaning that 5 is in the 3rd quantile of the randomly created array if we divide it in 7 quantiles*)

This is the result I want, but I find the function too complicated and slow for what it needs to do. Any improvement ideas?

-
Are you mapping this over many points with the same data, in order to histogram the result ? Using HistrogramList with specific bins may be more efficient in that case. – lalmei Feb 9 '14 at 20:31
Not really. I don't need a histogram. For each value of the input array I need to show the quantile it sits in, so the output array will have the same size as the input. – MA81 Feb 9 '14 at 20:37
If you are using the same data to buildup the quantiles regions, I would simply map the position part over the points. Computing the quantile regions takes about ~50% of the whole function atm. – lalmei Feb 9 '14 at 20:47
Alternatively, you could use the theoretical distribution of your data like so: CDF[UniformDistribution[{0, 10}], q] n // Floor, which for q=5 and n=7 yields 3. Should be OK if you have a large sample. – Sjoerd C. de Vries Feb 9 '14 at 21:41
There's nothing wrong with your idea, just take the extra step of making a NearestFunction instead of calling Nearest every time. End result is much faster in my timings than the machinations like turning it into an empirical and running over the CDF, etc. – ciao Feb 10 '14 at 0:00

As I commented, nothing wrong with your idea, assuming the behavior of nearest in your example function is what you want just eliminate the overhead of Nearest by utilizing a NearestFunction. Here's a framework to give you an idea, with some honest timings (including the time to create the needed function or distribution). Timings on my cigar-time netbook. You can see creating the NearestFunction is speedier than creating a distribution, and using it is order of magnitude+ faster than CDF.

rd = RandomReal[1000, 5000000];

(* using nearest function *)
znf = Nearest[Quantile[rd, Range[7]/7] -> Automatic]; // Timing
znf /@ RandomReal[100, 1000]; // Timing

(* using distribution *)
e = EmpiricalDistribution[rd]; // Timing
1 + CDF[e, RandomReal[100, 1000]] 7 // Floor; // Timing

(*
{8.704856,Null}
{0.031200,Null}
{11.076071,Null}
{0.374402,Null}
*)

-

Let's try to make it interesting using a large array of random reals:

rd = RandomReal[10, 1000000];


Do[FindQuantile3[rd, 5, 7], {100}] // AbsoluteTiming//First


23.138323

Now, the alternative:

e = EmpiricalDistribution[rd];

Do[1+ CDF[e, 5] 7 // Floor, {100}] // AbsoluteTiming//First


0.018001

There can be some slight differences between the results of the two methods that are probably caused by the method that Quantile uses to determine the exact quantile.

-