1
$\begingroup$

I am trying to discretize a long array of real numbers into a uniformly distributed integer array. To do this I am identifying the quantile where each real value sits and writing it out to the output array as shown in the code below.

But I am not sure this is the most efficient / optimised way to do this. Is there any more efficient code to discretize arrays?

Qsections = Range[1/#, 1, 1/#] &; 
FindQuantile3[data0_, qvalue0_, qts0_] := 
  Module[{data = data0, qvalue = qvalue0, qts = qts0},
   quantilesT = Quantile[data, Qsections[qts]];
   Position[quantilesT, Nearest[quantilesT, qvalue0][[1]]][[1, 1]]
   ];
FindQuantile3[RandomReal[10,1000],5,7]

out: 3 (* meaning that 5 is in the 3rd quantile of the randomly created array if we divide it in 7 quantiles*)

This is the result I want, but I find the function too complicated and slow for what it needs to do. Any improvement ideas?

$\endgroup$
5
  • $\begingroup$ Are you mapping this over many points with the same data, in order to histogram the result ? Using HistrogramList with specific bins may be more efficient in that case. $\endgroup$
    – lalmei
    Feb 9, 2014 at 20:31
  • $\begingroup$ Not really. I don't need a histogram. For each value of the input array I need to show the quantile it sits in, so the output array will have the same size as the input. $\endgroup$
    – MA81
    Feb 9, 2014 at 20:37
  • $\begingroup$ If you are using the same data to buildup the quantiles regions, I would simply map the position part over the points. Computing the quantile regions takes about ~50% of the whole function atm. $\endgroup$
    – lalmei
    Feb 9, 2014 at 20:47
  • $\begingroup$ Alternatively, you could use the theoretical distribution of your data like so: CDF[UniformDistribution[{0, 10}], q] n // Floor, which for q=5 and n=7 yields 3. Should be OK if you have a large sample. $\endgroup$ Feb 9, 2014 at 21:41
  • $\begingroup$ There's nothing wrong with your idea, just take the extra step of making a NearestFunction instead of calling Nearest every time. End result is much faster in my timings than the machinations like turning it into an empirical and running over the CDF, etc. $\endgroup$
    – ciao
    Feb 10, 2014 at 0:00

2 Answers 2

1
$\begingroup$

As I commented, nothing wrong with your idea, assuming the behavior of nearest in your example function is what you want just eliminate the overhead of Nearest by utilizing a NearestFunction. Here's a framework to give you an idea, with some honest timings (including the time to create the needed function or distribution). Timings on my cigar-time netbook. You can see creating the NearestFunction is speedier than creating a distribution, and using it is order of magnitude+ faster than CDF.

rd = RandomReal[1000, 5000000];

(* using nearest function *)
znf = Nearest[Quantile[rd, Range[7]/7] -> Automatic]; // Timing
znf /@ RandomReal[100, 1000]; // Timing

(* using distribution *)
e = EmpiricalDistribution[rd]; // Timing
1 + CDF[e, RandomReal[100, 1000]] 7 // Floor; // Timing

(*   
   {8.704856,Null}
   {0.031200,Null}
   {11.076071,Null}
   {0.374402,Null}
*)
$\endgroup$
1
$\begingroup$

Let's try to make it interesting using a large array of random reals:

rd = RandomReal[10, 1000000];

Timing of your routine:

Do[FindQuantile3[rd, 5, 7], {100}] // AbsoluteTiming//First

23.138323

Now, the alternative:

e = EmpiricalDistribution[rd];

Do[1+ CDF[e, 5] 7 // Floor, {100}] // AbsoluteTiming//First

0.018001

There can be some slight differences between the results of the two methods that are probably caused by the method that Quantile uses to determine the exact quantile.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.