Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I am trying to discretize a long array of real numbers into a uniformly distributed integer array. To do this I am identifying the quantile where each real value sits and writing it out to the output array as shown in the code below.

But I am not sure this is the most efficient / optimised way to do this. Is there any more efficient code to discretize arrays?

Qsections = Range[1/#, 1, 1/#] &; 
FindQuantile3[data0_, qvalue0_, qts0_] := 
  Module[{data = data0, qvalue = qvalue0, qts = qts0},
   quantilesT = Quantile[data, Qsections[qts]];
   Position[quantilesT, Nearest[quantilesT, qvalue0][[1]]][[1, 1]]
   ];
FindQuantile3[RandomReal[10,1000],5,7]

out: 3 (* meaning that 5 is in the 3rd quantile of the randomly created array if we divide it in 7 quantiles*)

This is the result I want, but I find the function too complicated and slow for what it needs to do. Any improvement ideas?

share|improve this question
    
Are you mapping this over many points with the same data, in order to histogram the result ? Using HistrogramList with specific bins may be more efficient in that case. –  lalmei Feb 9 at 20:31
    
Not really. I don't need a histogram. For each value of the input array I need to show the quantile it sits in, so the output array will have the same size as the input. –  MA81 Feb 9 at 20:37
    
If you are using the same data to buildup the quantiles regions, I would simply map the position part over the points. Computing the quantile regions takes about ~50% of the whole function atm. –  lalmei Feb 9 at 20:47
    
Alternatively, you could use the theoretical distribution of your data like so: CDF[UniformDistribution[{0, 10}], q] n // Floor, which for q=5 and n=7 yields 3. Should be OK if you have a large sample. –  Sjoerd C. de Vries Feb 9 at 21:41
    
There's nothing wrong with your idea, just take the extra step of making a NearestFunction instead of calling Nearest every time. End result is much faster in my timings than the machinations like turning it into an empirical and running over the CDF, etc. –  rasher Feb 10 at 0:00
add comment

2 Answers 2

up vote 0 down vote accepted

As I commented, nothing wrong with your idea, assuming the behavior of nearest in your example function is what you want just eliminate the overhead of Nearest by utilizing a NearestFunction. Here's a framework to give you an idea, with some honest timings (including the time to create the needed function or distribution). Timings on my cigar-time netbook. You can see creating the NearestFunction is speedier than creating a distribution, and using it is order of magnitude+ faster than CDF.

rd = RandomReal[1000, 5000000];

(* using nearest function *)
znf = Nearest[Quantile[rd, Range[7]/7] -> Automatic]; // Timing
znf /@ RandomReal[100, 1000]; // Timing

(* using distribution *)
e = EmpiricalDistribution[rd]; // Timing
1 + CDF[e, RandomReal[100, 1000]] 7 // Floor; // Timing

(*   
   {8.704856,Null}
   {0.031200,Null}
   {11.076071,Null}
   {0.374402,Null}
*)
share|improve this answer
add comment

Let's try to make it interesting using a large array of random reals:

rd = RandomReal[10, 1000000];

Timing of your routine:

Do[FindQuantile3[rd, 5, 7], {100}] // AbsoluteTiming//First

23.138323

Now, the alternative:

e = EmpiricalDistribution[rd];

Do[1+ CDF[e, 5] 7 // Floor, {100}] // AbsoluteTiming//First

0.018001

There can be some slight differences between the results of the two methods that are probably caused by the method that Quantile uses to determine the exact quantile.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.