3
$\begingroup$

I need to generate a lot of point according to a given distribution. I don't have an analytic form of this distribution so I am using an InterpolatingFunction of ~100 points computed numerically. I then normalize this interpolation and I define a ProbabilityDistribution from which I generate numbers using RandomVariate.

First of all this leads to a really slow evaluation of RandomVariate (compared to cases in which the ProbabilityDistribution is known analitically), but that's not the main point of this question (it might be related to the fact that my pdf is exponentially decreasing and defined on a quite wide interval and so is close to zero most of the time... maybe?).

Anyway, I use these points for some heavy computations and so I try to parallelize as much as possible. This lead me to the nice discovery that RandomVariate takes twice as long when done in parallel. Here is a short example code that reproduces what I want to say:

(* I create a random pdf just as an example *)
pdf = Table[{x, RandomReal[]}, {x, 1, 10, 0.5}];
intpdf = Interpolation@pdf;
norm = Integrate[intpdf[x], {x, 1, 10}];
dist = ProbabilityDistribution[intpdf[x]/norm, {x, 1, 10}];

(* no parallelization *)
AbsoluteTiming[RandomVariate[dist, 1000000]][[1]]
(* Out = 7.28052 *)

(* parallelization on 8 subkernels *)
ParallelTable[
 AbsoluteTiming[RandomVariate[dist, 1000000]][[1]], {i, 1, 8}]
(* Out = {13.021, 12.7993, 13.8139, 13.417, 12.8863, 13.0874, 13.3788, 13.7836} *)

As you can see, the evaluation of the same operation takes approximately twice as long when done in the subkernels. I have read that this is due to poor parallelization properties of interpolating functions, but I can't find a way to fix this and have my code run fast. What can I do?

$\endgroup$
  • 1
    $\begingroup$ Firstly, I can confirm a lower execution speed on the subkernels (6.22 s (main kernel) vs 7.95 s (subkernels)). However, it occurs to me that Nonparametric Statistical Distributions like SmoothKernelDistribution achieve what you need. See reference.wolfram.com/language/guide/… RandomVariate runs at least two orders of magnitude faster with them. $\endgroup$ – Theo Tiger May 27 at 20:28
  • $\begingroup$ @TheoTiger thanks for your answer. I will try when I have enough time but I am not sure if it will work. To use SmoothKernelDistribution it looks like I need to have a lot of points that follow my distribution, but my pdf (the real one, not the one I am showing in the example) is exponential and so I don't have enough points on the tail of the distribution to get a good fit $\endgroup$ – FrodCube May 28 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.