# conditional expectation doesn't work with SmoothKernelDistribution

I'm trying to calculate a conditional expectation E[ x | x > a ], where the distribution of x is inferred from a finite sample of data using SmoothKernelDistribution.

The data contain points {t,s} where t is the time of day, and s is the traffic speed at that time. Here's a small subsample:

speedData = {{17.1006,30.2979},{20.3583,15.1275},{17.8508,26.2334},{12.8106,19.9852},{7.71861,11.9718},{17.1233,14.7792},{14.95,10.4096},{20.2389,33.6072},{18.3989,13.4087},{18.2425,14.074},{7.74028,22.0739},{18.07,9.80947},{15.6636,18.9742},{15.6806,15.1147},{17.8603,7.06266},{16.5042,17.7862},{14.8319,14.5645},{15.8456,11.0942},{15.7133,11.4662},{14.3653,37.694},{15.5131,19.3896},{17.7636,18.0347}}


I define speedDistr = SmoothKernelDistribution[speedData].

If I understand the docs correctly, the following command should produce the result that I'm looking for:

Expectation[s \[Conditioned] s > 30, {t, s} \[Distributed] speedDistr]


However, the output just restates the command, and does not produce the desired result. At the same time, this command works perfectly well with built-in distributions (e.g. Expectation[x \[Conditioned] x > 2, x \[Distributed] NormalDistribution[]]).

Also, unconditional expectation Expectation[s, {t, s} \[Distributed] speedDistr] works fine.

What am I doing wrong?

• Use NExpectation, or perhaps give KernelMixtureDistribution as swing. As nice as the probability capabilities of MMA are, they have holes (ones I'd much rather WRI spend time on vs yet another superfluous function...) – ciao Apr 9 '15 at 21:36
• @rasher, NExpectation actually produced a number, but it was quite slow, and showed two error messages: NIntegrate::slwcon: Numerical integration converging too slowly; suspect one of the following: singularity, value of the integration is 0, highly oscillatory integrand, or WorkingPrecision too small. >>NIntegrate::eincr: The global error of the strategy GlobalAdaptive has increased more than 2000 times. The global error is expected to decrease monotonically after a number \ of integrand evaluations. – verse Apr 9 '15 at 21:40
• See work-arounds posted.... but yes, when using numeric versions, sometimes you have to fiddle with parameters/methods/etc. – ciao Apr 9 '15 at 21:41

As a work-around :

Expectation[x \[Conditioned] x > 30, x \[Distributed] MarginalDistribution[speedDistr, 2]]

(* 34.8138 *)

• beaten by 14 seconds !! (+1) – kglr Apr 9 '15 at 21:43
• it does work for x>30, but for some reason when you change the inequality to x<30 -- it doesn't. not sure what's causing it. – verse Apr 9 '15 at 21:45
• @verse: Likely one of the "holes" I alluded to in my comment. Consider filing a report with WRI support - it would be nice for the probability stuff to get some attention. Is using EmpiricalDistribution out of the question? It seems to work fine for these cases and your data. Also, using the marginal, Nexpectation works for both inequalities and is speedy... – ciao Apr 9 '15 at 21:53

I think the real question is why are you using SmoothKernelDistribution at all here, and what do you hope to achieve by doing so? In particular, why do you think that, given sample data:

1. inferring a smoothed pdf from the sample data (and using the default settings for bandwidth choice and kernel choice, when there are an infinite number of possibilities for same, without even checking the fit), and then calculating the conditional mean of the latter approximation ...

... is any better theoretically or otherwise than simply:

1. calculating the sample mean of $X | X> 30$, using the raw data.

Your speed data, conditional on speed > 30, is:

data = Cases[speedData[[All, 2]], x_ /; x > 30]


The conditional mean is then:

Mean[data]


33.8664

That is all there is to it.

On what basis do you think that first constructing a smoothed non-parametric kernel density estimate, and then calculating the conditional mean of the latter, will provide a 'better' estimator of the conditional mean?

• that calculation is only the first one in the series of calculations that i need to perform. while it is possible to answer that particular question without relying on any kind of smoothing, it wouldn't be possible to do so for questions like "what's the average speed at 12:00pm". There might not be any data points precisely at 12:00pm, but might be points around 12, and one needs to interpolate the average speed at 12. With SmoothKernelDistribution I'd expect to evaluate Expectation[s \[Conditioned] t == 12, {t, s} \[Distributed] speedDistr] – verse Apr 10 '15 at 15:33