5
$\begingroup$

I'm trying to calculate a conditional expectation E[ x | x > a ], where the distribution of x is inferred from a finite sample of data using SmoothKernelDistribution.

The data contain points {t,s} where t is the time of day, and s is the traffic speed at that time. Here's a small subsample:

speedData = {{17.1006,30.2979},{20.3583,15.1275},{17.8508,26.2334},{12.8106,19.9852},{7.71861,11.9718},{17.1233,14.7792},{14.95,10.4096},{20.2389,33.6072},{18.3989,13.4087},{18.2425,14.074},{7.74028,22.0739},{18.07,9.80947},{15.6636,18.9742},{15.6806,15.1147},{17.8603,7.06266},{16.5042,17.7862},{14.8319,14.5645},{15.8456,11.0942},{15.7133,11.4662},{14.3653,37.694},{15.5131,19.3896},{17.7636,18.0347}}

I define speedDistr = SmoothKernelDistribution[speedData].

If I understand the docs correctly, the following command should produce the result that I'm looking for:

Expectation[s \[Conditioned] s > 30, {t, s} \[Distributed] speedDistr]

However, the output just restates the command, and does not produce the desired result. At the same time, this command works perfectly well with built-in distributions (e.g. Expectation[x \[Conditioned] x > 2, x \[Distributed] NormalDistribution[]]).

Also, unconditional expectation Expectation[s, {t, s} \[Distributed] speedDistr] works fine.

What am I doing wrong?

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ Use NExpectation, or perhaps give KernelMixtureDistribution as swing. As nice as the probability capabilities of MMA are, they have holes (ones I'd much rather WRI spend time on vs yet another superfluous function...) $\endgroup$
    – ciao
    Apr 9, 2015 at 21:36
  • $\begingroup$ @rasher, NExpectation actually produced a number, but it was quite slow, and showed two error messages: NIntegrate::slwcon: Numerical integration converging too slowly; suspect one of the following: singularity, value of the integration is 0, highly oscillatory integrand, or WorkingPrecision too small. >>NIntegrate::eincr: The global error of the strategy GlobalAdaptive has increased more than 2000 times. The global error is expected to decrease monotonically after a number \ of integrand evaluations. $\endgroup$
    – verse
    Apr 9, 2015 at 21:40
  • $\begingroup$ See work-arounds posted.... but yes, when using numeric versions, sometimes you have to fiddle with parameters/methods/etc. $\endgroup$
    – ciao
    Apr 9, 2015 at 21:41

2 Answers 2

Reset to default
7
$\begingroup$

As a work-around :

Expectation[x \[Conditioned] x > 30, x \[Distributed] MarginalDistribution[speedDistr, 2]]

(* 34.8138 *)
Improve this answer
$\endgroup$
3
  • $\begingroup$ beaten by 14 seconds !! (+1) $\endgroup$
    – kglr
    Apr 9, 2015 at 21:43
  • $\begingroup$ it does work for x>30, but for some reason when you change the inequality to x<30 -- it doesn't. not sure what's causing it. $\endgroup$
    – verse
    Apr 9, 2015 at 21:45
  • 1
    $\begingroup$ @verse: Likely one of the "holes" I alluded to in my comment. Consider filing a report with WRI support - it would be nice for the probability stuff to get some attention. Is using EmpiricalDistribution out of the question? It seems to work fine for these cases and your data. Also, using the marginal, Nexpectation works for both inequalities and is speedy... $\endgroup$
    – ciao
    Apr 9, 2015 at 21:53
2
$\begingroup$

I think the real question is why are you using SmoothKernelDistribution at all here, and what do you hope to achieve by doing so? In particular, why do you think that, given sample data:

  1. inferring a smoothed pdf from the sample data (and using the default settings for bandwidth choice and kernel choice, when there are an infinite number of possibilities for same, without even checking the fit), and then calculating the conditional mean of the latter approximation ...

... is any better theoretically or otherwise than simply:

  1. calculating the sample mean of $X | X> 30$, using the raw data.

Your speed data, conditional on speed > 30, is:

data = Cases[speedData[[All, 2]], x_ /; x > 30]

The conditional mean is then:

Mean[data]

33.8664

That is all there is to it.

On what basis do you think that first constructing a smoothed non-parametric kernel density estimate, and then calculating the conditional mean of the latter, will provide a 'better' estimator of the conditional mean?

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ that calculation is only the first one in the series of calculations that i need to perform. while it is possible to answer that particular question without relying on any kind of smoothing, it wouldn't be possible to do so for questions like "what's the average speed at 12:00pm". There might not be any data points precisely at 12:00pm, but might be points around 12, and one needs to interpolate the average speed at 12. With SmoothKernelDistribution I'd expect to evaluate Expectation[s \[Conditioned] t == 12, {t, s} \[Distributed] speedDistr] $\endgroup$
    – verse
    Apr 10, 2015 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.