How to find analytical solution for integral from product of Bessel functions of second kind with different order?

Question

I need help with analytical solution for the following integral:

$Assumptions = 
 k \[Element] Reals && r \[Element] Reals && k > 0 && r > 0 && 
  n >= 0 && n \[Element] Integers && a \[Element] Reals && 
  b \[Element] Reals && a > 0 && b > 0 && b > a
tmpI = Integrate[BesselY[-1 + n, k*r] BesselY[n, k*r], {r, a, b}]

After simplification of result (tmpI) I get Indeterminate. It looks like Mathematica 11 uses transformations for BesselY corresponding to non-integer order despite of the assumptions I've specified.

Answer 1

Calculate the integral for certain values of n = 0,1,2,3,... and you will find a rule for it for n>=2

Here for n=0

 int[k_, a_, b_, 0] = 
 Assuming[k \[Element] Reals && r \[Element] Reals && k > 0 && r > 0 &&
 n >= 0 && n \[Element] Integers && a \[Element] Reals && 
 b \[Element] Reals && a > 0 && b > 0 && b > a, 
 Integrate[BesselY[-1 + n, k*r] BesselY[n, k*r] /. n -> 0, {r, a, b}]]

 (*   (-BesselY[0, a k]^2 + BesselY[0, b k]^2)/(2 k)   *)

Here for n=1

    int[k_, a_, b_, 1] = 
     Assuming[k \[Element] Reals && r \[Element] Reals && k > 0 && r >0 &&
     n >= 0 && n \[Element] Integers && a \[Element] Reals && 
     b \[Element] Reals && a > 0 && b > 0 && b > a, 
     Integrate[BesselY[-1 + n, k*r] BesselY[n, k*r] /. n -> 1, {r, a, b}]]

    (*  (BesselY[0, a k]^2 - BesselY[0, b k]^2)/(2 k)   *)

Here for n=2

    int[k_, a_, b_, 2] = 
    Assuming[k \[Element] Reals && r \[Element] Reals && k > 0 && r > 0 &&
    n >= 0 && n \[Element] Integers && a \[Element] Reals && 
    b \[Element] Reals && a > 0 && b > 0 && b > a, 
    Integrate[BesselY[-1 + n, k*r] BesselY[n, k*r] /. n -> 2, {r, a, b}]]

   (*  (1/(2 Sqrt[\[Pi]]))(2 a MeijerG[{{}, {-1, 0, 1/
       2}}, {{-(3/2), -(1/2), -(1/2), 3/2}, {-1}}, a k, 1/2] - 
      2 b MeijerG[{{}, {-1, 0, 1/2}}, {{-(3/2), -(1/2), -(1/2), 3/
      2}, {-1}}, b k, 1/2] - 
      a MeijerG[{{0, 1/2}, {}}, {{3/2}, {-(3/2), -(1/2), -(1/2)}}, a k, 1/
      2] + b MeijerG[{{0, 1/2}, {}}, {{3/2}, {-(3/2), -(1/2), -(1/2)}}, 
       b k, 1/2])    *)

Calculating for n=3,4,5,6,.. you can extract the general rule

    intn[k_, a_, b_, n_] = 
     1/(2 Sqrt[\[Pi]]) (2 a MeijerG[{{}, {1 - n, 0, 1/
     2}}, {{-((2 n - 1)/2), -(1/2), -(1/2), (2 n - 1)/2}, {1 - n}},
     a k, 1/2] - 
     2 b MeijerG[{{}, {1 - n, 0, 1/
     2}}, {{-((2 n - 1)/2), -(1/2), -(1/2), (2 n - 1)/2}, {1 - n}},
     b k, 1/2] - 
     a MeijerG[{{0, 1/2}, {}}, {{(2 n - 1)/
     2}, {-((2 n - 1)/2), -(1/2), -(1/2)}}, a k, 1/2] + 
     b MeijerG[{{0, 1/2}, {}}, {{(2 n - 1)/
     2}, {-((2 n - 1)/2), -(1/2), -(1/2)}}, b k, 1/2])

Test it with numerical integration and you will see, it works for all n>=2

    nint[k_, a_, b_, n_] := 
    NIntegrate[BesselY[-1 + n, k*r] BesselY[n, k*r], {r, a, b}]

    {nint[1, 2, 3, 5], intn[1., 2, 3, 5]}

    (* {8.11274, 8.11274}   *)