Calculate the integral for certain values of n = 0,1,2,3,... and you will find a rule for it for n>=2
Here for n=0
int[k_, a_, b_, 0] =
Assuming[k \[Element] Reals && r \[Element] Reals && k > 0 && r > 0 &&
n >= 0 && n \[Element] Integers && a \[Element] Reals &&
b \[Element] Reals && a > 0 && b > 0 && b > a,
Integrate[BesselY[-1 + n, k*r] BesselY[n, k*r] /. n -> 0, {r, a, b}]]
(* (-BesselY[0, a k]^2 + BesselY[0, b k]^2)/(2 k) *)
Here for n=1
int[k_, a_, b_, 1] =
Assuming[k \[Element] Reals && r \[Element] Reals && k > 0 && r >0 &&
n >= 0 && n \[Element] Integers && a \[Element] Reals &&
b \[Element] Reals && a > 0 && b > 0 && b > a,
Integrate[BesselY[-1 + n, k*r] BesselY[n, k*r] /. n -> 1, {r, a, b}]]
(* (BesselY[0, a k]^2 - BesselY[0, b k]^2)/(2 k) *)
Here for n=2
int[k_, a_, b_, 2] =
Assuming[k \[Element] Reals && r \[Element] Reals && k > 0 && r > 0 &&
n >= 0 && n \[Element] Integers && a \[Element] Reals &&
b \[Element] Reals && a > 0 && b > 0 && b > a,
Integrate[BesselY[-1 + n, k*r] BesselY[n, k*r] /. n -> 2, {r, a, b}]]
(* (1/(2 Sqrt[\[Pi]]))(2 a MeijerG[{{}, {-1, 0, 1/
2}}, {{-(3/2), -(1/2), -(1/2), 3/2}, {-1}}, a k, 1/2] -
2 b MeijerG[{{}, {-1, 0, 1/2}}, {{-(3/2), -(1/2), -(1/2), 3/
2}, {-1}}, b k, 1/2] -
a MeijerG[{{0, 1/2}, {}}, {{3/2}, {-(3/2), -(1/2), -(1/2)}}, a k, 1/
2] + b MeijerG[{{0, 1/2}, {}}, {{3/2}, {-(3/2), -(1/2), -(1/2)}},
b k, 1/2]) *)
Calculating for n=3,4,5,6,.. you can extract the general rule
intn[k_, a_, b_, n_] =
1/(2 Sqrt[\[Pi]]) (2 a MeijerG[{{}, {1 - n, 0, 1/
2}}, {{-((2 n - 1)/2), -(1/2), -(1/2), (2 n - 1)/2}, {1 - n}},
a k, 1/2] -
2 b MeijerG[{{}, {1 - n, 0, 1/
2}}, {{-((2 n - 1)/2), -(1/2), -(1/2), (2 n - 1)/2}, {1 - n}},
b k, 1/2] -
a MeijerG[{{0, 1/2}, {}}, {{(2 n - 1)/
2}, {-((2 n - 1)/2), -(1/2), -(1/2)}}, a k, 1/2] +
b MeijerG[{{0, 1/2}, {}}, {{(2 n - 1)/
2}, {-((2 n - 1)/2), -(1/2), -(1/2)}}, b k, 1/2])
Test it with numerical integration and you will see, it works for all n>=2
nint[k_, a_, b_, n_] :=
NIntegrate[BesselY[-1 + n, k*r] BesselY[n, k*r], {r, a, b}]
{nint[1, 2, 3, 5], intn[1., 2, 3, 5]}
(* {8.11274, 8.11274} *)