For general values of $b$ and $c$, the trivial solution is indeed the only solution. However, there are special values of $b$ and $c$ which can yield non-trivial solutions.
To find these, we can start by telling Mathematica to find the general solution to the ODE:
soln = DSolve[{op[r][op[r][f[r]]] == 0}, f, r]
(* {f -> Function[{r}, (r^2 C[1])/2 - (r^2 C[2])/4 + (r^4 C[3])/4 + C[4] + 1/2 r^2 C[2] Log[r]]} *)
We can then look at the boundary conditions and see what they imply about the coefficients C[i]
. We do this by storing the vanishing quantities in a list BCs
; the actual equations are then BCs == 0
.
BCs = { f[b],
f'[b],
(f''[r] - 1/r*f'[r] - f[r]/(c - 1) /. r -> (1 + b)),
(f'''[r] - 1/r*f''[r] + 1/r^2*f'[r] /. r -> (1 + b))};
quants = BCs /. First[soln]
The quantities in quants
are four linear combinations of the coefficients C[1]
, C[2]
, C[3]
, and C[4]
, all of which must vanish. We can think of this set of simultaneous equations as the result of a matrix $M$ multiplying the vector $\vec{v} = \{C_1, C_2, C_3, C_4\}$. From basic results in linear algebra, we know that the only way for there to be a non-trivial solution for $\vec{v}$ is for the matrix $M$ to have a non-zero determinant. So we construct this matrix and take its determinant:
mat = Outer[Coefficient, quants, {C[1], C[2], C[3], C[4]}];
Simplify[Det[mat]]
(* -((b (1 + b) (-3 + 2 b + 4 c + 2 (1 + b)^2 Log[b] - 2 (1 + b)^2 Log[1 + b]))/(-1 + c)) *)
This implies that the trivial solution is the only solution to the ODE unless this last quantity (in terms of the constants $b$ and $c$) is zero, which occurs when $b = 0$, $b = -1$, or
$$
c = \frac{3 - 2b + 2 (1+b)^2 \ln( (1+b)/b )}{4}.
$$