This is getting too long to keep using comments. You need to use Limits for the special cases, then it works
ClearAll[a, b, x]
sol = Integrate[Cos[x]^a*Sin[b*x], {x, 0, Pi/2}, Assumptions -> {a >= 0, b >= 0}]

Limit[sol, {a -> 0, b -> 0}]

i.e. do not do
sol /. {a -> 0, b -> 0}

Tried Maple 2023 and it can't solve it. Maxima 5.46 can't do it, giac can't do it. Sympy can't do it. Tried Fricas 1.3.8 and it also can't solve it, it says:
(1) -> integrate(cos(x)^a*sin(b*x),x=0..%pi/2)
(1) "potentialPole"
So far only Mathematica can integrate it and says the answer is zero when a->0,b->0
. Rubi is not meant to be used on definite integration.
see if we can get the answer of that limit as π/2
I do not know why you think the limit should be Pi/2
as a->0,b->0
Just plotting the integrand as a->0,b->0
shows clearly that area under the curve is shrinking more and more to zero. Since integration is the area under the curve, then zero seems like the correct result.

Manipulate[
Plot[Cos[x]^a*Sin[b*x], {x, 0, Pi/2},
PlotRange -> {Automatic, {0, 1}}],
{{a, 2, "a"}, 2, 0.01, -0.01, Appearance -> "Labeled",
ContinuousAction -> False},
{{b, 2, "b"}, 2, 0.01, -0.01, Appearance -> "Labeled",
ContinuousAction -> False},
TrackedSymbols :> {a, b}
]
ClearAll[a, b, x]; Integrate[Cos[x]^a*Sin[b*x], {x, 0, Pi/2}, Assumptions -> {a >= 0, b >= 0}]
It gives !Mathematica graphics $\endgroup$Hypergeometric2F1[1, (1/2)*(2 + a - b), (1/2)*(2 - a - b), -1]/(a + b) + Hypergeometric2F1[1, (1/2)*(2 + a + b), (1/2)*(2 - a + b), -1]/(-a + b) + (2^(-1 - a)*Pi*Gamma[1 + a]*Sin[b*Pi])/((-Cos[a*Pi] + Cos[b*Pi])*Gamma[(1/2)*(2 + a - b)]*Gamma[(1/2)*(2 + a + b)])
screen shot !Mathematica graphics $\endgroup$Limit
? -- "Please do 'full simplify'": Sheesh! Look it up. $\endgroup$