This is just a long comment:
One issue with the symbolic definite integral is that if a can be b complex, then you have to worry about the residues.
If we make the substitution in the integrand $z = e^{it}$ we get
fz = 1/(a^2 Cos[t]^2 + b^2 Sin[t]^2) /. {Cos[t] -> (z + 1/z)/2, Sin[t] -> (z - 1/z)/(2 I)} // Factor
-(4 z^2) / ((-a - b - a z^2 + b z^2) (a - b + a z^2 + b z^2))
There are four poles of this function of z:
poles = z /. Solve[(-a - b - a z^2 + b z^2) (a - b + a z^2 + b z^2) == 0, z]
{-(Sqrt[-a - b]/Sqrt[a - b]), Sqrt[-a - b]/Sqrt[a - b],
-(Sqrt[-a + b]/Sqrt[a + b]), Sqrt[-a + b]/Sqrt[a + b]}
The corresponding poles of the integrand are at (since $z = e^{it}$)
trigpoles = Log[poles]/I
{-I Log[-(Sqrt[-a - b]/Sqrt[a - b])], -I Log[Sqrt[-a - b]/Sqrt[a - b]],
-I Log[-(Sqrt[-a + b]/Sqrt[a + b])], -I Log[Sqrt[-a + b]/Sqrt[a + b]]}
plus multiples of $2\pi$. The residues at the poles are
res = SeriesCoefficient[-(1/(a^2 Cos[t]^2 + b^2 Sin[t]^2)), {t, #, -1}] & /@ trigpoles
{-(I/(2 a b)), -(I/(2 a b)), I/(2 a b), I/(2 a b)}
If Mathematica is using residues to compute the complex exponential integral,
then it has to decide which residues to use, which depends which the poles lie inside the contour of integration. (In this integral, it will be either the first two or the last two.)
I suspect Mathematica has trouble deciding. Perhaps it picks all four. Their total is zero, which agrees with the output.
Example
For what it's worth, here's an example of how to compute this integral via the Residue Theorem. I'll work with the integrand fz expressed in terms of $z = e^{it}$ (above). Thus our contour will be the unit circle. Since $dt = i\,z\;dz$, the corresponding complex integrand with respect to $dz$ will be fz/(I z). One can check the residues are the same:
res = SeriesCoefficient[fz/(I z), {z, #, -1}] & /@ poles
{I/(2 a b), I/(2 a b), -(I/(2 a b)), -(I/(2 a b))}
Let's pick an explicit a and b:
abexample = {a -> 1 + I, b -> 2 - I};
The poles are
Norm /@ N[poles /. abexample]
{1.15829, 1.15829, 0.86334, 0.86334}
Now we pick out the residues inside the unit circle and multiply their total by 2 π I:
2 π I Pick[res /. abexample, poles /. abexample, _?(Norm[#] < 1 &)] // Total
(3/5 - I/5) \[Pi]
Compare with the output of the first (trig) integral:
(2 Sqrt[b^2/a^2] \[Pi])/b^2 /. abexample
(3/5 - I/5) \[Pi]
Or with the definite integral:
Integrate[1/(a^2*Cos[t]^2 + b^2*Sin[t]^2) /. abexample, {t, 0, 2*Pi}]
(3/5 - I/5) \[Pi]
Integrate[1/TrigToExp[a^2*Cos[t]^2 + b^2*Sin[t]^2], {t, 0, 2*Pi}]gives zero. !Mathematica graphics So clearly changing the form should not have made a difference in result. – Nasser Mar 8 at 5:20aandbwith real numbers you get the correct answer instantly. So Mathematica knows how to do the integration either way, something is just off here. – RunnyKine Mar 8 at 5:58