First, no, I don't have a sound reason why ControlActive is so slow inside Animator. I can only guess and this doesn't help anyone. Additionally, I don't have the slightest idea how to tweak it in a simple way. Here is what tried anyway:
I first thought of hacking a simple version of ControlActive myself which is three lines of code
DynamicModule[{i, $active}, {Slider[Dynamic[i,
{($active = True; i = #) &, Automatic,($active = False; i = #) &}]],
Dynamic[i], Dynamic[$active]}]
The idea is to introduce a variable which is set to True when you start changing a slider and reset to False when you are finished. This is the equivalent of $ControlActiveSetting which is used by ControlActive.
This approach could be packed in a MyControlActive function and used inside dynamic code:
DynamicModule[
{i, $active, MyControlActive},
MyControlActive[a_, b_] := If[$active, a, b];
Panel[
Column[{
Slider[
Dynamic[i, {($active = True; i = #) &,
Automatic, ($active = False; i = #) &}], {1, 10}],
Dynamic@
Plot3D[Sin[i x + y^2], {x, -3, 3}, {y, -2, 2},
PlotStyle -> MyControlActive[None, Automatic], Mesh -> True,
PerformanceGoal -> "Quality"]
}]
]
]

The unexpected part is, that Animator is completely resistant to this approach.
DynamicModule[{i, $active},{Animator[Dynamic[i,
{($active = True; i = #) &, Automatic, ($active = False; i = #) &}]],
Dynamic[i], Dynamic[$active]}]
This shows always False. But why? The original idea assumes, that the 3 functions given to Dynamic are executed one after another. We made this assumption due to the documentation of Dynamic where we find
For interactive mouse operations Dynamic[expr,{fstart,f,fend}] typically evaluates once fstart when the mouse is pressed, then evaluates f whenever the mouse is moved, and then evaluates fend once when the mouse is released.
Lets build a small DynamicModule which keeps track of the Clock when moving a Slider or using Animator
DynamicModule[{t1, t2, t3, i},
Panel[Column[{
Dynamic[i],
Dynamic[{t1, t2, t3}],
Animator[
Dynamic[i, {(t1 = Clock[]; i = #) &, (t2 = Clock[];
i = #) &, (t3 = Clock[]; i = #) &}],
AnimationRunning -> False],
Slider[
Dynamic[i, {(t1 = Clock[]; i = #) &, (t2 = Clock[];
i = #) &, (t3 = Clock[]; i = #) &}]]
}]
]]
What we notice is that when we use the slider of Animator then fstart, f and fend are always executed. It doesn't matter whether I click or drag or stop dragging. Compare it to the behavior of the Slider at the bottom.

So it seems we cannot use such a simple approach. Another thing which could work is to continuously check whether the variable inside Animator is changing. In the simplest way one would use Refresh and check whether the animator-time is equal to the last checked value. If yes, we are not active. If it is different, we are active. In either way we save the current value of the animator time:
DynamicModule[{t = 0, tt = 0, MyControlActive},
MyControlActive[a_, b_] :=
If[Refresh[If[t != tt, tt = t; True, tt = t; False],
UpdateInterval -> .1, TrackedSymbols :> {}], a, b];
Panel[Column[{
Dynamic[{NumberForm[t, {1, 3}], MyControlActive[True, False]}],
Animator[Dynamic[t], AnimationRunning -> False]}]
]]
Works like a charm for this simple code and will, I'm sure, lead to difficulties in a real-life example :-) Here, we can set UpdateInterval to whatever value is needed but if it is too small, we get some False peaks during a running animation.