I tried this in MMA 10.01 (Mac) with a simple example from the documentation:
anim = Table[Plot[Sin[n x], {x, 0, 10}], {n, 5}]
It doesn't look terrible but I agree that the anti-aliasing is clearly better in MMA (I had previously modified my anti-aliasing quality settings via Preferences/Appearance/Graphics).
A possible reason for this is that MMA's default codec for the exported animation (at least on the Mac) is "Cinepak". Cinepak is one of the very earliest QuickTime codecs dating from 1992 and appears to struggle with anti-aliased lines. I got the best results by exporting to QuickTime using the "Apple Intermediate Codec" ("Animation" also works well).
Export["anim.mov", anim, ImageResolution -> 800,
Antialiasing -> True, "VideoEncoding" -> "Apple Intermediate Codec"]
You can then re-compress and export this to another format, if necessary, using QuickTime itself or ffmpeg (which recognizes both codecs).