Under Linux, Mathematica uses preinstalled QT themes, but I'm sure that it's possible to make it use system GTK style. How?
Preferences -> Advanced -> Open Options Inspector.
In the Options Inspector window:
Global Options -> PrivateFrontEndOptions
In the right pane: "XDefaultApplicationStyle": by default, it is "plastique". Change it to "windows", for example, or other styles defined in qt4-qtconfig. You can also change the font "Bitstream Vera Sans-10" to "Bitstream Vera Sans-8", for example, to reduce the font size a little.