I've set up my plot to look exactly how I like it. At least it does so on screen. Upon export though, only the bitmap formats look the same. For PDF and EPS, the FontSize seems to get ignored and the lines aren't outlined so when downscaling the whole plot the lines get thicker in relation.
Isn't there a simple way to just render the whole thing as a proper PDF with everything (including fonts) outlined and at correct size?!
EDIT: So there were two different problems I ran into: First was Adobe Illustrator somehow couldn't display all the text that Mathematica output into the PDF. Therefore the option to outline from within Mathematica is very convenient.
Second was my (newbish) mistake to specify the font size with
Scaled[0.02] wich was clearly not what I wanted. I now changed it to fixed point sizes.