Font family, size, etc. depends on "Graphics" style in the notebook stylesheet

    CurrentValue[{StyleDefinitions, "Graphics", "FontSize"}]
     (* 10 *)

    CurrentValue[{StyleDefinitions, "Graphics", "FontFamily"}]
     (* "Times" *)

You can manually change it: open Format->Edit Stylesheet..., add
     
     Cell[StyleData["Graphics"],FontSize->20]

and press Ctrl+Shift+E. You will see that the font in `Plot`,`LinePlot`, etc. instantly increase.