Font family, size etc. depends on "Graphics" style in the notebook stylesheet
CurrentValue[{StyleDefinitions, "Graphics", "FontSize"}]
(* 10 *)
CurrentValue[{StyleDefinitions, "Graphics", "FontFamily"}]
(* "Times" *)
You can manually change it: open Format->Edit Stylesheet..., add
Cell[StyleData["Graphics"],FontSize->20]
and press Ctrl+Shift+E. You will see that the font in Plot,LinePlot, etc. instantly increase.