Font family, size, etc. depends on "Graphics" style in the notebook stylesheet

    CurrentValue[{StyleDefinitions, "Graphics", "FontSize"}]
     (* 10 *)

    CurrentValue[{StyleDefinitions, "Graphics", "FontFamily"}]
     (* "Times" *)

You can manually change it: open Format->Edit Stylesheet..., add
     
     Cell[StyleData["Graphics"],FontSize->20]

and press Ctrl+Shift+E. You will see that the font in `Plot`,`LinePlot`, etc. instantly increase.

*Edit:* default values for `Plot`, `ListPlot`, etc. separately can be set by

    SetOptions[Plot, BaseStyle -> {FontFamily -> "Helvetica", FontSize -> 16}]