My current understanding about Mathematica is that everything, at the lowest level, ends up as replacement rules.
First question: is this true?
Second question: does Mathematica "gloss over" these rules by representing them as functions? 
doesn't seem to be expressed as a rule.
Third question: does the by value/by reference question have any when you "pass" a value to a Mathematica function?
When you define a function f, you are (a) applying Set or SetDelayed to a pattern, (b) creating a rule, (c) associating the rule with a symbol (f), and (d) storing that in DownValues (usually).
FullForm[Hold[f[x_] := x^2]]
(* Hold[SetDelayed[f[Pattern[x, Blank[]]], Power[x, 2]]] *)
f[x_] := x^2
DownValues@f
(* {HoldPattern[f[x_]] :> x^2} *)
No: for example 2 is not a rule. It's just 2. So not everything is a rule, just functions. Everything is an expression, ... some expressions (as a side-effect) create and store rules in DownValues.
Yes and no. They give you a glossy shortcut for input:
f[x_] := x^2
instead of requiring you to write something like
AppendTo[DownValues[f], HoldPattern[f[x_]] :> x^2]
There was one misunderstanding in your question; you asked about the output of
FullForm[f[x]]
This is not showing you the definition of the function f. It is showing you the result of applying f to x. See the following:
FullForm[f[2]]
(* 4 *)
If you want to see the definition of f you should either use Information (which is glossy and not very useful) or DownValues which contains what you're after.
Work on the assumption that Mathematica is pretty smart "under the hood" about pass by value/ pass by reference.
Sin[0.15], it calls the built-in numerical implementation ofSinto make a computation - at which point it arguably leaves the rule-based paradigm. And similarly for most other useful lower-level actions. $\endgroup$DownValues[f]that's where the rules are hiding in this case $\endgroup$Function). $\endgroup$