This is somewhat subtle, but really it's just a syntax issue. If you want to use the colon for pattern-matching, leave off the underscore, i.e.,
Clear[func]
func[x : {__List ..}] := "The argument was a list of lists"
func[x : {__Rule ..}] := "The argument was a list of rules"
Then these work correctly:
func[listOfLists]
func[listOfRules]
(* "The argument was a list of lists" *)
(* "The argument was a list of rules" *)
The issue is that x_ already stands for the pattern, and so doing x_ : patt doesn't really perform any sort of matching. However, using the syntax x : patt, then x names the pattern, and you can use it in later expressions. To be clear, under the definitions of these functions as given, you don't even need the x. That is,
func[{__List ..}] := "The argument was a list of lists"
func[{__Rule ..}] := "The argument was a list of rules"
work just like the above version with the x : patt. However, if you want to use the value of the input in the body of the function, then you need to name the pattern. For instance,
Clear[func]
func[x : {__List ..}] := x^2
func[x : {__Rule ..}] := {a, b^2, c^3, d^4} /. x
(* {{a^2, b^2}, {c^2, d^2, e^2}} *)
(* {b, b^2, d^3, d^4} *)
Finally, note that we can simplify the patterns {__List ..} and {__Rule ..} to just {__List} and {__Rule}, because the double-underscore (BlankSequence) automatically matches one or more objects, so {__List} will match any list of one or more Lists.
This in my opinion is the cleanest way of doing this, but there are alternatives. The following patterns using /; and ? also work. First, using ?, we have
func[x_?(MatchQ[#, {__List}] &)] := ...
func[x_?(MatchQ[#, {__Rule}] &)] := ...
or the more succinct
func[x_?(MatchQ[{__List}])] := ...
func[x_?(MatchQ[{__Rule}])] := ...
the latter of which takes advantage of the operator form of MatchQ (unfortunately, the parentheses are still required for strange precedence reasons). The ? notation means that the pattern will match provided the function following ? evaluates to True when applied to x.
Using /;, we have
func[x_ /; MatchQ[x, {__List}]] := ...
func[x_ /; MatchQ[x, {__Rule}]] := ...
The /; has a slightly more general functionality than ?, because x_ will match any pattern so long as what follows the /; evaluates to True, and what follows /; does not have to be applied to x (although it usually is). For instance,
gg[x_ /; True]
is essentially the same as just gg[x_]. In addition /; can be used inside or outside the arguments of the function or even on the right-hand side of the entire expression, i.e.,
hh[x_ /; x > 2] := ...
hh[x_] /; x > 2 := ...
hh[x_] := ... /; x > 2
all essentially do the same thing.
We usually prefer /; over ? when doing things like logical tests:
g[x_ /; x > 2] := ...
is nicer than
g[x_?(# > 2 &)] := ...
Finally, just a little bit of syntax notes. Optional and Pattern have very similar syntax, because they both use a colon :. You can tell the difference partly in the way that they are used, but the syntax coloring also gives it away. Compare the following two code snippets:
f[x_: "a"] := x+1
g[x : "a"] := x+1
and consider the following sequence of commands:
f[b] (* 1 + b *)
f["a"] (* 1 + "a" *)
f[] (* 1 + "a" *)
g[b] (* g[b] *)
g["a"] (* 1 + "a" *)
g[] (* g[] *)
The x_: default notation allows you to specify a default value for the function that will be returned if you provide no input. Otherwise, it will evaluate the function on whatever you're feeding to it. On the other hand, the x : patt notation makes it so that the function g will only evaluate if the input x matches patt, which in this case means that it returns unevaluated (e.g., g[b] returns g[b]) unless x is exactly "a".
