When you use something like:
{a, b} := {1, 2}
This gets interpreted as:
a := 1; b := 2
as long as both sides are lists of the same length. If they are not, then it doesn't work:
{a, b} := Range[2]
SetDelayed::shape: Lists {a,b} and Range[2] are not the same shape.
$Failed
Now consider your example:
Hold[{{f[n], g[n]}} := {{f[n], g[n]}} = rhs] //FullForm
Hold[SetDelayed[List[List[f[n],g[n]]],Set[List[List[f[n],g[n]]],rhs]]]
Note that the RHS of SetDelayed is a an object with head Set, not a nested list. This is why it doesn't work. You could try the following instead:
Clear[f,g]
{{f[0]}, {g[0]}} = {{1},{1}};
f[n_] := (
{{f[n]}, {g[n]}} = {{f[n]},{g[n]}}={{0.2 f[n-1]+g[n-1]+1},{f[n-1]+0.2 g[n-1]+1}}
)[[1, 1]]
g[n_] := (
{{f[n]}, {g[n]}} = {{f[n]},{g[n]}}={{0.2 f[n-1]+g[n-1]+1},{f[n-1]+0.2 g[n-1]+1}}
)[[1, 2]]
Then:
f[3]
5.368
and let's check that definitions for both f
and g
have been cached:
Definition[f]
f[0]=1
f[1]=2.2
f[2]=3.64
f[3]=5.368
f[n_]:=({{f[n]},{g[n]}}={{f[n]},{g[n]}}={{0.2 f[n-1]+g[n-1]+1},{f[n-1]+0.2 g[n-1]+1}})[[1,1]]
and:
Definition[f]
g[0]=1
g[1]=2.2
g[2]=3.64
g[3]=5.368
g[n_]:=({{f[n]},{g[n]}}={{f[n]},{g[n]}}={{0.2 f[n-1]+g[n-1]+1},{f[n-1]+0.2 g[n-1]+1}})[[1,2]]