You can get the desired result in two steps:
d1 = TransformedDistribution[x1^2 + x2^2,
{Distributed[x1, NormalDistribution[0, σ]],
Distributed[x2, NormalDistribution[0, σ]]}]
(* ExponentialDistribution[1/(2 σ^2)] *)
d2 = TransformedDistribution[Sqrt[z], Distributed[z, d1]]
(* RayleighDistribution[σ] *)
PDF[d2,r]
$\begin{cases}
\frac{r e^{-\frac{r^2}{2 \sigma ^2}}}{\sigma ^2} & r>0 \\
0 & \text{True}
\end{cases}$
You can also fold the two steps into a single step:
With[{d1 =TransformedDistribution[x1^2 + x2^2,
{Distributed[x1, NormalDistribution[0, σ]],
Distributed[x2, NormalDistribution[0, σ]]}]},
TransformedDistribution[Sqrt[z], Distributed[z, d1]]]
(* RayleighDistribution[σ] *)
Update: The results above are obtained in Version 9.0.1.0. As noted by @m0nhawk in the comments, in version 10, TransformedDistribution[Sqrt[z], Distributed[z, d1]]
gives the result
WeibullDistribution[2, Sqrt[2] σ]
which is equivalent to RayleighDistribution[σ]
:
PDF[WeibullDistribution[2, Sqrt[2] σ], r] == PDF[d2, r]
(* True *)
RayleighDistribution
? $\endgroup$