4
$\begingroup$

I want to backpropagate through random operations (e.g. learn the mean and variance of a Gaussian random variable). I assumed that

RandomArrayLayer[NormalDistribution[#\[Mu], #\[Sigma]] &];

implements the reparameterization trick. And therefore a RandomArrayLayer backpropagates a gradient to the mu and sigma inputs.

However, initial experiments seem to indicate that no such gradient is propagated.

Am I wrong in thinking that RandomArrayLayer is differentiable?

Thanks for any help.

$\endgroup$

1 Answer 1

6
$\begingroup$

It turns out the answer is no. However, Mathematica's great support for NNs means we can create our own. For example, to create a network layer of random variables that are normally distributed:

NormalReparameterizationLayer[initialMeans_List, initialStds_List]:=Module[
   {\[Mu]Parameters, \[Sigma]Parameters, rl, rv, bnl},
   rl = RandomArrayLayer[NormalDistribution[0,1],"Output"->Length[initialMeans]];
   \[Mu]Parameters = NetArrayLayer["Array"->initialMeans,"Output"->Length[initialMeans]];
   \[Sigma]Parameters = NetArrayLayer["Array"->initialStds,"Output"->Length[initialStds]];
   rv = FunctionLayer[#Means+#Stds*#Random&,"Output"->Length[initialMeans]];
   NetGraph[
      {\[Mu]Parameters, \[Sigma]Parameters, rl, rv},
      {1->4, 2->4, 3->4, NetPort[3,"Output"]->NetPort["Params"]}
   ]
]

Say that we want to create a reparameterization layer that generates 2 normally distributed variables with initial parameters {0.0, 10.0} and {30.0, 5.0}:

rpl = NormalReparamterizationLayer[{0.0, 30.0}, {10.0, 5.0}]

enter image description here

Running this layer yields a realisation of the two random variables:

In[]:= npl[]["Output"]

Out[]:= {4.42441, 23.87}

The layer is automatically differentiable. And so we can plug this layer into any NN architecture and learn the parameters of random variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.