3
$\begingroup$

Consider this simple network that computes the following function: $$ \left\{ \begin{matrix} xy & \mbox{if} \;x,y>0 \\ 0 & \mbox{otherwise} \end{matrix} \right.$$

The net looks like:

net = NetChain[{ElementwiseLayer[Ramp], AggregationLayer[Times, 1]}]

This works correctly as:

In[2]:= net[{4, 2}]

Out[2]= 8.    

In[3]:= net[{4, -2}]    

Out[3]= 0.    

In[4]:= net[{-4, -2}]    

Out[4]= 0.

However if try to evaluate the gradient in a given point, this returns a floating point error when either $x$ or $y$ are smaller than $0$.

In[5]:= net[{4, -2}, NetPortGradient["Input"]]

During evaluation of In[5]:= NetGraph::netnan: A floating-point overflow, underflow, or division by zero occurred while evaluating the net.

Out[5]= $Failed

Why this happens? Is there a way to avoid this?

PS: this is a very simplified example of the minimal network required to reproduce the error, extracted from a much more complicated loss function

EDIT: After posting the question, I discovered that the error can be reproduced also replacing Ramp with # - # &, but does not occur with # - # + 1 &

Link to question on Wolfram Community [link] [link updated 02/04/2019]

$\endgroup$
  • $\begingroup$ Please edit both this and your Wolfram Community post and cross-link them. $\endgroup$ – Szabolcs Apr 1 at 18:29
  • $\begingroup$ Which is the best way to do it? You insert the link? Is it considered "unpolite" to post the question in both places? $\endgroup$ – Luca Amerio Apr 1 at 18:32
  • 1
    $\begingroup$ Isn't this to do with the fact that Ramp will return 0 for negative numbers? Does NetPortGradient do some division somewhere? $\endgroup$ – Carl Lange Apr 1 at 18:39
  • 1
    $\begingroup$ If you replace the Times with Total it does not cause the error... $\endgroup$ – Carl Lange Apr 1 at 18:42
  • 2
    $\begingroup$ Yes - I suspect NetPortGradient is seeing the Times and reversing it (eg, dividing) in some way. $\endgroup$ – Carl Lange Apr 1 at 18:52
2
$\begingroup$

The error seems to occur when NetPortGradient tries to evaluate the gradient of a product in a point where both product's terms are null with null derivative. This is exactly what happens, for example, with the function $f(x,y) = x\cdot y$ in a region where both $x$ and $y$ are constantly $0$.

This is, at least in my opinion, a bug of NetPortGradient.

This cause also NetTrain to fail, as it fails to perform gradient descend.

To overcome the issue I rewrote the problem as $$ \left\{ \begin{matrix} (x+1)(y+1) - (x +y) -1 & \mbox{if} \;x,y>0 \\ 0 & \mbox{otherwise} \end{matrix} \right.$$

Mathematically, this is equal to the function above, as the function is just a convoluted way of writing $x\cdot y$. However, the product is perform between $(x+1)$ and $(y+1)$ that are non-zero everywhere (except for a point).

This allow the gradient to be correctly evaluated.

In the wolfram's Network language, this can be implemented as

net = NetGraph[
  {
   "ramp" -> ElementwiseLayer[Ramp],
   "x+1" -> ElementwiseLayer[# + 1 &],
   "times" -> AggregationLayer[Times, 1],
   "sum" -> AggregationLayer[Total, 1],
   "x-y-1" -> ThreadingLayer[#1 - #2 - 1 &]
   },
  {
   "ramp" -> "x+1",
   "x+1" -> "times",
   "ramp" -> "sum",
   {"times", "sum"} -> "x-y-1"
   }
  ]

and indeed

In[116]:= net[{4, 2}]

Out[116]= 8.

In[117]:= net[{-4, -2}]

Out[117]= 0.

In[118]:= net[{4, -2}, NetPortGradient["Input"]]

Out[118]= {0., 0.}

In[119]:= net[{-4, -2}, NetPortGradient["Input"]]

Out[119]= {0., 0.}

It works!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.