# NetPortGradient returns “floating-point overflow” for this simple network

Consider this simple network that computes the following function: $$\left\{ \begin{matrix} xy & \mbox{if} \;x,y>0 \\ 0 & \mbox{otherwise} \end{matrix} \right.$$

The net looks like:

net = NetChain[{ElementwiseLayer[Ramp], AggregationLayer[Times, 1]}]


This works correctly as:

In[2]:= net[{4, 2}]

Out[2]= 8.

In[3]:= net[{4, -2}]

Out[3]= 0.

In[4]:= net[{-4, -2}]

Out[4]= 0.


However if try to evaluate the gradient in a given point, this returns a floating point error when either $$x$$ or $$y$$ are smaller than $$0$$.

In[5]:= net[{4, -2}, NetPortGradient["Input"]]

During evaluation of In[5]:= NetGraph::netnan: A floating-point overflow, underflow, or division by zero occurred while evaluating the net.

Out[5]= \$Failed


Why this happens? Is there a way to avoid this?

PS: this is a very simplified example of the minimal network required to reproduce the error, extracted from a much more complicated loss function

EDIT: After posting the question, I discovered that the error can be reproduced also replacing Ramp with # - # &, but does not occur with # - # + 1 &

• Please edit both this and your Wolfram Community post and cross-link them. – Szabolcs Apr 1 at 18:29
• Which is the best way to do it? You insert the link? Is it considered "unpolite" to post the question in both places? – Luca Amerio Apr 1 at 18:32
• Isn't this to do with the fact that Ramp will return 0 for negative numbers? Does NetPortGradient do some division somewhere? – Carl Lange Apr 1 at 18:39
• If you replace the Times with Total it does not cause the error... – Carl Lange Apr 1 at 18:42
• Yes - I suspect NetPortGradient is seeing the Times and reversing it (eg, dividing) in some way. – Carl Lange Apr 1 at 18:52

The error seems to occur when NetPortGradient tries to evaluate the gradient of a product in a point where both product's terms are null with null derivative. This is exactly what happens, for example, with the function $$f(x,y) = x\cdot y$$ in a region where both $$x$$ and $$y$$ are constantly $$0$$.

This is, at least in my opinion, a bug of NetPortGradient.

This cause also NetTrain to fail, as it fails to perform gradient descend.

To overcome the issue I rewrote the problem as $$\left\{ \begin{matrix} (x+1)(y+1) - (x +y) -1 & \mbox{if} \;x,y>0 \\ 0 & \mbox{otherwise} \end{matrix} \right.$$

Mathematically, this is equal to the function above, as the function is just a convoluted way of writing $$x\cdot y$$. However, the product is perform between $$(x+1)$$ and $$(y+1)$$ that are non-zero everywhere (except for a point).

This allow the gradient to be correctly evaluated.

In the wolfram's Network language, this can be implemented as

net = NetGraph[
{
"ramp" -> ElementwiseLayer[Ramp],
"x+1" -> ElementwiseLayer[# + 1 &],
"times" -> AggregationLayer[Times, 1],
"sum" -> AggregationLayer[Total, 1],
"x-y-1" -> ThreadingLayer[#1 - #2 - 1 &]
},
{
"ramp" -> "x+1",
"x+1" -> "times",
"ramp" -> "sum",
{"times", "sum"} -> "x-y-1"
}
]


and indeed

In[116]:= net[{4, 2}]

Out[116]= 8.

In[117]:= net[{-4, -2}]

Out[117]= 0.