I am trying to implement a neural network where I output a vector of 5 parameters each of which I know to be positive. I have code that works where I end up with an `"Output" -> 5.'

However, after training the network, it is often finding good results setting one of these values to be negative. However, physically that is not possible.

Is there a way to change my cost function to enforce (or encourage) positive values?

  • $\begingroup$ There is probably a way, yes. $\endgroup$ – anderstood Feb 15 '18 at 20:37
  • 1
    $\begingroup$ why not add a term to the cost function that vanishes for positive outputs and penalized negative outputs? $\endgroup$ – yohbs Feb 15 '18 at 21:09
  • $\begingroup$ A penalty like Total[Ramp[-x]^2] should do, where x is the parameter vector. Also, a barrier like -Total[Log[-x]] might do it. Hard to say without detailed code... $\endgroup$ – Henrik Schumacher Feb 15 '18 at 22:18
  • $\begingroup$ Another option is to use an activation function for the last layer where the range of the function matches the range of the expected output. e.g. you would often see a sigmoid function if the outputs are probabilities with range 0..1 $\endgroup$ – Niki Estner Feb 16 '18 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.