Is a way to calculate the Hessian of the loss function in Mathematica. It's easy to define a loss function for neural nets but I couldn't figure out how to calculate the Hessian(more precisely I wanted to get access to eigenspectrum of Hessian). There are scientific papers, where they calculate Hessian of a loss using python(autograd (auto differentiation) algorithm).

The idea here to train a neural net for few epochs (say on a simple fully connected net or simplified LeNet) and then calculate the Hessian of the loss function.

Addendum: Here's one such paper: https://arxiv.org/pdf/1611.07476.pdf

  • $\begingroup$ Could you share DOI of the paper you mentioned for reference? $\endgroup$ – Xminer Jul 2 '19 at 21:52
  • 1
    $\begingroup$ @Xminer Paper link added. Thanks for the suggestion. $\endgroup$ – psimeson Jul 2 '19 at 22:38
  • 1
    $\begingroup$ AFAIK Wolfram's NN framework works on top of MXNet library, so until it fully supports 2nd order differentiation it's not possible. Relevant issue. $\endgroup$ – swish Jul 3 '19 at 15:40
  • $\begingroup$ @swish This is quite frustrating then. Mathematica for theoretical ML research seems quite restricting. $\endgroup$ – psimeson Jul 3 '19 at 18:15
  • $\begingroup$ @psimeson because we can get 1st gradient,How about an approximation of Hessian? $\endgroup$ – Xminer Jul 3 '19 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.