Is a way to calculate the Hessian of the loss function in Mathematica. It's easy to define a loss function for neural nets but I couldn't figure out how to calculate the Hessian(more precisely I wanted to get access to eigenspectrum of Hessian). There are scientific papers, where they calculate Hessian of a loss using python(autograd (auto differentiation) algorithm).

The idea here to train a neural net for few epochs (say on a simple fully connected net or simplified LeNet) and then calculate the Hessian of the loss function.

Addendum: Here's one such paper: https://arxiv.org/pdf/1611.07476.pdf

  • $\begingroup$ Could you share DOI of the paper you mentioned for reference? $\endgroup$
    – Xminer
    Jul 2, 2019 at 21:52
  • 1
    $\begingroup$ @Xminer Paper link added. Thanks for the suggestion. $\endgroup$
    – psimeson
    Jul 2, 2019 at 22:38
  • 1
    $\begingroup$ AFAIK Wolfram's NN framework works on top of MXNet library, so until it fully supports 2nd order differentiation it's not possible. Relevant issue. $\endgroup$
    – swish
    Jul 3, 2019 at 15:40
  • $\begingroup$ @swish This is quite frustrating then. Mathematica for theoretical ML research seems quite restricting. $\endgroup$
    – psimeson
    Jul 3, 2019 at 18:15
  • 1
    $\begingroup$ @psimeson connect the net and loss function,train it with NetTrain and then get 1st Gradient with NetGradient[{position of the layer,"Weights"}] $\endgroup$
    – Xminer
    Jul 3, 2019 at 21:21


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.