5
$\begingroup$

Is there any basic code available to get the symbolic expression of the cross entropy (and/or mean squared error) loss function for feed-forward neural network with one or more hidden layers in Mathematica? The symbolic expression cannot be extracted from Mathematica's deep learning toolbox (Can I extract symbolic expression of neural network loss function?), so one has to create it from scratch if we want to analyze the loss function symbolically.

It would be helpful if there is already some code available to get this in Mathematica.

$\endgroup$
  • $\begingroup$ I can't quite tell what you're asking for - do you just want an implementation of the cross entropy loss function? $\endgroup$ – Carl Lange Jun 13 at 10:18
2
+25
$\begingroup$

The following from the documentation for CrossEntropyLossLayer (Properties & Relations section).


CrossEntropyLossLayer["Binary"]:

bceLoss = -(#Target*Log[#Input] + (1 - #Target)*Log[1 - #Input]) &;

bceLoss[<|"Input" -> 0.2, "Target" -> 1|>]

1.60944

CrossEntropyLossLayer["Binary"][<|"Input" -> 0.2, "Target" -> 1|>]

1.60944


CrossEntropyLossLayer["Probabilities"]:

ceLoss = -Total[#Target*Log@#Input] &;

data = <|"Input" -> {0.1, 0.2, 0.7}, "Target" -> {0.12, 0.3, 0.6}|>;
ceLoss[data]

0.973147

CrossEntropyLossLayer["Probabilities"][data]

0.973147


With these, you can eg plot the gradient:

Plot[bceLoss[<|"Target" -> 1, "Input" -> n|>], {n, 0, 1}]

enter image description here

$\endgroup$
  • $\begingroup$ how does it help in analyzing the loss function symbolically? e.g., Factoring the expression, dividing by some other symbolic expression, taking symbolic gradients, etc.? $\endgroup$ – dbm Jun 13 at 15:51
  • $\begingroup$ @dbm Apparently I did not, and still don't, really understand what the question is here. $\endgroup$ – Carl Lange Jun 13 at 16:59
  • $\begingroup$ @dbm you can certainly take derivatives and factor the ceLoss and bceLoss expressions by doing something like: bceLoss[<|"Input" -> inp, "Target" -> targ|>] where inp and targ are symbolic. $\endgroup$ – b3m2a1 Jun 13 at 17:51
2
$\begingroup$

The old Neural Networks Toolbox by Jonas Sjöberg had such functionality - not for the loss function, but for the squashers and weights. Probably it wouldn't be much problem to wrap a loss function on top of these. The neural nets created by the old package were not of course near as complex as the modern deep networks.

You can see the examples of output of the functional form with the weights inserted near the bottom of this page: How can I implement neural networks in other programming languages?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.