# Finding the symbolic expression for the loss function of a basic feed forward network

Is there any basic code available to get the symbolic expression of the cross entropy (and/or mean squared error) loss function for feed-forward neural network with one or more hidden layers in Mathematica? The symbolic expression cannot be extracted from Mathematica's deep learning toolbox (Can I extract symbolic expression of neural network loss function?), so one has to create it from scratch if we want to analyze the loss function symbolically.

It would be helpful if there is already some code available to get this in Mathematica.

• I can't quite tell what you're asking for - do you just want an implementation of the cross entropy loss function? – Carl Lange Jun 13 '19 at 10:18

The following from the documentation for CrossEntropyLossLayer (Properties & Relations section).

CrossEntropyLossLayer["Binary"]:

bceLoss = -(#Target*Log[#Input] + (1 - #Target)*Log[1 - #Input]) &;

bceLoss[<|"Input" -> 0.2, "Target" -> 1|>]


1.60944

CrossEntropyLossLayer["Binary"][<|"Input" -> 0.2, "Target" -> 1|>]


1.60944

CrossEntropyLossLayer["Probabilities"]:

ceLoss = -Total[#Target*Log@#Input] &;

data = <|"Input" -> {0.1, 0.2, 0.7}, "Target" -> {0.12, 0.3, 0.6}|>;
ceLoss[data]


0.973147

CrossEntropyLossLayer["Probabilities"][data]


0.973147

With these, you can eg plot the gradient:

Plot[bceLoss[<|"Target" -> 1, "Input" -> n|>], {n, 0, 1}]


• how does it help in analyzing the loss function symbolically? e.g., Factoring the expression, dividing by some other symbolic expression, taking symbolic gradients, etc.? – dbm Jun 13 '19 at 15:51
• @dbm Apparently I did not, and still don't, really understand what the question is here. – Carl Lange Jun 13 '19 at 16:59
• @dbm you can certainly take derivatives and factor the ceLoss and bceLoss expressions by doing something like: bceLoss[<|"Input" -> inp, "Target" -> targ|>] where inp and targ are symbolic. – b3m2a1 Jun 13 '19 at 17:51

The old Neural Networks Toolbox by Jonas Sjöberg had such functionality - not for the loss function, but for the squashers and weights. Probably it wouldn't be much problem to wrap a loss function on top of these. The neural nets created by the old package were not of course near as complex as the modern deep networks.

You can see the examples of output of the functional form with the weights inserted near the bottom of this page: How can I implement neural networks in other programming languages?