# Symbolic Neural Networks in Mathematica

Mathematica is very well known about its symbolic capabilities/functionality. Heavy users usually say that “everything is a symbolic expression”.

Does this principle apply to neural networks? If not, why is this the case?

Here is an example:

I define my own linear layer

myLinLayer[w_, b_, ds_] := w.ds + b


and I consider the one of Mathematica (fixing the weights and biases)

SeedRandom; mathematicaLinLayer=NetInitialize@LinearLayer[2,"Input"->3]


For the numerical input data set {1,2,3} both results are almost identical (after the 6th-7th decimal the results are different but this might be a separate topic for discussion).

myLinLayer[NetExtract[mathematicaLinLayer,"Weights"],NetExtract[mathematicaLinLayer, "Biases"], {1, 2, 3}]
{-4.90982, 0.0722178}


and

mathematicaLinLayer[{1, 2, 3}]
{-4.90982, 0.0722178}


(you can check that the two results are not identical, but this is irrelevant for this discussion).

For the symbolic input {s1,s2,s3}, using the self-defined linear layer, we get:

myLinLayer[NetExtract[mathematicaLinLayer,"Weights"],NetExtract[mathematicaLinLayer,"Biases"],{a, b, c}]
{0. - 0.94985 s1 - 0.971035 s3, 0. - 0.57963 s1 + 0.0729379 s3}


and I would expect to get the same results from Mathematicas's linear layer

mathematicaLinLayer[{s1, s2, s3}]
LinearLayer::invindata2: Data supplied to port "Input" was not a length-3 vector.


but this doesn’t seem to be the case. Am I doing something wrong? In the Documentation Center it is written that the input to a LinearLayer should be a tensor. Based again on the Documentation Center tensors in Mathematica can be symbolic:

“The Wolfram Language's uniform representation of vectors and matrices as lists automatically extends to tensors of any rank, allowing the Wolfram Language's powerful list manipulation functions immediately to be applied to tensors, both numerical and symbolic.

Is it possible to get symbolic results for any type of built-in neural network layers e.g. LSTMs, RNNs, GRNNs? Or one needs to built the entire set of equations from scratch?

Thanks!

• Some of the functionality you are asking about should be added in 12.1, likely called "Net Functions" (perhaps not precisely what you're asking about, but to some degree). Some of the details can be found in the livestreams SW does with the Neural Network team if you want to spare a few hours :) Jan 8, 2019 at 11:42

When you call NetInitialize[...], you get a wrapper function of MXNet's executor. It's a compiled tensor function inside MXNet's tensor virtual machine. It won't ever recognize symbolic MMA expressions.