Symbolic Neural Networks in Mathematica

Question

Mathematica is very well known about its symbolic capabilities/functionality. Heavy users usually say that “everything is a symbolic expression”.

Does this principle apply to neural networks? If not, why is this the case?

Here is an example:

I define my own linear layer

myLinLayer[w_, b_, ds_] := w.ds + b

and I consider the one of Mathematica (fixing the weights and biases)

SeedRandom[123]; mathematicaLinLayer=NetInitialize@LinearLayer[2,"Input"->3]

For the numerical input data set {1,2,3} both results are almost identical (after the 6th-7th decimal the results are different but this might be a separate topic for discussion).

myLinLayer[NetExtract[mathematicaLinLayer,"Weights"],NetExtract[mathematicaLinLayer, "Biases"], {1, 2, 3}]
{-4.90982, 0.0722178}

and

mathematicaLinLayer[{1, 2, 3}]
{-4.90982, 0.0722178}

(you can check that the two results are not identical, but this is irrelevant for this discussion).

For the symbolic input {s1,s2,s3}, using the self-defined linear layer, we get:

myLinLayer[NetExtract[mathematicaLinLayer,"Weights"],NetExtract[mathematicaLinLayer,"Biases"],{a, b, c}]
{0. - 0.94985 s1 - 0.971035 s3, 0. - 0.57963 s1 + 0.0729379 s3}

and I would expect to get the same results from Mathematicas's linear layer

mathematicaLinLayer[{s1, s2, s3}]
LinearLayer::invindata2: Data supplied to port "Input" was not a length-3 vector.

but this doesn’t seem to be the case. Am I doing something wrong? In the Documentation Center it is written that the input to a LinearLayer should be a tensor. Based again on the Documentation Center tensors in Mathematica can be symbolic:

“The Wolfram Language's uniform representation of vectors and matrices as lists automatically extends to tensors of any rank, allowing the Wolfram Language's powerful list manipulation functions immediately to be applied to tensors, both numerical and symbolic.

Is it possible to get symbolic results for any type of built-in neural network layers e.g. LSTMs, RNNs, GRNNs? Or one needs to built the entire set of equations from scratch?

Thanks!

Answer 1

As of M11.3:

Mathematica's neural network implementation is a wrapper of Apache MXNet. While Mathematica's own matrix multiplication wraps Intel MKL. This can explain the numerical different you experience. The NN interface design is very similar to Keras. However it lacks support for full symbolic tensor expression manipulation. (Which I consider as a shortcoming)

When you call NetInitialize[...], you get a wrapper function of MXNet's executor. It's a compiled tensor function inside MXNet's tensor virtual machine. It won't ever recognize symbolic MMA expressions.

MXNet does have a symbolic API on itself, however it's not exposed to Mathematica as far as I know.