3
$\begingroup$

When a network is trained, is it possible to get the corresponding mathematical expression?

For instance consider this simple network:

data = {{-2., -2., -10.} -> {-2.2, -2.3}, {-2., -2., -9.5} -> {-2.2, -2.4}, {-2., -1., -9.} -> {-4., -1.23}}
net = NetChain[{2, LogisticSigmoid, 2, LogisticSigmoid, 2}, "Input" -> 3];
trained = NetTrain[net, data]

I would like to get an explicit definition of f such that f[{x,y,z}] == trained[{x,y,z}] for any values of x,y,z. Note: trained[{x,y,z}] only evaluates if the variables have values.

Improve this question
$\endgroup$
4
  • $\begingroup$ What do you mean by "an explicit definition of f "? $\endgroup$
    – user64494
    Commented Feb 22, 2023 at 19:05
  • $\begingroup$ @user64494 Something like f[{x_,y_,z_}] = {2x + 3z^2, 5.2y^3} or more realistically f[{x_,y_,z_}] = {{1,2,3},{4,5,6},{7,8,9}}.LogisticSigmoid[{x,y,z}] + ... Like a closed-form expression. $\endgroup$
    – anderstood
    Commented Feb 22, 2023 at 19:18
  • $\begingroup$ Are Piecewises allowed?The results of trained[{5, -4000, 30}] and trained[{10, 10^7, 100}] demonstrate that the dependance on x,y,z is very weak. $\endgroup$
    – user64494
    Commented Feb 22, 2023 at 19:25
  • $\begingroup$ @user64494 I guess Piecewises would appear with ReLU activation functions, for instance. Not sure where there would come from here. As per the weak dependence in x,y,z, it is only the case in the simplistic toy example (with arbitrary values). $\endgroup$
    – anderstood
    Commented Feb 22, 2023 at 19:43

1 Answer 1

Reset to default
5
$\begingroup$

I found two similar questions on this topic (#197422, #199360), none of which has an answer. Neural networks in Mathematica are implemented via external library, called MXNet. Using Trace[trained[{1,2,3}]], you can see that the input is numerically transferred to MXNet, which evaluates the function.

For simple networks, you can get the symbolic function by extracting the relevant information from NetChain. The following code handles only LinearLayer and ElementwiseLayer but can probably be adapted for other types of layers.

netApply[input_, layer_] := Switch[NetExtract[layer, "Type"],
   ElementwiseLayer, NetExtract[layer, "Function"]@input,
   LinearLayer, Normal[NetExtract[layer, "Weights"]] . input + 
    Normal[NetExtract[layer, "Biases"]]
   ];

symbolicNet[net_NetChain] := 
  Function[Evaluate@
    FunctionExpand@
     Fold[netApply, Slot /@ Range@NetExtract[trained, "Input"], 
      NetExtract[net, All]]];

f = symbolicNet[trained]

{-1.07891 - 1.28238/(1 + E^(-0.945874 - 0.183235/( 1 + E^(0.14805 - 0.903133 #1 - 2.16383 #2 + 0.280905 #3)) - 1.79141/( 1 + E^(-0.0879924 + 1.29072 #1 - 0.38487 #2 + 1.21005 #3)))) - 0.672629/(1 + E^(-0.735348 + 0.915794/( 1 + E^(0.14805 - 0.903133 #1 - 2.16383 #2 + 0.280905 #3)) - 0.477314/( 1 + E^(-0.0879924 + 1.29072 #1 - 0.38487 #2 + 1.21005 #3)))), -0.309332 - 0.897976/(1 + E^(-0.945874 - 0.183235/( 1 + E^(0.14805 - 0.903133 #1 - 2.16383 #2 + 0.280905 #3)) - 1.79141/( 1 + E^(-0.0879924 + 1.29072 #1 - 0.38487 #2 + 1.21005 #3)))) - 1.08833/(1 + E^(-0.735348 + 0.915794/( 1 + E^(0.14805 - 0.903133 #1 - 2.16383 #2 + 0.280905 #3)) - 0.477314/( 1 + E^(-0.0879924 + 1.29072 #1 - 0.38487 #2 + 1.21005 #3))))} &

Validating the function:

f[1, 2, 3] - trained[{1, 2, 3}]

{-1.33618*10^-7, 8.15666*10^-8}

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I confirm that Domen answered exactly my question, and I also have differences in the order of $10^{-7}$. "Using Trace[trained[{1,2,3}]], you can see that the input is numerically transferred to MXNet" -> how do you read that in the output? I get essentially things like NeuralNetworksPrivateNetApply. Thank you by the way! $\endgroup$
    – anderstood
    Commented Feb 22, 2023 at 20:06
  • 2
    $\begingroup$ @anderstood, click on show all, then you can see expressions like MXNetLink`MXExecutorForward[...] ... $\endgroup$
    – Domen
    Commented Feb 22, 2023 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.