3
$\begingroup$

I am attempting to use a Neural Network to solve differential equations based on a method I read in a paper. This involves constructing multiple separate NetGraph objects, where each one represents a derivative of the original NetChain that I would like to train.
To make this work, I had to use NetSharedArray for all of the weight and bias parameters so that the derivative network will be updated during each iteration of NetTrain as all of the parameters are updated.
The structure of my training network is as follows: enter image description here

The top section plugs the input as a "Real" into three networks: the network to be trained, its first derivative network, and its second derivative network. The output of these is then multiplied by coefficients to represent the Second Order ODE in question and Totaled. This is compared with a MeanSquaredLossLayer[] to a real number which represents the right hand side of the ODE, inputted through a ConstantArrayLayer[].

The bottom section handles the initial conditions by taking the target value used in training (which is not really needed for my training), multiplying it by 0, and then plugging it into the network and its first derivative to get the current y(0) and y'(0) in the training iteration. Finally, these are also compared to certain pre-chosen values sent through ConstantArrayLayer[]s and all of the Mean Squared Losses are summed up and sent to the "TotalLoss" port.

Here is the code I used to train this network:

data = RandomReal[{-1, 1}, 1000];
zeros = Table[0, {k, 1, 1000}];
trainingResult = NetTrain[trainingGraph, data -> zeros, LossFunction -> "TotalLoss"]

The training works fine and returns the trained NetGraph. But then when I extract the network to test it, I get this error:
enter image description here

For some reason, the NetSharedArray objects I used don't show any evidence of even being trained. When I actually check their values, only one has any:
enter image description here I'm not sure why this issue is happening. I tried this method with a First Order ODE and it worked fine and produced solutions, but now I'm seeing issues with it. Is this due to a problem with NetSharedArray or is there a problem with my training graph?

$\endgroup$
  • $\begingroup$ I'm the OP of this question (just added my account). I think this problem may be a glitch in Mathematica because it seems to go away randomly sometimes when I close and restart Mathematica. The glitch may just be in how the NetSharedArray objects are initialized during training. If you keep closing and reopening your notebook, it should eventually work again. Or try just resetting the Kernel. $\endgroup$ – JDComplex Sep 9 at 10:44
1
$\begingroup$

I think I am experiencing the same issue. I am trying out some experiments to see if I can train a Bayesian neural net using gradient descent, and I am trying an approach using the Mathematica neural net functionality.

My approach involves using a NetSharedArray in a couple of sub net graphs.

If I manually replace the values and run the net forward it seems to calculate correctly.

However when it comes to training using NetTrain, it doesn't appear to be learning very effectively at all. I am unable to evaluate the trained network:

NetTake[trained, {"pMarg", "pMarg"}][Association["x" -> 1, "y" -> 1]] returns NetGraph::nfspec: Cannot evaluate net: shared array "px/tabz0/1/Array" is not fully specified.

Interestingly if I try:

NetExtract[trained, NetSharedArray["px/tabz1/1/Array"]]

I get the output: Automatic which suggests the values have not been trained.

Thanks, Julian.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.