Recurrent neural network in 11.1 explicit examples?

Question

I heard that RNN was implemented in Mathematica as of 11.1. Trying to search online, I find some general information about neural networks in Mathematica, or a list of related functions. My trouble is that this list of functions lumps purely statistical machine learning functions like Classify and Predict together with neural network functions, as well as (I presume) recurrent neural network functions, so that it is really hard to tell what it is I actually need to do RNN.

Perhaps there is a resource I missed that shows an explicit example of how to tackle a time series forecasting problem with several inputs, making use of Mathematicas RNN functions?

If none are known, perhaps someone knowledgeable could write a short example, e.g. using example data from here?

Thanks for any suggestion!

Answer 1

Here is a simple example that may help you get started. In this example, we are going to a predict a simple time series of a sinusoid wave.

data = Table[Sin[x], {x, 0, 100, 0.04}];

We will cut the data into windows of 51 data points. The first 50 points as a whole is our X, and the last data point is our Y.

training = 
  RandomSample[
   List /@ Most[#] -> List@Last[#] & /@ (Partition[data, 51, 1])];

We use a single gated recurrent layer in our neural network

net = NetChain[{
   GatedRecurrentLayer[10],
   LinearLayer[1]}, "Input" -> {50, 1}, "Output" -> 1
  ]

and train with the training data

trained = NetTrain[net, training]

After training, we can use it to predict the time series. We first feed the neural network with 50 data points and then repeatedly use the data it generates to feed back into the neural network to generate the next data point. Here is a comparison between the ground truth and our predictions, which shows very good agreements.

ListPlot[{Flatten@
   NestList[Append[Rest[#], trained[#]] &, 
     List /@ Sin[Range[-49*0.04, 0, 0.04]], 500][[All, -1]], 
  Table[Sin[x], {x, 0, 500*0.04, 0.04}]}, Joined -> True, 
 PlotLegends -> {"predicted", "ground truth"}]

Answer 2

Taking inspiration from the answer by xslittlegrass, I came up with the following solution.

Recall the sample data from this question. We have an observable obs we are interested to predict:

and three parameters par1, par2, par3 that are correlated with the observable to some extent:

We only use the data for the first 700 time steps to train the model, and will try to predict the next 300 time steps.

We create a training set of tlen consecutive data points of length featn each, which have the respectively following data point of length featn as output. Then we train a model that returns featn outputs:

dat = Transpose[{par1/Max[par1], par2/Max[par2], par3/Max[par3], obs/Max[obs]}];
tlen = 300; featn = Length[dat[[1]]];
training = Table[dat[[i ;; i + tlen - 1]] -> dat[[i + tlen]], {i, 1, Length[dat] - tlen}];
net = NetChain[{GatedRecurrentLayer[tlen, "Dropout" -> {"VariationalInput" -> 0.1 , "VariationalState" -> 0.5}], LinearLayer[featn]}, "Input" -> {tlen, featn}, "Output" -> {featn}]
trained = NetTrain[net, training, Method -> {"ADAM", "InitialLearningRate" -> 0.0001}]

The training takes about two minutes. Finally, we can iteratively predict the future 300 steps

datt = dat;
Dynamic[i]
Do[
 start = datt[[-tlen ;;]];
 AppendTo[datt, trained[start]];
 , {i, 1, 300}]

Amazingly, the prediction is qualitatively correct, with amplitude deviation growing to about 15% over the course of 300 time steps!

ListPlot[{datt[[;; , 4]], tab}]

Any suggestions for how to improve upon the above?

Answer 3

For instance, let's assume you have a sequence of 3 and you have 8 input variables (features) X

Let Y be the output with values "yes" or "no" for each sequence of X

Let's X have a dimension of 195

You create a sequence of 3 for X

Xpartition = Partition[X, 3]

Now, you create your trainingData:

trainingData = MapThread[Rule, {Xpartition, Y}]

You build your model:

net = NetChain[
    { LongShortTermMemoryLayer[32]
    , SequenceLastLayer[]
    , LinearLayer[2]
    , SoftmaxLayer[]
    }
  , "Input" -> {3, 8}
  , "Output" -> NetDecoder[{"Class", {"no", "yes"}}]
]

where 3 is the number of vectors in the sequence, and 8 is the length of vectors.

Answer 4

Key applied RNN examples from developers are located in documentation at:

http://reference.wolfram.com/language/tutorial/NeuralNetworksOverview.html

See "Sequence Learning and NLP" section.

Answer 5

Here RNN in Mathematica? they are talking about RNN

In help section, if you for LongShortTermMemoryLayer you will also find other RNN models that have been implemented.