19
$\begingroup$

I heard that RNN was implemented in Mathematica as of 11.1. Trying to search online, I find some general information about neural networks in Mathematica, or a list of related functions. My trouble is that this list of functions lumps purely statistical machine learning functions like Classify and Predict together with neural network functions, as well as (I presume) recurrent neural network functions, so that it is really hard to tell what it is I actually need to do RNN.

Perhaps there is a resource I missed that shows an explicit example of how to tackle a time series forecasting problem with several inputs, making use of Mathematicas RNN functions?

If none are known, perhaps someone knowledgeable could write a short example, e.g. using example data from here?

Thanks for any suggestion!

$\endgroup$
  • 5
    $\begingroup$ Perhaps this YouTube video will help. youtu.be/dr3vAuFH5io $\endgroup$ – m_goldberg Aug 30 '17 at 22:55
  • 1
    $\begingroup$ @m_goldberg Thank you, that was very helpful indeed! Even though the video did not address any cases in which the sequence has continuity properties in the mathematical sense, I am starting to suspect that this property is implied to be captured by these techniques without doing anything extra. $\endgroup$ – Kagaratsch Aug 30 '17 at 23:48
14
$\begingroup$

Here is a simple example that may help you get started. In this example, we are going to a predict a simple time series of a sinusoid wave.

data = Table[Sin[x], {x, 0, 100, 0.04}];

enter image description here

We will cut the data into windows of 51 data points. The first 50 points as a whole is our X, and the last data point is our Y.

training = 
  RandomSample[
   List /@ Most[#] -> List@Last[#] & /@ (Partition[data, 51, 1])];

We use a single gated recurrent layer in our neural network

net = NetChain[{
   GatedRecurrentLayer[10],
   LinearLayer[1]}, "Input" -> {50, 1}, "Output" -> 1
  ]

and train with the training data

trained = NetTrain[net, training]

After training, we can use it to predict the time series. We first feed the neural network with 50 data points and then repeatedly use the data it generates to feed back into the neural network to generate the next data point. Here is a comparison between the ground truth and our predictions, which shows very good agreements.

ListPlot[{Flatten@
   NestList[Append[Rest[#], trained[#]] &, 
     List /@ Sin[Range[-49*0.04, 0, 0.04]], 500][[All, -1]], 
  Table[Sin[x], {x, 0, 500*0.04, 0.04}]}, Joined -> True, 
 PlotLegends -> {"predicted", "ground truth"}]

enter image description here

$\endgroup$
  • $\begingroup$ This looks interesting! What should be changed if we have more features than just the observable itself? Do I understand correctly, that we would need to predict each feature as well as the observable (plus 49 past steps) in order to make a second prediction step, and so forth, in this case? --- (for the sake of example, maybe you could just duplicate the data and feed it as several features in parallel, to illustrate the syntax?) $\endgroup$ – Kagaratsch Aug 30 '17 at 20:31
  • $\begingroup$ Also, which criterium tells you that you should use only one layer, and with exactly 10 nodes? $\endgroup$ – Kagaratsch Aug 30 '17 at 20:34
  • $\begingroup$ I decided to accept this answer because it made the syntax most clear and gave me the ability to write my own code. $\endgroup$ – Kagaratsch Aug 31 '17 at 12:58
  • 3
    $\begingroup$ @Kagaratsch For multiple inputs, the first thing you can try is to use CatenateLayer to join the three input into a long sequence before feeding it into the recurrent layers. For example, something like:NetGraph[{CatenateLayer[], GatedRecurrentLayer[10], LinearLayer[1]}, {{NetPort["Input1"], NetPort["Input2"], NetPort["Input3"]} -> 1, 1 -> 2 -> 3}, "Input1" -> {50, 1}, "Input2" -> {50, 1}, "Input3" -> {50, 1}, "Output" -> 1], where input1,2,3,4 can be your par1,2,3 and obs. $\endgroup$ – xslittlegrass Aug 31 '17 at 14:18
  • 1
    $\begingroup$ I set the number of layers and number of neurons by trail and error. But in general, increase the number of neurons will make the network more powerful to capture the features in the data, but it is also prone to overfitting and requires more data to train. $\endgroup$ – xslittlegrass Aug 31 '17 at 14:26
12
$\begingroup$

Taking inspiration from the answer by xslittlegrass, I came up with the following solution.

Recall the sample data from this question. We have an observable obs we are interested to predict:

enter image description here

and three parameters par1, par2, par3 that are correlated with the observable to some extent:

enter image description here

We only use the data for the first 700 time steps to train the model, and will try to predict the next 300 time steps.

We create a training set of tlen consecutive data points of length featn each, which have the respectively following data point of length featn as output. Then we train a model that returns featn outputs:

dat = Transpose[{par1/Max[par1], par2/Max[par2], par3/Max[par3], obs/Max[obs]}];
tlen = 300; featn = Length[dat[[1]]];
training = Table[dat[[i ;; i + tlen - 1]] -> dat[[i + tlen]], {i, 1, Length[dat] - tlen}];
net = NetChain[{GatedRecurrentLayer[tlen, "Dropout" -> {"VariationalInput" -> 0.1 , "VariationalState" -> 0.5}], LinearLayer[featn]}, "Input" -> {tlen, featn}, "Output" -> {featn}]
trained = NetTrain[net, training, Method -> {"ADAM", "InitialLearningRate" -> 0.0001}]

The training takes about two minutes. Finally, we can iteratively predict the future 300 steps

datt = dat;
Dynamic[i]
Do[
 start = datt[[-tlen ;;]];
 AppendTo[datt, trained[start]];
 , {i, 1, 300}]

Amazingly, the prediction is qualitatively correct, with amplitude deviation growing to about 15% over the course of 300 time steps!

ListPlot[{datt[[;; , 4]], tab}]

enter image description here

Any suggestions for how to improve upon the above?

$\endgroup$
  • 2
    $\begingroup$ Add dropout ("VariationalInput"-> 0.1 and "VariationalState"-> 0.5) ,(try different values) showed that also during training tweek with learningRate (try from 0.1 to 0.0001 and see you get an improvement. $\endgroup$ – user34018 Aug 31 '17 at 10:46
  • $\begingroup$ @user34018 Thanks! Those options helped improve the prediction quite a bit! $\endgroup$ – Kagaratsch Aug 31 '17 at 12:18
  • $\begingroup$ tab is not defined in the ListPlot code. You can include it for the sake of completeness. $\endgroup$ – PlatoManiac Aug 31 '17 at 13:50
  • 1
    $\begingroup$ @PlatoManiac I added tab to the data at gist.github.com/anonymous/8b65b221a1eef1afd9b8491d61283b0f $\endgroup$ – Kagaratsch Aug 31 '17 at 17:29
4
$\begingroup$

For instance, let's assume you have a sequence of 3 and you have 8 input variables (features) X

Let Y be the output with values "yes" or "no" for each sequence of X

Let's X have a dimension of 195

You create a sequence of 3 for X

Xpartition = Partition[X, 3]

Now, you create your trainingData:

trainingData = MapThread[Rule, {Xpartition, Y}]

You build your model:

net = NetChain[
    { LongShortTermMemoryLayer[32]
    , SequenceLastLayer[]
    , LinearLayer[2]
    , SoftmaxLayer[]
    }
  , "Input" -> {3, 8}
  , "Output" -> NetDecoder[{"Class", {"no", "yes"}}]
]

where 3 is the number of vectors in the sequence, and 8 is the length of vectors.

$\endgroup$
  • $\begingroup$ Thank you for your hints! May I ask a few questions? By sequence of 3 you mean, your time series has 3 consecutive time steps? What is the dimension 195 of X, I do not see it enter the code anywhere? You do not define any Y, what is it? What are all the SequenceLastLayer, LinearLayer and SoftmaxLayer for, why do they appear in this order in the syntax? A yes or no output is not really what I'd like to learn how to do, rather I'd like to predict the next value in a sequence of values of an observable, that also enters as one of the features. Could you use the data from my question? $\endgroup$ – Kagaratsch Aug 30 '17 at 11:41
  • $\begingroup$ Basically, I still do not see any consecutive time steps in the answer and how to predict the next one. What I am looking for is a worked example like the following, except for Mathematica: machinelearningmastery.com/… In the end, if no humanly comprehensible examples exist for how to use this functionality in Mathematica, I'll just end up using tensorflow, which would be sad... $\endgroup$ – Kagaratsch Aug 30 '17 at 11:45
  • 1
    $\begingroup$ In my problem, I have to predict device failure. I have a device is monitored by 8 variables. All these variables are connected to time. To use RNN, I have to decide my sequence that will predict failure (as I have more than one variables so instead of having xt-1, xt-2, xt-3 where x is just one data point, I have each xt with 8 data points, but I still want to predict failure using all these variables. Therefore, I have to provide RNN with my sequence that has all (xt-1, xt-2,xt-3). By the way, this gave me great results. $\endgroup$ – user34018 Aug 31 '17 at 10:37
  • 1
    $\begingroup$ 195 is my sample size $\endgroup$ – user34018 Aug 31 '17 at 10:47
2
$\begingroup$

Key applied RNN examples from developers are located in documentation at:

http://reference.wolfram.com/language/tutorial/NeuralNetworksOverview.html

See "Sequence Learning and NLP" section.

$\endgroup$
  • $\begingroup$ I have seen these examples before. integer addition, sorting sequences, question answering, language modeling, sentiment analysis - none of these address my questions, sadly. $\endgroup$ – Kagaratsch Aug 30 '17 at 21:09
  • $\begingroup$ As a note, this page does not exist anymore. $\endgroup$ – dearN May 17 at 14:30
  • $\begingroup$ @dearN URL corrected, thanks. $\endgroup$ – Vitaliy Kaurov May 17 at 16:24
-1
$\begingroup$

Here RNN in Mathematica? they are talking about RNN

In help section, if you for LongShortTermMemoryLayer you will also find other RNN models that have been implemented.

$\endgroup$
  • $\begingroup$ As far as I understand, a LongShortTermMemoryLayer is just a basic building block that can be part of a model. However, there does not seem to be a comprehensive example on how to use it in a concrete model e.g. to generate a forecast of several time steps of an observable based on several input parameters. $\endgroup$ – Kagaratsch Aug 30 '17 at 2:05
  • $\begingroup$ The example they present using LongShortTermMemoryLayer[20], SequenceLastLayer[], LinearLayer[1] $\endgroup$ – user34018 Aug 30 '17 at 2:31
  • $\begingroup$ There they are doing addition with strings, which has no time series properties. It is still unclear how to take a time flow into account and generate a forecast. $\endgroup$ – Kagaratsch Aug 30 '17 at 4:16
  • $\begingroup$ You are supposed to have a sequence of vectors of size k, where each vector contains your several inputs (predictors, also called independent variables or features). So your input will be defined as "input"-> {k,length_of_vector} $\endgroup$ – user34018 Aug 30 '17 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.