2
$\begingroup$

I would like to train a time-series data using GatedRecurrentLayer. The data consists of k scalar features mapping to a scalar number at every time step: i.e. {#1, #2, ..., #k}[[t]]->target[[t]], where t designates the time-step. I would like to create n recurrent neuron layer, such that an input sequence from time (t-n) to (t) is fed into the network to produce a predicted value for time t using both the current values of the features, as well as past history.

I created a [m,k] array (named featureset below) consisting of k columns corresponding to each feature and m rows (length of the total training series), as well as a [m, 1] target vector. I defined the training set and the netchain as:

trainset=Table[featureset[[i, All]]->target[[i]], {i, 1, Length[target]}];
net=NetChain[{GatedRecurrentLayer[n], SequenceLastLayer[], 1}];

When I try training this using NetTrain, I get the following error message:

NetTrain::nettinf2: Could not automatically find way to encode training data, which consists of length-16 vectors, to be compatible with port "Input", which expects n*[DottedSquare] matrices. Please specify shapes for the input and/or output ports of the net before calling NetTrain.

I think GatedRecurrentLayer[n] does not do what I want since n refers to the output vector size, but not the sequence length. And also, seems like I don't really understand sequences.

I have a similar problem with one of the examples in the NetTrain help document. The following code works:

seqs = Table[RandomReal[1, RandomInteger[{2, 5}]], 10000];
net = NetChain[{GatedRecurrentLayer[10], SequenceLastLayer[], 10, 
    Ramp, 1}];
trained = NetTrain[net, Map[# -> Max[#] &, seqs]]

but if you replace the first line with:

seqs = Table[RandomReal[1, 5], 10000];

it gives the same error message. As far as I can tell, all I have done is make the sequence length fixed as opposed to variable.

In any case, I am pretty new to all of this, so any help you can provide would be appreciated.

$\endgroup$

1 Answer 1

2
$\begingroup$

I think the confusion may come from the automatic inference of the input and output size of the neural network.

A recurrent layer always takes a sequence of vectors and turns it into a sequence of vectors with the same length. The input has form {x1,x2,x3,...,xn} and the output has the form {s1,s2,s3,...,sn} where both $x_i$ and $s_i$ are vectors. The size of the output vector $s_i$ is set by the parameter the to the recurrent layer, and the length of the output sequence n is set by the length of the input sequence. For example,

rnn = NetInitialize@BasicRecurrentLayer[3, "Input" -> {5, 2}]

takes a sequence of five vectors, each vector has 2 elements. An example input is

input = {{1, 2}, {3, 4}, {5, 6}, {7, 8}, {9, 10}}

The recurrent layer turns it into a sequence of five vectors, each has 3 elements:

rnn[input]
(* {{0.846091, 0.968395, -0.465709}, {0.945529, 
  0.998056, -0.910047}, {0.977855, 0.999987, -0.991245}, {0.9918, 
  1., -0.999365}, {0.99706, 1., -0.999957}} *)

A typical training data then may look like this:

training = Table[RandomReal[1, {5, 2}] -> target, 1000]

The recurrent layer also supports varying length sequence and the above example can be written as

rnn = NetInitialize@BasicRecurrentLayer[3, "Input" -> {"Varying", 2}]
training = Table[RandomReal[1, {RandomInteger[{1, 5}], 2}] -> target, 10]

Now the example in the documentation:

seqs = Table[RandomReal[1, RandomInteger[{2, 5}]], 10000];
net = NetChain[{GatedRecurrentLayer[10], SequenceLastLayer[], 10, 
   Ramp, 1}]
trained = NetTrain[net, Map[# -> Max[#] &, seqs]];

The input and output size is inferred from the neural network and the training data. If we fully specify the input and output, the example should be written like:

seqs = Table[RandomReal[1, {RandomInteger[{2, 5}], 1}], 10000];
net = NetChain[{GatedRecurrentLayer[10], SequenceLastLayer[], 10, 
    Ramp, 1}, "Input" -> {"Varying", 1}, "Output" -> {1}];
trained = NetTrain[net, Map[# -> {Max[#]} &, seqs]];

"Input" -> {"Varying", 1} specifies that the input should be a sequence of size 1 vectors, and "Output" -> {1} specifies that the ouput should be a size 1 vector:

Map[# -> {Max[#]} &, seqs][[1 ;; 2]]
(* {{{0.363334}, {0.998158}, {0.414586}} -> {0.998158},
   {{0.251652}, {0.518092}} -> {0.518092}} *)

In the document example, the training data has the form:

{{0.293984,0.748285,0.832492}->0.832492,{0.250352,0.280401}->0.280401}

In this form, NetTrain sees that the input data has a variable length, it then inferred that the numbers in the list should treated as size 1 vector. With the seqs = Table[RandomReal[1, 5], 10000];, NetTrain doesn't see variable length, thus unable to get a unique inferrence for the input dimension.

So in order for the fixed size training data to work, we need to specify the full dimensions of the input and output

seqs = Table[RandomReal[1, {5, 1}], 10000];
net = NetChain[{GatedRecurrentLayer[10], SequenceLastLayer[], 10, 
    Ramp, 1}, "Input" -> {5, 1}, "Output" -> {1}];
trained = NetTrain[net, Map[# -> {Max[#]} &, seqs]];
$\endgroup$
2
  • $\begingroup$ Thank you for answering my question. I did try to fully specify the input data as you suggested, but did not have much luck. For a simpler example, let's concentrate on the example from the documentation. I made the sequences exactly 5 long as before and added "Input"->{5,1} "Output"->{1} to the NetChain. The error I get is "Data provided to port "Input" should be a list of 5 x 1 matrices". The problem is that the list of sequences are in fact a list of 5 x 1 matrices, but when I associate a target using -> for the training set, they lose this dimensionality. $\endgroup$ Jul 24, 2017 at 15:48
  • $\begingroup$ @ArmanCingoz See the edits. The last block of code works for me. $\endgroup$ Jul 24, 2017 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.