Problem initializing training set on recurrent net

Question

I would like to train a time-series data using GatedRecurrentLayer. The data consists of k scalar features mapping to a scalar number at every time step: i.e. {#1, #2, ..., #k}[[t]]->target[[t]], where t designates the time-step. I would like to create n recurrent neuron layer, such that an input sequence from time (t-n) to (t) is fed into the network to produce a predicted value for time t using both the current values of the features, as well as past history.

I created a [m,k] array (named featureset below) consisting of k columns corresponding to each feature and m rows (length of the total training series), as well as a [m, 1] target vector. I defined the training set and the netchain as:

trainset=Table[featureset[[i, All]]->target[[i]], {i, 1, Length[target]}];
net=NetChain[{GatedRecurrentLayer[n], SequenceLastLayer[], 1}];

When I try training this using NetTrain, I get the following error message:

NetTrain::nettinf2: Could not automatically find way to encode training data, which consists of length-16 vectors, to be compatible with port "Input", which expects n*[DottedSquare] matrices. Please specify shapes for the input and/or output ports of the net before calling NetTrain.

I think GatedRecurrentLayer[n] does not do what I want since n refers to the output vector size, but not the sequence length. And also, seems like I don't really understand sequences.

I have a similar problem with one of the examples in the NetTrain help document. The following code works:

seqs = Table[RandomReal[1, RandomInteger[{2, 5}]], 10000];
net = NetChain[{GatedRecurrentLayer[10], SequenceLastLayer[], 10, 
    Ramp, 1}];
trained = NetTrain[net, Map[# -> Max[#] &, seqs]]

but if you replace the first line with:

seqs = Table[RandomReal[1, 5], 10000];

it gives the same error message. As far as I can tell, all I have done is make the sequence length fixed as opposed to variable.

In any case, I am pretty new to all of this, so any help you can provide would be appreciated.

Answer 1

I think the confusion may come from the automatic inference of the input and output size of the neural network.

A recurrent layer always takes a sequence of vectors and turns it into a sequence of vectors with the same length. The input has form {x1,x2,x3,...,xn} and the output has the form {s1,s2,s3,...,sn} where both $x_i$ and $s_i$ are vectors. The size of the output vector $s_i$ is set by the parameter the to the recurrent layer, and the length of the output sequence n is set by the length of the input sequence. For example,

rnn = NetInitialize@BasicRecurrentLayer[3, "Input" -> {5, 2}]

takes a sequence of five vectors, each vector has 2 elements. An example input is

input = {{1, 2}, {3, 4}, {5, 6}, {7, 8}, {9, 10}}

The recurrent layer turns it into a sequence of five vectors, each has 3 elements:

rnn[input]
(* {{0.846091, 0.968395, -0.465709}, {0.945529, 
  0.998056, -0.910047}, {0.977855, 0.999987, -0.991245}, {0.9918, 
  1., -0.999365}, {0.99706, 1., -0.999957}} *)

A typical training data then may look like this:

training = Table[RandomReal[1, {5, 2}] -> target, 1000]

The recurrent layer also supports varying length sequence and the above example can be written as

rnn = NetInitialize@BasicRecurrentLayer[3, "Input" -> {"Varying", 2}]
training = Table[RandomReal[1, {RandomInteger[{1, 5}], 2}] -> target, 10]

Now the example in the documentation:

seqs = Table[RandomReal[1, RandomInteger[{2, 5}]], 10000];
net = NetChain[{GatedRecurrentLayer[10], SequenceLastLayer[], 10, 
   Ramp, 1}]
trained = NetTrain[net, Map[# -> Max[#] &, seqs]];

The input and output size is inferred from the neural network and the training data. If we fully specify the input and output, the example should be written like:

seqs = Table[RandomReal[1, {RandomInteger[{2, 5}], 1}], 10000];
net = NetChain[{GatedRecurrentLayer[10], SequenceLastLayer[], 10, 
    Ramp, 1}, "Input" -> {"Varying", 1}, "Output" -> {1}];
trained = NetTrain[net, Map[# -> {Max[#]} &, seqs]];

"Input" -> {"Varying", 1} specifies that the input should be a sequence of size 1 vectors, and "Output" -> {1} specifies that the ouput should be a size 1 vector:

Map[# -> {Max[#]} &, seqs][[1 ;; 2]]
(* {{{0.363334}, {0.998158}, {0.414586}} -> {0.998158},
   {{0.251652}, {0.518092}} -> {0.518092}} *)

In the document example, the training data has the form:

{{0.293984,0.748285,0.832492}->0.832492,{0.250352,0.280401}->0.280401}

In this form, NetTrain sees that the input data has a variable length, it then inferred that the numbers in the list should treated as size 1 vector. With the seqs = Table[RandomReal[1, 5], 10000];, NetTrain doesn't see variable length, thus unable to get a unique inferrence for the input dimension.

So in order for the fixed size training data to work, we need to specify the full dimensions of the input and output

seqs = Table[RandomReal[1, {5, 1}], 10000];
net = NetChain[{GatedRecurrentLayer[10], SequenceLastLayer[], 10, 
    Ramp, 1}, "Input" -> {5, 1}, "Output" -> {1}];
trained = NetTrain[net, Map[# -> {Max[#]} &, seqs]];