# Mathematica 11.1 - Issue with Training Function for Sequence Data

I am attempting to use the new recurrent network functionality in Mathematica 11.1 Home Edition. I created a network that will attempt to predict future bytes in a raw audio file (using very small sequence length here for initial testing).

net = NetChain[{LongShortTermMemoryLayer[128],
LongShortTermMemoryLayer[128], BasicRecurrentLayer[1]},
"Input" -> NetEncoder["Scalar"], "Output" -> NetDecoder["Scalar"]];

initialized = NetInitialize[net];


This net works with a sequence as expected:

initialized[{1, 2, 3, 4, 5}]
{-0.0164315, -0.0766496, -0.212674, -0.439926, -0.697567}


I read the bytes from the file, and create input values:

bytes = BinaryReadList["C:\\Temp\\ParkerAudio.raw"];
values = N[bytes / 256];
RandomSample[values, 10]
{0.550781, 0.488281, 0.417969, 0.605469, 0.691406, 0.46875, 0.464844, \
0.445313, 0.542969, 0.640625}


I define generator function:

GetOneTrainRecord[unrolllen0_] := Module[
{unrolllen = unrolllen0, startpos, leftside, rightside, rule},
startpos = RandomInteger[{1, Length[values] - unrolllen - 1}];
leftside = Take[values, {startpos, startpos + unrolllen - 1}];
rightside = Take[values, {startpos + 1, startpos + unrolllen}];
rule = leftside -> rightside;
rule
]

generator[assoc0_] := Module[
{assoc = assoc0, batchsize},
batchsize = assoc["BatchSize"];
Table[GetOneTrainRecord[10], batchsize]
]


Generator works as expected:

generator[<|"BatchSize" -> 5|>]
{{0.523438, 0.523438, 0.527344, 0.527344, 0.519531, 0.527344, 0.53125,
0.523438, 0.523438, 0.535156} -> {0.523438, 0.527344, 0.527344,
0.519531, 0.527344, 0.53125, 0.523438, 0.523438, 0.535156,
0.542969}, {0.539063, 0.542969, 0.542969, 0.546875, 0.546875,
0.546875, 0.550781, 0.550781, 0.546875, 0.546875} -> {0.542969,
0.542969, 0.546875, 0.546875, 0.546875, 0.550781, 0.550781,
0.546875, 0.546875, 0.542969}, {0.503906, 0.460938, 0.480469,
0.433594, 0.527344, 0.484375, 0.433594, 0.523438, 0.507813,
0.46875} -> {0.460938, 0.480469, 0.433594, 0.527344, 0.484375,
0.433594, 0.523438, 0.507813, 0.46875, 0.46875}, {0.734375,
0.667969, 0.417969, 0.386719, 0.519531, 0.613281, 0.5, 0.335938,
0.390625, 0.53125} -> {0.667969, 0.417969, 0.386719, 0.519531,
0.613281, 0.5, 0.335938, 0.390625, 0.53125, 0.570313}, {0.523438,
0.550781, 0.554688, 0.527344, 0.523438, 0.554688, 0.542969,
0.511719, 0.523438, 0.546875} -> {0.550781, 0.554688, 0.527344,
0.523438, 0.554688, 0.542969, 0.511719, 0.523438, 0.546875,
0.546875}}


I generate some data with the generator, and successfully train with it:

trained = NetTrain[net, generator[<|"BatchSize" -> 100|>]]


However, when I attempt to train using the generator function itself, I get an error:

trained = NetTrain[net, generator]


Am I doing something stupid here, or is this a bug?

The generator NetTrain syntax doesn't currently support training nets with variable-length inputs. This is not well documented, so its a documentation bug.

One workaround for now: explicitly specify the input size and thus use sequences of fixed length.

• It's also an implementation bug (it should give an explicit complaint and fail with a helpful message, which it wasn't doing). It's fixed for 11.1. Commented Apr 10, 2017 at 15:40
• Sebastian says that NetTrain syntax does not support training nets with variable-length inputs, but what is above looks like a fixed sequence of 10 items. Taliesin: Says the bug will be fixed in 11.1, but the OP says this message occurs in 11.1. I'm seeing a similar internal error trying to use NetTrain with an LSTM, and this question seems like the most similar, but the responses here are confusing and inconsistent. Where is the variation in input-size and what version fixes what "bug"? Commented May 6, 2017 at 0:34

I think that this is a bug: generator function doesn't work with recurrent layers.

net1 = NetChain[
{
LinearLayer[100, "Input" -> {2, 3}],
2,
SoftmaxLayer[]
},
"Output" -> NetDecoder[{"Class", {0, 1}}]
]

net2 = NetChain[
{
LongShortTermMemoryLayer[512, "Input" -> {2, 3}],
SequenceLastLayer[],
2,
SoftmaxLayer[]
},
"Output" -> NetDecoder[{"Class", {0, 1}}]
]

generator = Function[
Thread[RandomReal[{-1, 1}, {#BatchSize, 2, 3}] -> RandomInteger[1, #BatchSize]]
];


With net1 generator works without problems.

trained = NetTrain[net1, generator, BatchSize -> 4]


But for net2 I see an error.

Possible workaround - insert LinearLayer before LongShortTermMemoryLayer:

net3 = NetChain[
{
LinearLayer[{2, 3}, "Input" -> {2, 3}],
LongShortTermMemoryLayer[512],
SequenceLastLayer[],
2,
SoftmaxLayer[]
},
"Output" -> NetDecoder[{"Class", {0, 1}}]
]

trained = NetTrain[net3, generator, BatchSize -> 4]


UPDATE

As Sebastian write: NetTrain syntax doesn't currently support training nets with variable-length inputs. net2 has fixed-length input. But "Input" should not be in the body of LongShortTermMemoryLayer function. This works with generator:

net2new = NetChain[
{
LongShortTermMemoryLayer[512],
SequenceLastLayer[],
2,
SoftmaxLayer[]
},
"Input" -> {2, 3},
"Output" -> NetDecoder[{"Class", {0, 1}}]
]