10
$\begingroup$

I am attempting to use the new recurrent network functionality in Mathematica 11.1 Home Edition. I created a network that will attempt to predict future bytes in a raw audio file (using very small sequence length here for initial testing).

net = NetChain[{LongShortTermMemoryLayer[128], 
   LongShortTermMemoryLayer[128], BasicRecurrentLayer[1]}, 
  "Input" -> NetEncoder["Scalar"], "Output" -> NetDecoder["Scalar"]];

initialized = NetInitialize[net];

This net works with a sequence as expected:

initialized[{1, 2, 3, 4, 5}]
{-0.0164315, -0.0766496, -0.212674, -0.439926, -0.697567}

I read the bytes from the file, and create input values:

bytes = BinaryReadList["C:\\Temp\\ParkerAudio.raw"];
values = N[bytes / 256];
RandomSample[values, 10]
{0.550781, 0.488281, 0.417969, 0.605469, 0.691406, 0.46875, 0.464844, \
0.445313, 0.542969, 0.640625}

I define generator function:

GetOneTrainRecord[unrolllen0_] := Module[   
  {unrolllen = unrolllen0, startpos, leftside, rightside, rule},   
  startpos = RandomInteger[{1, Length[values] - unrolllen - 1}];   
  leftside = Take[values, {startpos, startpos + unrolllen - 1}];
  rightside = Take[values, {startpos + 1, startpos + unrolllen}];   
  rule = leftside -> rightside;   
  rule   
]

generator[assoc0_] := Module[
  {assoc = assoc0, batchsize},
  batchsize = assoc["BatchSize"];
  Table[GetOneTrainRecord[10], batchsize]
  ]

Generator works as expected:

generator[<|"BatchSize" -> 5|>]
{{0.523438, 0.523438, 0.527344, 0.527344, 0.519531, 0.527344, 0.53125,
0.523438, 0.523438, 0.535156} -> {0.523438, 0.527344, 0.527344, 
0.519531, 0.527344, 0.53125, 0.523438, 0.523438, 0.535156, 
0.542969}, {0.539063, 0.542969, 0.542969, 0.546875, 0.546875, 
0.546875, 0.550781, 0.550781, 0.546875, 0.546875} -> {0.542969, 
0.542969, 0.546875, 0.546875, 0.546875, 0.550781, 0.550781, 
0.546875, 0.546875, 0.542969}, {0.503906, 0.460938, 0.480469, 
0.433594, 0.527344, 0.484375, 0.433594, 0.523438, 0.507813, 
0.46875} -> {0.460938, 0.480469, 0.433594, 0.527344, 0.484375, 
0.433594, 0.523438, 0.507813, 0.46875, 0.46875}, {0.734375, 
0.667969, 0.417969, 0.386719, 0.519531, 0.613281, 0.5, 0.335938, 
0.390625, 0.53125} -> {0.667969, 0.417969, 0.386719, 0.519531, 
0.613281, 0.5, 0.335938, 0.390625, 0.53125, 0.570313}, {0.523438, 
0.550781, 0.554688, 0.527344, 0.523438, 0.554688, 0.542969, 
0.511719, 0.523438, 0.546875} -> {0.550781, 0.554688, 0.527344, 
0.523438, 0.554688, 0.542969, 0.511719, 0.523438, 0.546875, 
0.546875}}

I generate some data with the generator, and successfully train with it:

trained = NetTrain[net, generator[<|"BatchSize" -> 100|>]]

However, when I attempt to train using the generator function itself, I get an error:

trained = NetTrain[net, generator]
NetTrain::interr: An internal error occurred. Please contact Wolfram Research.

Am I doing something stupid here, or is this a bug?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

The generator NetTrain syntax doesn't currently support training nets with variable-length inputs. This is not well documented, so its a documentation bug.

One workaround for now: explicitly specify the input size and thus use sequences of fixed length.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ It's also an implementation bug (it should give an explicit complaint and fail with a helpful message, which it wasn't doing). It's fixed for 11.1. $\endgroup$
    – Taliesin Beynon
    Apr 10, 2017 at 15:40
  • $\begingroup$ Sebastian says that NetTrain syntax does not support training nets with variable-length inputs, but what is above looks like a fixed sequence of 10 items. Taliesin: Says the bug will be fixed in 11.1, but the OP says this message occurs in 11.1. I'm seeing a similar internal error trying to use NetTrain with an LSTM, and this question seems like the most similar, but the responses here are confusing and inconsistent. Where is the variation in input-size and what version fixes what "bug"? $\endgroup$
    – Todd Johnson
    May 6, 2017 at 0:34
5
$\begingroup$

I think that this is a bug: generator function doesn't work with recurrent layers.

net1 = NetChain[
  {
   LinearLayer[100, "Input" -> {2, 3}],
   2,
   SoftmaxLayer[]
   },
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

net2 = NetChain[
  {
   LongShortTermMemoryLayer[512, "Input" -> {2, 3}],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

generator = Function[
   Thread[RandomReal[{-1, 1}, {#BatchSize, 2, 3}] -> RandomInteger[1, #BatchSize]]
   ];

With net1 generator works without problems.

trained = NetTrain[net1, generator, BatchSize -> 4]

But for net2 I see an error.

enter image description here

Possible workaround - insert LinearLayer before LongShortTermMemoryLayer:

net3 = NetChain[
  {
   LinearLayer[{2, 3}, "Input" -> {2, 3}],
   LongShortTermMemoryLayer[512],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

trained = NetTrain[net3, generator, BatchSize -> 4]

UPDATE

As Sebastian write: NetTrain syntax doesn't currently support training nets with variable-length inputs. net2 has fixed-length input. But "Input" should not be in the body of LongShortTermMemoryLayer function. This works with generator:

net2new = NetChain[
  {
   LongShortTermMemoryLayer[512],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Input" -> {2, 3},
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.