4
$\begingroup$

I'm a little uncertain as how to use NetBidirectionalOperator. In other frameworks, switching an LSTM to be bidirectional is an option but you have to double the dimensions of the linear layer directly after it.

There are no "application" examples in the documentation center so I'm hoping someone might provide one and show how it works with labels and targets during training. Do I have to reverse and concatenate the inputs?

Here's a toy example of a net (that predicts the last character in a word) to get started:

net = NetInitialize @ NetChain[{UnitVectorLayer[], 
    NetBidirectionalOperator @ BasicRecurrentLayer[20], 
    SequenceLastLayer[], LinearLayer[97], SoftmaxLayer[]}, 
   "Output" -> NetDecoder["Characters"], 
   "Input" -> NetEncoder["Characters"]];
data = Table[With[{r = RandomWord[]}, 
    StringTake[r, ;; -2] -> StringTake[r, -1]], 10];
NetTrain[NetStateObject[net], data, All]

enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ Does this example help. $\endgroup$ Sep 18, 2018 at 15:36
  • 1
    $\begingroup$ Thanks. It says that it doubles the sequence length - but that I did know. It's a good example but I'm looking for how it works with generating text. $\endgroup$
    – user5601
    Sep 18, 2018 at 18:05

1 Answer 1

3
$\begingroup$
net = NetGraph[
  <|
   "net" -> {
     UnitVectorLayer[],
     NetBidirectionalOperator@BasicRecurrentLayer[20],
     SequenceLastLayer[],
     LinearLayer[97],
     SoftmaxLayer[]
     },
   "last" -> SequenceLastLayer[],
   "loss" -> CrossEntropyLossLayer["Index"]
   |>,
  {
   NetPort["Input"] -> "net" -> NetPort["loss", "Input"],
   NetPort["Target"] -> "last" -> NetPort["loss", "Target"]
   },
  "Input" -> NetEncoder["Characters"],
  "Target" -> NetEncoder["Characters"]
  ]

enter image description here

data = Table[With[{r = RandomWord[]}, StringTake[r, ;; -2] -> StringTake[r, -1]],10]

{"ima" -> "m", "stoppin" -> "g", "digestiv" -> "e", "conventionalit" -> "y", "interactio" -> "n", "hedge" -> "r", "stabl" -> "y", "bowspri" -> "t", "bur" -> "p", "ingrainin" -> "g"}

netT = NetTrain[net, <|"Input" -> data[[;; , 1]], "Target" -> data[[;; , 2]]|>, All]

netT2 = NetReplacePart[
  NetExtract[netT["TrainedNet"], "net"],
  {
   "Input" -> NetEncoder["Characters"],
   "Output" -> NetDecoder["Characters"]
   }
  ]

enter image description here

netT2@data[[2, 1]]

"g"

Generating text

gen[start_String, n_Integer] :=
 Module[
  {sobj, next},
  Reap[
     Sow[start];
     sobj = NetStateObject[
       netT2,
       <|
        {2, "ForwardNet", "State"} -> ConstantArray[0, 20],
        {2, "BackwardNet", "State"} -> ConstantArray[0, 20]
        |>
       ];
     Sow[next = sobj@start];
     Do[
      sobj = NetStateObject[
        netT2,
        <|
         {2, "ForwardNet", "State"} -> 
          NetExtract[sobj, "States"][{2, "ForwardNet", "State"}][[1]],
         {2, "BackwardNet", "State"} -> ConstantArray[0, 20]
         |>
        ];
      Sow[next = sobj@next],
      {n - 1}
      ]
     ][[2, 1]] // StringJoin
  ]
gen["abc", 15]

"abcsssssssyyyyyyyy"

$\endgroup$
4
  • $\begingroup$ Thanks, but why is there an extra SequenceLastLayer $\endgroup$
    – user5601
    Sep 19, 2018 at 18:12
  • $\begingroup$ Output of NetEncoder["Characters"] is {"Varying", Restricted["Integer", 97]} and CrossEntropyLossLayer expects {Restricted["Integer", 97]} as input. We need to remove "Varying" dimension. $\endgroup$ Sep 20, 2018 at 6:51
  • $\begingroup$ @AlexeyGolyshev Thanks! Would you mind adding how things would change if we wanted to switch from "word -> next character" to "word -> next word" type training data (with some vocabulary of words)? $\endgroup$
    – M.R.
    Sep 20, 2018 at 13:38
  • 1
    $\begingroup$ @M.R. It's very easy to do. In net replace: LinearLayer[97] with LinearLayer[40236], "Target" -> NetEncoder["Characters"] with "Target" -> NetEncoder["Tokens"]. data = Table[RandomWord[] -> RandomWord[], 10]. In netT2 replace: "Output" -> NetDecoder["Characters"] with "Output" -> NetDecoder["Tokens"]. That's all! $\endgroup$ Sep 21, 2018 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.