RNN in Mathematica?

Question

Recurrent neural network is an important part of machine learning and its improved version: LSTM network is widely applied, but Mathematica seemingly is in lack of these two features and in my opinion, banned it totally as Mathematica's neural networks created by NetGraph will report bugs whenever I try to create a network with self loops or loops.

Are there anyway to solve this problem? Or will this feature be add in further versions of Mathematica?

Answer 1

This is actually a useful question, because it gives an opportunity for us to communicate with the community about what is coming, so we can get feedback. So here is some preliminary information about our plans (subject to change, of course).

First, we'll have the concept of a sequence of things (e.g. sequence of vectors, sequence of matrices, sequence of bounded integers).

We'll have NetSequenceEncoder and NetSequenceDecoder to manage creating and interpreting such sequences. Obviously, for strings, you want to encode the individual characters or words according to some codebook, that will be straightforward. Audio will also have such sequence encoders and decoders, that automatically do spectrograms etc.

In addition, we'll have the following 'ready-made' recurrent layers, that on GPU will go straight to CUDnn's super-optimized implementations:

• BasicRecurrentLayer, which takes a sequence and produces a sequence, via Subscript[s, t]=Tanh[Dot[W1,Subscript[x, t]]+Dot[W2,Subscript[s, t-1]]+B]

• GatedRecurrentLayer, which again takes a sequence and produces a sequence. This is a GRU, and I won't reproduce the formula here.

• LongShortTermMemoryLayer, which takes a sequence and produces TWO sequences.

All of these will take an optional initial state, if you don't specify it a zero vector will be used.

To be able to do things like regression on the sequence output of a recurrent layer, you'll typically only look at the last state (imagine sentiment prediction). So we'll have a SequenceLastLayer that extracts the last element from a sequence.

Crucially, if you have an operation you want to do on every element of a sequence, you'll be able to use a NetMapOperator to 'lift' a subnetwork so that it operates on all the elements individually, using shared weights. This actually unlocks some non-recurrent use-cases, like siamese networks, where you have replicas of a network whose outputs get compared, etc.

Now, when you actually apply a net that has these layers, we'll unroll the net to match the provided sequence length. It complicates training quite a bit, however, because to get good efficiency we have to sort training data into sequence length buckets (e.g. powers of 2), and then randomize within each bucket. All of this will be done automatically, which will be a really nice improvement on a lot of the other frameworks out there.

Already you can do a lot with this. But wait, there's more!

To help you play with recurrent networks that don't match the standard ones, we'll provide more 'net operators'. A partial list is:

• NetMapOperator (already mentioned)

• NetFoldOperator, which 'folds' the sequence into a subnetwork, passing the recurrent state. The subnetwork must take two inputs, one for the xi and one for the si-1. The output is the final state.

• NetFoldListOperator, which is like NetFoldOperator but emits a sequence of states and not just the final state.

• NetNest(List)Operator, which just repeats the subnetwork a fixed number of times, feeding its output back to its input.

• NetBidirectionalOperator, which runs two copies of the subnetwork, one forward on the sequence, and one in reverse, and concatenates their outputs. With this, it's easy to express e.g. a bidirectional LSTM, without duplicating the number of layer types.

Lastly, and this is most speculative, we plan on implementing another operator-like layer called SequenceTransformerLayer, which takes an encoder network and decoder network, and allows you to build a sequence-to-sequence type of network. It's important to have this be a special type of network, because for evaluation you have to feed its own output back into its input in a loop until it decides it is finished, and that would be cumbersome and slow to do by hand. So we'll do that for you.