8
$\begingroup$

According Keras blog,I find the Seq2Seq auto-encoder. But it didn't give any example only code.

enter image description here

To build a LSTM-based autoencoder, first use a LSTM encoder to turn your input sequences into a single vector that contains information about the entire sequence, then repeat this vector n times (where n is the number of timesteps in the output sequence), and run a LSTM decoder to turn this constant sequence into the target sequence.

So I try to make similar thing in Mathematica using MNIST database.

resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];
trainingSubset = Select[trainingData, Last[#] <= 4 &];
testSubset = Select[testData, Last[#] <= 4 &];
trainingImages = Keys[trainingSubset];
meanImage = Image[Mean@Map[ImageData, trainingImages]];

This is normal DNN auto-encoder of MNIST.

encoder = NetChain[{FlattenLayer[], 100, Ramp, 50, Ramp, 16}];
decoder = NetChain[{32, 50, Ramp, 100, Ramp, 784, ReshapeLayer[{1, 28, 28}]}];
net = NetGraph[{encoder, decoder, MeanSquaredLossLayer[]}, {1 -> 2 -> NetPort["Output"], 2 -> NetPort[3, "Input"], NetPort["Input"] -> NetPort[3, "Target"]}, 
               "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale", "MeanImage" -> meanImage}], 
               "Output" -> NetDecoder[{"Image", "Grayscale"}]]
{lossplot1, trained} = NetTrain[net, <|"Input" -> trainingImages|>, "Loss", 
                                {"LossEvolutionPlot", "TrainedNet"}, 
                                BatchSize -> 256, MaxTrainingRounds -> 10];

enter image description here

We could explore the visulization of it.

reconstructor = Take[trained, {NetPort["Input"], NetPort["Output"]}];
BlockRandom[
 Grid[{#, ImageAdd[reconstructor[#], meanImage] & /@ #} &@
   RandomSample[trainingImages, 10]], RandomSeeding -> 1234]

enter image description here

encoder = Take[trained, {NetPort["Input"], 1}];
testImages = Keys[testSubset];
coords = DimensionReduce[encoder[testImages], 2, Method -> "TSNE"];
labels = Values[testSubset];
ListPlot[Table[Extract[coords, Position[labels, i]], {i, 0, 4}], 
 PlotLegends -> PointLegend[96, Range[0, 4]], Axes -> None, 
 PlotStyle -> Map[ColorData[96], Range[1, 5]], AspectRatio -> 1, 
 ImageSize -> Small]

enter image description here

But using autoencoder based LSTM,the result isn't good.

encoder = NetChain[{ReshapeLayer[{28, 28}], LongShortTermMemoryLayer[16], 
                    SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], LongShortTermMemoryLayer[28], 
                    ReshapeLayer[{1, 28, 28}]}];
net = NetGraph[{encoder, decoder, MeanSquaredLossLayer[]}, {1 -> 2 -> NetPort["Output"], 2 -> NetPort[3, "Input"], NetPort["Input"] -> NetPort[3, "Target"]}, 
                "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale", "MeanImage" -> meanImage}], 
                "Output" -> NetDecoder[{"Image", "Grayscale"}]]
{lossplot2, trained} = NetTrain[net, <|"Input" -> trainingImages|>, "Loss", 
                                {"LossEvolutionPlot", "TrainedNet"}, 
                                BatchSize -> 256, MaxTrainingRounds -> 10];

enter image description here

reconstructor = Take[trained, {NetPort["Input"], NetPort["Output"]}];
BlockRandom[Grid[{#, ImageAdd[reconstructor[#], meanImage] & /@ #} &@
   RandomSample[trainingImages, 10]], RandomSeeding -> 1234]

enter image description here

encoder = Take[trained, {NetPort["Input"], 1}];
testImages = Keys[testSubset];
coords = DimensionReduce[encoder[testImages], 2, Method -> "TSNE"];
labels = Values[testSubset];
ListPlot[Table[Extract[coords, Position[labels, i]], {i, 0, 4}], 
 PlotLegends -> PointLegend[96, Range[0, 4]], Axes -> None, 
 PlotStyle -> Map[ColorData[96], Range[1, 5]], AspectRatio -> 1, 
 ImageSize -> Small]

enter image description here

The loss of LSTM(purple color) fall down slowly

Show[{lossplot1, lossplot2 /. Hue[__] -> Hue[0.75]}]

enter image description here

How to improve the result?

PS:Using LSTM to predict digits number in MNIST got a good result even data is image instead of temporal time-series in this case

Using LSTM,the dimensions of embedding vector after LSTM is also 16,it can be think a presentation or encoder of the image.

why in the above case,same embedding vector size can't get a pretty good result of auto-encoder?

lstmnet = NetChain[{ReshapeLayer[{28, 28}], LongShortTermMemoryLayer[16], 
                    SequenceLastLayer[], LinearLayer[5], SoftmaxLayer[]}, 
                    "Output" -> NetDecoder[{"Class", Range[0, 4]}], 
                    "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]];
trained = NetTrain[lstmnet, trainingSubset, ValidationSet -> testSubset, 
                   BatchSize -> 256, MaxTrainingRounds -> 5]
ClassifierMeasurements[trained, testSubset]["Accuracy"]
(*0.965947*)

enter image description here

Using CNN(lenet version)

lenet = NetChain[{ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2], 
                  ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2], FlattenLayer[],
                  500, Ramp, 5, SoftmaxLayer[]}, 
                  "Output" -> NetDecoder[{"Class", Range[0, 4]}], 
                  "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]];
trained = NetTrain[lenet, trainingSubset, ValidationSet -> testSubset,
                   BatchSize -> 256,MaxTrainingRounds -> 5]
ClassifierMeasurements[trained, testSubset]["Accuracy"]
(*0.997081*)
Improve this question
$\endgroup$
2
  • $\begingroup$ I think that there is no way to improve the result. Because LSTM is not suitable for images. See your screenshot from Keras blog: "If you inputs are sequences, rather than vectors or 2D images, then you may want to use as encoder and decoder a type of model that can capture temporal structure, such as a LSTM". Answer on the first revision of your question is the same: no way. Because your data are random. It's very hard to overfit random data. Have you tried LSTM autoencoder on your real data? $\endgroup$
    – Alexey Golyshev
    Commented Nov 29, 2017 at 10:00
  • $\begingroup$ @Alexey Golyshev I tried to use LSTM to classify digits in MNIST,and the result is good, I will update the question.And I try LSTM auto-encoder in my project.The loss can't be declined too much $\endgroup$
    – partida
    Commented Nov 29, 2017 at 10:38

1 Answer 1

Reset to default
6
$\begingroup$

I compare many net structure using LSTM.

Then finding multi-stacks LSTM will get a more accurate result.But it's not as effective as enlarge the embedding size.

Note:All the networks use this code as basic model,batch size 256,epochs 10

net = NetGraph[{encoder, decoder, MeanSquaredLossLayer[]}, 
               {1 -> 2 -> NetPort["Output"], 2 -> NetPort[3, "Input"], NetPort["Input"] -> NetPort[3, "Target"]}, 
               "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale", "MeanImage" -> meanImage}], 
               "Output" -> NetDecoder[{"Image", "Grayscale"}]]

Baseline network:

encoder = NetChain[{FlattenLayer[], 100, Ramp, 50, Ramp, 16}];
decoder = NetChain[{50, Ramp, 100, Ramp, 784, ReshapeLayer[{1, 28, 28}]}];

Using single LSTM(state size 16):

encoder = NetChain[{ReshapeLayer[{28, 28}], LongShortTermMemoryLayer[16], 
                    SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], LongShortTermMemoryLayer[28], 
                    ReshapeLayer[{1, 28, 28}]}];

Using single LSTM(state size 64):

encoder = NetChain[{ReshapeLayer[{28, 28}], LongShortTermMemoryLayer[64], 
                    SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], LongShortTermMemoryLayer[28], 
                    ReshapeLayer[{1, 28, 28}]}];

Using double LSTM(state size 16):

encoder = NetChain[{ReshapeLayer[{28, 28}], LongShortTermMemoryLayer[16], 
                    LongShortTermMemoryLayer[16], SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], LongShortTermMemoryLayer[28], 
                    LongShortTermMemoryLayer[28], ReshapeLayer[{1, 28, 28}]}];

Using double LSTM(state size 64):

encoder = NetChain[{ReshapeLayer[{28, 28}], LongShortTermMemoryLayer[64], 
    LongShortTermMemoryLayer[64], SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], LongShortTermMemoryLayer[28], 
    LongShortTermMemoryLayer[28], ReshapeLayer[{1, 28, 28}]}];

Using triple LSTM(state size 64):

encoder = NetChain[{ReshapeLayer[{28, 28}], 
                    LongShortTermMemoryLayer[64], 
                    LongShortTermMemoryLayer[64], 
                    LongShortTermMemoryLayer[64], SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], 
                    LongShortTermMemoryLayer[28], 
                    LongShortTermMemoryLayer[28], 
                    LongShortTermMemoryLayer[28], ReshapeLayer[{1, 28, 28}]}];

Using triple LSTM(state size 64) and DNN:

Contrary to my expectations, it didn't get a better result compare to the above net :)

encoder = NetChain[{ReshapeLayer[{28, 28}], 
                    LongShortTermMemoryLayer[64], 
                    LongShortTermMemoryLayer[64], 
                    LongShortTermMemoryLayer[64], 
                    SequenceLastLayer[], 32, ElementwiseLayer["SELU"], 16}];
decoder = NetChain[{32, ElementwiseLayer["SELU"], 64, ReplicateLayer[28], 
                    LongShortTermMemoryLayer[28], 
                    LongShortTermMemoryLayer[28], 
                    LongShortTermMemoryLayer[28], ReshapeLayer[{1, 28, 28}]}];

Using quadruple LSTM(state size 64) - Best result of LSTM family.

encoder = NetChain[{ReshapeLayer[{28, 28}], 
                    LongShortTermMemoryLayer[64], 
                    LongShortTermMemoryLayer[64], 
                    LongShortTermMemoryLayer[64], 
                    LongShortTermMemoryLayer[64], SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], 
                    LongShortTermMemoryLayer[28], 
                    LongShortTermMemoryLayer[28], 
                    LongShortTermMemoryLayer[28], 
                    LongShortTermMemoryLayer[28], ReshapeLayer[{1, 28, 28}]}];

enter image description here

Visualize the effect of auto-encoder of Bast result(using quadruple LSTM(state size 64)),the effect of clustering is pretty good.

enter image description here

enter image description here

Extreme Test:

Enlarge the number of LSTM-staks is not an efficency way, it will make the net hard to train. In other words, Increasing the number of LSTM-stacks may not be effective.

Using quadruple LSTM(state size 16):

encoder = NetChain[{ReshapeLayer[{28, 28}], LongShortTermMemoryLayer[16], 
    LongShortTermMemoryLayer[16], LongShortTermMemoryLayer[16], 
    LongShortTermMemoryLayer[16], LongShortTermMemoryLayer[16], 
    SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], LongShortTermMemoryLayer[28], 
    LongShortTermMemoryLayer[28], LongShortTermMemoryLayer[28], 
    LongShortTermMemoryLayer[28], LongShortTermMemoryLayer[28], 
    ReshapeLayer[{1, 28, 28}]}];

Using octuple LSTM(state size 16),it is very hard to train because too many parameters.

encoder = NetChain[{ReshapeLayer[{28, 28}], LongShortTermMemoryLayer[16], 
    LongShortTermMemoryLayer[16], LongShortTermMemoryLayer[16], 
    LongShortTermMemoryLayer[16], LongShortTermMemoryLayer[16], 
    LongShortTermMemoryLayer[16], LongShortTermMemoryLayer[16], 
    LongShortTermMemoryLayer[16], SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], LongShortTermMemoryLayer[28], 
    LongShortTermMemoryLayer[28], LongShortTermMemoryLayer[28], 
    LongShortTermMemoryLayer[28], LongShortTermMemoryLayer[28], 
    LongShortTermMemoryLayer[28], LongShortTermMemoryLayer[28], 
    LongShortTermMemoryLayer[28], ReshapeLayer[{1, 28, 28}]}];

enter image description here

Finally,Come back to this question.

We can simulate the process the auto-encoder of DNN, not only the result is not bad but also it can learn the embedding size(16) if using the structure of Best result of quadruple LSTM(state size 64)

encoder = NetChain[{ReshapeLayer[{28, 28}], LongShortTermMemoryLayer[64], 
    LongShortTermMemoryLayer[64], LongShortTermMemoryLayer[32], 
    LongShortTermMemoryLayer[16], SequenceLastLayer[]}];
decoder = NetChain[{ReplicateLayer[28], LongShortTermMemoryLayer[16], 
    LongShortTermMemoryLayer[32], LongShortTermMemoryLayer[28], 
    LongShortTermMemoryLayer[28], ReshapeLayer[{1, 28, 28}]}];

We can see the result is not bad.

enter image description here

The loss plot refer the answer from Adding legends when using Show

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.