# Use autoencoder for dimensionality reduction

I have been trying to follow this tutorial (https://www.wolfram.com/language/11/neural-networks/unsupervised-learning-with-autoencoders.html?product=mathematica) to train an autoencoder in Mathematica, to do dimensionality reduction. However, I must admit -- because I am quite unfamiliar with neural networks in Mathematica -- I am having trouble following it.

What I want to do is quite simple. I have a list of vectors of length 100, for example,

lData = Table[RandomVariate[
MultinormalDistribution[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10},
RandomVariate[WishartMatrixDistribution[10, IdentityMatrix[10]],
1][[1]]], 1][[1]], {i, 1, 100, 1}];


and I want to use an autoencoder with, say a network of layers: input -> 10 -> 3 -> 10 -> output=input, in order to reduce the dimensionality of the data. I would like the individual neurons to have logistic sigmoid activations.

This is much much simpler than the example that is in the tutorial, and I was hoping that someone might be able to explain to me how to do this?

At the moment I have tried constructing a NetGraph, although am getting errors about "the Input Target of vertex 4 does not exist". I fear this is because I don't really know what each bit of the following does,

NetGraph[{LogisticSigmoid, 10, LogisticSigmoid, 3, LogisticSigmoid, ReshapeLayer[{100, 10}],
MeanSquaredLossLayer[]}, {1 -> 2 -> 3 -> NetPort["Output"],
3 -> NetPort[4, "Input"], NetPort["Input"] -> NetPort[4, "Target"]},
"Input" -> NetEncoder[{"Scalar", {10, 1}}],
"Output" -> NetDecoder[{"Image", "Grayscale"}]]


Does anyone know how I can use an autoencoder as desired here?

Best,

Ben

The message from NetGraph message is saying: you're feeding the input to your entire network to the "Target" input of the 4th vertex. Read off the 4th vertex from the list you are giving to NetGraph: it's a DotPlusLayer[3]. That doesn't have a "Target" input. Only MeanSquaredLossLayer[], the 7th vertex, has a "Target" input.

Also, I'm not sure why you're using an "Image" output encoder and a "Scalar" input encoder, or what the ReshapeLayer is for. An autoencoder produces the same shape output as its input, because its job is to reproduce the actual values of the input as accurately as possible.

A "Scalar" NetEncoder is for encoding a single number as a 1-vector so that you can operate on it using DotPlusLayer, for example. You're giving a parameter to the "Scalar" input encoder, but it doesn't take any parameters.

Anyway, here's what you probably meant:

coder = NetChain[{LogisticSigmoid, 10, LogisticSigmoid, 3, LogisticSigmoid, 10}]

trainer = NetGraph[{coder, MeanSquaredLossLayer[]}, {1 -> NetPort[2, "Input"], NetPort["Input"] -> NetPort[2, "Target"]}]

lData = Table[
RandomVariate[
MultinormalDistribution[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10},
RandomVariate[WishartMatrixDistribution[10, IdentityMatrix[10]],
1][[1]]], 1][[1]], {i, 1, 10000, 1}];

trained = NetTrain[trainer, <|"Input" -> lData|>]

trainedCoder = NetExtract[trained, 1]


Notice that just having 100 examples from your manifold is very little data to train on, so I've upped that to 10000. We feed each vector in -- because it's already a tensor of numeric values, it doesn't need an NetEncoder. The trainer network gives it to the coder network, and then compares the output to the original input to come up with a loss.

Once we've finished training, we extract the coder network from inside the training network, because we don't want to calculate losses anymore, of course.

Then you can split the trained coder network into the actual encoder and decoder parts using Take:

encoder = Take[trainedCoder, 4]
decoder = Drop[trainedCoder, 4]


You would use these encoder and decoder networks as in the examples.

Hope this helps! And until there's a nice tutorial available inside Mathematica, I recommend reading http://www.deeplearningbook.org/

• Thank you very much! And also for the book reference. Shall read it. Best, Ben – ben18785 Oct 9 '16 at 10:52
• Sorry -- a quick question. How do I obtain the loss from my 'trained' object? – ben18785 Oct 9 '16 at 10:59
• @ben18785 I'm not sure what you mean by 'the loss'. A network has a loss on specific input data -- the job of training is to minimize the expected value of the loss. The trained network above (which is the trained version of the trainer we started with) calculates the loss of the autoencoder on a specific input. The NetExtract gets out the piece we care about, which is the autoencoder. The trained/trainer networks you see above are the "scaffolding" on which we trained the autoencoder. You throw them away afterwards. – Taliesin Beynon Oct 9 '16 at 19:04
• Take a look at PrintDefinitions[N]. What is that weird second definition with _[___, N, ___]? Is that a real thing or does PrintDefinitions just misformat Default[N, 2]? – Szabolcs Oct 11 '16 at 14:50
• @Szabolcs this is completely unrelated to this question, but that is because SystemPrivateHasUpCodeQ[N] is True, in other words there is an unknowable kernel upvalue definied against N. For better or worse that's how PrintDefinitions represents such a thing -- it has no actual LHS pattern or RHS, the 'upcode' as its called can do anything it wants via a C function. In this case the 'LHS' would make more sense if represented as _[___, _N, ___]. – Taliesin Beynon Oct 16 '16 at 16:37