Generative adversarial networks (GAN) is regarded as one of "the most interesting idea in the last ten years in machine learning" by Yann LeCun. It can be used to generate photo-realistic images that are almost indistinguishable from the real ones.

GAN trains two competing neural networks: a generator network which generates image, and a discriminator network that distinguishes the generated image and the real training image. For example, the images shown below are generated by the network from the texts above them (taken from Han Zhang, etc., StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks).

I'm wondering whether we can implement a simplified version of that in Mathematica, given that the neural network framework has be enhanced greatly in version 11.1.

I wrestled with this for a while and got some kind of results, but nowhere near the great performance for which GANs are famous. Ultimately, they're absurdly sensitive to hyperparameters and initialization, and if you don't exactly imitate the published settings, you are unlikely to get good results.

I figure I should post my attempt -- maybe the community can figure out a good set of parameters that works. Mine sort of trained, but suffered from mode collapse and often converged on a blob. However, for sets of all one digit, it did seem to work okay, although this is much easier and not really where GANs have any advantage.

I tried to implement the Wasserstein GAN (paper available here) to generate MNIST digits. The training procedure is to update the discriminator on a batch 5 times for every 1 time you show it to the generator. Because Mathematica doesn't yet allow preservation of optimizer parameters between calls to NetTrain, I couldn't get this to work. Instead, I trained the networks jointly as suggested by Taliesin Beynon, setting the learning rate on the generator to something like -1/5, because it seemed like a plausible approximation.

The paper also used RMSProp as an optimizer. Mathematica has an RMSProp option, but on the net I defined it immediately diverged no matter what learning rate I chose. I used ADAM instead.

To begin, let's get a big batch of MNIST digits.

mnist = ResourceData["MNIST"];
mnistDigits = First /@ mnist;


Let's give a 10-dimensional noise input to the generator, and define the generator and discriminator. Notice that the discriminator does not have an activation on its output -- this is specific to the WGAN, a normal GAN would have a LogisticSigmoid or something.

randomDim = 10;
generator =
NetChain[{128, Ramp, 128, Ramp, 28*28, LogisticSigmoid,
ReshapeLayer[{1, 28, 28}]}, "Input" -> randomDim]
discriminator =
NetChain[{128, Ramp, 128, Ramp, 128, Ramp, 1},
"Input" -> {1, 28, 28}]


Now the tricky part. We'll feed noise into the generator to produce a fake image, and also accept a real image as input. We want to apply the discriminator to both images, but with one set of weights, so we concatenate them and use NetMapOperator. Then, the loss function should be to maximize the score on the real image while minimizing the score on the fake image, so we negate the real score and then add them.

wganNet =
NetInitialize[
NetGraph[<|"gen" -> generator,
"discrimop" -> NetMapOperator[discriminator],
"cat" -> CatenateLayer[],
"reshape" -> ReshapeLayer[{2, 1, 28, 28}],
"flat" -> FlattenLayer[], "total" -> SummationLayer[],
"scale" ->
ConstantTimesLayer["Scaling" -> {-1, 1}]|>, {NetPort["random"] ->
"gen" -> "cat", NetPort["Input"] -> "cat",
"cat" ->
"reshape" -> "discrimop" -> "flat" -> "scale" -> "total"},
"Input" -> {1, 28, 28}]]


One of the strengths of Mathematica's neural networks framework is that it's really easy to watch the networks train. We'll feed the trainer a progress function that takes 4 fixed random inputs and shows the generator's output, so we can watch the generator evolve over time.

ClearAll[progressFuncCreator]
progressFuncCreator[rands_List] :=
Function[{reals},
ImageResize[
NetDecoder[{"Image", "Grayscale"}][
NetExtract[#Net, "gen"][reals]], 50]] /@ rands &


Finally, create the training data:

trainingData = <|"random" -> RandomReal[{-1, 1}, {randomDim}],
"Input" -> ArrayReshape[ImageData[#], {1, 28, 28}]|> & /@
mnistDigits;


And train, watching the generator make a bunch of vaguely number-shaped blobs. Notice the "WeightClipping" option on the discriminator -- this is the "secret sauce" in Wasserstein GANs that makes them learn an approximation of the Wasserstein/Earth-Mover's distance as opposed to the Jensen-Shannon distance, as explained in the paper.

NetTrain[wganNet, trainingData, "Output",
Method -> {"ADAM", "Beta1" -> 0.5, "LearningRate" -> 0.00005,
"WeightClipping" -> {"discrimop" -> 0.01}},
TrainingProgressReporting ->
progressFuncCreator[Table[RandomReal[{-1, 1}, {randomDim}], 4]],
LearningRateMultipliers -> {"scale" -> 0, "gen" -> -0.2},
TargetDevice -> "GPU", BatchSize -> 64]


Overall, my impression of the neural networks framework is very good. It's extremely flexible, coherently designed, and also extremely pretty. Crucially, it's easier to watch your net train than in any other framework. However, due to difficulties with staged training/saving optimizer parameters, it's not yet possible to replicate (in the sense of replicating a scientific experiment) some published results, like GANs, that use weirder architectures.

• Thanks for the post! It's very cool to watch the generator in action! – xslittlegrass Mar 29 '17 at 17:00
• Nice post! It is possible to get something a little less blob-looking. A convnet as the discriminator seems to help quite a bit. I'm also using a deconvnet as the generator. Latent distribution is an embedding layer. Here's my code: gist.github.com/anonymous/4cccd394f4f7954d577ceb2f75971094 – Taliesin Beynon Apr 5 '17 at 19:54
• @TaliesinBeynon The link is dead. – partida Nov 9 '17 at 4:40
• In order to make this work in 11.3, you have to : - Replace FlattenLayer[] with ReshapeLayer[{2}] - Change in the NetTrain function "Output" to LossFunction -> "Output" Thanks again for this tutorial ! :) – Krakopince Mar 29 '18 at 21:01
• alright, being a newby is hard: I got this all to run, but now that the trained net is ganNet, how do I run the trained generator on a new random sample, i.e. generate a good fake image? running "generator" alone it tells me that it hasn't been trained (no weights)... – dhohl May 28 '18 at 16:29

Yes it is possible. You can do alternating training manually by literally following the algorithm, so that you have a Do loop whose body contains two calls to NetTrain, but that suffers from overhead at each alternation (this could be overcome with clever caching, but we haven't done that yet). An approximation of this is to build a single network and optimize the D and G losses simultaneously by using a negative learning rate for the generator.

I have prototyped this, but only on a toy example.

I encourage you to try how to do it, it didn't take us more than a few hours of playing around to make a simple GAN in which the data distribution is a gaussian, the discriminator is an MLP, and the generator is a single EmbeddingLayer (just a fixed set of samples that can be moved around by gradient updates).

• to exactly replicate the code provided for a lot of the GAN papers, you'd need to do alternating training while preserving the optimizer state (momentum and learning rates, at least) between training steps. i can't find a way to do this in mathematica, and it does seem to make a difference -- i'm unable to replicate simple MNIST examples. i bet it's possible to get things to work but would require a lot more experimenting with parameters. – Michael Curry Mar 19 '17 at 18:55
• @TaliesinBeynon Is there an elegant way to do GAN training yet? – user5601 Mar 29 '18 at 21:04
• Is there any update on this in v12 – M.R. May 24 '19 at 18:22

I see the generation of images from classification models as a trivial idea, largely, because this kind of processes and algorithms are standard in natural language processing.

More to the point of the question, the following posts show generation of images with classification derived bases:

In some sense when using SVD or NNMF bases to recognize an image we reconstruct it by appropriate overlaying of basis images. Obviously such overlaying can be done without a recognition goal just to generate new images.

## Update, 2017-06-24

Looking at the answer of Michael Curry and running the code (and Taliesin Beynon's code) I kind of see using Neural Networks as some sort of a long route. The MNIST based images those codes generate can be generated in a much quicker and controllable way using SVD and NNMF.

As an example, examine this basis images of handwriting "5" obtained with NNMF:

With those kind of bases are generated the (re)constructed handwritten digit images from this MSE answer:

• It seems the NNMF basis is so powerful that it can reconstruct a perfectly clear 0 from a noisy image of a 4, and vice versa... (2nd column, 6th, 11th, 13th, 15th rows) – user484 Jun 25 '17 at 10:25
• @Rahul This actually illustrates the point that with the NNMF basis one can construct new, human interpretable images from a given image collection, which is the whole point of OP's question. (I have semi-hand-picked the NNMF basis for reconstruction, that is why there are some misses.) – Anton Antonov Jun 25 '17 at 11:16
• yeah, if you actually just want to generate relatively simple images like MNIST digits, there's way simpler generative models like this one that will train faster and work better. especially for people learning ML right now at peak deep learning hype, it's real important not to forget this -- deep learning should be your last resort, not your first! but hopefully this simple example should provide the "scaffolding" to train on more challenging datasets, like generating cats affinelayer.com/pixsrv – Michael Curry Jun 26 '17 at 15:51
• @MichaelCurry God points. I would like to say that I played quite a lot with your code (and Taliesin's) with several different datasets (like this experiment here). Thanks for posting them! – Anton Antonov Jun 26 '17 at 17:02
• @AntonAntonov oh dear. those are some... peculiar mandalas it's generating. – Michael Curry Jun 28 '17 at 1:38

This is a GAN using more accurate loss function,Thanks for @Michael Curry

mnist = ResourceData["MNIST"];
mnistDigits = First /@ mnist;

randomDim = 10;
generator = NetChain[{128, Ramp, 128, Ramp, 28*28, LogisticSigmoid,
ReshapeLayer[{1, 28, 28}]}, "Input" -> randomDim];
discriminator = NetChain[{128, Ramp, 128, Ramp, 128, Ramp,
1, LogisticSigmoid},
"Input" -> {1, 28, 28}];


The loss function follow this paper

Gloss = NetChain[{PartLayer[1], ElementwiseLayer[Log[1 - #] &]}];

Dloss = NetGraph[{PartLayer[1], ElementwiseLayer[Log[1 - #] &],
PartLayer[2], ElementwiseLayer[Log[#] &],
TotalLayer[],
ElementwiseLayer[-1 # &](*transfer Maximize to Minimize*)},
{1 -> 2, 3 -> 4, {2, 4} -> 5 -> 6}];


In the flat layer,the first part estimate the probability of fake,the second part estimate the probability of real.You can add another port Output2 to check this

ganNet = NetInitialize[
NetGraph[<|"gen" -> generator,
"discrimop" -> NetMapOperator[discriminator],
"cat" -> CatenateLayer[],
"reshape" -> ReshapeLayer[{2, 1, 28, 28}],
"flat" -> FlattenLayer[], "G_loss" -> Gloss,
"D_loss" -> Dloss|>,
{NetPort["random"] -> "gen" -> "cat", NetPort["Input"] -> "cat", "cat" -> "reshape" -> "discrimop" -> "flat" -> {"G_loss", "D_loss"},
"G_loss" -> NetPort["Loss1"], "D_loss" -> NetPort["Loss2"]},
"Input" -> NetEncoder[{"Image", {28, 28}, ColorSpace -> "Grayscale"}]]]


trainingData = <|"random" -> RandomVariate[NormalDistribution[], {Length@mnistDigits, randomDim}],
"Input" -> mnistDigits|>;


But when training,I want to reduce the loss of "Loss1" and "Loss2".But find nothing in docs,only got code like this trained = NetTrain[net, <|"Input" -> trainingImages|>, "Loss"].But here we have two loss function.

In the Pytorch,we can use two optimizer to reduce the two loss.But in MMA,I have no idea.

So this a simple way to train the net.

ganNet = NetTrain[ganNet, trainingData, {"Loss1", "Loss2"},
Method -> {"ADAM", "Beta1" -> 0.5, "LearningRate" -> 0.00005, "WeightClipping" -> {"discrimop" -> 0.01}},
BatchSize -> 64];

• I could not make it work. It just generates noises. How many rounds are required with batch size 64? – Joo-Haeng Lee May 9 '19 at 14:03