7
$\begingroup$

I have a set of images (a few hundred) that represent a certain style and I would like to train an unpaired image to image translator with CycleGAN.

I'm looking for a tutorial on how one would do this with NetTrain.

For example, in the Wolfram Neural Net repository there is a NetModel for Photo-to-Van Gogh Translation, but the construction notebook is missing any details about training:

enter image description here

Ok, so I'm trying to piece this back the missing training section. From the construction notebook we have these functions:

convIN[args___] := {ConvolutionLayer[args], 
  InstanceNormalizationLayer["Epsilon" -> 0.00001]}

convINRamp[args___] := {ConvolutionLayer[args], 
  InstanceNormalizationLayer["Epsilon" -> 0.00001], 
  ElementwiseLayer[Ramp]}

residualModule = NetGraph[
   {
    NetChain@
     Join[{PaddingLayer[{{0, 0}, {1, 1}, {1, 1}}, 
        "Padding" -> "Reflected"]}, 
      convINRamp[128, 
       3], {PaddingLayer[{{0, 0}, {1, 1}, {1, 1}}, 
        "Padding" -> "Reflected"]}, convIN[128, 3]],
    TotalLayer[]
    },
   {NetPort["Input"] -> {1, 2}, 1 -> 2}
   ];

cycleGAN = NetChain[
  Join[
   {ElementwiseLayer[2 # - 1 &],
    PaddingLayer[{{0, 0}, {3, 3}, {3, 3}}, 
     "Padding" -> "Reflected"]},
   convINRamp[32, 7],
   convINRamp[64, 3, "PaddingSize" -> 1, "Stride" -> 2],
   convINRamp[128, 3, "PaddingSize" -> 1, "Stride" -> 2],
   ConstantArray[residualModule, 9],
   {DeconvolutionLayer[64, 3, "Stride" -> 2],
    PartLayer[{All, 2 ;;, 2 ;;}],
    InstanceNormalizationLayer["Epsilon" -> 0.00001],
    ElementwiseLayer[Ramp],
    DeconvolutionLayer[32, 3, "Stride" -> 2],
    PartLayer[{All, 2 ;;, 2 ;;}],
    InstanceNormalizationLayer["Epsilon" -> 0.00001],
    ElementwiseLayer[Ramp],
    PaddingLayer[{{0, 0}, {3, 3}, {3, 3}}, "Padding" -> "Reflected"],
    ConvolutionLayer[3, 7],
    ElementwiseLayer[0.5*Tanh[#] + 0.5 &]}
   ],
  "Input" -> NetEncoder[{"Image", 512}],
  "Output" -> NetDecoder[{"Image"}]
  ]

Now let's grab some data in order to try training it:

styleFolder = "~/Downloads/style";
randomFolder = "~/Downloads/random";

CreateDirectory /@ {styleFolder, randomFolder};

URLDownload[#, styleFolder] & /@ 
  WebImageSearch["monet", "ImageHyperlinks", "MaxItems" -> 5];

URLDownload[#, randomFolder] & /@ 
  WebImageSearch["outdoors", "ImageHyperlinks", "MaxItems" -> 5];

styleImages = File /@ FileNames["*.jpg", styleFolder]
randomImages = File /@ FileNames["*.jpg", randomFolder]

This is where I'm stuck, and there are two main questions:

  1. How to write cycle-consistency loss
  2. How to format/feed the training data to the GAN

But I'm still a long way from filling in these last two lines:

trainingData = (* ? *)
NetTrain[cycleGAN, trainingData]

Links:

$\endgroup$
9
  • 2
    $\begingroup$ If I had to guess, I'd say NetTrain was never meant to support GAN training, maybe this will be a future feature in v11.4 or v12? $\endgroup$ – M.R. May 17 '18 at 1:44
  • 5
    $\begingroup$ @M.R. this is true, NetTrain is not designed for GANs. Generalizing NetTrain to support GANs and reinforcement learning is a high priority for us for the next version. $\endgroup$ – Sebastian May 17 '18 at 8:49
  • 1
    $\begingroup$ @Sebastian So do you train the CycleGAN based NetModels in pure MXNet? $\endgroup$ – user5601 May 17 '18 at 13:50
  • 2
    $\begingroup$ If no one can get this to work in v12 I’m giving up on Mma for ML, this is trivial in pytorch!! $\endgroup$ – user5601 May 19 '19 at 8:55
  • 2
    $\begingroup$ I added a bounty, hopefully that will help you @user5601 $\endgroup$ – M.R. May 19 '19 at 8:56
1
$\begingroup$

There is now a complete CycleGAN construction example in the Mathematica 12.1 documentation for NetGANOperator (under Applications)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.