7
$\begingroup$

I have a set of images (a few hundred) that represent a certain style and I would like to train an unpaired image to image translator with CycleGAN.

I'm looking for a tutorial on how one would do this with NetTrain.

For example, in the Wolfram Neural Net repository there is a NetModel for Photo-to-Van Gogh Translation, but the construction notebook is missing any details about training:

enter image description here

Ok, so I'm trying to piece this back the missing training section. From the construction notebook we have these functions:

convIN[args___] := {ConvolutionLayer[args], 
  InstanceNormalizationLayer["Epsilon" -> 0.00001]}

convINRamp[args___] := {ConvolutionLayer[args], 
  InstanceNormalizationLayer["Epsilon" -> 0.00001], 
  ElementwiseLayer[Ramp]}

residualModule = NetGraph[
   {
    NetChain@
     Join[{PaddingLayer[{{0, 0}, {1, 1}, {1, 1}}, 
        "Padding" -> "Reflected"]}, 
      convINRamp[128, 
       3], {PaddingLayer[{{0, 0}, {1, 1}, {1, 1}}, 
        "Padding" -> "Reflected"]}, convIN[128, 3]],
    TotalLayer[]
    },
   {NetPort["Input"] -> {1, 2}, 1 -> 2}
   ];

cycleGAN = NetChain[
  Join[
   {ElementwiseLayer[2 # - 1 &],
    PaddingLayer[{{0, 0}, {3, 3}, {3, 3}}, 
     "Padding" -> "Reflected"]},
   convINRamp[32, 7],
   convINRamp[64, 3, "PaddingSize" -> 1, "Stride" -> 2],
   convINRamp[128, 3, "PaddingSize" -> 1, "Stride" -> 2],
   ConstantArray[residualModule, 9],
   {DeconvolutionLayer[64, 3, "Stride" -> 2],
    PartLayer[{All, 2 ;;, 2 ;;}],
    InstanceNormalizationLayer["Epsilon" -> 0.00001],
    ElementwiseLayer[Ramp],
    DeconvolutionLayer[32, 3, "Stride" -> 2],
    PartLayer[{All, 2 ;;, 2 ;;}],
    InstanceNormalizationLayer["Epsilon" -> 0.00001],
    ElementwiseLayer[Ramp],
    PaddingLayer[{{0, 0}, {3, 3}, {3, 3}}, "Padding" -> "Reflected"],
    ConvolutionLayer[3, 7],
    ElementwiseLayer[0.5*Tanh[#] + 0.5 &]}
   ],
  "Input" -> NetEncoder[{"Image", 512}],
  "Output" -> NetDecoder[{"Image"}]
  ]

Now let's grab some data in order to try training it:

styleFolder = "~/Downloads/style";
randomFolder = "~/Downloads/random";

CreateDirectory /@ {styleFolder, randomFolder};

URLDownload[#, styleFolder] & /@ 
  WebImageSearch["monet", "ImageHyperlinks", "MaxItems" -> 5];

URLDownload[#, randomFolder] & /@ 
  WebImageSearch["outdoors", "ImageHyperlinks", "MaxItems" -> 5];

styleImages = File /@ FileNames["*.jpg", styleFolder]
randomImages = File /@ FileNames["*.jpg", randomFolder]

This is where I'm stuck, and there are two main questions:

  1. How to write cycle-consistency loss
  2. How to format/feed the training data to the GAN

But I'm still a long way from filling in these last two lines:

trainingData = (* ? *)
NetTrain[cycleGAN, trainingData]

Links:

$\endgroup$
  • 2
    $\begingroup$ If I had to guess, I'd say NetTrain was never meant to support GAN training, maybe this will be a future feature in v11.4 or v12? $\endgroup$ – M.R. May 17 '18 at 1:44
  • 5
    $\begingroup$ @M.R. this is true, NetTrain is not designed for GANs. Generalizing NetTrain to support GANs and reinforcement learning is a high priority for us for the next version. $\endgroup$ – Sebastian May 17 '18 at 8:49
  • 1
    $\begingroup$ @Sebastian So do you train the CycleGAN based NetModels in pure MXNet? $\endgroup$ – user5601 May 17 '18 at 13:50
  • 2
    $\begingroup$ If no one can get this to work in v12 I’m giving up on Mma for ML, this is trivial in pytorch!! $\endgroup$ – user5601 May 19 at 8:55
  • 2
    $\begingroup$ I added a bounty, hopefully that will help you @user5601 $\endgroup$ – M.R. May 19 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.