8
$\begingroup$

I have a set of images (a few hundred) that represent a certain style and I would like to train an unpaired image to image translator with CycleGAN.

I'm looking for a tutorial on how one would do this with NetTrain.

For example, in the Wolfram Neural Net repository there is a NetModel for Photo-to-Van Gogh Translation, but the construction notebook is missing any details about training:

enter image description here

Ok, so I'm trying to piece this back the missing training section. From the construction notebook we have these functions:

convIN[args___] := {ConvolutionLayer[args], 
  InstanceNormalizationLayer["Epsilon" -> 0.00001]}

convINRamp[args___] := {ConvolutionLayer[args], 
  InstanceNormalizationLayer["Epsilon" -> 0.00001], 
  ElementwiseLayer[Ramp]}

residualModule = NetGraph[
   {
    NetChain@
     Join[{PaddingLayer[{{0, 0}, {1, 1}, {1, 1}}, 
        "Padding" -> "Reflected"]}, 
      convINRamp[128, 
       3], {PaddingLayer[{{0, 0}, {1, 1}, {1, 1}}, 
        "Padding" -> "Reflected"]}, convIN[128, 3]],
    TotalLayer[]
    },
   {NetPort["Input"] -> {1, 2}, 1 -> 2}
   ];

cycleGAN = NetChain[
  Join[
   {ElementwiseLayer[2 # - 1 &],
    PaddingLayer[{{0, 0}, {3, 3}, {3, 3}}, 
     "Padding" -> "Reflected"]},
   convINRamp[32, 7],
   convINRamp[64, 3, "PaddingSize" -> 1, "Stride" -> 2],
   convINRamp[128, 3, "PaddingSize" -> 1, "Stride" -> 2],
   ConstantArray[residualModule, 9],
   {DeconvolutionLayer[64, 3, "Stride" -> 2],
    PartLayer[{All, 2 ;;, 2 ;;}],
    InstanceNormalizationLayer["Epsilon" -> 0.00001],
    ElementwiseLayer[Ramp],
    DeconvolutionLayer[32, 3, "Stride" -> 2],
    PartLayer[{All, 2 ;;, 2 ;;}],
    InstanceNormalizationLayer["Epsilon" -> 0.00001],
    ElementwiseLayer[Ramp],
    PaddingLayer[{{0, 0}, {3, 3}, {3, 3}}, "Padding" -> "Reflected"],
    ConvolutionLayer[3, 7],
    ElementwiseLayer[0.5*Tanh[#] + 0.5 &]}
   ],
  "Input" -> NetEncoder[{"Image", 512}],
  "Output" -> NetDecoder[{"Image"}]
  ]

Now let's grab some data in order to try training it:

styleFolder = "~/Downloads/style";
randomFolder = "~/Downloads/random";

CreateDirectory /@ {styleFolder, randomFolder};

URLDownload[#, styleFolder] & /@ 
  WebImageSearch["monet", "ImageHyperlinks", "MaxItems" -> 5];

URLDownload[#, randomFolder] & /@ 
  WebImageSearch["outdoors", "ImageHyperlinks", "MaxItems" -> 5];

styleImages = File /@ FileNames["*.jpg", styleFolder]
randomImages = File /@ FileNames["*.jpg", randomFolder]

This is where I'm stuck, and there are two main questions:

  1. How to write cycle-consistency loss
  2. How to format/feed the training data to the GAN

But I'm still a long way from filling in these last two lines:

trainingData = (* ? *)
NetTrain[cycleGAN, trainingData]

Links:

Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ If I had to guess, I'd say NetTrain was never meant to support GAN training, maybe this will be a future feature in v11.4 or v12? $\endgroup$
    – M.R.
    Commented May 17, 2018 at 1:44
  • 5
    $\begingroup$ @M.R. this is true, NetTrain is not designed for GANs. Generalizing NetTrain to support GANs and reinforcement learning is a high priority for us for the next version. $\endgroup$
    – Sebastian
    Commented May 17, 2018 at 8:49
  • 1
    $\begingroup$ @Sebastian So do you train the CycleGAN based NetModels in pure MXNet? $\endgroup$
    – user5601
    Commented May 17, 2018 at 13:50
  • 3
    $\begingroup$ If no one can get this to work in v12 I’m giving up on Mma for ML, this is trivial in pytorch!! $\endgroup$
    – user5601
    Commented May 19, 2019 at 8:55
  • 2
    $\begingroup$ I added a bounty, hopefully that will help you @user5601 $\endgroup$
    – M.R.
    Commented May 19, 2019 at 8:56

1 Answer 1

Reset to default
2
$\begingroup$

There is now a complete CycleGAN construction example in the Mathematica 12.1 documentation for NetGANOperator (under Applications)

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ the example you're referring to has very little to do with actual CycleGAN implementation as originally asked: everything is wrong from loss function to the training logic. It's actually impossible to recreate original CycleGAN in NetTrain settings (and even using undocumented MXNetLink functionality. MMA up to 13 is not suitable for nontrivial GANs and I suspect this will never change. $\endgroup$
    – Pavel Perikov
    Commented Jan 1, 2022 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.