10
$\begingroup$

NVIDIA recently published a new way to reconstruct images with holes. They use a new type of convolutional layer which is described in the paper Image Inpainting for Irregular Holes Using Partial Convolutions.

enter image description here

I'm trying to implement this in Mathematica. The crucial part is the Partial Convolutional Layer (described in section 3.1 on page 5) defined as follows:

enter image description here

What I've tried so far:

partialConv = NetInitialize @ NetGraph[
    <|
       "c1" -> ConvolutionLayer[10, {3, 3}, "Weights" -> NetSharedArray["c"]],
       "c2" -> ConvolutionLayer[10, {3, 3}, "Weights" -> NetSharedArray["c"]],
       "thread" -> ThreadingLayer[#1/#2 &]
    |>,
    { 
      NetPort["mask"] -> "c1", 
      NetPort["maskedImg"] -> "c2", 
      {"c1", "c2"} -> "thread" -> NetPort["Output"]
    }, 
      "maskedImg" -> NetEncoder["Image"], 
      "mask" -> NetEncoder["Image"], 
      "Output" -> NetDecoder["Image"]
   ]

img = ImageResize[ExampleData[{"TestImage", "Lena"}], 70];
mask = Binarize@RemoveBackground@img;
maskedImg = ImageSubtract[img, ColorNegate@mask];
Grid@{{img, maskedImg}}
partialConv[<|"mask" -> mask, "maskedImg" -> img|>]

enter image description here

$\endgroup$
  • 1
    $\begingroup$ Hard to say. What have you tried? $\endgroup$ – MarcoB Apr 27 '18 at 17:41
  • 2
    $\begingroup$ I just skimmed the paper - wouldn't you get the same result if you applied the same convolution filter (with shared weights) to the input image and the mask, both with masked pixels set to 0, then used a ThreadingLayer to divide the two results? $\endgroup$ – Niki Estner Apr 27 '18 at 17:52
  • 1
    $\begingroup$ I think you might be right, trying it now... $\endgroup$ – user5601 Apr 27 '18 at 17:54
  • 1
    $\begingroup$ @NikiEstner I'm trying to match things up with the paper using your idea for implementing the partialConv layer, but I'm not sure if it is right $\endgroup$ – user5601 Apr 30 '18 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.