2
$\begingroup$

In neural network segmentation task, the training data usually consists of pairs of images. In each training sample, the input image is the image we want to segment, and the target image contains the corresponding class index for each pixel of the input image.

For dealing with large dataset, the NetEncoder provides efficient loading of images files. However, the encoder doesn't seem to be compatible with the CrossEntropyLossLayer in the following example.

Consider the following neural network

net = NetGraph[
  <|"t1" -> TransposeLayer[{3 <-> 1, 1 <-> 2}], 
   "loss" -> CrossEntropyLossLayer["Index"], 
   "f" -> {PartLayer[1], ElementwiseLayer[Clip[Round[225 #], {1, 3}] &]}|>,
  {NetPort["Input1"] -> "t1" -> NetPort[{"loss", "Input"}],
   NetPort["Input2"] -> "f"}, "Input1" -> NetEncoder[{"Image", 8}], 
  "Input2" -> NetEncoder[{"Image", 8, ColorSpace -> "Grayscale"}]]

enter image description here

where input images (3 channel 8 by 8) feed into the "Input1" port and the target images (8 by 8) feed into the "Input2" port. The "Output" port from f is a matrix of size 8 by 8, each pixel of which indicates the class and is either 1,2 or 3.

When I connect the "Output" port from f to the "Target" port of the CrossEntropyLossLayer, NetTrain complains that the type doesn't match

NetGraph[
 <|"t1" -> TransposeLayer[{3 <-> 1, 1 <-> 2}], 
  "loss" -> CrossEntropyLossLayer["Index"], 
  "f" -> {PartLayer[1], ElementwiseLayer[Clip[Round[225 #], {1, 3}] &]}|>,
 {NetPort["Input1"] -> "t1" -> NetPort[{"loss", "Input"}],
  NetPort["Input2"] -> "f" -> NetPort[{"loss", "Target"}]}, 
 "Input1" -> NetEncoder[{"Image", 8}], 
 "Input2" -> NetEncoder[{"Image", 8, ColorSpace -> "Grayscale"}]]

NetGraph::tyfail2: Inferred inconsistent shapes for output of second layer of layer "f" (a tensor of bounded integers versus a tensor).

So is there a way to encode the image into index that is compatible with CrossEntropyLossLayer?

$\endgroup$
2
$\begingroup$

NetEncoder[{"Function", NetEncoder[{"Image"}]@# &, {}}]

enter image description here

net = NetGraph[
  <|
   "conv" -> {ConvolutionLayer[3, {1, 1}], TransposeLayer[1 <-> 3], SoftmaxLayer[]},
   "loss" -> CrossEntropyLossLayer["Index"]
   |>,
  {
   NetPort["Input"] -> "conv" -> NetPort["loss", "Input"],
   NetPort["Target"] -> NetPort["loss", "Target"]
   },
  "Input" -> NetEncoder[{"Image", {8, 8}, ColorSpace -> "RGB"}],
  "Target" -> NetEncoder[
    {
     "Function", 
     (Clip[Round[225 #], {1, 3}] &@ NetEncoder[{"Image", {8, 8}, ColorSpace -> "Grayscale"}]@#)[[1]] &,
     {8, 8, Restricted["Integer", 3]}
     }
    ]
  ]

enter image description here

X = Table[RandomImage[1, {8, 8}, ColorSpace -> "RGB"], {10}];
Y = Table[RandomImage[1, {8, 8}, ColorSpace -> "Grayscale"], {10}];

{X, Y}

enter image description here

NetTrain[net, <|"Input" -> X, "Target" -> Y|>]

enter image description here

Binary classification

net = NetGraph[
  <|
   "conv" -> {ConvolutionLayer[1, {1, 1}], LogisticSigmoid},
   "loss" -> CrossEntropyLossLayer["Binary"]
   |>,
  {
   NetPort["Input"] -> "conv" -> NetPort["loss", "Input"],
   NetPort["Target"] -> NetPort["loss", "Target"]
   },
  "Input" -> NetEncoder[{"Image", {8, 8}, ColorSpace -> "RGB"}],
  "Target" -> NetEncoder[{"Image", {8, 8}, ColorSpace -> "Grayscale"}]
  ]

enter image description here

NetTrain[net, <|"Input" -> X, "Target" -> Y|>]

enter image description here

$\endgroup$
  • $\begingroup$ Thanks! But would the nested NetEncoder impact the performance? In my case, it seems slightly (10%) slower than the out-of-core generator version. Consider a case where we want to do quite a few image operations (Part, Rescale, etc.) before converting it to the target port, in that case, it would be good if we can treat NetEncoder as an ordinary layer and include those image operations as neural network layers rather than as composite functions in NetEncoder. Any comments on that? $\endgroup$ – xslittlegrass Jun 19 '18 at 5:14
  • $\begingroup$ @xslittlegrass Yes, nested NetEncoder will be slower. "the image loader is written in C++ ... and each thread processes a batch of images". I am not sure that now it will process a batch of images. Your idea with generator is good. I see no way how we can convert real to integer inside of the network. Because CrossEntropyLossLayer["Index"] expects only integer "Target". $\endgroup$ – Alexey Golyshev Jun 19 '18 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.