9
$\begingroup$

I'm looking for someway to do what this syntax "should" do:

NetEncoder[{"Image", Automatic, ColorSpace -> "RGB"}] /@ imgs;

The documentation states that NetEncoder["Image"] is equivalent to NetEncoder[{"Image", {128,128}}]. But I don't want to resize the training images to a constant size (e.g. I'm training a fully convolutional network). I simply want like to encode them as 3-tensors with whatever ImageDimensions they actually are.

For example, say I have the following two images with dimensions {{4608, 3456}, {512, 512}}. Using NetEncoder forces us to resize both images to a fixed dimension, which is not what I want:

imgs = ExampleData /@ {{"TestImage", "Stall"}, {"TestImage", "F16"}};
NetDecoder["Image"]@*NetEncoder["Image"] /@ imgs

enter image description here

The documentation also states that a ConvolutionLayer can operate on tensors that contain "Varying" dimensions, which seems helpful:

imgsData = ImageData[#, Interleaving -> False] & /@ imgs;
conv = NetInitialize @ ConvolutionLayer[3, 2, "Input" -> {3, Automatic, Automatic}]
ImageAdjust@*NetDecoder["Image"]@*conv@*NetEncoder["Image"] /@ imgs
ImageAdjust@*NetDecoder["Image"]@*conv /@ imgsData

enter image description here

But then we also lose all the benefits of using a NetEncoder (out-of-core parallel file loading, etc..) Is there a way around this issue?

Updates:

Automatic doesn't work for dimensions in a custom encoder:

enc = NetEncoder[{"Function", 
   ImageData[FastImageImport@#, Interleaving -> False] &, {3, Automatic, Automatic}}]

enter image description here

not to mention importing images yourself is a terrible idea (slows down training 100x).

Here's the motivating example of a fully convolutional net one might want to train on images of varying size:

n = NetInitialize[NetChain[{
    ConvolutionLayer[32, {3, 3}, "Dilation" -> 1, "PaddingSize" -> 1],
    ConvolutionLayer[32, {3, 3}, "Dilation" -> 2, "PaddingSize" -> 2],
    ConvolutionLayer[32, {3, 3}, "Dilation" -> 4, "PaddingSize" -> 4], 
    ConvolutionLayer[32, {3, 3}, "Dilation" -> 1, "PaddingSize" -> 1],
    ConvolutionLayer[3, {1, 1}, "Dilation" -> 1]
},
"Input" -> NetEncoder[{"Image", {"Varying", "Varying"}, ColorSpace -> "RGB"}],
"Output" -> NetDecoder[{"Image", ColorSpace -> "RGB"}]], 
Method -> "Identity"]

enter image description here

$\endgroup$
  • $\begingroup$ Not possible. Because the the dimension of output of convolutional net is depend on the dimension of input. If the dimension of input is varying, the number of class in the output is changed. $\endgroup$ – partida Apr 27 '18 at 12:26
  • $\begingroup$ @partida there is no class in the output, the output is an image $\endgroup$ – M.R. Apr 27 '18 at 15:01
  • 1
    $\begingroup$ Shouldn't we be able to apply any convolution to any image of any size? If I'm training on image input-output pairs, all that matters is that the input and output have the same dimensions so that the MeanSquaredLossLayer will work. $\endgroup$ – M.R. Apr 27 '18 at 15:07
  • 1
    $\begingroup$ If the dimension of input and ouput is identity, it should be work. In the ref/netencoder/Image, it seems no options fotr this case $\endgroup$ – partida Apr 28 '18 at 2:21
5
+50
$\begingroup$

Automatic is not the same as "Varying".

Example 1

This works:

conv1 = NetInitialize@ConvolutionLayer[3, 2, "Input" -> {3, Automatic, Automatic}];
conv1@NetEncoder[{"Image", {128, 128}}]@RandomImage[1, {128, 128}, ColorSpace -> "RGB"]

enter image description here

This doesn't work at the stage of layer creation:

conv2 = NetInitialize@ConvolutionLayer[3, 2, "Input" -> {3, "Varying", "Varying"}]

enter image description here

Example 2

This doesn't work at the stage of neural network training:

SeedRandom[0];
X = Table[RandomReal[{0, 1}, {3, RandomInteger[{3, 64}], 64}], {10}];
Y = RandomInteger[{0, 1}, 10];

net1 = NetChain[
   {
    ConvolutionLayer[32, {3, 3}],
    AggregationLayer[Mean],
    2,
    SoftmaxLayer[]
    },
   "Input" -> {3, Automatic, Automatic},
   "Output" -> NetDecoder[{"Class", {0, 1}}]
   ];

NetTrain[net1, X -> Y]

enter image description here

This doesn't work at the stage of network creation:

net2 = NetChain[
   {
    ConvolutionLayer[32, {3, 3}],
    AggregationLayer[Mean],
    2,
    SoftmaxLayer[]
    },
   "Input" -> {3, "Varying", "Varying"},
   "Output" -> NetDecoder[{"Class", {0, 1}}]
   ];

enter image description here

Conclusions

ConvolutionLayer supports "Varying" only as the first dimension and if "Interleaving" -> True.

Do[
  Print[{i, ConvolutionLayer[32, {3, 3}, "Interleaving" -> False, "Input" -> i]}],
  {i, Tuples[{64, "Varying"}, 3]}
  ] // Quiet

enter image description here

Do[
  Print[{i, ConvolutionLayer[32, {3, 3}, "Interleaving" -> True, "Input" -> i]}],
  {i, Tuples[{64, "Varying"}, 3]}
  ] // Quiet

enter image description here

This works:

SeedRandom[0];
X = Table[RandomReal[{0, 1}, {RandomInteger[{3, 64}], 64, 3}], {10}];
Y = RandomInteger[{0, 1}, 10];

net = NetChain[
   {
    ConvolutionLayer[32, {3, 3}, "Interleaving" -> True],
    AggregationLayer[Mean, 1;;2],
    2,
    SoftmaxLayer[]
    },
   "Input" -> {"Varying", 64, 3},
   "Output" -> NetDecoder[{"Class", {0, 1}}]
   ];

NetTrain[net, X -> Y]

![enter image description here

As I understand, support of "Varying" dimension was added in ConvolutionLayer to work with text of varying size (ConvS2S etc.), not with images.

NetChain[
 {
  EmbeddingLayer[10],
  ConvolutionLayer[32, {3}, "Interleaving" -> True],
  AggregationLayer[Mean, 1],
  2,
  SoftmaxLayer[]
  },
 "Input" -> "Varying",
 "Output" -> NetDecoder[{"Class", {0, 1}}]
 ]

enter image description here

So your problem is not only in the NetEncoder that doesn't support varying dimensions.

Probably support of more than one "Varying" dimension will be added in the next releases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.