9
$\begingroup$

I would like to train a network that can parse clothing:

enter image description here

Does anyone know of a fashion segmentation network with or without pre-trained weights available that I can use in Mathematica?

Update:

I haven't been able to find a pre-trained network that I might port into Mathematica via MXNetLink, so I've decided to train one.

Here is a link to the fashionista dataset of pixel level labels for clothing. The first issue I've encountered is blocking my progress: I can't import the .mat file:

enter image description here

If someone can help me parse this dataset, I'd like to train a semantic segmentation net like this example from Wolfram Community. Or perhaps there is another source of training data that is accessible from Mathematica?

References:

Fashion related segmentation is described in these papers, blogs, and projects:

$\endgroup$
2
  • 1
    $\begingroup$ Looking at the file "fashionista-v0.2.1/fashionista_v0.2.1.mat" and after a cursory reading of the explanations, it seems to me that you might be better off using the images in the older version "fashionista-v0.2.tgz". (At least at first...) $\endgroup$ Commented May 29, 2018 at 17:32
  • $\begingroup$ Related: How can I separate the dress from the background? $\endgroup$
    – C. E.
    Commented Jan 8, 2019 at 18:51

1 Answer 1

9
+200
$\begingroup$

Yes, but that particular dataset is not extremely great.

Instead of using the Fashionista dataset, let's use this dataset instead: https://github.com/bearpaw/clothing-co-parsing. The Fashionista dataset is too much inside the Matlab walled garden (I am living in a glass house and throwing stones).

This dataset is much more usable. 1004 images have masks (a matlab matrix, but one we can read). They look like this:

mask

Let's load the images and masks.

images = (File /@ 
     FileNames["~/Downloads/clothing-co-parsing-master/photos/*"])[[;;
      1004]];

masks = Import[
  "~/Downloads/clothing-co-parsing-master/annotations/pixel-level/*"]

Now we'll turn resize the masks to something much smaller (my poor laptop isn't beefy enough for larger image dimensions). Input images will get resized by the network NetEncoder. We add 1 to the masks so that they are the lie in the same range as the output.

dims = {104, 157};
amasks = Round[
  ArrayResample[First[#], {157, 104}, "Bin", 
   Resampling -> "NearestRight"]] + 1 & /@ masks;

So we can generate image->mask data for the neural net like so:

data = Thread[images -> amasks];

Now we'll build an extremely quick-and-dirty pixel-level segmentation network as a proof-of-concept.

This network is of my own design, which means it's probably terrible. However, it does go fast. We chop off the last few layers of Squeezenet and attach a few skip connections. This network does a decent job of binary prediction - let's see how it does with 59-class prediction.

A quick improvement to this network is to use more than just the first few layers of squeezenet. However, I don't want to watch my computer train a network slowly forever, so I have settled for about the first half of squeezenet.

squeeze = 
  NetModel["SqueezeNet V1.1 Trained on ImageNet Competition Data"];

tnet = NetGraph[
  Join[Normal@
    NetFlatten[
     NetTake[NetReplacePart[squeeze, 
       "Input" -> NetEncoder[{"Image", dims, "ColorSpace" -> "RGB"}]],
       "fire5"]],
   <|
    
    "f2u" -> {BatchNormalizationLayer[], 
      ResizeLayer[{Scaled[2], Scaled[2]}, "Resampling" -> "Nearest"], 
      ConvolutionLayer[59, 1], ElementwiseLayer["ELU"]},
    "f3u" -> {BatchNormalizationLayer[], 
      ResizeLayer[{Scaled[2], Scaled[2]}, "Resampling" -> "Nearest"], 
      ConvolutionLayer[59, 1], ElementwiseLayer["ELU"]},
    "f4u" -> {BatchNormalizationLayer[], 
      ResizeLayer[{Scaled[4], Scaled[4]}, "Resampling" -> "Nearest"], 
      ConvolutionLayer[59, 1, "PaddingSize" -> 1], 
      ElementwiseLayer["ELU"]},
    "f5u" -> {BatchNormalizationLayer[], 
      ResizeLayer[{Scaled[4], Scaled[4]}, "Resampling" -> "Nearest"], 
      ConvolutionLayer[59, 1, "PaddingSize" -> 1], 
      ElementwiseLayer["ELU"]},
    "cat" -> CatenateLayer[],
    "drop" -> DropoutLayer[],
    "sig" -> ElementwiseLayer["ReLU"],
    "con" -> {ResizeLayer[Reverse@dims], ConvolutionLayer[59, 1], 
      TransposeLayer[{1 <-> 3, 1 <-> 2}], SoftmaxLayer[]}
    |>
   ],
  {Fold[#2 -> #1 &, 
    Reverse@Keys@Normal@NetFlatten[NetTake[squeeze, "fire5"]]],
   "fire2" -> "f2u",
   "fire3" -> "f3u",
   "fire4" -> "f4u",
   "fire5" -> "f5u",
   {"f2u", "f3u", "f4u", "f5u"} -> "cat" -> "drop" -> "sig" -> "con"
   },
  "Input" -> NetEncoder[{"Image", dims, "ColorSpace" -> "RGB"}],
  "Output" -> NetDecoder[{"Class", Range[59], "InputDepth" -> 3}]
  ]

Now we can train this network. I have frozen the squeezenet weights, but it's not necessary (and not doing this will definitely result in a more accurate network).

net = NetTrain[tnet, data[[;; -30]], ValidationSet -> data[[-30 ;;]], 
  LearningRateMultipliers -> {"conv1" -> 0, "fire2" -> 0, 
    "fire3" -> 0, "fire4" -> 0}]

I trained this for a short time (5 rounds), let's see what it looks like:

test output

That's not bad at all for such a short time training. Especially because there are many improvements you could make to this extremely off-the-cuff network:

Good luck!

$\endgroup$
13
  • $\begingroup$ For your”exercise”, are you just taking the decoded image and finding the nearest color for each label per pixel? $\endgroup$
    – M.R.
    Commented Jan 9, 2019 at 22:27
  • $\begingroup$ Or are you combing the networks output with the input image and growing the regions with some sort of clustering components? What’s the method $\endgroup$
    – M.R.
    Commented Jan 9, 2019 at 22:37
  • $\begingroup$ Yes, probably the first one (I would say it's significantly easier if you train for much longer with that network). The better method is to figure out how to train a pixel-level class predictor network. I'll work on it when I get the chance. $\endgroup$
    – Carl Lange
    Commented Jan 9, 2019 at 22:49
  • $\begingroup$ Is the last rule of images you show the output -> decoded output or are you just showing the mask from the file? $\endgroup$
    – M.R.
    Commented Jan 9, 2019 at 23:10
  • $\begingroup$ I'm showing output->original data mask (I'm showing how poorly this network did - it was more an illustration than a working example) $\endgroup$
    – Carl Lange
    Commented Jan 9, 2019 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.