Image Segmentation

Question

I have what should be a pretty simple question for Mathematica gurus. I have a set of images containing cars and a background. I am trying to create a black and white image where the white part is the car and the background is black. I am linking a sample input image and what the output should be. Thanks in advance.

input= https://s3.amazonaws.com/albertoaiml/car.jpg

output = https://s3.amazonaws.com/albertoaiml/outputcar.png

Answer 1

The Neural Net Repository has several useful Neural Nets and examples of exactly this type of problem.

Using one of the Neural Networks is quite easy. The first time you evaluate a NetModel, it downloads a fairly large amount of data from the internet. I'll use the Ademxapp Model A1 Trained on ADE20K Data since that's one I already have downloaded.

First, we set up a function to deal with reshaping the input and output. Basically, the network has been trained on images of a certain size and type, so this function changes whatever image you give it into the size and type the network expects.

netevaluate[img_, device_: "CPU"] := 
 Block[{net, resized, encData, dec, mean, var, prob}, 
  net = NetModel["Ademxapp Model A1 Trained on ADE20K Data"];
  resized = ImageResize[img, {504}];
  encData = Normal@NetExtract[net, "Input"];
  dec = NetExtract[net, "Output"];
  {mean, var} = Lookup[encData, {"MeanImage", "VarianceImage"}];
  prob = NetReplacePart[
     net, {"Input" -> 
       NetEncoder[{"Image", ImageDimensions@resized, 
         "MeanImage" -> mean, "VarianceImage" -> var}], 
      "Output" -> Automatic}][resized, TargetDevice -> device];
  prob = ArrayResample[prob, Append[Reverse@ImageDimensions@img, 150]];
  dec[prob]]

Then, we create a list of labels, one for each class that the neural network knows about. This is in the documentation for each Neural Network.

labels = {"wall", "building", "sky", "floor", "tree", "ceiling", 
   "road", "bed", "windowpane", "grass", "cabinet", "sidewalk", 
   "person", "earth", "door", "table", "mountain", "plant", "curtain",
    "chair", "car", "water", "painting", "sofa", "shelf", "house", 
   "sea", "mirror", "rug", "field", "armchair", "seat", "fence", 
   "desk", "rock", "wardrobe", "lamp", "bathtub", "railing", 
   "cushion", "base", "box", "column", "signboard", "chest", 
   "counter", "sand", "sink", "skyscraper", "fireplace", 
   "refrigerator", "grandstand", "path", "stairs", "runway", "case", 
   "pool", "pillow", "screen", "stairway", "river", "bridge", 
   "bookcase", "blind", "coffee", "toilet", "flower", "book", "hill", 
   "bench", "countertop", "stove", "palm", "kitchen", "computer", 
   "swivel", "boat", "bar", "arcade", "hovel", "bus", "towel", 
   "light", "truck", "tower", "chandelier", "awning", "streetlight", 
   "booth", "television", "airplane", "dirt", "apparel", "pole", 
   "land", "bannister", "escalator", "ottoman", "bottle", "buffet", 
   "poster", "stage", "van", "ship", "fountain", "conveyer", "canopy",
    "washer", "plaything", "swimming", "stool", "barrel", "basket", 
   "waterfall", "tent", "bag", "minibike", "cradle", "oven", "ball", 
   "food", "step", "tank", "trade", "microwave", "pot", "animal", 
   "bicycle", "lake", "dishwasher", "screen", "blanket", "sculpture", 
   "hood", "sconce", "vase", "traffic", "tray", "ashcan", "fan", 
   "pier", "crt", "plate", "monitor", "bulletin", "shower", 
   "radiator", "glass", "clock", "flag"};

Now we can create a mask that segments all the objects in the image (that is, all the objects the neural network is able to see).

You can visualise this mask easily by using Colorize, and you can easily get a list of all the items the neural network can see with labels[[DeleteDuplicates@Flatten@mask]].

Now that we have this mask, we can get rid of all the classes that aren't the object you're looking for - in this case, cars.

We first find the index of the object we're looking for in the labels.

index = Position[labels, "car"]

In the case of this network, it's 21.

Now we set all the pixels in the mask to be 0 unless they have the value 21. (This isn't great code, sorry!)

ReplaceAll[{21 -> 1, _ -> 0}] /@ # & /@ mask

Hope that helps.