# Image Segmentation

I have what should be a pretty simple question for Mathematica gurus. I have a set of images containing cars and a background. I am trying to create a black and white image where the white part is the car and the background is black. I am linking a sample input image and what the output should be. Thanks in advance.

The Neural Net Repository has several useful Neural Nets and examples of exactly this type of problem.

Using one of the Neural Networks is quite easy. The first time you evaluate a NetModel, it downloads a fairly large amount of data from the internet. I'll use the Ademxapp Model A1 Trained on ADE20K Data since that's one I already have downloaded.

First, we set up a function to deal with reshaping the input and output. Basically, the network has been trained on images of a certain size and type, so this function changes whatever image you give it into the size and type the network expects.

netevaluate[img_, device_: "CPU"] :=
Block[{net, resized, encData, dec, mean, var, prob},
resized = ImageResize[img, {504}];
encData = Normal@NetExtract[net, "Input"];
dec = NetExtract[net, "Output"];
{mean, var} = Lookup[encData, {"MeanImage", "VarianceImage"}];
prob = NetReplacePart[
net, {"Input" ->
NetEncoder[{"Image", ImageDimensions@resized,
"MeanImage" -> mean, "VarianceImage" -> var}],
"Output" -> Automatic}][resized, TargetDevice -> device];
prob = ArrayResample[prob, Append[Reverse@ImageDimensions@img, 150]];
dec[prob]]


Then, we create a list of labels, one for each class that the neural network knows about. This is in the documentation for each Neural Network.

labels = {"wall", "building", "sky", "floor", "tree", "ceiling",
"road", "bed", "windowpane", "grass", "cabinet", "sidewalk",
"person", "earth", "door", "table", "mountain", "plant", "curtain",
"chair", "car", "water", "painting", "sofa", "shelf", "house",
"sea", "mirror", "rug", "field", "armchair", "seat", "fence",
"desk", "rock", "wardrobe", "lamp", "bathtub", "railing",
"cushion", "base", "box", "column", "signboard", "chest",
"counter", "sand", "sink", "skyscraper", "fireplace",
"refrigerator", "grandstand", "path", "stairs", "runway", "case",
"pool", "pillow", "screen", "stairway", "river", "bridge",
"bookcase", "blind", "coffee", "toilet", "flower", "book", "hill",
"bench", "countertop", "stove", "palm", "kitchen", "computer",
"swivel", "boat", "bar", "arcade", "hovel", "bus", "towel",
"light", "truck", "tower", "chandelier", "awning", "streetlight",
"booth", "television", "airplane", "dirt", "apparel", "pole",
"land", "bannister", "escalator", "ottoman", "bottle", "buffet",
"poster", "stage", "van", "ship", "fountain", "conveyer", "canopy",
"washer", "plaything", "swimming", "stool", "barrel", "basket",
"waterfall", "tent", "bag", "minibike", "cradle", "oven", "ball",
"food", "step", "tank", "trade", "microwave", "pot", "animal",
"bicycle", "lake", "dishwasher", "screen", "blanket", "sculpture",
"hood", "sconce", "vase", "traffic", "tray", "ashcan", "fan",
"pier", "crt", "plate", "monitor", "bulletin", "shower",


Now we can create a mask that segments all the objects in the image (that is, all the objects the neural network is able to see).

You can visualise this mask easily by using Colorize, and you can easily get a list of all the items the neural network can see with labels[[DeleteDuplicates@Flatten@mask]].

Now that we have this mask, we can get rid of all the classes that aren't the object you're looking for - in this case, cars.

We first find the index of the object we're looking for in the labels.

index = Position[labels, "car"]


In the case of this network, it's 21.

Now we set all the pixels in the mask to be 0 unless they have the value 21. (This isn't great code, sorry!)

ReplaceAll[{21 -> 1, _ -> 0}] /@ # & /@ mask


Hope that helps.

• Wow, this is great. Thank you so much. I will mark this question as answered. – Alberto Artasanchez Apr 21 '18 at 19:41
• Just one quick correction, if you want to edit your answer. I think the last step should be: Colorize[ReplaceAll[{21 -> 1, _ -> 0}] /@ # & /@ mask] – Alberto Artasanchez Apr 21 '18 at 20:14
• Colorize will display an image in more or less the same way that Image will, if mask is a list of zeroes or ones, so either would be correct :) In any case, mask can be used in many ways, not just for display - for instance you could make a word cloud of different translations for "car" in the shape of the mask :) Check the Neural Net Repository for many other examples of how to do this - your mileage will vary with different nets. – Carl Lange Apr 21 '18 at 21:22