I have an image of a product on a poorly made green screen and need to segment out just the product:

enter image description here

The problem is that it contains a mirror, so simple color-based methods are not enough.

I tried playing with the function RemoveBackground using markers, but no luck. Here's what I tried so far:

RemoveBackground[img, {"Background", Green}]
RemoveBackground[img, {"Background", {"Uniform", 0.1}}]

enter image description here enter image description here


With python and opencv can do it easily using the Grabcut algorithm referenced in the comments, but I can't find the way to do it with MMA.

%matplotlib inline

import numpy as np
import cv2
import skimage
from matplotlib import pyplot as plt
img = cv2.imread(path_to_img)
print "img", img.shape

# resize
side = 600
ratio = float(side) / max(img.shape)
img = skimage.img_as_ubyte(
        img, (int(img.shape[0] * ratio), int(img.shape[1] * ratio))))

s = (img.shape[0] / 10, img.shape[1] / 10)
rect = (s[0], s[1], img.shape[0] - 2 * s[0], img.shape[1] - 2 * s[1])

mask = np.zeros(img.shape[:2],np.uint8)

bgdModel = np.zeros((1,65),np.float64)
fgdModel = np.zeros((1,65),np.float64)

mask2 = np.where((mask==2)|(mask==0),0,1).astype('uint8')
img = img*mask2[:,:,np.newaxis]


enter image description here

  • $\begingroup$ You may try something like colors = DominantColors@i; RemoveBackground[i, {"Background", {colors[[1]], 0.05}}] $\endgroup$ – s.s.o Dec 5 '16 at 22:08
  • $\begingroup$ Also related ... mathematica.stackexchange.com/questions/9449/… $\endgroup$ – s.s.o Dec 5 '16 at 22:09
  • $\begingroup$ @s.s.o that doesn't help with the problem of the mirror... $\endgroup$ – M.R. Dec 5 '16 at 22:56
  • $\begingroup$ Does mathematica have the grabcut algorithm implemented? I could do this with opencv pretty easy: docs.opencv.org/3.1.0/d8/d83/tutorial_py_grabcut.html $\endgroup$ – M.R. Dec 6 '16 at 2:31
  • 1
    $\begingroup$ Almost: mask = FillingTransform@ DeleteBorderComponents@ DeleteSmallComponents@ColorNegate@ContourDetect[#, 0.4] &@ ImageAdjust@img and ImageMultiply[mask, img]. If only I could enclose the region of interest in mask :( $\endgroup$ – corey979 Dec 6 '16 at 10:42

1.This method is from documentation of function ClusterClassify

image = Import["https://i.stack.imgur.com/zP5xF.jpg"];
imageData = Flatten[ImageData[ColorConvert[image, "LAB"]], 1];
c = ClusterClassify[imageData, 4, Method -> "KMedoids"];
decision = c[imageData];
mask = Image /@ 
     Partition[decision, First@ImageDimensions[image]]}, "Mask"][[All,

enter image description here

allMask = FillingTransform[Dilation[ColorNegate[mask[[4]]], 1]];
SetAlphaChannel[image, Blur[allMask, 8]]

enter image description here

2.Based on machine learning

Method one,Classify the pixel by chain a nerve

I have to say this is worthless method in real life,because it is very very very low efficiency(Maybe when you have a CUDA feature GPU, it will be more faster).I don't remember how long I have run it.Well,Just for fun.

First we select a range that you need,which just is a selection roughly that mean you can include some singular point in your trained data.Of course you can make yourself trained data.This is what I select that arbitrarily

Then define a net and train it

image = Import["https://i.stack.imgur.com/zP5xF.jpg"];
trainData = Join[Thread[Rule[no, False]], Thread[Rule[yes, True]]];
net = NetChain[{20, Tanh, 2, 
    SoftmaxLayer["Output" -> NetDecoder[{"Class", {True, False}}]]}, 
   "Input" -> 3];
ringQ = NetTrain[net, trainData, MaxTrainingRounds -> 20]

Be patient and wait some minutes,then you can get your ring.The final effect is depened on your training data and some luck.


We can use my above method to refine it in following step.

Method two,use the built-in function of Classify

This method is not bad as the result effect,but actually I will not tell you this code cost my one night to run,which mean this method is slower than that NetChain. Firstly,make some sample data

match = Classify[<|False -> Catenate[ImageData[no]], 
    True -> Catenate[ImageData[yes]]|>];
ImageApply[If[match[#], #, {1, 1, 1}] &, image]

Be more patient please,after just one night,the result will show you.like this:

3.Above answer for another motivation or just fun,but in this part,I will post some method for image-processing

image = Import["https://i.stack.imgur.com/zP5xF.jpg"];

Method one

    FillingTransform[Binarize[GradientFilter[image, 1], 0.035]]], 10],

Method two

   Image[WatershedComponents[GradientFilter[image, 2], 
      Method -> {"MinimumSaliency", 0.2}] - 1]], 5]]

Method three

     First[ColorSeparate[ColorConvert[image, "CMYK"]]]], {.6, .93}]], 

Last but not least,this method do some principal component decomposition of color channels,which can face more situation commonly

  ColorCombine /@ Tuples[ColorSeparate[image], {3}]]]

Note that picture from 2 to 5,every picture have more strong contrast then origin.Than we can use fist three method do next step.

| improve this answer | |
  • $\begingroup$ our are missing a HUGE chuck of the mirror. $\endgroup$ – M.R. Dec 8 '16 at 3:46
  • $\begingroup$ Great answer. Methods 1 and 3 seem to work fine. $\endgroup$ – corey979 Dec 15 '16 at 11:19
  • $\begingroup$ @corey979 Thanks. :) $\endgroup$ – yode Dec 15 '16 at 11:22

Here's a method that could be iterated and refined to replicate the opencv result, I think.

First we use the ClusterClassify method of yobe then we simply fill in the holes by generating a mask that gets the frame we need and combine this into a single mask.

First the boiler plate:

img = Import["https://i.stack.imgur.com/zP5xF.jpg"];

clusterGet[image_] :=
  Module[{imageData, c, decision},
   imageData = Flatten[ImageData[ColorConvert[image, "LAB"]], 1]; 
   c = ClusterClassify[imageData, 4, Method -> "KMedoids"]; 
   decision = c[imageData]; 
   Image /@ 
       Partition[decision, First@ImageDimensions[img]]}, "Mask"][[All,

maskCombine[{base_, others__}] :=

  Block[{root = base, 
    alphas = SetAlphaChannel[#, ColorNegate@#] & /@ {others}},
   Do[root = ImageCompose[base, a], {a, alphas}];

then figure out which mask we want:

baseMask = clusterGet[img][[4]]

base mask

then we need to create a filling mask for that:

fillingMask = Closing[
    MeanShiftFilter[ImageAdjust[Lighter@img, 2], 1, .01, 
     MaxIterations -> 5],

filling mask

then set the composite mask as the overal image:

 ColorNegate@maskCombine@{baseMask, fillingMask}]


Using more sophisticated filters I've been able to build a better filling mask that minimizes the amount of lost green space/frame but I can't remember exactly which set of filters I combined. For those looking to extend this, the edge-preserving filters such as PeronaMalikFilter appear to be the place to start. There are tons of filters to apply, so I'm sure trial and error can give you the results you want.

You could also use Java and opencv, doing more or less what Leonid Shifrin does here or write your own simple boundary detection code. I did some of the latter, but it's generally just too slow to be properly workable and figuring out the appropriate pixel distance function is, again, a matter of trial and error.

| improve this answer | |
  • $\begingroup$ This result is great, but may be it can be further improved by some DeleteSmallComponents or come edge refining? $\endgroup$ – Wjx Dec 11 '16 at 11:47
  • $\begingroup$ Funny,I solve the problem,then you get the bounty. :) $\endgroup$ – yode Dec 18 '16 at 20:44
  • $\begingroup$ @yobe I have literally no idea why, honestly. $\endgroup$ – b3m2a1 Dec 18 '16 at 22:08
  • $\begingroup$ @yode I was on vacation, so sorry, I don't know how why you didn't get the bounty $\endgroup$ – M.R. Dec 19 '16 at 22:19

With helpful ideas of KAI:

img = Import["https://i.stack.imgur.com/zP5xF.jpg"]

mask = FillingTransform@
       ContourDetect[#, 0.4]&@
        ImageAdjust @ img

back = MorphologicalPerimeter@
        Dilation[Closing[mask, DiskMatrix[15]], DiskMatrix[2]]

c = Binarize @ Colorize @ GrowCutComponents[img, {mask, back}]

ImageMultiply[img, c] // RemoveBackground // ImageCrop

enter image description here

Not a complete solution, but maybe could serve as a starting point.

| improve this answer | |
  • $\begingroup$ Thanks but I really need it to be a complete object without any missing parts - just like how python does it. $\endgroup$ – M.R. Dec 7 '16 at 16:31
  • $\begingroup$ @M.R. virtually all of the functions have options (e.g. check out the Method options of Binarize for example). So does DeleteSmallComponents. Why not try those yourself? $\endgroup$ – dr.blochwave Dec 7 '16 at 16:51
  • $\begingroup$ @blochwave I was hoping the underlying algorithm choice for grabcut would be accessible $\endgroup$ – M.R. Dec 7 '16 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.