I have an image of a product on a poorly made green screen and need to segment out just the product:

enter image description here

The problem is that it contains a mirror, so simple color-based methods are not enough.

I tried playing with the function RemoveBackground using markers, but no luck. Here's what I tried so far:

RemoveBackground[img, {"Background", Green}]
RemoveBackground[img, {"Background", {"Uniform", 0.1}}]

enter image description here enter image description here


With python and opencv can do it easily using the Grabcut algorithm referenced in the comments, but I can't find the way to do it with MMA.

%matplotlib inline

import numpy as np
import cv2
import skimage
from matplotlib import pyplot as plt
img = cv2.imread(path_to_img)
print "img", img.shape

# resize
side = 600
ratio = float(side) / max(img.shape)
img = skimage.img_as_ubyte(
        img, (int(img.shape[0] * ratio), int(img.shape[1] * ratio))))

s = (img.shape[0] / 10, img.shape[1] / 10)
rect = (s[0], s[1], img.shape[0] - 2 * s[0], img.shape[1] - 2 * s[1])

mask = np.zeros(img.shape[:2],np.uint8)

bgdModel = np.zeros((1,65),np.float64)
fgdModel = np.zeros((1,65),np.float64)

mask2 = np.where((mask==2)|(mask==0),0,1).astype('uint8')
img = img*mask2[:,:,np.newaxis]


enter image description here

  • $\begingroup$ You may try something like colors = DominantColors@i; RemoveBackground[i, {"Background", {colors[[1]], 0.05}}] $\endgroup$
    – s.s.o
    Commented Dec 5, 2016 at 22:08
  • $\begingroup$ Also related ... mathematica.stackexchange.com/questions/9449/… $\endgroup$
    – s.s.o
    Commented Dec 5, 2016 at 22:09
  • $\begingroup$ @s.s.o that doesn't help with the problem of the mirror... $\endgroup$
    – M.R.
    Commented Dec 5, 2016 at 22:56
  • $\begingroup$ Does mathematica have the grabcut algorithm implemented? I could do this with opencv pretty easy: docs.opencv.org/3.1.0/d8/d83/tutorial_py_grabcut.html $\endgroup$
    – M.R.
    Commented Dec 6, 2016 at 2:31
  • 1
    $\begingroup$ Almost: mask = FillingTransform@ DeleteBorderComponents@ DeleteSmallComponents@ColorNegate@ContourDetect[#, 0.4] &@ ImageAdjust@img and ImageMultiply[mask, img]. If only I could enclose the region of interest in mask :( $\endgroup$
    – corey979
    Commented Dec 6, 2016 at 10:42

3 Answers 3


1.This method is from documentation of function ClusterClassify

image = Import["https://i.sstatic.net/zP5xF.jpg"];
imageData = Flatten[ImageData[ColorConvert[image, "LAB"]], 1];
c = ClusterClassify[imageData, 4, Method -> "KMedoids"];
decision = c[imageData];
mask = Image /@ 
     Partition[decision, First@ImageDimensions[image]]}, "Mask"][[All,

enter image description here

allMask = FillingTransform[Dilation[ColorNegate[mask[[4]]], 1]];
SetAlphaChannel[image, Blur[allMask, 8]]

enter image description here

2.Based on machine learning

Method one,Classify the pixel by chain a nerve

I have to say this is worthless method in real life,because it is very very very low efficiency(Maybe when you have a CUDA feature GPU, it will be more faster).I don't remember how long I have run it.Well,Just for fun.

First we select a range that you need,which just is a selection roughly that mean you can include some singular point in your trained data.Of course you can make yourself trained data.This is what I select that arbitrarily

Then define a net and train it

image = Import["https://i.sstatic.net/zP5xF.jpg"];
trainData = Join[Thread[Rule[no, False]], Thread[Rule[yes, True]]];
net = NetChain[{20, Tanh, 2, 
    SoftmaxLayer["Output" -> NetDecoder[{"Class", {True, False}}]]}, 
   "Input" -> 3];
ringQ = NetTrain[net, trainData, MaxTrainingRounds -> 20]

Be patient and wait some minutes,then you can get your ring.The final effect is depened on your training data and some luck.


We can use my above method to refine it in following step.

Method two,use the built-in function of Classify

This method is not bad as the result effect,but actually I will not tell you this code cost my one night to run,which mean this method is slower than that NetChain. Firstly,make some sample data

match = Classify[<|False -> Catenate[ImageData[no]], 
    True -> Catenate[ImageData[yes]]|>];
ImageApply[If[match[#], #, {1, 1, 1}] &, image]

Be more patient please,after just one night,the result will show you.like this:

3.Above answer for another motivation or just fun,but in this part,I will post some method for image-processing

image = Import["https://i.sstatic.net/zP5xF.jpg"];

Method one

    FillingTransform[Binarize[GradientFilter[image, 1], 0.035]]], 10],

Method two

   Image[WatershedComponents[GradientFilter[image, 2], 
      Method -> {"MinimumSaliency", 0.2}] - 1]], 5]]

Method three

     First[ColorSeparate[ColorConvert[image, "CMYK"]]]], {.6, .93}]], 

Last but not least,this method do some principal component decomposition of color channels,which can face more situation commonly

  ColorCombine /@ Tuples[ColorSeparate[image], {3}]]]

Note that picture from 2 to 5,every picture have more strong contrast then origin.Than we can use fist three method do next step.

  • $\begingroup$ our are missing a HUGE chuck of the mirror. $\endgroup$
    – M.R.
    Commented Dec 8, 2016 at 3:46
  • $\begingroup$ Great answer. Methods 1 and 3 seem to work fine. $\endgroup$
    – corey979
    Commented Dec 15, 2016 at 11:19
  • $\begingroup$ @corey979 Thanks. :) $\endgroup$
    – yode
    Commented Dec 15, 2016 at 11:22

Here's a method that could be iterated and refined to replicate the opencv result, I think.

First we use the ClusterClassify method of yobe then we simply fill in the holes by generating a mask that gets the frame we need and combine this into a single mask.

First the boiler plate:

img = Import["https://i.sstatic.net/zP5xF.jpg"];

clusterGet[image_] :=
  Module[{imageData, c, decision},
   imageData = Flatten[ImageData[ColorConvert[image, "LAB"]], 1]; 
   c = ClusterClassify[imageData, 4, Method -> "KMedoids"]; 
   decision = c[imageData]; 
   Image /@ 
       Partition[decision, First@ImageDimensions[img]]}, "Mask"][[All,

maskCombine[{base_, others__}] :=

  Block[{root = base, 
    alphas = SetAlphaChannel[#, ColorNegate@#] & /@ {others}},
   Do[root = ImageCompose[base, a], {a, alphas}];

then figure out which mask we want:

baseMask = clusterGet[img][[4]]

base mask

then we need to create a filling mask for that:

fillingMask = Closing[
    MeanShiftFilter[ImageAdjust[Lighter@img, 2], 1, .01, 
     MaxIterations -> 5],

filling mask

then set the composite mask as the overal image:

 ColorNegate@maskCombine@{baseMask, fillingMask}]


Using more sophisticated filters I've been able to build a better filling mask that minimizes the amount of lost green space/frame but I can't remember exactly which set of filters I combined. For those looking to extend this, the edge-preserving filters such as PeronaMalikFilter appear to be the place to start. There are tons of filters to apply, so I'm sure trial and error can give you the results you want.

You could also use Java and opencv, doing more or less what Leonid Shifrin does here or write your own simple boundary detection code. I did some of the latter, but it's generally just too slow to be properly workable and figuring out the appropriate pixel distance function is, again, a matter of trial and error.

  • $\begingroup$ This result is great, but may be it can be further improved by some DeleteSmallComponents or come edge refining? $\endgroup$
    – Wjx
    Commented Dec 11, 2016 at 11:47
  • $\begingroup$ Funny,I solve the problem,then you get the bounty. :) $\endgroup$
    – yode
    Commented Dec 18, 2016 at 20:44
  • $\begingroup$ @yobe I have literally no idea why, honestly. $\endgroup$
    – b3m2a1
    Commented Dec 18, 2016 at 22:08
  • $\begingroup$ @yode I was on vacation, so sorry, I don't know how why you didn't get the bounty $\endgroup$
    – M.R.
    Commented Dec 19, 2016 at 22:19

With helpful ideas of KAI:

img = Import["https://i.sstatic.net/zP5xF.jpg"]

mask = FillingTransform@
       ContourDetect[#, 0.4]&@
        ImageAdjust @ img

back = MorphologicalPerimeter@
        Dilation[Closing[mask, DiskMatrix[15]], DiskMatrix[2]]

c = Binarize @ Colorize @ GrowCutComponents[img, {mask, back}]

ImageMultiply[img, c] // RemoveBackground // ImageCrop

enter image description here

Not a complete solution, but maybe could serve as a starting point.

  • $\begingroup$ Thanks but I really need it to be a complete object without any missing parts - just like how python does it. $\endgroup$
    – M.R.
    Commented Dec 7, 2016 at 16:31
  • $\begingroup$ @M.R. virtually all of the functions have options (e.g. check out the Method options of Binarize for example). So does DeleteSmallComponents. Why not try those yourself? $\endgroup$ Commented Dec 7, 2016 at 16:51
  • $\begingroup$ @blochwave I was hoping the underlying algorithm choice for grabcut would be accessible $\endgroup$
    – M.R.
    Commented Dec 7, 2016 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.