How to detect rectangles with certain colors in an image?

Question

I'd like to find the red bordered rectangles in this image:

Here are my example images:

examples = CloudGet @ "https://www.wolframcloud.com/obj/cb4a73b7-5108-4c46-985c-769aa1eef3aa";

It’s tricky because they can be nested. There are at least seven of them in the above image, my current code only detects one of them so far:

img = examples[[3]];
mf1 = MeanFilter[img, 5];
ht1 = HistogramTransform[mf1];
i = SelectComponents[SelectComponents[Binarize[ht1, .2], Small], Large];
bb = Last /@ ComponentMeasurements[i, "BoundingBox"] // First;
bbc = Mean /@ bb;
Show[i, Graphics[{Red, Opacity[0.5], Rectangle[bb[[1]], bb[[2]]]}]]

Related Questions:

• How can I detect a rectangular region of interest in a picture?
• Detecting rectangle shape from an image
• How can I capture a rectangular area from an image?

Answer 1

Here's my take at the issue (some comments are inlined)

extractRectangles[i_Image?ImageQ] :=
Block[
    {dims, i0, i1, masks1, masks2, masks3, tfun, coords},
    (* downsampling (just to make it faster) *) 
    i0 = ImageResize[RemoveAlphaChannel[i], 500];
    dims = ImageDimensions[i0];
    (* smoothing *)
    i1 = MeanShiftFilter[i0, 3, .05, MaxIterations->5];
    (* color based segmentation *)
    masks1 = DominantColors[i1, All, "CoverageImage",
        MinColorDistance -> .02,
        ColorCoverage -> .02
    ];
    (* mask cleaning *)
    masks2 = FillingTransform @ Opening[#,2] & /@ masks1;
    (* mask selection *)
    masks3 = Flatten[Map[
        Values@ComponentMeasurements[
            #, "BoundingBox",
            And[#Rectangularity > .5, #Area > .005 Times @@ dims] &
        ]&, masks2], 1];

    (* scaling to the original image size *)
    tfun = ScalingTransform[ImageDimensions[i] / dims];
    coords = tfun /@ masks3;
    (* rectangles *)
    Rectangle @@@ coords
]

Compared to your attempt I used a MeanShiftFilter in place of the MeanFilter to better preserve the hard edges and DominantColors to get a more controlled colour segmentation.

Once that's done I want to have just one more utility to get consistent colouring for the rectangles

exprColor[expr_] := 
 RGBColor["#" <> IntegerString[Hash[Unevaluated[expr]], 16, 6]]

And now this is the result.

res = extractRectangles /@ examples;

MapThread[
 Function[{image, rectangles}, 
  HighlightImage[
   image, {{"Boundary", 10}, exprColor[#], #} & /@ rectangles, 
   ImageSize -> 400]
  ],
 {examples, res}
 ] // Column

The main issues I have still are:

• it gets thrown off by the text and thus some rectangles are smaller than they should be
• it does not deals very well with nested rectangles
• it has no notion of overlapping rectangles

Unfortunately, I don't have much time to dig into this more in depth.

Cheers!