# Ignoring text to compute regions in image?

Given this image (from Sculley et al "Hidden Technical Debt in Machine Learning Systems"), how can the relative areas of the boxes be computed?

In particular, the text interferes with most morphological and other operators:

ImageLines didn't work even with "Segments"->True options.

ImageCorners identifies most - though not all - corners of rectangles (desired) but also tags most areas of text even with large minimum spacing parameter, eg 20.

corners = debt[0] // Binarize // ImageCorners[#, 2, 0, 20] &


The strategy, given only the box corners, would be to computationally fit them to rectangles and then solve for the areas.

Is there a more direct, feasible approach that does not require manual removal of text?

ComponentMeasurements has a handy measurement EnclosingComponentCount that's 0 for the (outermost) boxes, and >= 1 for text inside the boxes:

img = Import["https://i.stack.imgur.com/5vrE2.png"];
comp = ComponentMeasurements[
ColorNegate[Binarize[img]], {"BoundingBox",
"EnclosingComponentCount"}, #EnclosingComponentCount == 0 &]


Which finds all boxes:

HighlightImage[img,
comp /. (n_ -> {bbox_, ___}) :> Rectangle @@ bbox]


# Response to comment:

what's the correct way of doing ImageTake based on the individual output box coordinates?

The easiest way is to use ImageTrim:

comp /. (n_ -> {bbox_, ___}) :> ImageTrim[img, bbox]


and you can use TextRecognize to read the text inside the box:

comp /. (n_ -> {bbox_, ___}) :>
Labeled[ImageTrim[img, bbox], TextRecognize[ImageTrim[img, bbox]]]


• Awesome, thanks. Incidentally, here are the area ratios: (comp /. (n_ -> {bbox_, ___}) :> Rectangle @@ bbox ) // Map[Area] // Sort // Query[{Identity, First} /* Apply[Divide]] . results (* {1., 6.375, 10.237, 10.6074, 10.8121, 11.1833, 13.2904, 13.7338, \ 17.1404, 27.0344} *) – alancalvitti Mar 1 '18 at 7:37
• Would be killer if the rectangles could be associated to each text as label. – alancalvitti Mar 1 '18 at 7:38
• what's the correct way of doing ImageTake based on the individual output box coordinates? I tried for example ImageTake[img, Sequence @@ ({{1232., 6.}, {1506., 568.}} // Transpose)] but got an out of range error. – alancalvitti Mar 1 '18 at 18:04
• @alancalvitti: ImageTake doesn't use coordinates, it uses indices. Here's a good explanation: reference.wolfram.com/language/tutorial/… – Niki Estner Mar 5 '18 at 6:20
• Using the coordinate xform in the tutorial, indexToImage[height_][{x_, y_}] := { height - y + 1/2, x + 1/2}; this image is 584px high, but when I try an example rectangle: r1 = rectangles[1] // Map[indexToImage[debt[0] // ImageDimensions // Last]], (where r1 is 1 -> {{1232., 6.}, {1506., 568.}}) and then ImageTake[img, Sequence @@ Transpose[Reverse[r1]]] the rectangle is returned but in reverse (ie the text is reversed). What's the appropriate way to package arguments here? – alancalvitti Mar 6 '18 at 3:21