2
$\begingroup$

Given this image (from Sculley et al "Hidden Technical Debt in Machine Learning Systems"), how can the relative areas of the boxes be computed?

In particular, the text interferes with most morphological and other operators:

enter image description here

ImageLines didn't work even with "Segments"->True options.

ImageCorners identifies most - though not all - corners of rectangles (desired) but also tags most areas of text even with large minimum spacing parameter, eg 20. 

corners = debt[0] // Binarize // ImageCorners[#, 2, 0, 20] &

enter image description here

The strategy, given only the box corners, would be to computationally fit them to rectangles and then solve for the areas.

Is there a more direct, feasible approach that does not require manual removal of text?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

ComponentMeasurements has a handy measurement EnclosingComponentCount that's 0 for the (outermost) boxes, and >= 1 for text inside the boxes:

img = Import["https://i.stack.imgur.com/5vrE2.png"];    
comp = ComponentMeasurements[
  ColorNegate[Binarize[img]], {"BoundingBox", 
   "EnclosingComponentCount"}, #EnclosingComponentCount == 0 &]

Which finds all boxes:

HighlightImage[img,
 comp /. (n_ -> {bbox_, ___}) :> Rectangle @@ bbox]

enter image description here

Response to comment:

what's the correct way of doing ImageTake based on the individual output box coordinates?

The easiest way is to use ImageTrim:

comp /. (n_ -> {bbox_, ___}) :> ImageTrim[img, bbox]

enter image description here

and you can use TextRecognize to read the text inside the box:

comp /. (n_ -> {bbox_, ___}) :> 
  Labeled[ImageTrim[img, bbox], TextRecognize[ImageTrim[img, bbox]]]

enter image description here

Improve this answer
$\endgroup$
5
  • $\begingroup$ Awesome, thanks. Incidentally, here are the area ratios: (comp /. (n_ -> {bbox_, ___}) :> Rectangle @@ bbox ) // Map[Area] // Sort // Query[{Identity, First} /* Apply[Divide]] . results (* {1., 6.375, 10.237, 10.6074, 10.8121, 11.1833, 13.2904, 13.7338, \ 17.1404, 27.0344} *) $\endgroup$
    – alancalvitti
    Mar 1, 2018 at 7:37
  • $\begingroup$ Would be killer if the rectangles could be associated to each text as label. $\endgroup$
    – alancalvitti
    Mar 1, 2018 at 7:38
  • $\begingroup$ what's the correct way of doing ImageTake based on the individual output box coordinates? I tried for example ImageTake[img, Sequence @@ ({{1232., 6.}, {1506., 568.}} // Transpose)] but got an out of range error. $\endgroup$
    – alancalvitti
    Mar 1, 2018 at 18:04
  • $\begingroup$ @alancalvitti: ImageTake doesn't use coordinates, it uses indices. Here's a good explanation: reference.wolfram.com/language/tutorial/… $\endgroup$
    – Niki Estner
    Mar 5, 2018 at 6:20
  • $\begingroup$ Using the coordinate xform in the tutorial, indexToImage[height_][{x_, y_}] := { height - y + 1/2, x + 1/2}; this image is 584px high, but when I try an example rectangle: r1 = rectangles[1] // Map[indexToImage[debt[0] // ImageDimensions // Last]], (where r1 is 1 -> {{1232., 6.}, {1506., 568.}}) and then ImageTake[img, Sequence @@ Transpose[Reverse[r1]]] the rectangle is returned but in reverse (ie the text is reversed). What's the appropriate way to package arguments here? $\endgroup$
    – alancalvitti
    Mar 6, 2018 at 3:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.