2
$\begingroup$

Given this image (from Sculley et al "Hidden Technical Debt in Machine Learning Systems"), how can the relative areas of the boxes be computed?

In particular, the text interferes with most morphological and other operators:

enter image description here

ImageLines didn't work even with "Segments"->True options.

ImageCorners identifies most - though not all - corners of rectangles (desired) but also tags most areas of text even with large minimum spacing parameter, eg 20.

corners = debt[0] // Binarize // ImageCorners[#, 2, 0, 20] &

enter image description here

The strategy, given only the box corners, would be to computationally fit them to rectangles and then solve for the areas.

Is there a more direct, feasible approach that does not require manual removal of text?

| improve this question | | | | |
$\endgroup$
6
$\begingroup$

ComponentMeasurements has a handy measurement EnclosingComponentCount that's 0 for the (outermost) boxes, and >= 1 for text inside the boxes:

img = Import["https://i.stack.imgur.com/5vrE2.png"];    
comp = ComponentMeasurements[
  ColorNegate[Binarize[img]], {"BoundingBox", 
   "EnclosingComponentCount"}, #EnclosingComponentCount == 0 &]

Which finds all boxes:

HighlightImage[img,
 comp /. (n_ -> {bbox_, ___}) :> Rectangle @@ bbox]

enter image description here

Response to comment:

what's the correct way of doing ImageTake based on the individual output box coordinates?

The easiest way is to use ImageTrim:

comp /. (n_ -> {bbox_, ___}) :> ImageTrim[img, bbox]

enter image description here

and you can use TextRecognize to read the text inside the box:

comp /. (n_ -> {bbox_, ___}) :> 
  Labeled[ImageTrim[img, bbox], TextRecognize[ImageTrim[img, bbox]]]

enter image description here

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Awesome, thanks. Incidentally, here are the area ratios: (comp /. (n_ -> {bbox_, ___}) :> Rectangle @@ bbox ) // Map[Area] // Sort // Query[{Identity, First} /* Apply[Divide]] . results (* {1., 6.375, 10.237, 10.6074, 10.8121, 11.1833, 13.2904, 13.7338, \ 17.1404, 27.0344} *) $\endgroup$ – alancalvitti Mar 1 '18 at 7:37
  • $\begingroup$ Would be killer if the rectangles could be associated to each text as label. $\endgroup$ – alancalvitti Mar 1 '18 at 7:38
  • $\begingroup$ what's the correct way of doing ImageTake based on the individual output box coordinates? I tried for example ImageTake[img, Sequence @@ ({{1232., 6.}, {1506., 568.}} // Transpose)] but got an out of range error. $\endgroup$ – alancalvitti Mar 1 '18 at 18:04
  • $\begingroup$ @alancalvitti: ImageTake doesn't use coordinates, it uses indices. Here's a good explanation: reference.wolfram.com/language/tutorial/… $\endgroup$ – Niki Estner Mar 5 '18 at 6:20
  • $\begingroup$ Using the coordinate xform in the tutorial, indexToImage[height_][{x_, y_}] := { height - y + 1/2, x + 1/2}; this image is 584px high, but when I try an example rectangle: r1 = rectangles[1] // Map[indexToImage[debt[0] // ImageDimensions // Last]], (where r1 is 1 -> {{1232., 6.}, {1506., 568.}}) and then ImageTake[img, Sequence @@ Transpose[Reverse[r1]]] the rectangle is returned but in reverse (ie the text is reversed). What's the appropriate way to package arguments here? $\endgroup$ – alancalvitti Mar 6 '18 at 3:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.