# How to detect and split sections of an image

I'm new to Mathematica and I was trying to split the following image into horizontal sections, just above every square box. I have done some pre-processing but I can't figure out how to "locate" the boxes and split them at their topmost boundaries, until the next box's topmost boundary.

EDIT - 2 ( Better info about previous edits can be gleamed from the accepted answer )

This image has the proper text segment I wish to extract, I just need to create a corpus with the sentences with the corresponding boxes, which I'm choosing as a guide for comp. vision.

I need to chunk it to components segments

1. The sentence lengths and thus their spacing on a page is variable.
2. There are variable number of such sentence groups in a page. ( As shown in Full Page )

Could you help me out?

### Update 2:

We'll work this case by first finding the most common left-most pixel. Note that this won't work for the previous update, as that vertical separator will be it. However it should be possible to detect and strip that vertical separator.

img = Import["https://i.stack.imgur.com/APXIy.png"];

norgba = ColorCombine@Take[ColorSeparate@img, UpTo[3]];
binarized = Binarize@norgba;
data = ImageData@binarized;

leftmosts =
FirstPosition[#, 0, Nothing] & /@ data // Counts // ReverseSort //
KeyMap[First];

blockstart = First@Keys@leftmosts;


Then we find the maximum contiguous block of black pixels (in our binarized image) starting from there.

contiguous =
Split[
Position[#, 0, {1}] & /@ data // Flatten // Sort //
DeleteDuplicates,
Abs[# - #2] == 1 &
];

pixelBlock = SelectFirst[contiguous, MemberQ[blockstart]];


Then we'll apply the same row splitting algorithm, but this time allowing much longer stretches of white pixels (so that the wrapped lines are captured). Then we'll find the same set of rows defined by these blocks, then select out those that don't have a marker:

splits =
Block[{block = False, counts = 0, countmax = 50},
SplitBy[data, (If[Length@Counts[#] > 1, block = True;
counts = 0;, counts++;
block = (block && counts <= countmax)];
block) &]];

rowsRaw =
Partition[Riffle[Most@#, Rest@#] &@Accumulate[Length /@ splits], 2];

rows =
Select[
rowsRaw,
MemberQ[
Take[data, #][[All, pixelBlock]] // Flatten,
0
] &
];


Finally we take from the maximum right position of a marker and crop out whitespace:

fullcolordata = ImageData@img;

chunks =
ImageCrop@Image@
Take[fullcolordata, #][[All, Last@pixelBlock + 1 ;; -1]] & /@
rows;

chunks


### Update

This is for the updated version of the question.

First we need to get an appropriate image to work on. First we'll binarize and then take only the chunk to the left of the vertical divider and below the horizontal one:

img = Import@"https://i.stack.imgur.com/s8eZx.png";

norgba = ColorCombine@Most@ColorSeparate@img;

binarized = Binarize@norgba;
data = ImageData@binarized;
flipped = Transpose@data;
rowseparator =
Max@
Position[
data,
Alternatives @@ MinimalBy[data, Total],
{1}
];
rowdropped =
Take[data, {
rowseparator + 5 (* a fudge factor could be much bigger, too*),
-1}];
columnseparator =
Min@
Position[
flipped,
Alternatives @@ MinimalBy[flipped, Total],
{1}
];

chopped =
Transpose@Take[Transpose@rowdropped, columnseparator - 1];

Take[chopped, 1000] // Image


Then we split as before:

splits =
Block[{block = False, counts = 0, countmax = 3},
SplitBy[chopped,
(If[Length@Counts[#] > 1, block = True;
counts = 0;, counts++;
block = (block && counts <= countmax)];
block) &]];


Then we need the row spans each of the chunks is in (rows) and the vertical range that the indicator is in (subblocks):

rows =
Partition[
Riffle[Most@#,
Rest@#] &@
Accumulate[
Length /@ splits
],
2
];

subblocks =
If[Length@# > 0,
{0, 10} + Through[{Min, Max}[Keys@#]],
None] &@
(Function[Through[{Min, Max}[#]]] /@

GroupBy[First -> Last]@Position[#, 0]
) & /@ splits;


Then we thread over these together, first taking the row range, then taking the subrange:

fullcolordata = ImageData@img;

chunks =
Take[
Take[fullcolordata, rowseparator + #],
#2
] &, {
rows,
DeleteCases[subblocks, None]
}];


And then we view:

ImageCrop@Image[#, ColorSpace -> "RGB"] & /@ chunks


And if you just want the line text you can use this:

fullcolorrowchopped =
Transpose@
Take[
Transpose@Take[fullcolordata, {rowseparator + 5, -1}],
{columnseparator + 5, -1}
];

fchopped =
Select[
Take[
Take[fullcolorrowchopped, #],
#2
] &, {
rows,
DeleteCases[subblocks, None]
}],
Length@Counts[#] > 1 &
];

ImageCrop@Image[#, ColorSpace -> "RGB"] & /@ fchopped


### Original

So we can do this by looking at the pixel values and just splitting where we see whitespace.

First get the image and its pixel rows:

img = Import["https://i.stack.imgur.com/TPX7R.png"];
data = ImageData@img;


Then we'll use a procedural style loop to chunk our data by the following criterion:

Either 1. There's more than a single color pixel in it

Or 1. Up to n rows previous there was more than a single pixel color

Here's my loop. It's not optimized and a pretty lazy, off the top of my head way to do it, but hopefully it's good enough:

splits =
Block[{
block = False,
counts = 0,
countmax = 3
},
SplitBy[data,
(
If[Length@Counts[#] > 1,
block = True;
counts = 0;,
counts++;
block = (block && counts <= countmax)
];
block
) &
]
];


Then we'll crop whitespace and format images:

chunks = ImageCrop@*Image /@ splits;


And then delete those images that are just whitespace:

imgs =
Select[chunks,
Length@Counts[Flatten[ImageData[#], 1]] > 1 &
];


And let's see how we did:

imgs[[;; 10]]


There's probably a cleaner way to do this, but this was the first thing that came to mind.

• This is a great solution to the problem - I'll accept it:) But sadly, I mis-framed my question as I realize now. The thing is I need to split the images in the page image above. If possible, could you help me find a solution to the updated question above. – Abhinav Sharma Jun 3 '17 at 7:32
• @AbhinavSharma No need to accept so soon. Someone might come along with something better. You mean just split at the 175, 176, etc? I'll give that a bit of thought. It's a harder thing to do -- although presumably one of the many machine learning experts on this site will have something. – b3m2a1 Jun 3 '17 at 7:35
• @AbhinavSharma will all of your pages look like that? (i.e. the number bar will be thin and everything else will be thick?) If so I can do this in a way similar to what I did here – b3m2a1 Jun 3 '17 at 7:40
• Nope, actually the only thing common to the corpus is that there are big boxes, signifying the script name next to the text. The number line etc are not there in most other texts. – Abhinav Sharma Jun 3 '17 at 7:42
• @AbhinavSharma OHHH. (Just saw your update) This one I can do. Give me a moment. – b3m2a1 Jun 3 '17 at 7:44

To extract the sentences, you could use TextRecognize if the text is what you really want.

In:

img = Import["https://i.stack.imgur.com/s8eZx.png"] // Rasterize;
ts = Block[{\$Language = #},
TextRecognize[img]] & /@ {"ChineseTraditional", "Chinese"};
ts // Map[StringSplit[#, "\n"] &] // Map[MatrixForm]
Transpose[{ts, {{5, 12}, {8, 15}}}] //
Map[StringSplit[First@#, "\n"][[Last@#]] &]


Out:

{{"他在廚房 。 他在煮束西 口",
"你踩 到 我的 腳 了 。\[LongDash] 對不起"},
{"他在厨房 \[Degree] 他正在做饭 。",
"你踩着我脚了 口 一对不起 口"}}

• Hey, actually the trouble is I don't have that kind of a bandwidth to send this across. And conversion to smaller image chunks is necessary since we have a pipeline to process them via tesseract - on the campus. – Abhinav Sharma Jun 3 '17 at 8:42
• All right, I just give another way of thinking. If it's done in the cloud, the bandwidth problem could be alleviated. – UnchartedWorks Jun 3 '17 at 8:47
• I agree, thanks for this one though :) – Abhinav Sharma Jun 3 '17 at 8:51