# How can I separate a scanned text to its letters?

I am looking for a simple way to recognize musical Unicode text (written in byzantine musical notation). My idea is to compare each musical symbol with the scanned text and find all the coordinates(of the surrounding box) of this symbol in the scanned text. My basic problem is that TextRecognize does not takes in account these symbols. The second answer In textrecognize not up to pa

suggest the use of the command ImageCorrelate. Also Highlightimage is another good candidate. Both commands find all the occurrences of a template(in my case of a letter) in the scanned image. But I don’t know how to use these commands to extract some useful information that will help me recognize the text. Could you please give some ideas? For testing purposes I upload three different letters the (U+1D046) the (U+1D047) and (U+1D051) plus a random piece of musical in .jpg format

This is not a full answer, but it's too extensive to be a comment. Here's a good way to get all of the components using the wonderful MorphologicalComponents, paired with ChanVeseBinarize:

img = Import["https://i.stack.imgur.com/rXlxu.jpg"];

binarized =
ChanVeseBinarize@img;

components =
MorphologicalComponents[binarized];

componentNumbers = Rest@Sort[DeleteDuplicates@Flatten@components];

slices =
Table[
ImageCrop@
Image[Map[If[# == i, 0., 1.] &] /@ components,
"Real", ColorSpace -> "Grayscale"],
{i, componentNumbers[[;; 100]]}
];


Then we can see the effect of this:

Partition[slices, 10] // Grid[#, Dividers -> All] &


Note that we could get bounding boxes instead by supplying Method->"BoundingBox" to MorphologicalComponents.

Hope this is enough to get you started.

### On getting image coordinates

We can slightly modify the code that generated our slices to also get the coordinate bbox:

slices =
Table[
With[{component = Map[If[# == i, 0., 1.] &] /@ components},
ImageCrop@
Image[component, "Real", ColorSpace -> "Grayscale"] ->
CoordinateBoundingBox@Position[component, 0.]
],
{i, componentNumbers[[;; 100]]}
];


This way each image is paired with its coordinate bounding box. Then we can operate on the images themselves to figure out which ones are the same. This'll take a tiny bit more work, but shouldn't be bad at this point as the component images are of pretty high quality already.

### On getting matching images

So ImageDistance seems to ID the images pretty well if you use the "EarthMoverDistance". I'm sure this could be played with more to make it more solid though.

Here's what I found from a quick survey:

Start by importing your test images and prepping them:

chars =
ImageCrop@*
ColorNegate@*ChanVeseBinarize@*Import /@ {
"https://i.stack.imgur.com/xVqDd.jpg",
"https://i.stack.imgur.com/Uci9a.jpg",
"https://i.stack.imgur.com/1LtJm.jpg"
};


Then the general form will be:

KeySort@GroupBy[
Select[slices,
ImageDistance[First@#, chars[[i]],
DistanceFunction -> "EarthMoverDistance"] < <dissimilarity-max> &
],
ImageDistance[First@#, chars[[i]],
DistanceFunction -> "EarthMoverDistance"] &
]


And here .05 seems to work pretty well for <dissimilarity-max> although you'll want to tune this. By pre-processing the pulled slice images (filling in gaps and whatnot using, e.g. Dilation) you can improve this matching process yet further.

### Using ImageDimensions as a criterion

As each of these characters has a pretty unique size to it we can use the ImageDimensions as matching criteria (to scale this we could use the ImageDimensions ratio).

distanceFunctions[im1_, im2_] :=
{
Norm[ImageDimensions@im1 - ImageDimensions@im2],
ImageDistance[im1, chars[[1]],
DistanceFunction -> "EarthMoverDistance"]
};

matches =
Table[
KeySort@GroupBy[
Select[slices,
# < #2 &, {
distanceFunctions[First@#, chars[[i]]],
{15, 1}
}] &
],
Last@distanceFunctions[First@#, chars[[i]]] &
],
{i, Length@chars}]


This is near perfect (note that we're not really using the ImageDistance but we could tune the factors for each match so I'm leaving it in there).

• @MB 1965 the command MorphologicalComponents is indeed very good. I would like also to have the actual coordinates of these components in the text. Is it possible? Mar 9, 2017 at 11:59
• Sure. Where I generate those slices, use Position instead / as well. Mar 9, 2017 at 13:07
• Could you please give the code? I cant understand how I will generate all the positions in which the first element slices[[1]] (for example) does appears ...In others words at which positions of the text the ( could be found? Mar 9, 2017 at 15:44
• Yeah, by the way I don't think the answer is worth accepting as it is. Leave the question open in case someone else can do it better. Mar 9, 2017 at 16:49
• I have not tested the last code yet but I am sure it works very well. If you have time please enter some pre-processing techniques. I think that one must have much experience with Dilation or other Image Processing commands for good results. I am newbie on this topic ...Thank you! Mar 9, 2017 at 18:25