How can I separate a scanned text to its letters?

Question

I am looking for a simple way to recognize musical Unicode text (written in byzantine musical notation). My idea is to compare each musical symbol with the scanned text and find all the coordinates(of the surrounding box) of this symbol in the scanned text. My basic problem is that TextRecognize does not takes in account these symbols. The second answer In textrecognize not up to pa

suggest the use of the command ImageCorrelate. Also Highlightimage is another good candidate. Both commands find all the occurrences of a template(in my case of a letter) in the scanned image. But I don’t know how to use these commands to extract some useful information that will help me recognize the text. Could you please give some ideas? For testing purposes I upload three different letters the (U+1D046) the (U+1D047) and (U+1D051) plus a random piece of musical in .jpg format

Answer 1

This is not a full answer, but it's too extensive to be a comment. Here's a good way to get all of the components using the wonderful MorphologicalComponents, paired with ChanVeseBinarize:

img = Import["https://i.stack.imgur.com/rXlxu.jpg"];

binarized =
  ChanVeseBinarize@img;

components =
  MorphologicalComponents[binarized];

componentNumbers = Rest@Sort[DeleteDuplicates@Flatten@components];

slices =
  Table[
   ImageCrop@
    Image[Map[If[# == i, 0., 1.] &] /@ components,
     "Real", ColorSpace -> "Grayscale"],
   {i, componentNumbers[[;; 100]]}
   ];

Then we can see the effect of this:

Partition[slices, 10] // Grid[#, Dividers -> All] &

Note that we could get bounding boxes instead by supplying Method->"BoundingBox" to MorphologicalComponents.

Hope this is enough to get you started.

On getting image coordinates

We can slightly modify the code that generated our slices to also get the coordinate bbox:

slices =
  Table[
   With[{component = Map[If[# == i, 0., 1.] &] /@ components},
    ImageCrop@
      Image[component, "Real", ColorSpace -> "Grayscale"] ->
     CoordinateBoundingBox@Position[component, 0.]
    ],
   {i, componentNumbers[[;; 100]]}
   ];

This way each image is paired with its coordinate bounding box. Then we can operate on the images themselves to figure out which ones are the same. This'll take a tiny bit more work, but shouldn't be bad at this point as the component images are of pretty high quality already.

On getting matching images

So ImageDistance seems to ID the images pretty well if you use the "EarthMoverDistance". I'm sure this could be played with more to make it more solid though.

Here's what I found from a quick survey:

Start by importing your test images and prepping them:

chars =
  ImageCrop@*
    ColorNegate@*ChanVeseBinarize@*Import /@ {
    "https://i.stack.imgur.com/xVqDd.jpg",
    "https://i.stack.imgur.com/Uci9a.jpg",
    "https://i.stack.imgur.com/1LtJm.jpg"
    };

Then the general form will be:

KeySort@GroupBy[
  Select[slices,
   ImageDistance[First@#, chars[[i]],
      DistanceFunction -> "EarthMoverDistance"] < <dissimilarity-max> &
   ],
  ImageDistance[First@#, chars[[i]],
    DistanceFunction -> "EarthMoverDistance"] &
  ]

And here .05 seems to work pretty well for <dissimilarity-max> although you'll want to tune this. By pre-processing the pulled slice images (filling in gaps and whatnot using, e.g. Dilation) you can improve this matching process yet further.

Using ImageDimensions as a criterion

As each of these characters has a pretty unique size to it we can use the ImageDimensions as matching criteria (to scale this we could use the ImageDimensions ratio).

distanceFunctions[im1_, im2_] :=
  {
   Norm[ImageDimensions@im1 - ImageDimensions@im2],
   ImageDistance[im1, chars[[1]],
    DistanceFunction -> "EarthMoverDistance"]
   };

matches =
 Table[
  KeySort@GroupBy[
    Select[slices,
     And @@ MapThread[
        # < #2 &, {
         distanceFunctions[First@#, chars[[i]]],
         {15, 1}
         }] &
     ],
    Last@distanceFunctions[First@#, chars[[i]]] &
    ],
  {i, Length@chars}]

This is near perfect (note that we're not really using the ImageDistance but we could tune the factors for each match so I'm leaving it in there).