4
$\begingroup$

I am looking for a simple way to recognize musical Unicode text (written in byzantine musical notation). My idea is to compare each musical symbol with the scanned text and find all the coordinates(of the surrounding box) of this symbol in the scanned text. My basic problem is that TextRecognize does not takes in account these symbols. The second answer In textrecognize not up to pa

suggest the use of the command ImageCorrelate. Also Highlightimage is another good candidate. Both commands find all the occurrences of a template(in my case of a letter) in the scanned image. But I don’t know how to use these commands to extract some useful information that will help me recognize the text. Could you please give some ideas? For testing purposes I upload three different letters the ison(U+1D046) the oligon(U+1D047) and apostrofos (U+1D051) plus a random piece of musical text in .jpg format

$\endgroup$
6
$\begingroup$

This is not a full answer, but it's too extensive to be a comment. Here's a good way to get all of the components using the wonderful MorphologicalComponents, paired with ChanVeseBinarize:

img = Import["https://i.stack.imgur.com/rXlxu.jpg"];

binarized =
  ChanVeseBinarize@img;

components =
  MorphologicalComponents[binarized];

componentNumbers = Rest@Sort[DeleteDuplicates@Flatten@components];

slices =
  Table[
   ImageCrop@
    Image[Map[If[# == i, 0., 1.] &] /@ components,
     "Real", ColorSpace -> "Grayscale"],
   {i, componentNumbers[[;; 100]]}
   ];

Then we can see the effect of this:

Partition[slices, 10] // Grid[#, Dividers -> All] &

char grid

Note that we could get bounding boxes instead by supplying Method->"BoundingBox" to MorphologicalComponents.

Hope this is enough to get you started.

On getting image coordinates

We can slightly modify the code that generated our slices to also get the coordinate bbox:

slices =
  Table[
   With[{component = Map[If[# == i, 0., 1.] &] /@ components},
    ImageCrop@
      Image[component, "Real", ColorSpace -> "Grayscale"] ->
     CoordinateBoundingBox@Position[component, 0.]
    ],
   {i, componentNumbers[[;; 100]]}
   ];

This way each image is paired with its coordinate bounding box. Then we can operate on the images themselves to figure out which ones are the same. This'll take a tiny bit more work, but shouldn't be bad at this point as the component images are of pretty high quality already.

On getting matching images

So ImageDistance seems to ID the images pretty well if you use the "EarthMoverDistance". I'm sure this could be played with more to make it more solid though.

Here's what I found from a quick survey:

Start by importing your test images and prepping them:

chars =
  ImageCrop@*
    ColorNegate@*ChanVeseBinarize@*Import /@ {
    "https://i.stack.imgur.com/xVqDd.jpg",
    "https://i.stack.imgur.com/Uci9a.jpg",
    "https://i.stack.imgur.com/1LtJm.jpg"
    };

Then the general form will be:

KeySort@GroupBy[
  Select[slices,
   ImageDistance[First@#, chars[[i]],
      DistanceFunction -> "EarthMoverDistance"] < <dissimilarity-max> &
   ],
  ImageDistance[First@#, chars[[i]],
    DistanceFunction -> "EarthMoverDistance"] &
  ]

And here .05 seems to work pretty well for <dissimilarity-max> although you'll want to tune this. By pre-processing the pulled slice images (filling in gaps and whatnot using, e.g. Dilation) you can improve this matching process yet further.

Using ImageDimensions as a criterion

As each of these characters has a pretty unique size to it we can use the ImageDimensions as matching criteria (to scale this we could use the ImageDimensions ratio).

distanceFunctions[im1_, im2_] :=
  {
   Norm[ImageDimensions@im1 - ImageDimensions@im2],
   ImageDistance[im1, chars[[1]],
    DistanceFunction -> "EarthMoverDistance"]
   };

matches =
 Table[
  KeySort@GroupBy[
    Select[slices,
     And @@ MapThread[
        # < #2 &, {
         distanceFunctions[First@#, chars[[i]]],
         {15, 1}
         }] &
     ],
    Last@distanceFunctions[First@#, chars[[i]]] &
    ],
  {i, Length@chars}]

This is near perfect (note that we're not really using the ImageDistance but we could tune the factors for each match so I'm leaving it in there).

$\endgroup$
  • $\begingroup$ @MB 1965 the command MorphologicalComponents is indeed very good. I would like also to have the actual coordinates of these components in the text. Is it possible? $\endgroup$ – kornaros Mar 9 '17 at 11:59
  • $\begingroup$ Sure. Where I generate those slices, use Position instead / as well. $\endgroup$ – b3m2a1 Mar 9 '17 at 13:07
  • $\begingroup$ Could you please give the code? I cant understand how I will generate all the positions in which the first element slices[[1]] (for example) does appears ...In others words at which positions of the text the ( could be found? $\endgroup$ – kornaros Mar 9 '17 at 15:44
  • $\begingroup$ Yeah, by the way I don't think the answer is worth accepting as it is. Leave the question open in case someone else can do it better. $\endgroup$ – b3m2a1 Mar 9 '17 at 16:49
  • $\begingroup$ I have not tested the last code yet but I am sure it works very well. If you have time please enter some pre-processing techniques. I think that one must have much experience with Dilation or other Image Processing commands for good results. I am newbie on this topic ...Thank you! $\endgroup$ – kornaros Mar 9 '17 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.