62
$\begingroup$

From time to time I have to extract data from scanned papers or rasterized PDFs. Imagine a table of the following kind (but much longer):

enter image description here

A naive application of TextRecognize on the image fails spectacularly:

"OOE-
20B-
70E~
20E-
90E-
70E-
70B-
30B-
30B-
ZOE
.50E+O2
.50E+02
.0OE+02
.O0E+O2
.00E+O2
.OOE+02
18E+O4
.30E+O3
.70E+03
10E+03

Did anybody use TextRecognize successfully on similar images?

$\endgroup$
7
  • 6
    $\begingroup$ TextRecognize doesn't do it letter by letter, but tries to identify words from a dictionary. This is a known limitation... See this answer by Arnoud $\endgroup$
    – rm -rf
    Jan 29, 2013 at 15:08
  • $\begingroup$ and "When recognizing formatted numbers, whitespace may cause OCR errors", according to the documentation... $\endgroup$
    – cormullion
    Jan 29, 2013 at 15:14
  • 4
    $\begingroup$ TextRecognize is based on Tesseract, which is free and open source and will give you a lot more flexibility than using TextRecognize. For example it lets you constrain the set of characters you are interested in. That said, some time ago I tried to use it to read a huge matrix of 0s and 1s (again from an old paper), and ended up needing to do many manual corrections in each row ... It's probably not worth the time. $\endgroup$
    – Szabolcs
    Jan 29, 2013 at 17:43
  • 2
    $\begingroup$ Interesting this Tesseract. I saw a simple nice guide for Mac OS here $\endgroup$
    – Murta
    Jan 29, 2013 at 22:39
  • 2
    $\begingroup$ I wish someone would write an article on how to customize Tesseract for MMA. A while back I went digging and discovered grammar/dictionary files---but I'd have no clue how to customize these, or if the version distributed with MMA would even allow for it. $\endgroup$ Feb 3, 2013 at 20:12

3 Answers 3

40
$\begingroup$

Here is a way of extracting the positions of the various characters in your image by using ImageCorrelate.

Define the image to be worked on.

image = <your image goes here>

Define a set of kernel images to correlate with the image. I use {0,1,2,3,4,5,6,7,8,9,+,-,E,.} scaled to an appropriate size and then rasterised.

kernels =
  Map[
    (Style[#, FontSize -> 28, FontWeight -> Bold] // Rasterize // 
    ColorNegate // Binarize) &,
    Join[Range[0, 9], {"+", "-", "E", "."}]
  ]

Mathematica graphics

Prepare the image for use in ImageCorrelate. Adding 0.01 to the 0 background avoids problems with it being “detected” - try omitting this step to see what happens.

imageneg = image // ColorNegate // ImageAdd[#, 0.01] &

Mathematica graphics

Compute the CosineDistance image correlation for each kernel image. Threshold the results to detect instances of each kernel image, and Dilate the resulting small bright detections so you can easily see them.

imagecorrels = 
  Map[
    (ImageCorrelate[imageneg, #, CosineDistance] // ColorNegate //
     Binarize[#, 0.85] &) &,
    kernels
  ];

MapThread[Column[{#1, #2}] &, {Dilation[#, 3] & /@ imagecorrels, kernels}]

Mathematica graphics

The results are very good for this set of kernel images. However, some of the “+” and “-“ are missed, which is probably because their kernel images differ too much from what appears in the image to be processed, and the “.” at the bottom of the image are missed, but this looks like an “edge effect” that would go away if you didn’t crop the image so tightly. Also, the “I” of “SI” is mistaken for a “1”.

The image processing could be tweaked to improve this performance. For instance, you could make the correlation results "compete" with each other to determine the "detection" at each image location - i.e. identify before detect - rather than simply threshold the correlations independently.

The locations of the detected image kernels could then be post-processed to create a representation of your image in terms of kernel images.

$\endgroup$
2
  • 2
    $\begingroup$ Nice (and got the +1 when you posted), but it's a bit unfortunate that we need to write our OCR engine from scratch (what you practically did here!) because TextRecognize won't allow configuring Tesserract (the engine it uses) with enough flexibility. One important step in your solution is restricting the possible set of characters. Tesserract has options for this, but TextRecognize doesn't expose them. $\endgroup$
    – Szabolcs
    Feb 3, 2013 at 21:43
  • $\begingroup$ I have just checked what happens when you include the whole alphabet by using Join[Range[0,9],CharacterRange["A","Z"],{"+","-","."}]. “B” detects “8” as well as “B”, and similarly “I” detects “1” as well as “I”, but otherwise there are no false alarms or missed detections of alphabetic characters. I haven't checked, but I assume that the false alarm problem can be fixed by forcing the detections to "compete" with each other, as I suggested in my answer. $\endgroup$ Feb 4, 2013 at 0:41
30
$\begingroup$

I get perfect reading using Tesseract using pagesegmode value 6.

enter image description here

I did this test after install Tesseract

SetDirectory@NotebookDirectory[];
Run["/usr/local/bin/tesseract hex.png hex -psm 6 "]

on as hex.png

enter image description here

and get

enter image description here

with no need to restrict character set (you can see how to do it here)

The only mistake are between 0 and O that can be easily corrected as @belisarius shows here.

Update

Now let's do it inside Mathematica. In this post from Wolfram Community, Ilian Gachevski shows me how to control page segmentation from inside Mathematica. It's as simple as:

i=Import[hex.png]
TextRecognize[i, "SegmentationMode" -> 6]

You can check that options are equivalent executing:

Map[Row[#,"  "]&,Image`ExternalOCRDump`$TextRecognizeSegmentationModes,{-2}]//TableForm

enter image description here

With all the comfort of doing all inside Mathematica. Just don't forget that this is an undocumented option.

$\endgroup$
1
  • $\begingroup$ Now that's cool. Thanks for figuring that out. $\endgroup$ Sep 20, 2013 at 8:27
25
$\begingroup$

As I've shown here, TextRecognize[] is a very basic tool and you usually need to tweak it to fit your needs.

In your case, you could do something like:

i = Binarize@ColorNegate@Import["https://i.stack.imgur.com/lP6Yt.jpg"];

(* Group the chars line by line *)
ifc = ImageForestingComponents[ColorNegate@i, Automatic, {First@ImageDimensions@i, 1}];

Mathematica graphics

ClearAll[im];
(* A function for dropping blank spaces between letters *)
rule = {h__, {0 ..}, w : {0 ..}, {0 ..}, t__} -> {h, w, w, t};
im[x_] := Transpose[ Transpose[Replace[ifc, Except[x] -> 0, {2}] //. rule] //. rule];

(* French periodic tables are better understood :) *)
tr = TextRecognize[Image[im[#]], Language -> "French"] & /@  Range[2, Max@ifc];

(* Split column blocks *)
strs = {StringTake[#, {1, -17}], StringTake[#, {-16, -9}], StringTake[#, -8]} & /@ tr

$\left( \begin{array}{ccc} \text{H} & \text{6.00E-O9} & \text{3.5OE+O2} \\ \text{H2} & \text{l.20E-09} & \text{4.50E+02} \\ \text{I-lE} & \text{5.70E-08} & \text{l.00E+02} \\ \text{C} & \text{4.2OE-l2} & \text{8.00E+O2} \\ \text{N} & \text{3.90E-l2} & \text{8.00E+02} \\ 0 & \text{3.70E-12} & \text{B.00E+02} \\ \text{NA} & \text{6.70E-80} & \text{1.18E+04} \\ \text{MG} & \text{9.3OE-40} & \text{5.30E+03} \\ \text{SI} & \text{8.3OE-24} & \text{2.70E+03} \\ \text{S} & \text{4.20E-l4} & \text{1.10E+O3} \end{array} \right)$

You may see that there are problems (easy to fix knowing the context) with the number zero and the letter "o", and with the number eight and the letter "B". Also, one letter "H" got read as "I-1"

Now, Here you have a bunch of transformation rules designed to get the correct answer based upon the errors observed in TextRecognize[]

base = "(\\+|[0-9]|E|\\.|\-)";
MatrixForm[FixedPoint[
  StringReplace[#, {"\[Euro]" -> "E", "I-l" -> "H",
       WordBoundary ~~ "0" ~~ WordBoundary -> "O",
       RegularExpression[base ~~ "O"] :> "$1" ~~ "0",
   RegularExpression[base ~~ "B"] :> "$1" ~~ "8",
       RegularExpression["B" ~~ base] :> "8" ~~ "$1"}] & /@ # &, strs]] 
  • $\left( \begin{array}{ccc} \text{H} & \text{6.00E-09} & \text{3.50E+02} \\ \text{H2} & \text{l.20E-09} & \text{4.50E+02} \\ \text{HE} & \text{5.70E-08} & \text{l.00E+02} \\ \text{C} & \text{4.20E-l2} & \text{8.00E+02} \\ \text{N} & \text{3.90E-l2} & \text{8.00E+02} \\ \text{O} & \text{3.70E-12} & \text{8.00E+02} \\ \text{NA} & \text{6.70E-80} & \text{1.18E+04} \\ \text{MG} & \text{9.30E-40} & \text{5.30E+03} \\ \text{SI} & \text{8.30E-24} & \text{2.70E+03} \\ \text{S} & \text{4.20E-l4} & \text{1.10E+03} \end{array} \right)$

For performance issues, please see the companion question Performance: Collapsing repeated contiguous rows & cols from a matrix

$\endgroup$
4
  • $\begingroup$ The periodic table must sound very French :P This "tweaking" must have taken you a while to get right $\endgroup$
    – Szabolcs
    Feb 3, 2013 at 21:43
  • $\begingroup$ @Szabolcs Not really. As of v8.0 Mma recognized only 6 languages :) $\endgroup$ Feb 3, 2013 at 21:48
  • $\begingroup$ @belisarius Brilliant! Working like a charm (with some manual tweaking still to do). Is there an easy way to automatically determine the number of table lines in the figure, or do I have to count manually and modify the code accordingly? $\endgroup$ Feb 5, 2013 at 13:20
  • $\begingroup$ @MarkusRoellig In the code above Max@ifc - 1 is the number of lines (I'm updating the code). Glad to hear it works. The only way to avoid the manual tweaks is by having a "trainable" OCR, and Mma doesn't offer that. $\endgroup$ Feb 5, 2013 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.