Applying TextRecognize on alpha-numerical table

From time to time I have to extract data from scanned papers or rasterized PDFs. Imagine a table of the following kind (but much longer):

A naive application of TextRecognize on the image fails spectacularly:

"OOE-
20B-
70E~
20E-
90E-
70E-
70B-
30B-
30B-
ZOE
.50E+O2
.50E+02
.0OE+02
.O0E+O2
.00E+O2
.OOE+02
18E+O4
.30E+O3
.70E+03
10E+03


Did anybody use TextRecognize successfully on similar images?

-
TextRecognize doesn't do it letter by letter, but tries to identify words from a dictionary. This is a known limitation... See this answer by Arnoud –  rm -rf Jan 29 '13 at 15:08
and "When recognizing formatted numbers, whitespace may cause OCR errors", according to the documentation... –  cormullion Jan 29 '13 at 15:14
TextRecognize is based on Tesseract, which is free and open source and will give you a lot more flexibility than using TextRecognize. For example it lets you constrain the set of characters you are interested in. That said, some time ago I tried to use it to read a huge matrix of 0s and 1s (again from an old paper), and ended up needing to do many manual corrections in each row ... It's probably not worth the time. –  Szabolcs Jan 29 '13 at 17:43
Interesting this Tesseract. I saw a simple nice guide for Mac OS here –  Murta Jan 29 '13 at 22:39
I wish someone would write an article on how to customize Tesseract for MMA. A while back I went digging and discovered grammar/dictionary files---but I'd have no clue how to customize these, or if the version distributed with MMA would even allow for it. –  telefunkenvf14 Feb 3 '13 at 20:12

Here is a way of extracting the positions of the various characters in your image by using ImageCorrelate.

Define the image to be worked on.

image = <your image goes here>


Define a set of kernel images to correlate with the image. I use {0,1,2,3,4,5,6,7,8,9,+,-,E,.} scaled to an appropriate size and then rasterised.

kernels =
Map[
(Style[#, FontSize -> 28, FontWeight -> Bold] // Rasterize //
ColorNegate // Binarize) &,
Join[Range[0, 9], {"+", "-", "E", "."}]
]


Prepare the image for use in ImageCorrelate. Adding 0.01 to the 0 background avoids problems with it being “detected” - try omitting this step to see what happens.

imageneg = image // ColorNegate // ImageAdd[#, 0.01] &


Compute the CosineDistance image correlation for each kernel image. Threshold the results to detect instances of each kernel image, and Dilate the resulting small bright detections so you can easily see them.

imagecorrels =
Map[
(ImageCorrelate[imageneg, #, CosineDistance] // ColorNegate //
Binarize[#, 0.85] &) &,
kernels
];

MapThread[Column[{#1, #2}] &, {Dilation[#, 3] & /@ imagecorrels, kernels}]


The results are very good for this set of kernel images. However, some of the “+” and “-“ are missed, which is probably because their kernel images differ too much from what appears in the image to be processed, and the “.” at the bottom of the image are missed, but this looks like an “edge effect” that would go away if you didn’t crop the image so tightly. Also, the “I” of “SI” is mistaken for a “1”.

The image processing could be tweaked to improve this performance. For instance, you could make the correlation results "compete" with each other to determine the "detection" at each image location - i.e. identify before detect - rather than simply threshold the correlations independently.

The locations of the detected image kernels could then be post-processed to create a representation of your image in terms of kernel images.

-
Nice (and got the +1 when you posted), but it's a bit unfortunate that we need to write our OCR engine from scratch (what you practically did here!) because TextRecognize won't allow configuring Tesserract (the engine it uses) with enough flexibility. One important step in your solution is restricting the possible set of characters. Tesserract has options for this, but TextRecognize doesn't expose them. –  Szabolcs Feb 3 '13 at 21:43
I have just checked what happens when you include the whole alphabet by using Join[Range[0,9],CharacterRange["A","Z"],{"+","-","."}]. “B” detects “8” as well as “B”, and similarly “I” detects “1” as well as “I”, but otherwise there are no false alarms or missed detections of alphabetic characters. I haven't checked, but I assume that the false alarm problem can be fixed by forcing the detections to "compete" with each other, as I suggested in my answer. –  Stephen Luttrell Feb 4 '13 at 0:41

As I've shown here, TextRecognize[] is a very basic tool and you usually need to tweak it to fit your needs.

In your case, you could do something like:

i = Binarize@ColorNegate@Import["http://i.stack.imgur.com/lP6Yt.jpg"];

(* Group the chars line by line *)
ifc = ImageForestingComponents[ColorNegate@i, Automatic, {First@ImageDimensions@i, 1}];


ClearAll[im];
(* A function for dropping blank spaces between letters *)
rule = {h__, {0 ..}, w : {0 ..}, {0 ..}, t__} -> {h, w, w, t};
im[x_] := Transpose[ Transpose[Replace[ifc, Except[x] -> 0, {2}] //. rule] //. rule];

(* French periodic tables are better understood :) *)
tr = TextRecognize[Image[im[#]], Language -> "French"] & /@  Range[2, Max@ifc];

(* Split column blocks *)
strs = {StringTake[#, {1, -17}], StringTake[#, {-16, -9}], StringTake[#, -8]} & /@ tr


$\left( \begin{array}{ccc} \text{H} & \text{6.00E-O9} & \text{3.5OE+O2} \\ \text{H2} & \text{l.20E-09} & \text{4.50E+02} \\ \text{I-lE} & \text{5.70E-08} & \text{l.00E+02} \\ \text{C} & \text{4.2OE-l2} & \text{8.00E+O2} \\ \text{N} & \text{3.90E-l2} & \text{8.00E+02} \\ 0 & \text{3.70E-12} & \text{B.00E+02} \\ \text{NA} & \text{6.70E-80} & \text{1.18E+04} \\ \text{MG} & \text{9.3OE-40} & \text{5.30E+03} \\ \text{SI} & \text{8.3OE-24} & \text{2.70E+03} \\ \text{S} & \text{4.20E-l4} & \text{1.10E+O3} \end{array} \right)$

You may see that there are problems (easy to fix knowing the context) with the number zero and the letter "o", and with the number eight and the letter "B". Also, one letter "H" got read as "I-1"

Now, Here you have a bunch of transformation rules designed to get the correct answer based upon the errors observed in TextRecognize[]

base = "(\\+|[0-9]|E|\\.|\-)";
MatrixForm[FixedPoint[
StringReplace[#, {"\[Euro]" -> "E", "I-l" -> "H",
WordBoundary ~~ "0" ~~ WordBoundary -> "O",
RegularExpression[base ~~ "O"] :> "$1" ~~ "0", RegularExpression[base ~~ "B"] :> "$1" ~~ "8",
RegularExpression["B" ~~ base] :> "8" ~~ "$1"}] & /@ # &, strs]]  •$\left( \begin{array}{ccc} \text{H} & \text{6.00E-09} & \text{3.50E+02} \\ \text{H2} & \text{l.20E-09} & \text{4.50E+02} \\ \text{HE} & \text{5.70E-08} & \text{l.00E+02} \\ \text{C} & \text{4.20E-l2} & \text{8.00E+02} \\ \text{N} & \text{3.90E-l2} & \text{8.00E+02} \\ \text{O} & \text{3.70E-12} & \text{8.00E+02} \\ \text{NA} & \text{6.70E-80} & \text{1.18E+04} \\ \text{MG} & \text{9.30E-40} & \text{5.30E+03} \\ \text{SI} & \text{8.30E-24} & \text{2.70E+03} \\ \text{S} & \text{4.20E-l4} & \text{1.10E+03} \end{array} \right)$For performance issues, please see the companion question Performance: Collapsing repeated contiguous rows & cols from a matrix - The periodic table must sound very French :P This "tweaking" must have taken you a while to get right – Szabolcs Feb 3 '13 at 21:43 @Szabolcs Not really. As of v8.0 Mma recognized only 6 languages :) – belisarius Feb 3 '13 at 21:48 @belisarius Brilliant! Working like a charm (with some manual tweaking still to do). Is there an easy way to automatically determine the number of table lines in the figure, or do I have to count manually and modify the code accordingly? – Markus Roellig Feb 5 '13 at 13:20 @MarkusRoellig In the code above Max@ifc - 1 is the number of lines (I'm updating the code). Glad to hear it works. The only way to avoid the manual tweaks is by having a "trainable" OCR, and Mma doesn't offer that. – belisarius Feb 5 '13 at 13:24 I get perfect reading using Tesseract using pagesegmode value 6. I did this test after install Tesseract SetDirectory@NotebookDirectory[]; Run["/usr/local/bin/tesseract hex.png hex -psm 6 "]  on as hex.png and get with no need to restrict character set (you can see how to do it here) The only mistake are between 0 and O that can be easily corrected as @belisarius shows here. Update Now let's do it inside Mathematica. In this post from Wolfram Community, Ilian Gachevski shows me how to control page segmentation from inside Mathematica. It's as simple as: i=Import[hex.png] TextRecognize[i, "SegmentationMode" -> 6]  You can check that options are equivalent executing: Map[Row[#," "]&,ImageExternalOCRDump$TextRecognizeSegmentationModes,{-2}]//TableForm


With all the comfort of doing all inside Mathematica. Just don't forget that this is an undocumented option.

-
Now that's cool. Thanks for figuring that out. –  Markus Roellig Sep 20 '13 at 8:27