12
$\begingroup$

I want to extract table from the image bellow as a list of strings. I tried the following but it does not work.

im=Import["https://i.sstatic.net/Tt6AH.png"];
TextRecognize[im]
FindImageText[im,"Character","Image"]

(* "" *)
(* {} *)

enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ TextRecognize[im, RecognitionPrior -> "SparseText"] returns most of the text, although it misses the column/row headings. $\endgroup$
    – Domen
    Commented Oct 28, 2022 at 17:32
  • $\begingroup$ It at least outputs something... but "3 9 AL AN AND..." is not very satisfactory. $\endgroup$ Commented Oct 28, 2022 at 17:37

4 Answers 4

5
$\begingroup$
im = Import["https://i.sstatic.net/Tt6AH.png"]
im2 = Dilation[Binarize@im, DiskMatrix[0.3]] // ImageAdjust;
im3 = ImageDifference[im2, FillingTransform[im2]] // ColorNegate // 
  Sharpen

This recognizes most of the characters successfully other than 0 and 'O'.

chars = TextRecognize[im3, "Character", RecognitionPrior -> "Block"]

{"1", "2", "3", "4", "5", "6", "7", "8", "9", "0", "1", "A", "1", \
"A", "L", "A", "N", "A", "N", "D", "A", "R", "A", "R", "E", "A", "S", \
"A", "T", "A", "T", "E", "2", "A", "T", "I", "B", "2", "B", "E", "C", \
"3", "C", "A", "C", "E", "C", "0", "C", "O", "M", "3", "D", "4", "D", \
"A", "D", "E", "E", "5", "E", "A", "E", "D", "E", "N", "E", "N", "T", \
"4", "E", "R", "E", "R", "E", "E", "R", "S", "E", "S", "E", "S", "T", \
"F", "6", "G", "7", "H", "5", "8", "H", "A", "S", "H", "E", "I", "9", \
"I", "N", "I", "N", "G", "I", "O", "N", "I", "S", "I", "T", "6", "I", \
"V", "E", "J", "0", "K", "L", "L", "A", "L", "E", "M", "M", "E", "N", \
"7", "N", "D", "N", "E", "N", "T", "0", "O", "F", "O", "N", "O", "R", \
"O", "U", "P", "Q", "8", "R", "R", "A", "R", "E", "R", "E", "D", "R", \
"E", "S", "R", "I", "R", "O", "S", "S", "E", "S", "H", "9", "S", "T", \
"S", "T", "O", "T", "T", "E", "T", "E", "D", "T", "E", "R", "T", "H", \
"T", "H", "E", "T", "H", "I", "T", "H", "R", "0", "T", "I", "T", "O", \
"U", "V", "V", "E", "W", "W", "E", "X", "Y", "Z"}

But there is a problem. To visualize:

res = TextRecognize[im3, "Character", {"Text", "BoundingBox"}, 
   RecognitionPrior -> "Block"];
HighlightImage[im3, res]

enter image description here

Split on Euclidean distance between centroids of rectangles in res. Prepend 0 to the list since the top left box is empty. After that, StringJoin can be used.

lens = {0}~
  Join~(Length /@ 
    Split[RegionCentroid /@ Last /@ res, 
     EuclideanDistance[First@#1, First@#2] < 20 &])

The color in the title bar is found using:

bkgColorTitle = First@DominantColors@ImageTake[im, {5, 10}, {6, 10}]

Finally collect characters and do an ArrayReshape:

{im, ArrayReshape[StringJoin @@@ TakeList[chars, lens], {11, 11}] //
  Grid[#, Frame -> All, ItemSize -> {2.3, 1.8}
    , Background -> {{bkgColorTitle, None}
      , {bkgColorTitle, None}
      }
    ] &}

enter image description here


Other comments and queries

  1. It was pointed out (and observed) that 0 and O were not being faithfully recognized.

  2. I thought scaling the image up would fix everything. Surely, I went for im2 = Binarize[ImageResize[im, Scaled[2]], 0.6] while making manual changes to the EuclideanDistance metric to group rectangles together and at this level of magnification, let's say, glasses were not required. But the entries at (7,4) and (6,3) got identified inversely.

  3. It seemed strange that the (7,4) entry was a standalone O and the engine recognized it as 0. On the same row, it detected four other Os successfully. A 0is shaped like an ellipse, whereas the O is almost circular.

  4. To investigate further, I converted the original picture to pdf and used Acrobat to do a character recognition on the original image. Perfect recognition or so it seemed. But when I exported to MS word, the 0 reappeared in (7,4) although on the Acrobat, it looked very much like good enough to be an O. It hints that this OCR engine is closely related to the one found in Acrobat.

  5. Do I have a systematic method for a EuclideanDistance metric in order to separate characters?

It was trial and error. A more systematic way would be to bring font size into consideration. This strategy would fail quickly during recognition of subscripted variables or where there are multiple fonts in the document.

$\endgroup$
8
  • $\begingroup$ Nice! Just one incorrectly recognized letter at position (2, 9): it should be CO not C0 :) $\endgroup$
    – Domen
    Commented Oct 29, 2022 at 16:34
  • $\begingroup$ I think it can be fixed more easily with a replacement rule or by fiddling with the parameters some more. This O definitely does not look like the zero in the top right and bottom left corners, but it is this uncertainty of AI algorithms that gives me hope. Also look at (4,7) and (3,6). Thanks for identifying. $\endgroup$
    – Syed
    Commented Oct 29, 2022 at 16:49
  • $\begingroup$ That looks great. Which parts improved the answer by Anton Antonov the most ? Are the results sensitive to the parameter choice DiskMatrix[0.3] ? I might rasterize some random tables to test how well this method works. It looks quite nice. $\endgroup$ Commented Oct 29, 2022 at 17:41
  • $\begingroup$ I am not sure how image scaling works with text recognition, Naively I imagine that the scale would not matter just the pixel content. I used you options for TextRecognize in the answer I gave and it does work better now. It does not confuse 0 and O but it was not able to find the I character. when it was alone. $\endgroup$ Commented Oct 29, 2022 at 18:38
  • 3
    $\begingroup$ There is resource function CharacterArrayPlot that has other examples to test in the documentation if ever you happen to be interested $\endgroup$ Commented Oct 29, 2022 at 19:08
14
$\begingroup$

This seems to work to a point:

im = Import["https://i.sstatic.net/Tt6AH.png"];
im2 = Binarize[im, 0.6];
im3 = ImageDifference[im2, FillingTransform[im2]];
im4 = Dilation[ColorNegate[im3], 0.4]

You might have to play with parameters of the functions in order to get better results.


enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ Most impressive! +1 $\endgroup$
    – Hans Olo
    Commented Oct 28, 2022 at 19:09
  • $\begingroup$ @HansOlo Thanks! I was at WTC-2022 last week, so, I am still on my "Mathematica-beat." $\endgroup$ Commented Oct 28, 2022 at 19:23
8
$\begingroup$

See links at the end for documentation of functions used in this answer.

EDIT4

This might not work for some tables but one can obtain the regions of the cells in the table using the following steps:

Step 1 :

Obtain a mesh and decompose the mesh components:

mesh = im // ImageMesh // ConnectedMeshComponents;

Step 2 :

Among the mesh components, select only the elements that have holes. These are the ones that have the outline of text inside them:

mesh2=Select[mesh, 
        Length@ConnectedMeshComponents@RegionBoundary@# > 1 &]

Here is a subset of the components:

mesh_comp

The empty grid in the top left is missing in that list but there is no text to recognize anyway. One may just insert a "" in the grid.

With the above regions one can find their locations in the the table of the image using

mesh2 // Map[BoundingRegion]

Then one can either use ImageTrim or the masking option of TextRecognize and follow the steps of the previous edits.


EDIT3

I came to the conclusion that TextRecognize and FindImageText are not reliable enough to get good results. The code below with FindImageText included should give nice results once these functions become more accurate.

To use the codes below one has to first detect the column lines and row lines. Maybe this can be done automatically but I used Manipulate to tweak parameters while extracting horizontal and vertical lines using:

gridlines=Select[lines, 
  Abs[Sin[2*
       VectorAngle[# /. Line -> Identity // Apply[Subtract], {1, 
         0}]]] < 0.01 &] // Graphics

the value 0.01 might need adjusting depending on the image. An example is given in the previous edit.

Then one can use the maketable function in the code below

maketable[Binarize@im, gridlines, $numberOfColumns,type]

where $numberOfColumns is the number of columns and type is either "Character" or "Word". If type is omitted then "Word" is assumed.

The table cell detection and image extraction works well once the lines are found with the method above. The issue is the text recognition.


Code latest version

imageStringDetection[im_,type_:"Word"]:=

Module[{s,blankQ},

If[type==="Word",
    s=TextRecognize[im,"Word",{"Text","Strength"},RecognitionPrior->"Block"];
    If[Not[s==={}],Return[s]]
];

(* Search for characters in image *)

s=TextRecognize[im,"Character",{"Text","Strength"},RecognitionPrior->"Block"];
(*If no character is detected check if the image is blank*)(* ****START:checking case of blank character (see Finish delimeter below)*****)If[s=={},blankQ=1==(im//ImageData//Flatten//DeleteDuplicates//Length)];
If[blankQ,Return[Column@{im,"",Style[1,RGBColor[0.0737601,0.865914,0.132145]]}];,s=TextRecognize[im,"Character",{"Text","Strength"},RecognitionPrior->"Character"];];
(*If still no character has been detected then consider the character to be undetermined*)If[s=={},Print[{im,"Undetermined Character"}];
Return[Column@{im,"$Undetermined",Style[0,Red]}]];
(* ****FINISH:checking case of blank character*****)(*If only one character detected*)If[Depth@s==2,Return[Column@Prepend[{s[[1]],Style[NumberForm[s[[2]],2],Blend[{Red,RGBColor[0.0737601,0.865914,0.132145]},s[[2]]]]},im]]];
(*If multiple characters are detected join them and take the average strength*)Column[{im}~Join~Through@{StringJoin@*Map[First],Style[NumberForm[#,2],Blend[{Red,RGBColor[0.0737601,0.865914,0.132145]},#]]&@*Mean@*Map[Last]}@s]];

grid=ResourceFunction["PrettyGrid"];

removeBorderLines[im_]:=ImageTake[#,{2,-2},{2,-2}]&@First@First@MaximalBy[Last][Values@ComponentMeasurements[im,{"Image","BoundingBoxArea"}]];

makeassoc[im_,gridlines_]:=
Module[{gridCells},

gridCells=gridlines//Apply[RegionUnion]//MeshPrimitives[#,1]&
               //ReplaceAll[Line->Identity]//MapApply[UndirectedEdge]
               //FindCycle[#,{4},All]&//ReplaceAll[UndirectedEdge->List]
               //Map[CrossingPolygon];
               
Association[(#[[1]]->removeBorderLines[#[[2]]]&)/@({RegionCentroid[#],ImageTrim[im,#]}&)/@gridCells]
               
    ];
Clear[maketable];
maketable[im_,gridlines_,numColumn_,type_:"Word"]:=Module[{assoc},assoc=makeassoc[im,gridlines];
grid@Map[imageStringDetection[#,type]&@*assoc,SortBy[#,First]&/@Partition[ReverseSortBy[#,Last]&@Keys@assoc,numColumn],{2}]] 

EDIT2: The major weakness of the code below was that it relied on finding clusters of morphological components of the image to find which components correspond to cells of the table. The method I used did not work for a table "in the wild" such as this table:

|nutrition

The major change to the code is that it will first find the cells of the table by detecting image lines then it will try to recognize the text within the bounding region from the table. I imagine that this is what most algorithms for searching text in a table do. The image recognition seems to be bothered by borders of the image that have pieces of the line of the table. As such I have not yet posted any results and instead explain the method and show parts of the code.

Step 1

Detect lines in image:

lines = ImageLines[EdgeDetect@im2];

lines

Step 2

Select only lines that are approximately horizontal or vertical within a tolerance value:

gridlines = Select[lines, Min@Abs[Subtract @@ #[[1]]] < 10 &];

gridlines

Step 3

Find the rectangles within the lines using the answer here I thank @Domen again for the help.

com2 = gridlines // Apply[RegionUnion] // MeshPrimitives[#, 1] & // 
       ReplaceAll[Line -> Identity] // MapApply[UndirectedEdge] // 
     FindCycle[#, {4}, All] & // ReplaceAll[UndirectedEdge -> List] //
    Map[CrossingPolygon];

Step 4

Use text recognition on each of the regions delimited by the detected rectangles and make a grid. This step can use the masking option of TextRecognize for each rectangle (using all rectangles at once will join all the words) or using ImageTrim to take parts of the image and use image detection on each part. The borders might be affecting the results so I will not post the results yet.


Previous version

EDIT: The code now matches perfectly the characters in the table provided by OP. The other answers here helped to improve the code. The code below has no parameter fine tuning and so is hopefully robust. That said, in another example below, the text recognition algorithm gets confused at times.

Table with elements of the form {image of table component, predicted string, confidence (color coded from red (bad) to green (good))

table

Notice that the algorithm is less confident about the character "1" as the strength is lower and the color is darker (closer to red).


Code

NOTE !!: I changed Norm /@ to N/@ Norm in the FindCluster line below as I realized the sorting does not necessarily work well without the N. However, I have not checked thta the code works with the added N/@ as that code does not feel robust enough anyway. Indeed, it relies on the hypothesis that there are many cells with the same dimensions which might not be true for certain images of tables.

explanation in next section

imageStringDetection[im_]:=

Module[{s,blankQ},

s=TextRecognize[im,"Character",{"Text","Strength"},RecognitionPrior->"Block"];

(* If no character is detected check if the image is blank*)

(* ****  START: checking case of blank character (see Finish delimeter below ) **** *)

If[s=={}, blankQ = 1==(im//ImageData//Flatten//DeleteDuplicates//Length)];

If[blankQ,
    
        Return[Column@{im,"",Style[1,RGBColor[0.0737601, 0.865914, 0.132145]]}];
        ,   
        s=TextRecognize[im,"Character",{"Text","Strength"},RecognitionPrior->"Character"];
];

(* If still no character has been detected then consider the character to be undetermined *)

If[s=={},

    Print[{im, "Undetermined Character"}];
    
    Return[Column@{im,"$Undetermined",Style[0,Red]}]
 ];
 
 (* **** FINISH: checking case of blank character **** *)

(* If only one character detected *)

If[Depth@s==2,Return[Column@Prepend[{s[[1]],
                                    Style[NumberForm[s[[2]],2],
                                    Blend[{Red,RGBColor[0.0737601, 0.865914, 0.132145]},s[[2]]]]
                                    },
                             im]]];

(* If multiple characters are detected join them and take the average strength *)

Column[{im}~Join~Through@{StringJoin@*Map[First],Style[NumberForm[#,2],Blend[{Red,RGBColor[0.0737601, 0.865914, 0.132145]},#]]&@*Mean@*Map[Last]}@s]];

makeassoc[tableim_] :=
   Module[{auxassoc},
   auxassoc = 
   Association[#[[1]] -> #[[2]] & /@ 
     Values@ComponentMeasurements[
       Binarize@tableim, {"Centroid", "Image"}]];
  
   # -> auxassoc[#] & /@ 
    First@MaximalBy[Length]@
      FindClusters[N/@ Norm /@ ImageDimensions /@ auxassoc] // 
   Association
  ];

grid = ResourceFunction["PrettyGrid"];

maketable[im_, numColumn_] :=
Module[{assoc},
assoc = makeassoc[im];
grid@Map[imageStringDetection@ assoc[#] &, 
SortBy[#, First] & /@ 
 Partition[ReverseSortBy[#, Last] &@Keys@assoc, numColumn], {2}]]

Explanation of code

Here is an approach that:

  • extracts image components and their positions

  • removes unwanted components

  • applies text recognition to each component

  • makes a table using the image component positions. An element of the constructed table has an image that corresponds to the original table cell and the predicted text.

Step 1: Make an association between components of an image and their positions in the image. This association has unwanted images that will be removed in the following step.

assoc = Association[#[[1]] -> #[[2]] & /@ 
Values@ComponentMeasurements[Binarize@im, {"Centroid", "Image"}]];

Step 2: In the association there are images that do not correspond to components of the table. They are either too small or too big. They can be removed using FindClusters and, in this case, taking the largest cluster.

Note : Taking the biggest cluster worked here but that might be image dependent and one should check the validity each time.

betterassoc = # -> assoc[#] & /@ 
First@MaximalBy[Length]@
  FindClusters[Norm /@ ImageDimensions /@ assoc] // Association;

The two steps above are summarized into the function makeassoc in the code above.

Step 3: Use the positions in the association to make a table where an element of the new table is the original image,the predicted string and the level of confidence.

This step requires the user to know the number of columns in the table in order to shape it. Perhaps this could be done automatically using Gather with a good test function.

The function maketable follows the procedure above.


Examples

The first image can be generated using (first argument is the image and the second argument is the number of columns)

maketable[im, 11]

To create random strings for the table one may use the resource function RandomString.

rands = ResourceFunction["RandomString"];

Now we can make an image of a table of random characters:

im2 = Rasterize@Grid[#, Frame -> All] &@
Partition[Table[rands[RandomInteger[{1, 3}]], 20], 10]

table2

maketable[im2, 10]

res

Notice that there was an error with the image nw although the text recognition was fairly confident. There is also an error with "or". That said, the predictions do kind of look like the images.


Links below are generated automatically using Mathematica on the text of this answer. May contain errors.

{ImageMesh,ConnectedMeshComponents,Select,Length,RegionBoundary,Here,With,Map,BoundingRegion,ImageTrim,TextRecognize,FindImageText,Manipulate,Abs,Sin,VectorAngle,Line,Identity,Apply,Subtract,Graphics,Binarize,Character,Word,If,Module,Text,RecognitionPrior,Block,Not,Return,ImageData,Flatten,DeleteDuplicates,Column,Style,RGBColor,Print,Red,Depth,Prepend,NumberForm,Blend,Join,Through,StringJoin,First,Mean,Last,ResourceFunction,ImageTake,MaximalBy,Values,ComponentMeasurements,Image,RegionUnion,MeshPrimitives,ReplaceAll,MapApply,UndirectedEdge,FindCycle,All,List,CrossingPolygon,Association,RegionCentroid,Clear,SortBy,Partition,ReverseSortBy,Keys,ImageLines,EdgeDetect,Min,Find,Previous,Table,Norm,N,FindClusters,ImageDimensions,In,Gather,Now,Rasterize,Grid,Frame,RandomInteger,Links,Polygon,RegionBounds,Transpose,Rectangle,StringSplit,StringRiffle,Masking,Reverse,TableForm,Row,For}

$\endgroup$
6
  • $\begingroup$ Two things I can not understand - 1. how evident 1 can be recognized as 4 - 2. why high quality image must be first degraded to such a poor quality to do the recognition on it. I am sure that even humans would have difficulties recognizing some of the characters after such degradation - on the other hand they would not have any problems to recognize original image. (except of the well known issues - distinguishing between 0 - O and l - I (lowercase 'L' and uppercase i) ). $\endgroup$ Commented Oct 29, 2022 at 9:33
  • $\begingroup$ @azerbajdzan I am not sure why it mistakes 1 for 4 although 1 kind of looks like a 4 that has been shrinked at the top. I removed the Colorize which made the images look messy (see update). The results still are not that good. Maybe there is a Python code that does ocr/text recognition for tables. $\endgroup$ Commented Oct 29, 2022 at 10:47
  • $\begingroup$ @azerbajdzan I also tried vectorizing the image with ImageGraphics as a preprocessing part but the result were not really better. $\endgroup$ Commented Oct 29, 2022 at 10:56
  • $\begingroup$ @azerbajdzan I included an image that uses RecognitionPrior -> "Block" as in Anton Antonov's answer. The 1 is no longer mistaken for 4 but there are still mistakes. $\endgroup$ Commented Oct 29, 2022 at 11:07
  • $\begingroup$ I used Syed's options for TextRecognize and it works except for an "I" that was not detected. $\endgroup$ Commented Oct 29, 2022 at 18:54
2
$\begingroup$

The other answer I wrote handles the case where the table has line delimiters. This answer addresses the case where there are no frames.

Method 1

I found out that you can get columns using TextRecognize by transposing the image and then use the line option. The resulting boxes have to be transposed again for the original image without the transposition.

The usual line code:

HighlightImage[im, 
 TextRecognize[im, "Line", "BoundingBox", 
  RecognitionPrior -> "Block"]]

rows

To get the columns we first transpose the image:

im2 = Image@Transpose@ImageData[Binarize@im];

Then we get the lines of the transposed image which after transposition corresponds to the columns

HighlightImage[im2, 
 TextRecognize[im2, "Line", "BoundingBox", 
  RecognitionPrior -> "Block"]]

side

To obtain the columns for the original image without the transposition we have to transpose the bounding boxes:

columns = 
  TextRecognize[im2, "Line", "BoundingBox", 
    RecognitionPrior -> "Block"] /. 
   Rectangle[a__] :> Rectangle @@ Reverse /@ {a};

Now we may visualize both the rows and columns

HighlightImage[im, rows~Join~columns]

grid

The intersection of the lines and columns corresponds to a cell only if the cell has one line. This is the case of the table above. To obtain the cell boxes we can use:

cells = Outer[RegionIntersection, rows, columns];

These cells are ordered. We can obtain a left to right up to down ordering using:

cells2 = Flatten[Reverse /@ cells]; 

To visualize the order in the table one can use

HighlightImage[im, 
 Association[Thread[Range[Length[cells2]] -> cells2]]]

The labels at the top of the first row do not appear

labels

One can then uses TextRecognize on each cell but the borders should be removed using the method provided by @AntonAntonov

Method 2

In the case where there are multiple lines in a cell one would have to group lines belonging to a cell. If the table cells are clearly separated one could use FindClusters.

First we extract characters from the image (one might consider working directly with lines but it would be, subjectively, less pretty for the plot at the end).

boxes = TextRecognize[im, "Character", "BoundingBox", 
RecognitionPrior -> "Block"];

Then, as was done by @Syed we can extract the center of each box:

posbox = boxes // Map[RegionCentroid];

We can define the y (vertical) coordinate of the center of a box as a feature of the list so that elements with similar y coordinates will be grouped together.

s = FeatureExtraction[posbox, Last]

Then we may request 11 features for the 11 rows:

FindClusters[posbox, 11, FeatureExtractor -> s] // ListPlot

points

Perhaps that could also be used for a table where one knows the number of rows and that has multiple lines in some cells but I did not try.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.