# Importing a grid of numbers from an image (sudoku like)

Can anyone see a way to import a grid of numbers from

http://www.mit.edu/~puzzle/2019/puzzle/lantern_festival.html

into Mathematica? TextRecognize doesn't seem to directly work

# 1 - Summary of a simple solution

In this particular DIGIT case there is a very simple solution based on neural nets (NNs)trained on MNIST Data. It is just a few lines of code:

i=Import["https://i.stack.imgur.com/LC2c2.png"];
imageGRID = ImagePartition[i, Scaled[1/22]];
lenet = NetModel["LeNet Trained on MNIST Data"];
test[x_] := If[ImageDistance[imageGRID[[2, 2]], x] > 10, lenet[x], "-"]
Grid[imageGRID /. x_Image :> test[x] /. 7 -> 1, Frame -> All]


# 2 - How it wroks

Now let's go in detail about it. In Wolfram NN repo there are 2 directly relevant NNs (as of today):

I will go with the simplest - LeNet, let's get it from the repo:

lenet = NetModel["LeNet Trained on MNIST Data"];


Next get this image:

i=Import["https://i.stack.imgur.com/LC2c2.png"];


Now - partition it into an a matrix of sub-images -- one sub-image per digit. Your image got 22 boxes vertically and horizontally - so this is how you do it:

imageGRID = ImagePartition[i, Scaled[1/22]]


Now we can run LeNet on recognizing the digits, but we got a few little problems here.

• LeNet is not trained on blank images - images without digits - it always expects a digit. So if you feed it blank it will make up a closest possible digit it thinks it corresponds to. So we need a way to test for blanks. THere are many ways - but let's just use a this test (where imageGRID[[2, 2]] is a sample blank image):

test[x_] := If[ImageDistance[imageGRID[[2, 2]], x] > 10, lenet[x], "-"]

• Another problem - LeNet can get confused with some of the typed digits. It will think 1 is a 7 actually due to the font chosen in your original image. This depends on specific images and fonts and can be customary hot-fixed. To avoid hacks I use here, you can train your own LeNet easily on the digits fo your type. Docs have a lot of examples about it.

So here is your final result:

Grid[imageGRID /. x_Image :> test[x] /. 7 -> 1, Frame -> All]


So simple with modern AI :-) And actually you can train a NN to take your original image grid and return a matrix of values. Maybe image2image nets' architecture would be interesting to try to adopt for this, as matrix is just another image; you can find those nets in Wolfram NN repo.

• Thanks! BTW, SVHN trained digit recognizer in wolfram library may be useful -- ufldl.stanford.edu/housenumbers Jan 5 '20 at 16:52
• I like your solution! If the original image had a 7, your last replacement would cause problems. It would be great to start with pre-trained MNIST and extend it in this case for the blank and for the 7 versus 1. Jan 6 '20 at 21:38

Here is a semi-manual way to do it :

Importation of the image, cutting it in a 48X48 array of small images, removing the borders :

imageArray = img  //
RightComposition[
ImagePartition[#, 40, 40] &
, Map[Binarize, #, {2}] &
, Map[ImageCrop[#, 38] &, #, {2}] &
];
(* a view of a piece of the array :  *)
imageArray[[10 ;; 15, 5 ;; 10]] // Grid[#, Dividers -> All] &


Then regrouping with FindCluster[#,5] (5 because we want 5 groups), removing exact duplicates (with Union) and see the result :

imageArray //
RightComposition[
Flatten
, FindClusters[#, 5] &
, (Union /@ # &)
, Column[Row /@ #, Dividers -> All] &
]


There's no errors, so one can manually create the correspondances between the groups of images and the numbers :

  rules = {1 -> "-", 2 -> 1, 3 -> 3, 4 -> 2, 5 -> 0}


The final result :

 imageArray //
RightComposition[
ClusteringComponents[#, 5] &
, # /. rules  &
, Grid]


]

• I don't know how FindClusters works, but it's probably less "black box magic" than NN. Jan 5 '20 at 16:50

Inspired by the solution of Vitaliy Kaurov and using his initial code to import the image and create the imageGRID, I came up with the following:

representativeDigits =
Association[{imageGRID[[1, 1]] -> "-", imageGRID[[1, 7]] -> 1,
imageGRID[[1, 8]] -> 3, imageGRID[[1, 10]] -> 2,
imageGRID[[1, 12]] -> 0}];
Grid[Partition[
representativeDigits[
First@Nearest[Keys@representativeDigits, #]] & /@
Flatten@imageGRID, 22], Frame -> All]


No neural net, just using "Nearest" and giving it some values that show what we are seeing. Not nearly as fun as using a neural network but kind of nice too. Of course, it won't work if there aren't examples of each of the digits we want.