14
$\begingroup$

Can anyone see a way to import a grid of numbers from

http://www.mit.edu/~puzzle/2019/puzzle/lantern_festival.html

into Mathematica? TextRecognize doesn't seem to directly work

enter image description here

$\endgroup$
18
$\begingroup$

1 - Summary of a simple solution

In this particular DIGIT case there is a very simple solution based on neural nets (NNs)trained on MNIST Data. It is just a few lines of code:

i=Import["https://i.stack.imgur.com/LC2c2.png"];
imageGRID = ImagePartition[i, Scaled[1/22]];
lenet = NetModel["LeNet Trained on MNIST Data"];
test[x_] := If[ImageDistance[imageGRID[[2, 2]], x] > 10, lenet[x], "-"]
Grid[imageGRID /. x_Image :> test[x] /. 7 -> 1, Frame -> All]

enter image description here

2 - How it wroks

Now let's go in detail about it. In Wolfram NN repo there are 2 directly relevant NNs (as of today):

I will go with the simplest - LeNet, let's get it from the repo:

lenet = NetModel["LeNet Trained on MNIST Data"];

Next get this image:

enter image description here

i=Import["https://i.stack.imgur.com/LC2c2.png"];

Now - partition it into an a matrix of sub-images -- one sub-image per digit. Your image got 22 boxes vertically and horizontally - so this is how you do it:

imageGRID = ImagePartition[i, Scaled[1/22]]

enter image description here

Now we can run LeNet on recognizing the digits, but we got a few little problems here.

  • LeNet is not trained on blank images - images without digits - it always expects a digit. So if you feed it blank it will make up a closest possible digit it thinks it corresponds to. So we need a way to test for blanks. THere are many ways - but let's just use a this test (where imageGRID[[2, 2]] is a sample blank image):

    test[x_] := If[ImageDistance[imageGRID[[2, 2]], x] > 10, lenet[x], "-"]
    
  • Another problem - LeNet can get confused with some of the typed digits. It will think 1 is a 7 actually due to the font chosen in your original image. This depends on specific images and fonts and can be customary hot-fixed. To avoid hacks I use here, you can train your own LeNet easily on the digits fo your type. Docs have a lot of examples about it.

So here is your final result:

Grid[imageGRID /. x_Image :> test[x] /. 7 -> 1, Frame -> All]

enter image description here

So simple with modern AI :-) And actually you can train a NN to take your original image grid and return a matrix of values. Maybe image2image nets' architecture would be interesting to try to adopt for this, as matrix is just another image; you can find those nets in Wolfram NN repo.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks! BTW, SVHN trained digit recognizer in wolfram library may be useful -- ufldl.stanford.edu/housenumbers $\endgroup$ – Yaroslav Bulatov Jan 5 at 16:52
  • $\begingroup$ I like your solution! If the original image had a 7, your last replacement would cause problems. It would be great to start with pre-trained MNIST and extend it in this case for the blank and for the 7 versus 1. $\endgroup$ – Mark R Jan 6 at 21:38
4
$\begingroup$

Here is a semi-manual way to do it :

Importation of the image, cutting it in a 48X48 array of small images, removing the borders :

imageArray = img  //
   RightComposition[
    ImagePartition[#, 40, 40] &
    , Map[Binarize, #, {2}] &
    , Map[ImageCrop[#, 38] &, #, {2}] &
    ];
(* a view of a piece of the array :  *)  
imageArray[[10 ;; 15, 5 ;; 10]] // Grid[#, Dividers -> All] &  

enter image description here

Then regrouping with FindCluster[#,5] (5 because we want 5 groups), removing exact duplicates (with Union) and see the result :

imageArray //
 RightComposition[
  Flatten
  , FindClusters[#, 5] &
  , (Union /@ # &)
  , Column[Row /@ #, Dividers -> All] &
  ]   

enter image description here

There's no errors, so one can manually create the correspondances between the groups of images and the numbers :

  rules = {1 -> "-", 2 -> 1, 3 -> 3, 4 -> 2, 5 -> 0}  

The final result :

 imageArray //
 RightComposition[
  ClusteringComponents[#, 5] &
  , # /. rules  &
  , Grid]  

enter image description here ]

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I don't know how FindClusters works, but it's probably less "black box magic" than NN. $\endgroup$ – andre314 Jan 5 at 16:50
1
$\begingroup$

Inspired by the solution of Vitaliy Kaurov and using his initial code to import the image and create the imageGRID, I came up with the following:

representativeDigits = 
 Association[{imageGRID[[1, 1]] -> "-", imageGRID[[1, 7]] -> 1, 
   imageGRID[[1, 8]] -> 3, imageGRID[[1, 10]] -> 2, 
   imageGRID[[1, 12]] -> 0}];
Grid[Partition[
  representativeDigits[
     First@Nearest[Keys@representativeDigits, #]] & /@ 
   Flatten@imageGRID, 22], Frame -> All]

No neural net, just using "Nearest" and giving it some values that show what we are seeing. Not nearly as fun as using a neural network but kind of nice too. Of course, it won't work if there aren't examples of each of the digits we want.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.