# Trying to build a code to translate handwritten formulas using ML

I'm lazy writing equations in Latex so I want a code to translate handwritten equations. This question is related to an old experiment I'm reviewing during vacations, here. I'm trying to use Classify and although I was told it won't work, it does not seem impossible to me. Wouldn't it be great to just take a photo of the blackboard with a cell phone and just translate everything into Latex?

Suppose we have the Schrödinger equation, handwritten:

We must be able to identify each symbol in the equation. For this purpose I used Classify:

where I trained the machine to recognize the symbols in the equation (plus two arbitrary constants):

It worked fine recognizing individual symbols:

However the problem comes when I want to identify the full equation:

The problem to solve is to separate the equations into its components. It is needed a function like ImageCases, but more specifically a function of the type:

IComponents[equation_image,{symbols_categories}]


where in the first argument it is the input equation, and in the second argument there is the set of symbols' categories.

(Of course once we get the translation to Mathematica it is possible to get latex code through TeXForm or something else).

---------------UPDATE 5/1/2020 ----------------

I don`t know if I must open another thread, but the question is that I have an idea to build the function IComponents which I mention above. However, I'm stalled since using handwritten symbols don't work as expected:

In other words that piece of code must separate the symbol "A" handwritten from "A B". I don't know how to fine tune it to do it.

Doing this is an important step forward achiving the goal of the post.

• Your question is a very good one, and the answer to it will be extremely useful for people like me dreaming of such an application translating hand-written mathematical equations into LaTeX automatically. Jan 2, 2021 at 22:51
• @Tugrul Temel, thank you, I think we are close... Jan 2, 2021 at 22:55
• Please keep me informed if you make any progress in this endeavor. Let us hope some genius people will react to this. Jan 2, 2021 at 22:59
• Use this: mathpix.com - I tried it on your top example and it gave me $$i \hbar \frac{\partial|\psi\rangle}{\partial t} \hat{H}|\psi\rangle$$ so I'm pleased to find out it also almost works on handwritten formulae too, but it did miss out the = sign. Jan 3, 2021 at 13:58
• @flinty, yes I knew it, thankx, but it is paid, and doing it with Mathematica has another taste Jan 3, 2021 at 14:59