# What's the principle behind the new MoleculeRecognize function?

I played a bit with the new MoleculeRecognize function implemented in the newly released WM 12.1:
which simply recognizes a molecule in image and returns it as a Molecule object.

MoleculeRecognize[image]


It is really an amazing tool but what I really want to know is the algorithm behind this cool function? Is it Machine-learning engine or a classical Image-analysis-based pipeline! Also I want to know its accuracy and eventual limitations?

• In fact, molecule OCR has precedent (e.g. this and this), and MoleculeRecognize[] just happens to be piggybacking on software like those. – J. M.'s torpor Mar 23 '20 at 4:55

Using ResourceFunction["PrintDefinitions"][MoleculeRecognize], one can determine that the source implementation is MolVec. The specific source code is:

ChemistryMolVecLinkPrivatemolStringFromImageFile[
Pattern[path,
Blank[String]]] := Block[
{cleanup, jFile, molString},
ChemistryMolVecLinkPrivateinitialize[];
JLinkJavaBlock[
molString =
Quiet[ChemistryMolVecLinkPrivaterunMolvec @ jFile,
cleanup[];
molString
]
];


which is pretty clear. You can check out this demo to see an example of MolVec outside of Wolfram Language.

Looking at the MolVec code, it seems like it uses a variety different methods to attempt to extract molecular information, but they all seem to be OCR related (ie Image-analysis-based) rather than a machine-learning-based pipeline.

Information about MolVec accuracy and performance can be found in this presentation, linked from the demo above. There does not seem to be any specific detail about its limitations on the web, and I would posit that after a point, this becomes no longer a Mathematica question :)

• thank you @Carl for answering the How? part.. It remains the accuracy and limitations one. – Youphyso Mar 22 '20 at 11:12
• I added a further link with more details, but I would have to say that this might be a little too broad for a Mathematica-specific forum. I would suggest you perhaps email the author of MolVec to gather more details! – Carl Lange Mar 22 '20 at 11:34
• Hi @Carl I took a look at the MolVec code which was large and complicated as expected from a classical approach ,I wonder why not implementing a Maching-Learning pipeline with probably less and manageable code I think?! may be the nature of this problem is not suitable for ML! we need a ML expert opinion here. – Youphyso Mar 22 '20 at 11:37
• @Youphyso - have a look at this paper for more on the ML approach to this problem. – Jason B. Mar 23 '20 at 14:02

I am one of the authors of Molvec. It was designed in a classical approach using various image processing heuristics to guess the structure. While it's on my todo list, we still haven't published a Molvec paper. However there is a recent 3rd party comparison paper which I did not have any part in that compared Molvec to other similar tools and it's available here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7541205/ which includes accuracy and performance metrics. We have used the results in this paper to make improvements in newer versions as well.

As far as limitations, (as of this writing that's Molvec version 0.9.8) there are some things molvec does really well and somethings it doesn't do well. First, it tries to really hard to turn any image you provide it into some kind of structure even when the image is not a structure. Furthermore text from captions or nearby paragraphs might confuse Molvec as well.
From a Machine Learning perspective having something sit in front using ML to find areas that aren't structures would help, for now I would just recommend being careful how you crop your image you submit to molvec.

Aside from that, Molvec has problems with structures that have brackets or variable groups like Markush structures. It thinks those lines for the brackets and labels are meant to be bonds. But this should be fixable in future versions.

These issues haven't been a high priority for us because Molvec's main usecase is for structure registration and curation during regulatory data entry. These registrars use Molvec as a first pass entering structure information and carefully review the results and make edits.

Thanks for your question and it's great it's been incorporated into Mathmatica.