Comparing natural language strings

Question

I have a large spreadsheet with thousands of rows/columns of text. Some are saying essentially the same thing but in different ways.

When you enter a natural language question Wolfram Alpha makes a good stab at 'understanding' what you enter. For example, 'how far is the sun from the earth' has the same answer as 'what is the distance from the earth to the sun'. Mathematica converts it to [Sun][ Distance from the Earth].

Does MMA have built-in (undocumented) functions that would allow us to compare text strings and test for equivalence (or fuzzy equivalence)?

Answer 1

You should take a look at:

• New in the Wolfram Language: GrammarRules
• Programmable Linguistic Interface
• Sequence Alignment & Comparison

You can also use Machine Learning to help classify strings. Here is how to train a classifier to understand if a string talks about a cat a or a dog.

cat = ToLowerCase[Import["http://en.wikipedia.org/wiki/Cat"]];
catTR = StringCases[
   StringTake[
    cat, {2000, 
     5000}], (x__ /; StringLength[x] == 15) ~~ ("cat" | 
      "cats") ~~ (y__ /; StringLength[y] == 15), Overlaps -> All];

dog = ToLowerCase[Import["http://en.wikipedia.org/wiki/Dog"]];
dogTR = StringCases[
   StringTake[
    dog, {2000, 
     5000}], (x__ /; StringLength[x] == 15) ~~ ("dog" | 
      "dogs") ~~ (y__ /; StringLength[y] == 15), Overlaps -> All];

catdog = Classify[<|"CAT" -> catTR, "DOG" -> dogTR|>];

Now check - note classifier is not confused by presence of dog in cat and vice versa:

catdog["dogma is not a cat friend"]

"CAT"

catdog["strange dog is in a good location"]

"DOG"

Answer 2

You could measure the similarity between two strings.

One simple approach is to convert all strings into the bag-of-words model and then compare the resulting vectors. This could work well if the strings contain the same words, but not in the same order.

Nearest[vectors, x, DistanceFunction -> CosineDistance]

Will give you a measure of how close vector x is to each of the other vectors. This is called vector distance classification. (Normally one would define classes and have sample vectors for each class, to which new vectors are compared. The new vector would then be added to the same class as the nearest sample vector.)

It would probably help if you included some example strings, so that one could see in what way they're similar (and how long they are).