11
$\begingroup$

I have a large spreadsheet with thousands of rows/columns of text. Some are saying essentially the same thing but in different ways.

When you enter a natural language question Wolfram Alpha makes a good stab at 'understanding' what you enter. For example, 'how far is the sun from the earth' has the same answer as 'what is the distance from the earth to the sun'. Mathematica converts it to [Sun][ Distance from the Earth].

Does MMA have built-in (undocumented) functions that would allow us to compare text strings and test for equivalence (or fuzzy equivalence)?

$\endgroup$
  • $\begingroup$ Unfortunately I can't offer much in the way of practical insight, but I wonder if you could work something out starting from the SemanticInterpretation and WordData functions. In particular, WordData seems to be able to give the general family of meaning to which a particular word belongs. Maybe that could be a start for building a similarity measure? $\endgroup$ – MarcoB Apr 28 '15 at 4:52
5
$\begingroup$

You could measure the similarity between two strings.

One simple approach is to convert all strings into the bag-of-words model and then compare the resulting vectors. This could work well if the strings contain the same words, but not in the same order.

Nearest[vectors, x, DistanceFunction -> CosineDistance]

Will give you a measure of how close vector x is to each of the other vectors. This is called vector distance classification. (Normally one would define classes and have sample vectors for each class, to which new vectors are compared. The new vector would then be added to the same class as the nearest sample vector.)

It would probably help if you included some example strings, so that one could see in what way they're similar (and how long they are).

$\endgroup$
7
$\begingroup$

You should take a look at:

You can also use Machine Learning to help classify strings. Here is how to train a classifier to understand if a string talks about a cat a or a dog.

cat = ToLowerCase[Import["http://en.wikipedia.org/wiki/Cat"]];
catTR = StringCases[
   StringTake[
    cat, {2000, 
     5000}], (x__ /; StringLength[x] == 15) ~~ ("cat" | 
      "cats") ~~ (y__ /; StringLength[y] == 15), Overlaps -> All];

dog = ToLowerCase[Import["http://en.wikipedia.org/wiki/Dog"]];
dogTR = StringCases[
   StringTake[
    dog, {2000, 
     5000}], (x__ /; StringLength[x] == 15) ~~ ("dog" | 
      "dogs") ~~ (y__ /; StringLength[y] == 15), Overlaps -> All];

catdog = Classify[<|"CAT" -> catTR, "DOG" -> dogTR|>];

Now check - note classifier is not confused by presence of dog in cat and vice versa:

catdog["dogma is not a cat friend"]

"CAT"

catdog["strange dog is in a good location"]

"DOG"

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.