11
$\begingroup$

Let say I have three blocks of text like the following.

{"first block of text with random content","different block of text","1 2 3 4"}

Now let's say I want to determine which block of text matches "content random with" closest what would I do?

Apologies if this question is to broad thanks again!

I want to distinguish between 2 blocks of text that are identical except for order and ideally a block of text that has more appearances of the same word then another. I was hoping mathematica had some in built function that would help make this easier. I wonder how documentation search work.s

$\endgroup$
  • $\begingroup$ Nearest[(StringPadRight[#, 50] & /@ {"first block of text with random content", "different block of text", "1 2 3 4"}), "content random with"] $\endgroup$ – Conor Cosnett Aug 28 '16 at 22:57
9
+50
$\begingroup$

As of Mathematica 11:

filenames = Table[CreateFile[], 3];
content = {"first block of text with random content", "different block of text", "1 2 3 4"};

MapThread[Put, {content, filenames}];
index = CreateSearchIndex[filenames];

Perform searches using TextSearch:

Snippet /@ Normal@TextSearch[index, "block"]

In order to rank search results, score them using SearchAdjustment. You can also experiment with SearchQueryString to get a series of search results which have different priorities. Finally, you can post process and rank according to measurements you were not able to capture by the following methods. Perhaps counting the number of times certain words appear as you indicate that you would like to do.

$\endgroup$
9
$\begingroup$
    strs = {"first block of text with random content", 
            "different block of text", "1 2 3 4"};

    Nearest[(StringPadRight[#, 50] & /@ strs), "content random with"]

This is a deep and complex question apparently:



Mathematica has a menagerie of built in goodies to assemble your own variant.

  • EditDistance
  • DamerauLevenshteinDistance
  • NeedlemanWunschSimilarity
  • SmithWatermanSimilarity

using Padding to make the strings the same length and then using Levenshtein distance (EditDistance):

Module[{},

    strs = {"first block of text with random content",
            "different block of text", 
            "1 2 3 4"};

    pattern = "content random with";


    lengthOfLongestString = 
      StringLength@First@MaximalBy[strs, StringLength];



    vectors = (PadRight[#, lengthOfLongestString] & /@ToCharacterCode /@ strs);

    EditDistance[#, ToCharacterCode@"content random with"] & /@ vectors

 ]

    (*{25, 32, 39}*)

Using this test the first string is now closest to the pattern


SmithWatermanSimilarity does it :)

strs = {"first block of text with random content", "different block of text", "1 2 3 4"};
metric3 = SmithWatermanSimilarity["content random with", #] &

MaximalBy[strs, metric3]

(* {"first block of text with random content"} *)

How about checking if the string under question matches each of the three words individually. If it matches all 3 words it passes.

In this first version of my code I have done the parsing manually so you can see what it does:

  Cases[{"first block of text with random content", 
         "different block of text", 
         "1 2 3 4"}, 
        a_ /;
    StringMatchQ[a, "*with*"] && 
    StringMatchQ[a, "*content*"] && 
    StringMatchQ[a, "*random*"]]   

>

Automated version

 Cases[
    {"first block of text with random content", 
     "different block of text", 
     "1 2 3 4"}, 

  a_ /; Fold[And, 
           Quiet[StringMatchQ[a, #] & /@ (("*" ~~ # ~~ "*") & /@StringSplit["content random with"])]]]

Another test of greater complexity would be to test if the three words occur adjacently in a string...


$\endgroup$
  • 1
    $\begingroup$ This is what I was just trying to write up myself. The only problem with this approach is it doesn't distinguish 2 blocks of text that are identical except for order and ideally a block of text that has more appearances of the same word then another. I was hoping mathematica had some in built function that would help make this easier. I wonder how documentation search work.s $\endgroup$ – William Aug 28 '16 at 20:46
  • 1
    $\begingroup$ @Liam Try StringCount to rank strings by the number of appearances of the search words. You could then test if the words are in the right order or not and combine these two tests into a search rank. $\endgroup$ – Conor Cosnett Aug 28 '16 at 21:08
  • $\begingroup$ I'll add a major bounty in a couple days I hope you will consider posting another answer then. $\endgroup$ – William Aug 28 '16 at 21:09
3
$\begingroup$

If we define

strs = {"first block of text with random content", 
   "different block of text", "1 2 3 4"};
metric = EditDistance["content random with", #] &
(* EditDistance["content random with", #1] & *)

We can do

MinimalBy[strs, metric]
(* {"different block of text", "1 2 3 4"} *)

This tells us that the 2nd and 3rd string are equal closest by edit distance. A different metric could give a different answer.

EDIT

For matching by words rather than characters, you could use (suggested by @Conor Cosnett)

metric2 = -StringCount[#, StringSplit["content random with"]] &;
MinimalBy[strs, metric2]
(* {"first block of text with random content"} *)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.