Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

One could call this a pattern finding question with dictionary (or WolframAlpha) lookup, but let me give you some background first.

I have a couple of friends who are trying to improve their English listening skills by watching TV shows in English, with English subtitles. I have access to the subtitles in text format which I can import and process as I like. Now it is easy to pick out common words from a list of words using something like Tally, but what I would like to do is more complicated. I want to pick out phrases (and perhaps some longer or slang-like words) that a non-native speaker may not recognize. I understand I need some sort of a lookup for this, and I thought of using WolframAlpha's Phrase:WordData pod, but it doesn't show up for every word, and didn't exactly do what I really wanted. I'll provide an example block of text below if anybody would like to go ahead and try it using whatever method they would like.

myStr = "Wow that must have been an awful result. That was not even close! People didn't even find out until they reviewed the footage.";
myStrList = StringCases[myStr, WordCharacter ..]

And this gives

{"Wow", "that", "must", "have", "been", "an", "awful", "result", "That", "was", "not", "even", "close", "People", "didn", "t", "even", "find", "out", "until", "they", "reviewed", "the", "footage"}

Now I'd like an output that might look like something like

{"awful", "not even close", "find out", "reviewed", "footage"}

I know this is rather vague, but I think determining search patterns and how the search would be executed would be a good start.

share|improve this question

2 Answers 2

You need decent frequency data, which is hard to come by. This data is pretty good; it was collected from Wikipedia. I don't have it installed on this machine and the file is gigantic, so I'll show you how to do it using crappy data from here.

First let's clean up the text.

myStr="Wow that must have been an awful result. That was not even close! People didn't even find out until they reviewed the footage.";
phrase=Fold[
  StringSplit,
  myStr,
  { RegularExpression["[.!?]"], RegularExpression["\\s+"] } ]

{{Wow,that,must,have,been,an,awful,result},{That,was,not,even,close},{People,didn't,even,find,out,until,they,reviewed,the,footage}}

Alright. Out of loyalty, one (abominably slow) option is

rare=
  With[{
    freq=WolframAlpha[#,{{"Frequency:WordData",1},"ComputableData"}]},
    If[
      MatchQ[freq,_Missing],True,
      StringCases[
        freq[[1,1]],
        d:DigitCharacter.. :> ToExpression[d]][[1]] //If[#>2000,True,False]]]&;

Cases[phrase, s_String?rare, Infinity]

{"Wow", "been", "People", "didn't", "reviewed", "footage"}

Doing it with our own data (emphatically, use the better data):

freqData = Import["...\\en-2012.zip", {"en.txt"}];
freqs=
  Append[
    MapAt[
      ToExpression,
      Fold[ StringSplit, freqData, {"\n"," "..} ],
      {All,2} ] ,
    _String->0];
freqs[[All,2]] = Power[Rescale@freqs[[All,2]], 1/3] ;
freqs = Dispatch[ Rule@@@ freqs ] ;

Cases[phrase,w_String/;(ToLowerCase[w]/.freqs)<.1,Infinity]

{"result", "didn't", "reviewed", "footage"}

The data I recommended includes "n-grams" up to 7. Those are sequences of words, for example "not even close". You would test for them in much the same way by chopping up the text into segments of appropriate length:

phrase/.l:{__String}:>ReplaceList[l,{___,a_,b_,c_,___}:>{a,b,c}]

{{{Wow,that,must},{that,must,have},{must,have,been},{have,been,an},{been,an,awful},{an,awful,result}},{{That,was,not},{was,not,even},{not,even,close}},{{People,didn't,even},{didn't,even,find},{even,find,out},{find,out,until},{out,until,they},{until,they,reviewed},{they,reviewed,the},{reviewed,the,footage}}}

You probably want to IgnoreCase and may want to make other decisions concerning, for example, punctuation.

See also this corpus, built on movie subtitles.

share|improve this answer

Think about making a watch-list per say, to run through a string to weed out words you add to your list.

Im sure someone can put some shine on this dull code but it works for me.

myStr="Wow that must have been an awful result. That was not even close! People didn't even find out until they reviewed the footage.";
watchlist={"awful","worst","dont even","not even close"};
DeleteCases[Table[StringCases[myStr,watchlist[[i]]],{i,1,Length[watchlist],1}],{}]
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.