I want to create a bag of bigrams in a set of documents and calculate the TF-IDF vector of each document. To calculate the bigram of the text I used the following code: The small example of the data (each element in the list is a different document)

data = {"The food at snack is a selection of popular Greek dishes. 
The appetizer tray is good as is the Greek salad. We were underwhelmed with 
the main courses. There are 4-5 tables here so it's 
sometimes hard to get seated.","This little place in Soho is wonderful. I 
a lamb sandwich and a glass of wine. The price shocked me for how small the 
serving was, but then again, this is Soho. The staff can be a little snotty 
and rude, but the food is great, just don't expect world-class service.", 
"ordered lunch for 15 from Snack last Friday.  On time, nothing 
missing and the food was great.  I have added it to the regular 
company lunch list, as everyone enjoyed their meal."}

The way that I create the bigrams in the file (set of documents) and calculate the TF for each bigram in specific document:

bigram =
      <|First[#] <> " " <> Last[#] -> 1|> & /@ 
          StringSplit[StringReplace[data[[i]], PunctuationCharacter ->""]],
          2, 1],
    {i, 1, Length@data}]

The way that I calculate the frequency of the bigrams in the file (ITF):

bigramUniqe = <||>
Scan[(If[MissingQ[bigramUniqe[#]], AssociateTo[bigramUniqe, # -> 1], 
 AssociateTo[bigramUniqe, # -> (bigramUniqe[#] + 1)]]) &, bigram];

But in this way, I do not succeed to count the frequency of document that contains the specific bigrams( I have some issue with the level specification of the Associate). Anyway, I look for a more efficient way to implement this task.Thank in advance for any suggestions.

  • $\begingroup$ Consider that removing punctuation will make things like "it's" and "its" the same, although they obviously do not serve the same purpose. Is that intended / acceptable? $\endgroup$ – MarcoB Apr 29 '18 at 17:52

What do you think about skipping the StringJoin and storing a bigram as a pair of strings?

getbigrams[text_String] := Module[{words},
  words = 
    ToLowerCase[StringDelete[text, PunctuationCharacter]]];
  Counts[Partition[words, 2, 1]]

That can save about 40 % of time:

data = ExampleData /@ ExampleData["Text"];

a = Table[
     Merge[<|First[#] <> " " <> Last[#] -> 1|> & /@ 
          PunctuationCharacter -> ""]], 2, 1], Total], {i, 1, 
      Length@data}]; // AbsoluteTiming // First

b = getbigrams /@ data; // AbsoluteTiming // First

Values[a] == Values[b]





I've implemented the Bigrams and TF-IDF calculation according to @Henrik Schumacher suggestion (see the attached code). Thanks a lot, @Henrik Schumacher.

 bigramUniqe = <||>

 getbigrams[text_String] := Module[{words, res, temp},
 words = StringSplit[StringDelete[text, PunctuationCharacter]];
 temp = Partition[words, 2, 1];
 Scan[(If[MissingQ[bigramUniqe[#]], AssociateTo[bigramUniqe, # -> 1],
   AssociateTo[bigramUniqe, # -> (bigramUniqe[#] + 1)]]) &, temp];
 res = Counts[temp]


 bigram = Table[getbigrams[data[[i]]], {i, 1, Length@data}]; 


 KeyValueMap[# ->  Log[2, Length@data/bigramUniqe[[Key[#]]]] &, bigramUniqe]
  • 2
    $\begingroup$ If Henrik’s answer solved your problem the thing to do is accept it to show that this problem is solved and also give him a bit of a reward for being helpful. $\endgroup$ – b3m2a1 Apr 30 '18 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.