Issues with a Counter that is tallying term appearances

Question

I am working with an array of textfiles, trying to see how frequent certain terms are. However, I am running into some difficulty with my running counts of bigram frequency. Here is a snippet of my code that has been giving me trouble:

SetDirectory[
  "/users/ianmilligan/desktop/project/blah/blah/"];

filelist = FileNames["*bigrams.txt"];
citizenCounter97 = {};

Do[
  citizenchecker97 = Get[file];
  Clear[bigramHash];
  (bigramHash[Sequence @@ #1] = #2) & @@@ citizenchecker97;
  count = bigramHash["ottawa", "citizen"];
  Print[ToString[file] <> ":" <> ToString[count]];
  AppendTo[citizenCounter97, count];
  , {file, filelist}];

The file that is being checked looks like this citizenchecker97=

  {{{"owner", "further"}, 262}, {{"on", "the"}, 206}, {{"the", 
  "women"}, 204}, {{"women", "s"}, 197}, {{"maritime", 
  "baptist"}, 195}, {{"and", "the"}, 193}, {{"for", 
  "the"}, 189}, {{"the", "maritime"}, 186}, {{"by", 
  "the"}, 180}, {{"with", "the"}, 170}, {{"the", 
  "missionaries"}, 146}, {{"that", "the"}, 142}, {{"the", 
  "mission"}, 141}...}}

As it stands, my bigramHash (which stems from an earlier exceptionally helpful SO answer) generates a number if the file contains that bigram (if it appears twice, it generates '2', which I want to AppendTo citizenCounter97. If it doesn't contain the bigram, however, it generates bigramHash[ottawa, citizen].

My plan was to just generate a list, subsequently just Cases[citizenCounter97,_Integer]; the thing, and have a sense of how often the given term appeared.

Problem is, the results did not make sense. When I went in and started de-bugging, I realized that the counts that come out of Print were completley different than the lists it was generating! The result for citizenCounter97 would be like this:

{2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, \
2, 2, 2, 2, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, \
2, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 
 bigramHash["ottawa", "citizen"], bigramHash["ottawa", "citizen"], 
 bigramHash["ottawa", "citizen"]}

Whereas the real Print output would show that the first 12 results were nothing, the 13th was 2, and then nothing again until 27, when it became 1.

So basically, I am confused as to what's happening, why my counter is completely divergent from the reality as I do it manually/debug with Print.

Thanks in advance, and happy to clarify as best I can.

---- edited to add ----

I suspect one solution lies in an If statement - i.e. If count is an integer, then append that; if the 'If' ends up being not an integer, then return 0. I've tried a variety of calls using Head however, and am still hitting a brick wall.

Answer 1

If I am following your code, I think the problem is that you are collecting expressions such as bigramHash["ottawa", "citizen"] which will evaluate differently depending on the DownValue associated with bigramHash. Therefore, these expression will evaluate to whatever the final DownValue of bigramHash["ottawa", "citizen"] is, if present.

Here is a simplified example of the problem:

filelist = {
   {{"a", 1}, {"b", 2}},
   {{"cat", 5}, {"dog", 3}},
   {{"a", 4}, {"b", 6}}
 };

Reap[Do[
   Clear[bigramHash];
   (bigramHash[#] = #2) & @@@ file;
   count = bigramHash["a"];
   Print[count];
   Sow[count],
   {file, filelist}
]][[2, 1]]

During evaluation of In[26]:= 1

During evaluation of In[26]:= bigramHash[a]

During evaluation of In[26]:= 4

{1, 4, 4}

One simple fix is to define a default DownValue e.g. bigramHash[___] = 0:

Reap[Do[
   Clear[bigramHash];
   bigramHash[___] = 0;
   (bigramHash[#] = #2) & @@@ file;
   count = bigramHash["a"];
   Print[count];
   Sow[count],
   {file, filelist}
]][[2, 1]]

During evaluation of In[36]:= 1

During evaluation of In[36]:= 0

During evaluation of In[36]:= 4

{1, 0, 4}