# Issues with a Counter that is tallying term appearances

I am working with an array of textfiles, trying to see how frequent certain terms are. However, I am running into some difficulty with my running counts of bigram frequency. Here is a snippet of my code that has been giving me trouble:

SetDirectory[
"/users/ianmilligan/desktop/project/blah/blah/"];

filelist = FileNames["*bigrams.txt"];
citizenCounter97 = {};

Do[
citizenchecker97 = Get[file];
Clear[bigramHash];
(bigramHash[Sequence @@ #1] = #2) & @@@ citizenchecker97;
count = bigramHash["ottawa", "citizen"];
Print[ToString[file] <> ":" <> ToString[count]];
AppendTo[citizenCounter97, count];
, {file, filelist}];


The file that is being checked looks like this citizenchecker97=

  {{{"owner", "further"}, 262}, {{"on", "the"}, 206}, {{"the",
"women"}, 204}, {{"women", "s"}, 197}, {{"maritime",
"baptist"}, 195}, {{"and", "the"}, 193}, {{"for",
"the"}, 189}, {{"the", "maritime"}, 186}, {{"by",
"the"}, 180}, {{"with", "the"}, 170}, {{"the",
"missionaries"}, 146}, {{"that", "the"}, 142}, {{"the",
"mission"}, 141}...}}


As it stands, my bigramHash (which stems from an earlier exceptionally helpful SO answer) generates a number if the file contains that bigram (if it appears twice, it generates '2', which I want to AppendTo citizenCounter97. If it doesn't contain the bigram, however, it generates bigramHash[ottawa, citizen].

My plan was to just generate a list, subsequently just Cases[citizenCounter97,_Integer]; the thing, and have a sense of how often the given term appeared.

Problem is, the results did not make sense. When I went in and started de-bugging, I realized that the counts that come out of Print were completley different than the lists it was generating! The result for citizenCounter97 would be like this:

{2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, \
2, 2, 2, 2, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, \
2, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2,
bigramHash["ottawa", "citizen"], bigramHash["ottawa", "citizen"],
bigramHash["ottawa", "citizen"]}


Whereas the real Print output would show that the first 12 results were nothing, the 13th was 2, and then nothing again until 27, when it became 1.

So basically, I am confused as to what's happening, why my counter is completely divergent from the reality as I do it manually/debug with Print.

Thanks in advance, and happy to clarify as best I can.

I suspect one solution lies in an If statement - i.e. If count is an integer, then append that; if the 'If' ends up being not an integer, then return 0. I've tried a variety of calls using Head however, and am still hitting a brick wall.

-
I'm confused about what you are wanting and what the file citizenchecker97 contains. Could you describe what you want as a black box, i.e. what goes in and what do you hope to come out? –  Andy Ross Mar 29 '12 at 0:59

If I am following your code, I think the problem is that you are collecting expressions such as bigramHash["ottawa", "citizen"] which will evaluate differently depending on the DownValue associated with bigramHash. Therefore, these expression will evaluate to whatever the final DownValue of bigramHash["ottawa", "citizen"] is, if present.

Here is a simplified example of the problem:

filelist = {
{{"a", 1}, {"b", 2}},
{{"cat", 5}, {"dog", 3}},
{{"a", 4}, {"b", 6}}
};

Reap[Do[
Clear[bigramHash];
(bigramHash[#] = #2) & @@@ file;
count = bigramHash["a"];
Print[count];
Sow[count],
{file, filelist}
]][[2, 1]]

During evaluation of In[26]:= 1

During evaluation of In[26]:= bigramHash[a]

During evaluation of In[26]:= 4

{1, 4, 4}


One simple fix is to define a default DownValue e.g. bigramHash[___] = 0:

Reap[Do[
Clear[bigramHash];
bigramHash[___] = 0;
(bigramHash[#] = #2) & @@@ file;
count = bigramHash["a"];
Print[count];
Sow[count],
{file, filelist}
]][[2, 1]]

During evaluation of In[36]:= 1

During evaluation of In[36]:= 0

During evaluation of In[36]:= 4

{1, 0, 4}

-
Fantastic, thank you for deciphering my code. The issue laid in not having the right terminology to describe my problem! The bigramHash[__]=0 addition works swimmingly, and the results look like they should. –  programming_historian Mar 29 '12 at 13:34