Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I am working with an array of textfiles, trying to see how frequent certain terms are. However, I am running into some difficulty with my running counts of bigram frequency. Here is a snippet of my code that has been giving me trouble:

SetDirectory[
  "/users/ianmilligan/desktop/project/blah/blah/"];

filelist = FileNames["*bigrams.txt"];
citizenCounter97 = {};

Do[
  citizenchecker97 = Get[file];
  Clear[bigramHash];
  (bigramHash[Sequence @@ #1] = #2) & @@@ citizenchecker97;
  count = bigramHash["ottawa", "citizen"];
  Print[ToString[file] <> ":" <> ToString[count]];
  AppendTo[citizenCounter97, count];
  , {file, filelist}];

The file that is being checked looks like this citizenchecker97=

  {{{"owner", "further"}, 262}, {{"on", "the"}, 206}, {{"the", 
  "women"}, 204}, {{"women", "s"}, 197}, {{"maritime", 
  "baptist"}, 195}, {{"and", "the"}, 193}, {{"for", 
  "the"}, 189}, {{"the", "maritime"}, 186}, {{"by", 
  "the"}, 180}, {{"with", "the"}, 170}, {{"the", 
  "missionaries"}, 146}, {{"that", "the"}, 142}, {{"the", 
  "mission"}, 141}...}}

As it stands, my bigramHash (which stems from an earlier exceptionally helpful SO answer) generates a number if the file contains that bigram (if it appears twice, it generates '2', which I want to AppendTo citizenCounter97. If it doesn't contain the bigram, however, it generates bigramHash[ottawa, citizen].

My plan was to just generate a list, subsequently just Cases[citizenCounter97,_Integer]; the thing, and have a sense of how often the given term appeared.

Problem is, the results did not make sense. When I went in and started de-bugging, I realized that the counts that come out of Print were completley different than the lists it was generating! The result for citizenCounter97 would be like this:

{2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, \
2, 2, 2, 2, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, \
2, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 
 bigramHash["ottawa", "citizen"], bigramHash["ottawa", "citizen"], 
 bigramHash["ottawa", "citizen"]}

Whereas the real Print output would show that the first 12 results were nothing, the 13th was 2, and then nothing again until 27, when it became 1.

So basically, I am confused as to what's happening, why my counter is completely divergent from the reality as I do it manually/debug with Print.

Thanks in advance, and happy to clarify as best I can.

---- edited to add ----

I suspect one solution lies in an If statement - i.e. If count is an integer, then append that; if the 'If' ends up being not an integer, then return 0. I've tried a variety of calls using Head however, and am still hitting a brick wall.

share|improve this question
1  
I'm confused about what you are wanting and what the file citizenchecker97 contains. Could you describe what you want as a black box, i.e. what goes in and what do you hope to come out? –  Andy Ross Mar 29 '12 at 0:59
add comment

1 Answer 1

up vote 4 down vote accepted

If I am following your code, I think the problem is that you are collecting expressions such as bigramHash["ottawa", "citizen"] which will evaluate differently depending on the DownValue associated with bigramHash. Therefore, these expression will evaluate to whatever the final DownValue of bigramHash["ottawa", "citizen"] is, if present.

Here is a simplified example of the problem:

filelist = {
   {{"a", 1}, {"b", 2}},
   {{"cat", 5}, {"dog", 3}},
   {{"a", 4}, {"b", 6}}
 };

Reap[Do[
   Clear[bigramHash];
   (bigramHash[#] = #2) & @@@ file;
   count = bigramHash["a"];
   Print[count];
   Sow[count],
   {file, filelist}
]][[2, 1]]
During evaluation of In[26]:= 1

During evaluation of In[26]:= bigramHash[a]

During evaluation of In[26]:= 4

{1, 4, 4}

One simple fix is to define a default DownValue e.g. bigramHash[___] = 0:

Reap[Do[
   Clear[bigramHash];
   bigramHash[___] = 0;
   (bigramHash[#] = #2) & @@@ file;
   count = bigramHash["a"];
   Print[count];
   Sow[count],
   {file, filelist}
]][[2, 1]]
During evaluation of In[36]:= 1

During evaluation of In[36]:= 0

During evaluation of In[36]:= 4

{1, 0, 4}
share|improve this answer
    
Fantastic, thank you for deciphering my code. The issue laid in not having the right terminology to describe my problem! The bigramHash[__]=0 addition works swimmingly, and the results look like they should. –  programming_historian Mar 29 '12 at 13:34
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.