# Creating word histograms from lists of strings

Consider the following:

StringData1={"Dog","Dog","Dog","Dog","Duck","Duck","Duck","Rabbit","House"};
StringData2={"A dog is in the house.","A dog is outside.",
"Why is everybody talking about a dog?", "Dog", "The duck died in the house.",
"The duck is actually a rabbit.", "Duck", "And the rabbit in fact is a dog",
"I have a dog in my house."};


Using the following function, it is quite easy to plot a histogram for StringData1. However the options remain fixed.

StringHistogram[list_] :=
Module[{counter, strings, numbers},
counter = {First@#, Length@#} & /@ (GatherBy@list);
strings = First /@ counter;
numbers = Last /@ counter;

BarChart[numbers, ChartLabels -> strings]
]


In case of StringData2 I was wondering whether it would be possible to chop up each sentence into single words (e.g.{Sentence1,...} -> {{Word1, Word2,...},...}) and then to run StringHistogram on Flatten@StringData2New, whereas StringData2New is of the form {{Word1, Word2,...},...}.

-

I have a slightly different strategy for splittling the string than rcollyer, because you can stick the pattern into StringSplit directly instead of needing to do a StringReplace first to prepare the string for splitting. For instance:

In[141]:= StringSplit["A dog is in the house.", Except[WordCharacter]]
Out[141]= {"A", "dog", "is", "in", "the", "house"}


Also, you can make StringHistogram simpler by using the built-in Tally function, which you've reimplemented (pretty darn well, I must say):

StringHistogram[list_, opts : OptionsPattern[]] :=
With[{tally = Tally@list},
BarChart[tally[[All, 2]],
BarOrigin -> Left,
ChartLabels -> tally[[All, 1]],
FilterRules[{opts}, Options[BarChart]]
]]


I use the BarOrigin option to make the bars come from the side, which makes everything much easier to read when you have many strings, and I pass options to BarChart in order to make tweaking things easier:

StringHistogram[
StringSplit[StringData2, Except[WordCharacter]] // Flatten,
ImageSize -> 500, BaseStyle -> "Label"]


-
Except[WordCharacter] is a very good alternative, and I'm ashamed to admit I never think of Except, +1. – rcollyer Apr 27 '12 at 21:51

You're looking for StringSplit. In your case, I would do this

StringSplit[{"A dog is in the house.","A dog is outside."}]


which returns

(* {{"A", "dog", "is", "in", "the", "house."},
{"A", "dog", "is", "outside."}} *)


Unfortunately, that leaves in the punctuation. To rid yourself of the punctuation, I would use StringReplace first, as follows

StringSplit @ StringReplace[{"A dog is in the house.", "A dog is outside."},
RegularExpression["[[:punct:]]"]


which gives

(* {{"A", "dog", "is", "in", "the", "house"},
{"A", "dog", "is", "outside"}} *)


Note the use of the character class in the RegularExpression; it makes specifying all punctuation easier.

-

Regarding the first part of your question, I would rewrite StringHistogram like this:

StringHistogram[list_] := BarChart[#2, ChartLabels -> #1] & @@ Transpose[Tally@list]


If you want to supply additional options, you can extend this as:

StringHistogram[list_, opts : OptionsPattern[]] :=
BarChart[#2, ChartLabels -> #1, opts] & @@ Transpose[Tally@list]


I think Pillsy's suggestion to split the strings is very clean. Here's another equally clean approach using StringCases:

StringCases[StringData2, x : WordCharacter .. :> x]

-