Analyzing frequency of individual letters in a large body of text

Question

There are interesting histograms about the English language:

BarChart[Counts[StringTake[WordList[], 1]], ChartLabels -> Automatic]

Histogram[StringLength[WordList[]]]

I want to do something like this with the Holy Bible.

I tried:

numberofwords = 
Take[WolframAlpha[
"Bible", {{"Word:FamousTextData", 1}, "ComputableData"}], 1]

There are lists of the most frequent words:

WolframAlpha["Bible", IncludePods -> "MostCommonWords:FamousTextData",
AppearanceElements -> {"Pods"}, 
TimeConstraint -> {30, Automatic, Automatic, Automatic}]

But I don't know how to determine the most frequent letters (alphabet).

How could I do this by using Mathematica?

Thank you in advance to any one who may be able to give me some ideas

Answer 1

Something like this perhaps?

bibOriginal = 
  Import["http://www.gutenberg.org/cache/epub/10/pg10.txt"]; (* load the bible *)
bib = StringDrop[bibOriginal, 658]; (* drop the introductory text *)
bib = StringDrop[bib, -18733];(* drop the legalistic closeout at the end*)
bibC = Map[{#[[1]], Length[#]} &, Gather[Sort[ToLowerCase[Characters[bib]]]]];
bibC = Select[bibC, MemberQ[CharacterRange["a", "z"], #[[1]]] &]
  (*ditch punctuation and numbers*);
BarChart[bibC[[All, 2]], ChartLabels -> bibC[[All, 1]]]

TableForm[Sort[bibC, #1[[2]] > #2[[2]] &]]

The same basic approach would probably work with the original text, so long as you were sensitive to the shift from Hebrew to Aramaic where appropriate.

Answer 2

We can import the bible directly from the web and convert it to a list of words fairly easily. Here I'm going to remove the chapter/verse headings as well as extraneous punctuation marks.

biblewords = 
  Import["http://www.gutenberg.org/cache/epub/10/pg10.txt"] // 
          StringSplit[#, {"*** START OF THIS PROJECT GUTENBERG EBOOK THE KING JAMES BIBLE ***", 
             "*** END OF THIS PROJECT GUTENBERG EBOOK THE KING JAMES BIBLE ***"}] & //
         StringTrim //
        StringSplit[#, {" ", ".", "\n", ","}][[2]] & //
        Flatten //
        DeleteCases[#, ""] & //
        Select[#, (Not@
          StringMatchQ[#, 
           DigitCharacter .. ~~ ":" ~~ DigitCharacter ..] &)] & //
        StringReplace[#, (Alternatives @@ Characters[",.?:;'\"!-()*"]) -> 
          ""] & // ToLowerCase;

We can ask how many words and how many unique words are in the bible,

Length@biblewords
Length@DeleteDuplicates@biblewords
(* 790146 *)
(* 12703 *)

This page lists 783,137 total words and 12,143 unique words, so the above is pretty close. Any refinements on the above importing scheme are most appreciated. We can use Tally to get the 30 most common words and give their prevalence:

TableForm[
 {#1, #2/Length[biblewords] 100.0} & @@@ (Sort[
     Tally@biblewords, #1[[2]] > #2[[2]] &][[;; 30]]), 
 TableHeadings -> {None, {"Word", "Frequency"}}]

And, finally, the character count,

TableForm[Tally[Characters@StringJoin@biblewords], 
 TableHeadings -> {None, {"Character", "Frequency"}}]

Answer 3

I will start by just copying how Jason B imported the bible:

bible = Import["http://www.gutenberg.org/cache/epub/10/pg10.txt"];
bible = StringSplit[bible, {
     "*** START OF THIS PROJECT GUTENBERG EBOOK THE KING JAMES BIBLE \
***", "*** END OF THIS PROJECT GUTENBERG EBOOK THE KING JAMES BIBLE ***"
     }][[2]];

After that I will use WordCounts and LetterCounts, which are two functions that were introduced in Mathematica 10.1:

wordcounts = WordCounts[bible];
lettercounts = LetterCounts[bible];

Then you can do whatever you want with those counts:

top30 = Take[Transpose[{Keys@wordcounts, Values@wordcounts}], 30];
TableForm[top30, TableHeadings -> {None, {"Words", "Count"}}]

Note that WordCount has the option IgnoreCase. If you add IgnoreCases -> True it will in effect convert all letters into lowercase before counting the words. WordCount can also be used to count n-grams.