I have a dataset of strings and I would like to take only the first two letters of each string and find the conditional probability P(Second Letter|First Letter) while only looking at the words of the dataset. I can split the words and make another list with only the first two letters of each word but I'm having difficulty figuring out a way to say for example, all of the two-letter word snippets that start with 'a' what is the probability that a,b,c...z will follow as the second letter. I want to do the probability for every letter pair and put it into a 3D histogram where the x-axis will be the first letter chosen, the y-axis will be the second letter and the height will be the probability that that pair occurs within the chosen dataset.

Example: Original dataset {cat,dog,human} dataset split to only include the first two letters {ca,do,hu}

I've been trying to figure out a way to do it and I just can't seem to get it. I attempted to search to see if anyone has asked a question like mine but I didn't see anything like it. Any help is thoroughly appreciated.

  • $\begingroup$ Build a matrix of frequencies of digrams? $\endgroup$
    – flinty
    Apr 12, 2021 at 21:20
  • $\begingroup$ @flinty Let me explore that. I appreciate it the help. $\endgroup$
    – WornOut
    Apr 12, 2021 at 21:22

1 Answer 1

(* English words, lowercase, length > 2 *)
words = ToLowerCase@Select[WordList[], StringLength[#] > 2 &];

(* Get first two chars, throw out any pairs containing non-letters/punctuation *)
pairs = Select[Characters[words][[All, ;; 2]], AllTrue[#, LetterQ] &];

(* Probability function *)
countProbs[list_] := Map[#/Length[list] &, Counts[list]];

(* Gather the pairs by common first letter and prob-tally each grouping *)
probAssoc = Association[#[[1,1]] -> countProbs[#[[All,2]]]& /@ GatherBy[pairs, First]];

(* Generate the 3D histogram *)
hist = Histogram3D[LetterNumber[pairs]]

(* Get the probability of a H following an S *)

3d histogram

  • $\begingroup$ Also if you want the height to scale with probability then do: Histogram3D[LetterNumber[pairs], Automatic, "Probability"] - I used LetterNumber to convert the letters to numbers that the histogram would like - I'm not sure how you'd change the ticks to a,b,c,d,....,z but that would be nice. $\endgroup$
    – flinty
    Apr 12, 2021 at 21:47
  • $\begingroup$ This is exactly what I was looking for! I appreciate the time you put into this. I will figure out how to incorporate it for every letter combination as well as convert the labels from number value to alphabet values. Again, thank you! I also have to do it for other languages but I will incorporate that as well. I just needed a beginning. $\endgroup$
    – WornOut
    Apr 12, 2021 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.