# How do I plot words by frequency for a text not in English?

I need to obtain Zipf plots (Frequency vs Rank) of non-English texts (a utf-8 encoded text file) using Mathematica. How do I do it?

• Check this example for the Zipf part.
– user31159
Commented Nov 30, 2016 at 19:59
• 1. Can you do it for English? 2. How is the task in a non-English language different? It's impossible to answer without more details. The procedure is exactly the same for French as for English. I guess for Chinese it's different because the word boundaries aren't clear. Commented Nov 30, 2016 at 20:04
• i'm working with a tamil utf encoded file. i need to obtain a plot of frequency of words versus their rank in the tamil text. Commented Dec 1, 2016 at 7:36
• As someone not familiar with Tamil: Is there anything that will make tokenizing Tamil different than English, i.e. are word boundaries delimited by spaces? Does Tamil rely on pre- or suffixes- to indicate tense, mood, etc. on words that you would otherwise want to consider the 'same word' for the sake of your analysis? Those could both complicate the answer. Commented Dec 2, 2016 at 17:41

It's pretty simple:

text = ToLowerCase@Import["C:\\Input\\Moby Dick.txt"];
text = StringReplace[
text, {"." -> "", ";" -> "", ":" -> "", "-" -> "", "," -> "", "*" -> "", "" -> ""}];
(*Get rid of extra whitespace*)
text = StringReplace[text, " " .. -> " "];
(*Make a list of words*)
words = StringSplit[text];
(*Tally the frequency of each word*)
tally = SortBy[Tally[words], -#[[2]] &];


Now we have a tally of the frequency of each word. Plotting it is simple:

Index[x_] := Table[{i, x[[i]]}, {i, Length@x}];
points = Index[tally[[All, 2]]];
ListLogLogPlot[points, PlotRange -> Full]