1
$\begingroup$

I need to obtain Zipf plots (Frequency vs Rank) of non-English texts (a utf-8 encoded text file) using Mathematica. How do I do it?

$\endgroup$
4
  • 2
    $\begingroup$ Check this example for the Zipf part. $\endgroup$
    – user31159
    Commented Nov 30, 2016 at 19:59
  • 4
    $\begingroup$ 1. Can you do it for English? 2. How is the task in a non-English language different? It's impossible to answer without more details. The procedure is exactly the same for French as for English. I guess for Chinese it's different because the word boundaries aren't clear. $\endgroup$
    – Szabolcs
    Commented Nov 30, 2016 at 20:04
  • $\begingroup$ i'm working with a tamil utf encoded file. i need to obtain a plot of frequency of words versus their rank in the tamil text. $\endgroup$ Commented Dec 1, 2016 at 7:36
  • 1
    $\begingroup$ As someone not familiar with Tamil: Is there anything that will make tokenizing Tamil different than English, i.e. are word boundaries delimited by spaces? Does Tamil rely on pre- or suffixes- to indicate tense, mood, etc. on words that you would otherwise want to consider the 'same word' for the sake of your analysis? Those could both complicate the answer. $\endgroup$
    – N.J.Evans
    Commented Dec 2, 2016 at 17:41

1 Answer 1

3
$\begingroup$

It's pretty simple:

text = ToLowerCase@Import["C:\\Input\\Moby Dick.txt"];
text = StringReplace[
text, {"." -> "", ";" -> "", ":" -> "", "-" -> "", "," -> "", "*" -> "", "" -> ""}];
(*Get rid of extra whitespace*)
text = StringReplace[text, " " .. -> " "];
(*Make a list of words*)
words = StringSplit[text];
(*Tally the frequency of each word*)
tally = SortBy[Tally[words], -#[[2]] &];

Now we have a tally of the frequency of each word. Plotting it is simple:

Index[x_] := Table[{i, x[[i]]}, {i, Length@x}];
points = Index[tally[[All, 2]]];
ListLogLogPlot[points, PlotRange -> Full]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.