5
$\begingroup$

Mathematica contains the 31801 most frequent Russian words. These can be obtained using:

WordList["KnownWords", Language -> "Russian"]

Furthermore, the function WordFrequencyData returns the frequency of a word as a percentage:

a sample output from WordFrequencyData

How could I write a similar function russianWordFrequencyRank that returns an integer corresponding to the word’s rank?

For instance the most frequent word in the Russian language is "и" (which means "and"). My desired function should return 1 when fed this word.

russianWordFrequencyRank["и"] = 1

Here are some Russian words to use as examples:

russianWords = {"машина", "привет", "товарищ"}
$\endgroup$
3
  • 2
    $\begingroup$ Sadly since WordFrequencyData sends an individual request to a server for each word you give it, you will be rate-limited using that function. The easiest way is just to find the data elsewhere and use it offline, unfortunately. $\endgroup$
    – Carl Lange
    Feb 3 '21 at 11:55
  • 1
    $\begingroup$ Ah, I may have been too hasty in my assumptions about "individual requests" but in general my point stands - WordFrequencyData uses Wolfram servers and has some caveats because of that. $\endgroup$
    – Carl Lange
    Feb 3 '21 at 11:56
  • 1
    $\begingroup$ Someone more well-versed in statistics might be able to give a sensible answer to the question of whether you could use the distribution of frequencies to estimate the rank - it looks like the frequencies of words is a LogNormalDistribution, so I wondered if you could use Probability to guess the rank of a word in some way. But I don't know what I'm talking about - it could be total nonsense. $\endgroup$
    – Carl Lange
    Feb 3 '21 at 12:36
5
$\begingroup$

Pursuant to my comments, here is a method to get all the frequencies by paginating requests to the WRI servers.

pages[l_, d_] := 
 Span @@@ NestList[{#[[2]], #[[2]] + d} &, {1, d}, Round[Length@l/d]]

words = WordList["KnownWords", Language -> "Russian"]

freqs = Join @@ 
   Table[WordFrequencyData[words[[p]], Language -> "Russian"], {p, 
     pages[words, 100]}];

This will get all the frequencies, 100 at a time (to prevent request timeouts). Sadly this takes approximately forever (which is why I suggest finding and downloading the source dataset for yourself, if you can).

I ran this for the first 1000 words in words with a page size of 100, which took about two minute, by doing

freqs = Join @@ 
   Table[WordFrequencyData[words[[p]], Language -> "Russian"], {p, 
     pages[words[[ ;; 1000]], 100]}];

Now your function is fairly simple: simply order the results by the frequency data and get the position of a word:

Position[Keys@ReverseSort@DeleteMissing@freqs, "была"][[1, 1]]

Of the first 1000 words, that is the 18th most common.

Or in function form:

russianWordFrequencyRank[w_] := 
 Position[Keys@ReverseSort@DeleteMissing@freqs, w][[1, 1]]

russianWordFrequencyRank["дан"]

which gives 357 - that is, it is the 357th most-common word in the Russian language that exists in the first 1000 words in words. If you create freqs based on all of words, you should get roughly the correct result.

$\endgroup$
2
  • $\begingroup$ Off topic a bit, but I am surprised that the word "дан" is so high in the rankings, or even at all made it in the 1000 most frequent words, given that it is a rather specialized term, used for the type of black belt one can have in certain martial arts like karate or taekwondo. $\endgroup$ Feb 3 '21 at 14:55
  • $\begingroup$ I'm not sure, I don't know if WordList["KnownWords", Language -> "Russian"] is ordered any particular way, so it is just that it happened to be the most popular word in a selection of 1000 of the 31801 "known" words. I would imagine "dan" (means the same, I guess it's just a different transliteration) is just as likely to be the most popular in that kind of subset in if we did the same in English. $\endgroup$
    – Carl Lange
    Feb 3 '21 at 15:27
4
$\begingroup$

Using Carl Lange's excellent idea of "paginating requests to prevent timeouts" and some psychological balm from Monitor, I downloaded all the data from Wolfram's server.

I then sorted it by rank and uploaded it github for future convenience.

data = Import["https://raw.githubusercontent.com/ccosnett/russianWordFrequenciesRanks/main/russianWordFrequencies.mx"]

enter image description here

russianWordFrequencyRank[w_] := Position[Import["https://raw.githubusercontent.com/ccosnett/russianWordFrequenciesRanks/main/russianWordFrequencies.mx"], {w, _, _}, 3][[1, 1]];

.

russianWordFrequencyRank["товарищ"]

1239

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.