I want the WordFrequencyData of the whole WordList[]
However it takes too long to get the result.
WordFrequencyData /@ WordList[Language -> "English"]
Is there a faster workaround?
$\begingroup$
$\endgroup$
1
1 Answer
$\begingroup$
$\endgroup$
You can sped it up a bit. Do NOT Map. If function can take lists - go with that functionality. Take 100 words sample:
sample:=RandomSample[WordList[],100]
and compare timings on it:
WordFrequencyData[sample];//AbsoluteTiming
{11.2825,Null}
WordFrequencyData/@sample;//AbsoluteTiming
{75.1206,Null}
If you have multiple cores you can try to get it in batches and parallelize. For instance:
dict = WordList[];
Merge[ParallelTable[WordFrequencyData[dict[[k;;k+9]]],{k,1,20,10}],Total]
answered Apr 23, 2022 at 3:38
WordFrequencyDatauses an external database (Google Ngrams) to retrieve the information so it is naturally much slower than a local function. Consider that you are trying to run roughly 40,000 such searches with your wordlist, and you'll see that it's unlikely you can make this fast. Finally, consider that you could feed the whole list toWordFrequencyDataand obtain an association as the output, rather than mapping the function of the list. I don't expect the latter to be much faster, but it's probably a better way to store the results. $\endgroup$