7
$\begingroup$

I'm trying to plot, given a list of words, how many words belongs to each language.

I'm using the following code:

Total /@ Transpose@
   ParallelMap[
    Function[word, 
     Length@DictionaryLookup[{#, word}] & /@ DictionaryLookup[All]] , 
    wordList]

I am finding it way too slow, the lengths of the lists of words which I am using are about 45000 (sample file: https://dl.dropboxusercontent.com/u/35192406/words.txt)

Is DictionaryLookup really slow? Does it use Internet? How can I speed up my function?

Improve this question
$\endgroup$
1
  • $\begingroup$ If each word appears only one, you could try and see if Intersection is faster than whatever pattern matching function DictionaryLookup uses: (Length@Intersection[{"hello", "undead", "fksjdlfs"}, DictionaryLookup[{#, All}]] & /@ DictionaryLookup[All]) // Timing $\endgroup$
    – C. E.
    Jun 25, 2013 at 8:43

2 Answers 2

Reset to default
7
$\begingroup$

Using Intersection as suggested by Anon seems to give good performance, provided I exclude Hebrew from the languages.

First define the list of all languages except Hebrew, and get the complete dictionary for each language:

lang = DeleteCases[DictionaryLookup[All], "Hebrew"];
allwords = DictionaryLookup[{#, __}] & /@ lang;

Now make a list of 45000 randomly chosen words:

wordList = RandomChoice[Flatten @ allwords, 45000];

The computation takes about 10 seconds:

Timing [Length[Intersection[wordList, #]] & /@ allwords]
(*  {10.094, {394, 5752, 367, 1763, 6793, 2530, 4004, 2408, 1850, 212, 
  1232, 6580, 1645, 7273, 762, 121, 2249, 701, 1367, 142, 2565, 7802, 
  286, 176, 1389, 1415}}  *)

If I don't remove Hebrew from the list of languages it is far slower. I have no idea why this might be.

For word lists which may contain repeated words, you could use this:

wordcounts = Dispatch[Rule @@@ Tally[wordList]];
Total[Intersection[wordList, #] /. wordcounts] & /@ allwords
Improve this answer
$\endgroup$
5
  • $\begingroup$ This is very nice. How can I make it work for wordList in which words may be reppeated? $\endgroup$
    – José D.
    Jun 25, 2013 at 9:13
  • $\begingroup$ @Simon - how did you spot that it was Hebrew that was causing the evaluation to take more time? $\endgroup$
    – Vincent Tjeng
    Jun 25, 2013 at 9:21
  • 1
    $\begingroup$ @VincentTjeng, I tested it first on a single language and it seemed fast, then it was terribly slow when I mapped it over all the languages. So I used a Do loop with Print to see the results coming out one by one and it was clear that it got stuck on Hebrew. $\endgroup$
    – Simon Woods
    Jun 25, 2013 at 10:13
  • 1
    $\begingroup$ @Trollkemada, see my update. $\endgroup$
    – Simon Woods
    Jun 25, 2013 at 10:19
  • $\begingroup$ @SimonWoods nice, thank you! I was having problems with Arabic and Hebrew too :) $\endgroup$
    – Vincent Tjeng
    Jun 26, 2013 at 5:46
6
$\begingroup$

If you're seeking to find the number of words from each language that are represented, you can simplify your code (although you lose the ability to parallelize the search process).

Using the following sample list of words:

wordList = {"ab", "aba", "abá", "abaá", "abab", "ababa", "abábades", 
   "ababillarse", "ababol", "abaca", "abacá", "abacà", "abacais", 
   "abacallan", "abacallana", "abacallanà", "abacallanada", "abacas", 
   "abacate", "abacateiro", "abacateiros", "abacates", "abacaxi", 
   "abacaxis", "abacht", "abaci", "aback", "abactis", "abactissen", 
   "abactor", "abacus", "abacuses", "abad", "abadański", "abadenn", 
   "abadennad", "abadh", "abádszalóki", "abafájai", "abafalai", 
   "abafalvai", "abaft", "abaich", "abaid", "abaissa", "abaissaient", 
   "abaissais", "abaissait", "abaissant", "abak", "abaka", 
   "abakańczyk", "abako", "abakus", "abakusen", "abakusens", 
   "abakuser", "abakuserna", "abalienato", "abänderlich", "abändern", 
   "abandon", "abandonner", "abandonnér", "abandonnere", 
   "abandonnerede", "abarbeiten", "abartig", "abate", "abætere", 
   "abati", "abato", "abavus", "abba", "abbabarn", "abbabarna", 
   "abbabarnanna", "abbabarni", "abbacinati", "abbagli", "abbaglia", 
   "abdicirah", "abdicirahu", "abdicirala", "abdicirali", "abdiciram",
    "abdiki", "abdomeno", "abela", "abessiivi", "abessiivia", 
   "abessiivimuotojen", "abessiivin", "abi"};

We can then code as follows:

wordListLanguage = DictionaryLookup[{All, wordList}];
Tally[#[[1]] & /@ wordListLanguage]

(*{{"BrazilianPortuguese", 11}, {"Breton", 7}, {"BritishEnglish", 8}, {"Catalan", 8}, {"Croatian", 5}, {"Danish", 5}, {"Dutch", 5}, {"English", 7}, {"Esperanto", 5}, {"Faroese", 5}, {"Finnish", 5}, {"French", 7}, {"Galician", 12}, {"German", 5}, {"Hungarian", 6}, {"IrishGaelic", 5}, {"Italian", 5}, {"Latin", 5}, {"Polish", 7}, {"Portuguese", 8}, {"ScottishGaelic", 5}, {"Spanish", 7}, {"Swedish", 6}}*)

For my sample word list, the search took 22.9 seconds. I'm not sure if it represents a performance improvement over your code. However, this compares favourably to Anon's code (123.6 s) as implemented as follows:

{#, Length@Intersection[wordList, DictionaryLookup[{#, All}]]} & /@ DictionaryLookup[All]

Of course, you should be able to parallelize the above process, but it appears that each of the kernels you start will download the data from Wolfram again.

Improve this answer
$\endgroup$
4
  • $\begingroup$ btw, I code in MMA 9.0 - not sure whether similar functionality is present in previous versions. $\endgroup$
    – Vincent Tjeng
    Jun 25, 2013 at 8:53
  • $\begingroup$ I am using MMA 9.0 too. This seems to be faster than my code, but still way too slow. BTW, in my list some words appear more than once, can i take advantage of that? $\endgroup$
    – José D.
    Jun 25, 2013 at 8:54
  • $\begingroup$ I'm not sure how you would be able to handle that, because if one uses Union to eliminate the words that appear multiple times then one would be unable to count them. It all depends on how often the duplicates appear though - if they appear very often, then you might want to code in a different way. $\endgroup$
    – Vincent Tjeng
    Jun 25, 2013 at 9:07
  • $\begingroup$ Not sure about using Parallelize :S i.imgur.com/Jow1ytO.png $\endgroup$
    – José D.
    Jun 25, 2013 at 9:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.