How to get a list of all words from an unabridged dictionary?

Question

I wanted to do something that Donald Knuth did. He entered a contest to make as many words as possible out of the phrase/string "Ziegler's Giant Bar". The judges master list had 2 500 words. In 8th grade, Donald Knuth pretended to be sick from stomach ache so he could get some time to work at home. He went through an unabridged dictionary and identified 4 500 words that could be formed. He won chocolate bars for all his class mates and a new television set. There are two interpretations. One is say you have palace. You cannot form keep because you have 1 e in palace and 2 e in keep. The other interpretation says you can form keep from palace. I found 2031 words where you can't form keep from palace, and 4 703 words if you can get keep from palace.

Here's the code I used:

zieglersGiantChocolateBar = "Ziegler's Giant Bar"
lowercaseZieglersGiantChocolateBar = 
 ToLowerCase[zieglersGiantChocolateBar]
lowercaseZieglersGiantChocolateBarCharacters = 
 Characters[lowercaseZieglersGiantChocolateBar]
ResourceFunction["EchoPerformance"][
 dictionary = Union[WordData[], DictionaryLookup[], WordList[]]]
ResourceFunction["EchoPerformance"][
 wordsPossibleWithSubsetQ = 
  ResourceFunction["MonitorProgress"][
   Select[dictionary, 
    SubsetQ[lowercaseZieglersGiantChocolateBarCharacters, 
      Characters[#]] &]]]

I also did

wordsPossibleWithMultisetInclusionQ = 
 ResourceFunction["MonitorProgress"][
  Select[dictionary, 
   ResourceFunction["MultisetInclusionQ"][
     lowercaseZieglersGiantChocolateBarCharacters, Characters[#]] &]]

I will note that SubsetQ is a lot faster than the ResourceFunction MultisetInclusionQ.

I'm not sure which set of rules Knuth and the judges used. If they used the harder set of rules, the only way I could see beating 2031 words is with a larger dictionary.

I want to form a dictionary with as many words as possible. The three main functions I use are

• WordList. This can be used with WordList[] which defaults to WordList["CommonWords"] with between 30,000 and 40,000 words. I got 39176 words, though this might be different in the future if more words are added or removed. You can also do WordList["KnownWords"]. There are 84923 with this. I use Length to count the words. I will not consider inflections as different words. That means dog and dogs are one word, not two. You can get more words with the option WordList["KnownWords", IncludeInflections->True]. I think this is a reasonable idea to state.
• DictionaryWordQ has 92518 words.
• Length[WordData[]] returns 149191.

I created a dictionary by combining all these words.

ResourceFunction["EchoPerformance"][
 dictionary = Union[WordData[], DictionaryLookup[], WordList[]]]

Then I do

Length[dictionary]

and I get 196529.

This is a lot of words, but I think there is room to expand the dictionary. For example, chalcogen is not in dictionary.

MemberQ[dictionary, "chalcogen"]

returns False. chalcogen refers to a class of chemical elements in chemistry. It is not an obscure word. I don't want to allow expanding the dictionary by obscure words and misspellings.

One idea is to use the entity word, though I'm pretty sure this also does not have chalcogen.

Now so far the question of how to expand the dictionary has been vague.

I will now make the question more specific.

There is something called a pentaquark. My idea to get it in the dictionary is to list all ?5.7 million Wikipedia articles and add them as words. Some of them will proper names, like people. In this case, I do not want to add a name like John Smith to the dictionary. But I would add pentaquark. This could be analyzed by seeing if the article title is used in lowercase as a noun in a sentence in the article summary plaintext. I can do WikipediaData["pentaquark", "SummaryPlaintext"] for this.

Then I can do

TextCases[WikipediaData["pentaquark", "SummaryPlaintext"], "Noun"]

which returns

{"pentaquark", "particle", "quarks", "antiquark", "experiments", \
"quarks", "baryon", "number", "antiquarks", "pentaquark", "baryon", \
"number", "baryon", "quarks", "baryons", "triquarks", "baryon", \
"name", "pentaquark", "possibility", "particles", "existence", \
"quarks", "decades", "pentaquarks", "physicists", "law", "nature", \
"production", "claim", "pentaquark", "discovery", "experiments", \
"mid-2000s", "discoveries", "pentaquark", "states", "researchers", \
"results", "pentaquark", "discoveries", "data", "analysis", "LHCb", \
"collaboration", "results", "pentaquark", "states", "decay", \
"baryons", "\[CapitalLambda]0b", "LHCb", "collaboration", \
"discovery", "pentaquark", "LHCb", "collaboration", "discovery", "P\
\[CapitalLambda]\[Psi]s", "pentaquark", "particle", "research", \
"laboratories", "pentaquarks", "processes", "formation", "neutron", \
"stars"}

I can then further analyze this to know for sure that pentaquark is a valid word.

Here's a case where a proper noun returns False and pentaquark returns True.

MemberQ[TextCases[WikipediaData["pentaquark", "SummaryPlaintext"], 
  "Noun"], "pentaquark"]
MemberQ[TextCases[WikipediaData["Donald Knuth", "SummaryPlaintext"], 
  "Noun"], "Donald Knuth"]

How can I list all Wikipedia articles to start my search for more words?

I am also aware of Wiktionary. I have edited it some. I am also aware of the ability to download an unabridged dictionary from Project Gutenberg.

Unfortunately, the 43.6 megabyte .txt file from Project Gutenberg has formatting that is not understandable to me. I did however find something in JSON format on GitHub.

The second specific part of my question is how could I parse this JSON file? Here is a sample:

"hydrolysis" : "A chemical process involving the addition of the elements of water.",

When there are multiple definitions, you have something like this.

"politics" : "1. The science of government; that part of ethics which has to do with the regulation and government of a nation or state, the preservation of its safety, peace, and prosperity, the defense of its existence and rights against foreign control or conquest, the augmentation of its strength and resources, and the protection of its citizens in their rights, with the preservation and improvement of their morals. 2. The management of a political party; the conduct and contests of parties with reference to political measures or the administration of public affairs; the advancement of candidates to office; in a bad sense, artful or dishonest management to secure the success of political candidates or parties; political trickery. When we say that two men are talking politics, we often mean that they are wrangling about some mere party question. F. W. Robertson.",

For now, I would just like a list of all the words in the file, though.

One thing I have realized is that there is a k in keep. So even if you could get two es you wouldn't have the k from palace. You could get lee from palace, for example.

Let's say our source word is palace instead of Ziegler's Giant Bar. We can form ace because there is two As, one C, and one E in palace and one A, one C, and one E in ace. Let's say we want to make lee. There is 2 Es and 1 L in lee. There is one E and one L in palace. We would need one extra e to make lee if we paid attention to multiplicity and treated the characters of palace--{"p","a","l","a","c","e"} as a multiset. If we count multiplicity, we would need something like palace to make lee. If we only require that the letter be present at least once, we could form lee from palace. I hope this helps.

Answer 1

For the JSON data, you can do this. Open in VS Code and press Shift+Alt+C to copy as (absolute) path.

JSONData = 
 Association[Import["C:\\Users\\peter\\Downloads\\dictionary.json"]]
JSONWords = Keys[JSONData]
Length[JSONWords]

I have 102 217 words. The size in memory is 38 megabytes, so you can click store full expression in notebook so the data persists in the variable even after the kernel quits.

Then you can do

dictionary = Union[JSONWords, dictionary]

Now we have

Length[dictionary]

returns 256 356.

To add the entities do:

ResourceFunction["EchoPerformance"][
 entityWords = EntityValue[EntityClass["Word", All], "Name"]]

It took 200.838 seconds. Length[entityWords] returns 262 181. Then do dictionary=Union[dictionary, entityWords].

Now Length[dictionary] is 299396. MemberQ[dictionary, "chalcogen"] returns True, but MemberQ[dictionary, "pentaquark"] returns False.

To download a list of words from Wiktionary, go to the dump download page. I downloaded the list of page titles in the main namespace.

The file has to be extracted. When you open it in, say, Notepad++, then there is a lot of information. pentaquark and chalcogen are both there. However, there are other languages that I don't want to include.

With the larger set of words, I was able to get 7141 words where you can get keep out of palace. I was able to get 3137 words where you cannot get keep out of palace. This surpasses the judges record of 2500 but is still less than what Knuth got. I will note however, that some of the words were plural like garages and I didn't want to include these so the majority of the increase might be due to inflected forms and not new words. There also seemed to be more proper nouns that I would not include that were added.