Removing banal words from a list

Question

NB: I am aware of the DeleteStopwords symbol.

I want to remove from a list of words the ones that are likely to be more banal. So for instance if we have

{"exegesis", "mystification", "bread", "synthesis", "dog", "autonomy", "develop", "enthusiastic", "house"}

then I would like a function that is likely to remove "bread", "dog", and "house" while leaving the other words in the list.

Any ideas for implementing such a thing? It doesn't have to be incredibly robust; I'm generating lists of words for games where the object is to guess the word.

Please add any useful tags; I couldn't think of any that applied very well.

Answer 1

Please also see a related problem here, it might give you more ideas. Words can be banal wrt all words in the dictionary or wrt some particular list of words. Let's consider the later case. WordFrequencyData gives the frequency of word in typical published English text. Within a given list of words you can find the max-frequency word and rescale the rest of the words according to it. Then a threshold to cut off more popular words will be always between 0 and 1.

banal[w_,t_]:=With[{freq=WordFrequencyData[w]},
Keys[Select[freq/Max[freq],#<=t&]]]

For your set of words:

words={"exegesis", "mystification", "bread", "synthesis", 
"dog", "autonomy", "develop", "enthusiastic", "house"};

and threshold $0.1$ we get

In[]:= banal[words,.1]
Out[]= {exegesis,mystification,synthesis,autonomy,enthusiastic}

with the dropped words being:

In[]:= Complement[words,%]
Out[]= {bread,develop,dog,house}

Answer 2

Whether a word is "banal" is a subjective judgment. If you want to approach it in a statistical way, I'd try a TF/IDF approach. You'll find no shortage of references about it on the web.

Here's the basic idea. Suppose you're building a search index over a bunch of documents.

TF = term frequency: How many times a word appears in one particular document

IDF = inverse document frequency: 1 over the number of documents where the word appears. So, the more documents the word appears in, the smaller this factor becomes. 1 over 10 documents is a bigger number than 1 over 100,000 documents.

You can think of this roughly as a generalization of the stopword concept. Rather than manually list out your stopwords, you use automated means to count words over a set of documents and come up with what you might think of as a distinctiveness score. The word "the" might appear many times within one document, but it also appears across many documents, so it's not a very distinctive word.

So suppose you're building a system like Google, and the user searches for "the green xylophone". You probably don't want to rank documents very high just because they contain "the". However, "xylophone" is a more distinctive word in the sense that it appear in far fewer documents, so you might consider a match on that word to be more important in ranking a document up. "green" would be somewhere in the middle, probably.