7
$\begingroup$

What is the simplest way to filter out profanity from Word Data without manually removing the words?

Improve this question
$\endgroup$
  • 1
    $\begingroup$ There is a build-in and ready to use classifier function for recognizing profanity in text. If you give a example of how you would like to use WordData I will write up an more specific answer to that if you like. $\endgroup$ – Sascha Aug 17 '16 at 7:15
  • 2
    $\begingroup$ Functions like TextCases and TextPosition can also recognize profanity. $\endgroup$ – Szabolcs Aug 17 '16 at 7:56
  • 3
    $\begingroup$ You'll make several clbuttic mistakes if you try. And then you'll either delete "prick my finger" or fail to delete "finger my prick" . $\endgroup$ – Carl Witthoft Aug 17 '16 at 19:11
7
$\begingroup$

From here:

newWordData = Complement[WordData[], Import["https://raw.githubusercontent.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/master/en", "List"]];
Improve this answer
$\endgroup$
  • 7
    $\begingroup$ Whoever compiled this list has been watching too much fetish porn. There are many extremely uncommon words/phrases in there, most of them not really classifiable as "profanity". It's more like a list of search terms to be avoided. $\endgroup$ – Oleksandr R. Aug 17 '16 at 10:38
  • 1
    $\begingroup$ @OleksandrR. Pssh, obviously the likes of "xx", "butt", "twinkie", "mr hands" and many other phrases that without context are totally harmless should be outright banned from human language entirely $\endgroup$ – Trotski94 Aug 17 '16 at 10:53
  • 4
    $\begingroup$ At alabama hot pocket I grew suspicious and by the time I scrolled down the list of excluded words to cleveland steamer I knew the author basically crawled all of Urban Dictionary and just took the result as words to filter out. $\endgroup$ – Sascha Aug 17 '16 at 15:43
  • 1
    $\begingroup$ @Sascha, that, or the curator is/was a /b/ habitué. $\endgroup$ – J. M.'s ennui♦ Aug 17 '16 at 17:16
11
$\begingroup$

As an example I take the following list of words

testdata = WordData[All, "Verb"] // Select[#, StringContainsQ[#, "fu"] &] &

{"befuddle", "be full", "bifurcate", "blow a fuse", "centrifugate", "centrifuge", "circumfuse", "coiffure", "confuse", "confute", "defuse", "desulfurize", "diffuse", "effuse", "fuck", "fuck off", "fuck up", "fuddle", "fudge", "fudge together", "fuel", "fulfill", "fulfill", "full", "fullback", "fulminate", "fumble", "fume", "fumigate", "function", "fund", "fund-raise", "fund raise", "fundraise", "funk", "funnel", "furbish", "furbish up", "furcate", "furl", "furlough", "furnish", "furrow", "further", "fuse", "fusillade", "fuss", "fustigate", "get a noseful", "give full measure", "infuriate", "infuscate", "infuse", "make full", "make fun", "malfunction", "misfunction", "obfuscate", "perfume", "perfuse", "poke fun", "refuel", "refund", "refurbish", "refurnish", "refuse", "refute", "snafu", "suffuse", "sulfur", "sulfurette", "transfuse", "travel purposefully", "trifurcate", "ultracentrifuge", "unfurl"}

and use the build-in classifier function "Profanity" to obfuscate bad language (you could just filter out words too)

Clear@profanityFilter
profanityFilter[word_] := StringReplace[word, {_ :> 
RandomChoice[{"&", "#", "!", "%", "@", "$", "^", "%", "*"}]}] 
/; Classify[ "Profanity", word]

profanityFilter[word_] := word

Now you can apply the filter to individual words or map it to a list of words to make your language squeaky clean:)

profanityFilter /@ testdata

{"befuddle", "be full", "bifurcate", "blow a fuse", "centrifugate", "centrifuge", "circumfuse", "coiffure", "confuse", "confute", "defuse", "desulfurize", "diffuse", "effuse", "%*!^", "&%#^&^%!", "%!!%!%!", "fuddle", "fudge", "fudge together", "fuel", "fulfil", "fulfill", "full", "fullback", "fulminate", "fumble", "fume", "fumigate", "function", "fund", "fund-raise", "fund raise", "fundraise", "funk", "funnel", "furbish", "furbish up", "furcate", "furl", "furlough", "furnish", "furrow", "further", "fuse", "fusillade", "fuss", "fustigate", "get a noseful", "give full measure", "infuriate", "infuscate", "infuse", "make full", "make fun", "malfunction", "misfunction", "obfuscate", "perfume", "perfuse", "poke fun", "refuel", "refund", "refurbish", "refurnish", "refuse", "refute", "snafu", "suffuse", "sulfur", "sulfurette", "transfuse", "travel purposefully", "trifurcate", "ultracentrifuge", "unfurl"}

Improve this answer
$\endgroup$
  • 2
    $\begingroup$ To identify "fuck off" in its own right rather than because it contains "fuck" would be impressive. It makes me wonder, does this pass the "Scunthorpe test"? $\endgroup$ – Oleksandr R. Aug 17 '16 at 17:26
  • $\begingroup$ @OleksandrR. one could probably build some classifier that takes into account not only the literal meaning of a word but also the actual meaning via something like WordData["fuck off"] which gives you {{"fuck off", "Verb", "Excite"}, {"fuck off", "Verb", "GoAway"}, {"fuck off", "Verb", "Slug"}}. Together with some syntactic analysis to figure out the context one could use that additional information to discern whether the user intended it as profanity or not. $\endgroup$ – Sascha Aug 17 '16 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.