15
$\begingroup$

The documentation of DeleteStopwords only says that it "uses a standard, built-in list of stopwords".

So what is it exactly?


Update

Now that it is said "standard", does that standard have a name?

$\endgroup$
15
$\begingroup$

A little spelunking of the code for DeleteStopwords[] yields the internally used stopword list:

DeleteStopwords; (* force auto-load *)
AlphabeticSort[List @@ TextProcessing`TextModificationDump`$stopWords["English"]] // Short
   {"a", "A", "about", "above", "across", "after", "again", "against", "all", "almost",
    "alone", "along", "already", "also", "although", <<240>>,
    "within", "without", "won't", "would", "wouldn't", "yet", "you", "you'd", "you'll",
    "you're", "you've", "your", "yours", "yourself", "yourselves"}
$\endgroup$
  • $\begingroup$ Wow, these are some undocumented functions? $\endgroup$ – Αλέξανδρος Ζεγγ Sep 25 '18 at 6:22
  • $\begingroup$ But your codes merely return {"English"} as the result on my machine. Did I miss something? $\endgroup$ – Αλέξανδρος Ζεγγ Sep 25 '18 at 6:24
  • 4
    $\begingroup$ @ΑλέξανδροςΖεγγ One has to evaluate DeleteStopwords first. The actual code for DeleteStopwords and TextProcessing`TextModificationDump`$stopWords is stored in a file and is loaded only after DeleteStopwords is evaluated in the current session. $\endgroup$ – Henrik Schumacher Sep 25 '18 at 6:29
  • $\begingroup$ Ah, forgot the autoload. Thanks @Henrik! $\endgroup$ – J. M. will be back soon Sep 25 '18 at 6:31
  • 1
    $\begingroup$ @hanshenrik note the use of // Short at the end of the code provided, since I didn't want to paste the entire list. Thus, only the first few and last few list elements are shown, and the rest are elided. Remove the // Short to see the entire list. $\endgroup$ – J. M. will be back soon Sep 25 '18 at 8:33
14
$\begingroup$

The stop-words are documented and can be looked up as (see Details section of WordList):

WordList["Stopwords"]

enter image description here

As it might be subject to change so the output below can get outdated, but you can always run the function WordList["Stopwords"]:

{"0","1","2","3","4","5","6","7","8","9","a","A","about","above","across","after","again","against","all","almost","alone","along","already","also","although","always","am","among","an","and","another","any","anyone","anything","anywhere","are","aren't","around","as","at","b","B","back","be","became","because","become","becomes","been","before","behind","being","below","between","both","but","by","c","C","can","cannot","can't","could","couldn't","d","D","did","didn't","do","does","doesn't","doing","done","don't","down","during","e","E","each","either","enough","even","ever","every","everyone","everything","everywhere","f","F","few","find","first","for","four","from","full","further","g","G","get","give","go","h","H","had","hadn't","has","hasn't","have","haven't","having","he","he'd","he'll","her","here","here's","hers","herself","he's","him","himself","his","how","however","how's","i","I","i'd","if","i'll","i'm","in","interest","into","is","isn't","it","it's","its","itself","i've","j","J","k","K","keep","l","L","last","least","less","let's","m","M","made","many","may","me","might","more","most","mostly","much","must","mustn't","my","myself","n","N","never","next","no","nobody","noone","nor","not","nothing","now","nowhere","o","O","of","off","often","on","once","one","only","or","other","others","ought","our","ours","ourselves","out","over","own","p","P","part","per","perhaps","put","q","Q","r","R","rather","s","S","same","see","seem","seemed","seeming","seems","several","shan't","she","she'd","she'll","she's","should","shouldn't","show","side","since","so","some","someone","something","somewhere","still","such","t","T","take","than","that","that's","the","their","theirs","them","themselves","then","there","therefore","there's","these","they","they'd","they'll","they're","they've","this","those","though","three","through","thus","to","together","too","toward","two","u","U","under","until","up","upon","us","v","V","very","w","W","was","wasn't","we","we'd","we'll","well","we're","were","weren't","we've","what","what's","when","when's","where","where's","whether","which","while","who","whole","whom","who's","whose","why","why's","will","with","within","without","won't","would","wouldn't","x","X","y","Y","yet","you","you'd","you'll","your","you're","yours","yourself","yourselves","you've","z","Z"}

$\endgroup$
  • 2
    $\begingroup$ Interestingly, the complement of this list and the list in my answer yields only a list of letters and numbers, so that list certainly captures all the words. However, applying DeleteStopwords[] seems to only remove "i" from the list. $\endgroup$ – J. M. will be back soon Sep 25 '18 at 18:06
  • $\begingroup$ @J.M.issomewhatokay. a few inconsistencies are known, hopefully will be polished. Important thing is --- the docs is open about the subject, good thing to know :-) The docs is an attractor of stability in terms of known things of internal architecture. $\endgroup$ – Vitaliy Kaurov Sep 25 '18 at 20:13
8
$\begingroup$

Here is a longer list using the following commands:

A = WordList["KnownWords"];
B = DeleteStopwords[A];
c = Complement[A, B]

{ a, A, about, above, across, A.D., add-in, add-on, A.E., after, again, against, a.k.a., all, all-around, all-or-nothing, all-out, almost, alone, along, al-Qur'an, already, also, although, always, a.m., A.M., Am, AM, among, an, AN, and, and/or, another, any, anyone, anything, anywhere, A-one, are, around, as, As, AS, at, At, back, back-to-back, balls-up, bang-up, be, Be, beaten-up, beat-up, because, become, beefed-up, before, behind, being, belly-up, below, between, bicycle-built-for-two, blown-up, booze-up, born-again, both, bottom-up, boxed-in, break-in, bride-to-be, broken-down, brush-off, built-in, built-up, bundled-up, burned-out, burned-over, burnt-out, bust-up, but, button-down, buttoned-down, buttoned-up, by, by-and-by, call-back, caller-out, caller-up, call-in, call-out, camp-made, can, can-do, cared-for, carry-over, cast-off, change-up, check-in, ch'i, Ch'in, chin-up, chock-full, choke-full, chucker-out, chuck-full, churned-up, climb-down, clip-on, coach-and-four, comb-out, come-on, cover-up, crack-up, cure-all, custom-made, cut-in, cut-up, D.A., dead-on, derring-do, do, DO, d.o.a., do-it-yourself, done, do-nothing, do-si-do, down, Down, down-and-out, drawn-out, dressed-up, dried-out, dried-up, drive-in, drop-off, during, each, eighty-four, eighty-one, eighty-three, eighty-two, either, end-all, enough, even, ever, every, everyone, everything, everywhere, eyes-only, face-off, factory-made, fare-thee-well, far-off, far-out, few, fifty-four, fifty-one, fifty-three, fifty-two, fill-in, find, first, F.I.S.C., flame-out, flare-up, fly-by, follow-on, follow-through, follow-up, for, force-out, fore-and-after, forget-me-not, form-only, forty-first, forty-four, forty-one, forty-three, forty-two, foul-up, four, frame-up, free-for-all, from, fucked-up, full, further, get, get-go, get-up-and-go, G.I., gill-less, give, give-and-go, give-and-take, go, go-around, go-between, going-over, good-by, good-for-nothing, goof-off, groom-to-be, half-seas-over, hand-down, handed-down, hand-me-down, hands-down, hands-off, hands-on, hanger-on, hang-up, hard-on, has-been, have, have-not, Hawai'i, he, He, head-on, heads-up, heart-whole, her, here, Here, hers, herself, higher-up, high-interest, high-up, him, himself, his, hold-down, hollow-back, hoped-for, hopped-up, how, how-do-you-do, how-d'ye-do, however, HSV-I, hundred-and-first, hushed-up, i, I, I.D., i.e., I.E.D., if, If, ill-being, in, In, IN, in-between, inside-out, interest, into, I.Q., it, IT, its, itself, I.W.W., jerk-off, Johnny-jump-up, jumped-up, keep, knock-down, knock-down-and-drag-out, knocked-out, know-all, know-how, know-it-all, ladder-back, laid-back, laid-off, lash-up, last, lay-by, lay-up, lead-in, lean-to, least, less, lie-in, lighting-up, lights-out, log-in, longed-for, looker-on, look-over, low-down, low-interest, lying-in, ma'am, machine-made, made, made-up, make-do, make-up, man-made, many, marked-up, match-up, matt-up, may, May, me, ME, mess-up, mid-May, mid-off, mid-on, might, might-have-been, mixed-up, mix-up, mock-up, more, More, most, mostly, much, must, my, myself, ne'er-do-well, never, never-never, Never-Never, new-made, next, next-to-last, ninety-four, ninety-one, ninety-three, ninety-two, no, No, no., nobody, no-go, no-one, nor, no-show, nosh-up, not, nothing, now, nowhere, odds-on, of, off, off-and-on, often, on, once, once-over, one, one-and-one, one-off, one-on-one, one-to-one, only, OR, other, our, ours, ourselves, out, out-and-out, over, own, p.a., P.A., paid-up, part, passer-by, pass-through, paste-up, pegged-down, pent-up, per, perhaps, phase-out, phone-in, pick-me-up, pick-off, pig-a-back, pin-up, piss-up, plug-in, pop-up, Post-It, press-up, pull-in, pull-off, pull-through, pull-up, pumped-up, punch-up, purpose-made, put, put-down, put-on, put-put, put-up, put-upon, rake-off, raree-show, rather, rave-up, read-out, ready-made, right-down, right-side-out, right-side-up, rip-off, roll-on, run-down, run-in, runner-up, run-on, run-through, run-up, same, Same, Sana'a, sauce-alone, save-all, sawed-off, sawn-off, say-so, schoolma'am, see, seem, seeming, see-through, self-interest, self-made, self-will, send-off, set-back, set-to, seventy-four, seventy-one, seventy-three, seventy-two, seven-up, several, shake-up, shape-up, share-out, she, shell-less, shoo-in, shoot-down, shoot-'em-up, show, show-off, shut-in, side, side-to-side, since, sit-down, sit-in, sit-up, sixty-four, sixty-one, sixty-three, sixty-two, slap-up, slip-on, slip-up, smash-up, snarl-up, so, so-and-so, sold-out, some, someone, something, somewhere, so-so, sought-after, spaced-out, spend-all, spin-off, spread-out, stand-alone, stand-down, stand-in, stand-up, start-off, step-down, step-in, step-up, stick-on, still, stock-still, stock-take, stopped-up, straight-out, stripped-down, strung-out, stuck-up, such, sum-up, sure-enough, tailor-made, take, take-in, take-up, tap-off, teach-in, than, that, the, their, theirs, them, themselves, then, there, therefore, these, they, thing-in-itself, thirty-first, thirty-four, thirty-one, thirty-something, thirty-three, thirty-two, this, those, though, three, through, throw-in, thus, tie-in, tie-on, tie-up, time-out, tip-off, tip-up, to, to-do, toe-in, together, too, top-down, top-up, toss-up, touch-and-go, touch-me-not, toward, trade-in, trade-last, trade-off, tricked-out, trip-up, trumped-up, try-on, tumble-down, tune-up, turn-on, twenty-first, twenty-four, twenty-one, twenty-three, twenty-two, two, two-by-four, two-part, uncalled-for, uncared-for, under, unheard-of, unhoped-for, unlooked-for, unthought-of, until, unwished-for, up, up-and-down, upon, upside-down, us, US, U.S.A., very, walk-in, walk-on, walk-through, walk-to, walk-up, warm-up, washed-out, washed-up, washing-up, wave-off, way-out, we, well, well-being, well-done, well-made, well-off, well-thought-of, well-to-do, what, when, where, where's, whether, which, while, whipper-in, white-out, who, WHO, whole, whom, whose, why, will, wished-for, with, within, with-it, without, work-in, worn-out, would-be, write-down, write-in, write-off, wrong-side-out, year-around, yearned-for, yet, you, you-all, your, yours, yourself }

$\endgroup$
  • $\begingroup$ Why uppercase A and B, but lowercase c? $\endgroup$ – Peter Mortensen Sep 25 '18 at 17:09
  • 1
    $\begingroup$ What this longer list actually demonstrates is what could be called a peculiarity in DeleteStopwords[]: DeleteStopwords["ne'er-do-well wearing buttoned-down vests"] $\endgroup$ – J. M. will be back soon Sep 25 '18 at 17:27
6
$\begingroup$

Another way

WordData[All, "Stopwords"] == WordList["Stopwords"]

True

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.