16
$\begingroup$

The documentation of DeleteStopwords only says that it "uses a standard, built-in list of stopwords".

So what is it exactly?


Update

Now that it is said "standard", does that standard have a name?

$\endgroup$

4 Answers 4

17
$\begingroup$

A little spelunking of the code for DeleteStopwords[] yields the internally used stopword list:

DeleteStopwords; (* force auto-load *)
AlphabeticSort[List @@ TextProcessing`TextModificationDump`$stopWords["English"]] // Short
   {"a", "A", "about", "above", "across", "after", "again", "against", "all", "almost",
    "alone", "along", "already", "also", "although", <<240>>,
    "within", "without", "won't", "would", "wouldn't", "yet", "you", "you'd", "you'll",
    "you're", "you've", "your", "yours", "yourself", "yourselves"}
$\endgroup$
7
  • $\begingroup$ Wow, these are some undocumented functions? $\endgroup$ Commented Sep 25, 2018 at 6:22
  • $\begingroup$ But your codes merely return {"English"} as the result on my machine. Did I miss something? $\endgroup$ Commented Sep 25, 2018 at 6:24
  • 4
    $\begingroup$ @ΑλέξανδροςΖεγγ One has to evaluate DeleteStopwords first. The actual code for DeleteStopwords and TextProcessing`TextModificationDump`$stopWords is stored in a file and is loaded only after DeleteStopwords is evaluated in the current session. $\endgroup$ Commented Sep 25, 2018 at 6:29
  • $\begingroup$ Ah, forgot the autoload. Thanks @Henrik! $\endgroup$ Commented Sep 25, 2018 at 6:31
  • 1
    $\begingroup$ @hanshenrik note the use of // Short at the end of the code provided, since I didn't want to paste the entire list. Thus, only the first few and last few list elements are shown, and the rest are elided. Remove the // Short to see the entire list. $\endgroup$ Commented Sep 25, 2018 at 8:33
15
$\begingroup$

The stop-words are documented and can be looked up as (see Details section of WordList):

WordList["Stopwords"]

enter image description here

As it might be subject to change so the output below can get outdated, but you can always run the function WordList["Stopwords"]:

{"0","1","2","3","4","5","6","7","8","9","a","A","about","above","across","after","again","against","all","almost","alone","along","already","also","although","always","am","among","an","and","another","any","anyone","anything","anywhere","are","aren't","around","as","at","b","B","back","be","became","because","become","becomes","been","before","behind","being","below","between","both","but","by","c","C","can","cannot","can't","could","couldn't","d","D","did","didn't","do","does","doesn't","doing","done","don't","down","during","e","E","each","either","enough","even","ever","every","everyone","everything","everywhere","f","F","few","find","first","for","four","from","full","further","g","G","get","give","go","h","H","had","hadn't","has","hasn't","have","haven't","having","he","he'd","he'll","her","here","here's","hers","herself","he's","him","himself","his","how","however","how's","i","I","i'd","if","i'll","i'm","in","interest","into","is","isn't","it","it's","its","itself","i've","j","J","k","K","keep","l","L","last","least","less","let's","m","M","made","many","may","me","might","more","most","mostly","much","must","mustn't","my","myself","n","N","never","next","no","nobody","noone","nor","not","nothing","now","nowhere","o","O","of","off","often","on","once","one","only","or","other","others","ought","our","ours","ourselves","out","over","own","p","P","part","per","perhaps","put","q","Q","r","R","rather","s","S","same","see","seem","seemed","seeming","seems","several","shan't","she","she'd","she'll","she's","should","shouldn't","show","side","since","so","some","someone","something","somewhere","still","such","t","T","take","than","that","that's","the","their","theirs","them","themselves","then","there","therefore","there's","these","they","they'd","they'll","they're","they've","this","those","though","three","through","thus","to","together","too","toward","two","u","U","under","until","up","upon","us","v","V","very","w","W","was","wasn't","we","we'd","we'll","well","we're","were","weren't","we've","what","what's","when","when's","where","where's","whether","which","while","who","whole","whom","who's","whose","why","why's","will","with","within","without","won't","would","wouldn't","x","X","y","Y","yet","you","you'd","you'll","your","you're","yours","yourself","yourselves","you've","z","Z"}

$\endgroup$
2
  • 2
    $\begingroup$ Interestingly, the complement of this list and the list in my answer yields only a list of letters and numbers, so that list certainly captures all the words. However, applying DeleteStopwords[] seems to only remove "i" from the list. $\endgroup$ Commented Sep 25, 2018 at 18:06
  • $\begingroup$ @J.M.issomewhatokay. a few inconsistencies are known, hopefully will be polished. Important thing is --- the docs is open about the subject, good thing to know :-) The docs is an attractor of stability in terms of known things of internal architecture. $\endgroup$ Commented Sep 25, 2018 at 20:13
9
$\begingroup$

Here is a longer list using the following commands:

A = WordList["KnownWords"];
B = DeleteStopwords[A];
c = Complement[A, B]

{ a, A, about, above, across, A.D., add-in, add-on, A.E., after, again, against, a.k.a., all, all-around, all-or-nothing, all-out, almost, alone, along, al-Qur'an, already, also, although, always, a.m., A.M., Am, AM, among, an, AN, and, and/or, another, any, anyone, anything, anywhere, A-one, are, around, as, As, AS, at, At, back, back-to-back, balls-up, bang-up, be, Be, beaten-up, beat-up, because, become, beefed-up, before, behind, being, belly-up, below, between, bicycle-built-for-two, blown-up, booze-up, born-again, both, bottom-up, boxed-in, break-in, bride-to-be, broken-down, brush-off, built-in, built-up, bundled-up, burned-out, burned-over, burnt-out, bust-up, but, button-down, buttoned-down, buttoned-up, by, by-and-by, call-back, caller-out, caller-up, call-in, call-out, camp-made, can, can-do, cared-for, carry-over, cast-off, change-up, check-in, ch'i, Ch'in, chin-up, chock-full, choke-full, chucker-out, chuck-full, churned-up, climb-down, clip-on, coach-and-four, comb-out, come-on, cover-up, crack-up, cure-all, custom-made, cut-in, cut-up, D.A., dead-on, derring-do, do, DO, d.o.a., do-it-yourself, done, do-nothing, do-si-do, down, Down, down-and-out, drawn-out, dressed-up, dried-out, dried-up, drive-in, drop-off, during, each, eighty-four, eighty-one, eighty-three, eighty-two, either, end-all, enough, even, ever, every, everyone, everything, everywhere, eyes-only, face-off, factory-made, fare-thee-well, far-off, far-out, few, fifty-four, fifty-one, fifty-three, fifty-two, fill-in, find, first, F.I.S.C., flame-out, flare-up, fly-by, follow-on, follow-through, follow-up, for, force-out, fore-and-after, forget-me-not, form-only, forty-first, forty-four, forty-one, forty-three, forty-two, foul-up, four, frame-up, free-for-all, from, fucked-up, full, further, get, get-go, get-up-and-go, G.I., gill-less, give, give-and-go, give-and-take, go, go-around, go-between, going-over, good-by, good-for-nothing, goof-off, groom-to-be, half-seas-over, hand-down, handed-down, hand-me-down, hands-down, hands-off, hands-on, hanger-on, hang-up, hard-on, has-been, have, have-not, Hawai'i, he, He, head-on, heads-up, heart-whole, her, here, Here, hers, herself, higher-up, high-interest, high-up, him, himself, his, hold-down, hollow-back, hoped-for, hopped-up, how, how-do-you-do, how-d'ye-do, however, HSV-I, hundred-and-first, hushed-up, i, I, I.D., i.e., I.E.D., if, If, ill-being, in, In, IN, in-between, inside-out, interest, into, I.Q., it, IT, its, itself, I.W.W., jerk-off, Johnny-jump-up, jumped-up, keep, knock-down, knock-down-and-drag-out, knocked-out, know-all, know-how, know-it-all, ladder-back, laid-back, laid-off, lash-up, last, lay-by, lay-up, lead-in, lean-to, least, less, lie-in, lighting-up, lights-out, log-in, longed-for, looker-on, look-over, low-down, low-interest, lying-in, ma'am, machine-made, made, made-up, make-do, make-up, man-made, many, marked-up, match-up, matt-up, may, May, me, ME, mess-up, mid-May, mid-off, mid-on, might, might-have-been, mixed-up, mix-up, mock-up, more, More, most, mostly, much, must, my, myself, ne'er-do-well, never, never-never, Never-Never, new-made, next, next-to-last, ninety-four, ninety-one, ninety-three, ninety-two, no, No, no., nobody, no-go, no-one, nor, no-show, nosh-up, not, nothing, now, nowhere, odds-on, of, off, off-and-on, often, on, once, once-over, one, one-and-one, one-off, one-on-one, one-to-one, only, OR, other, our, ours, ourselves, out, out-and-out, over, own, p.a., P.A., paid-up, part, passer-by, pass-through, paste-up, pegged-down, pent-up, per, perhaps, phase-out, phone-in, pick-me-up, pick-off, pig-a-back, pin-up, piss-up, plug-in, pop-up, Post-It, press-up, pull-in, pull-off, pull-through, pull-up, pumped-up, punch-up, purpose-made, put, put-down, put-on, put-put, put-up, put-upon, rake-off, raree-show, rather, rave-up, read-out, ready-made, right-down, right-side-out, right-side-up, rip-off, roll-on, run-down, run-in, runner-up, run-on, run-through, run-up, same, Same, Sana'a, sauce-alone, save-all, sawed-off, sawn-off, say-so, schoolma'am, see, seem, seeming, see-through, self-interest, self-made, self-will, send-off, set-back, set-to, seventy-four, seventy-one, seventy-three, seventy-two, seven-up, several, shake-up, shape-up, share-out, she, shell-less, shoo-in, shoot-down, shoot-'em-up, show, show-off, shut-in, side, side-to-side, since, sit-down, sit-in, sit-up, sixty-four, sixty-one, sixty-three, sixty-two, slap-up, slip-on, slip-up, smash-up, snarl-up, so, so-and-so, sold-out, some, someone, something, somewhere, so-so, sought-after, spaced-out, spend-all, spin-off, spread-out, stand-alone, stand-down, stand-in, stand-up, start-off, step-down, step-in, step-up, stick-on, still, stock-still, stock-take, stopped-up, straight-out, stripped-down, strung-out, stuck-up, such, sum-up, sure-enough, tailor-made, take, take-in, take-up, tap-off, teach-in, than, that, the, their, theirs, them, themselves, then, there, therefore, these, they, thing-in-itself, thirty-first, thirty-four, thirty-one, thirty-something, thirty-three, thirty-two, this, those, though, three, through, throw-in, thus, tie-in, tie-on, tie-up, time-out, tip-off, tip-up, to, to-do, toe-in, together, too, top-down, top-up, toss-up, touch-and-go, touch-me-not, toward, trade-in, trade-last, trade-off, tricked-out, trip-up, trumped-up, try-on, tumble-down, tune-up, turn-on, twenty-first, twenty-four, twenty-one, twenty-three, twenty-two, two, two-by-four, two-part, uncalled-for, uncared-for, under, unheard-of, unhoped-for, unlooked-for, unthought-of, until, unwished-for, up, up-and-down, upon, upside-down, us, US, U.S.A., very, walk-in, walk-on, walk-through, walk-to, walk-up, warm-up, washed-out, washed-up, washing-up, wave-off, way-out, we, well, well-being, well-done, well-made, well-off, well-thought-of, well-to-do, what, when, where, where's, whether, which, while, whipper-in, white-out, who, WHO, whole, whom, whose, why, will, wished-for, with, within, with-it, without, work-in, worn-out, would-be, write-down, write-in, write-off, wrong-side-out, year-around, yearned-for, yet, you, you-all, your, yours, yourself }

$\endgroup$
2
  • $\begingroup$ Why uppercase A and B, but lowercase c? $\endgroup$ Commented Sep 25, 2018 at 17:09
  • 1
    $\begingroup$ What this longer list actually demonstrates is what could be called a peculiarity in DeleteStopwords[]: DeleteStopwords["ne'er-do-well wearing buttoned-down vests"] $\endgroup$ Commented Sep 25, 2018 at 17:27
6
$\begingroup$

Another way

WordData[All, "Stopwords"] == WordList["Stopwords"]

True

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.