The documentation of DeleteStopwords only says that it "uses a standard, built-in list of stopwords".

So what is it exactly?


Update

Now that it is said "standard", does that standard have a name?

A little spelunking of the code for DeleteStopwords[] yields the internally used stopword list:

DeleteStopwords; (* force auto-load *)
AlphabeticSort[List @@ TextProcessing`TextModificationDump`$stopWords["English"]] // Short
   {"a", "A", "about", "above", "across", "after", "again", "against", "all", "almost",
    "alone", "along", "already", "also", "although", <<240>>,
    "within", "without", "won't", "would", "wouldn't", "yet", "you", "you'd", "you'll",
    "you're", "you've", "your", "yours", "yourself", "yourselves"}
  • Wow, these are some undocumented functions? – Αλέξανδρος Ζεγγ Sep 25 at 6:22
  • But your codes merely return {"English"} as the result on my machine. Did I miss something? – Αλέξανδρος Ζεγγ Sep 25 at 6:24
  • 4
    @ΑλέξανδροςΖεγγ One has to evaluate DeleteStopwords first. The actual code for DeleteStopwords and TextProcessing`TextModificationDump`$stopWords is stored in a file and is loaded only after DeleteStopwords is evaluated in the current session. – Henrik Schumacher Sep 25 at 6:29
  • Ah, forgot the autoload. Thanks @Henrik! – J. M. is computer-less Sep 25 at 6:31
  • 1
    @hanshenrik note the use of // Short at the end of the code provided, since I didn't want to paste the entire list. Thus, only the first few and last few list elements are shown, and the rest are elided. Remove the // Short to see the entire list. – J. M. is computer-less Sep 25 at 8:33

The stop-words are documented and can be looked up as (see Details section of WordList):

WordList["Stopwords"]

enter image description here

As it might be subject to change so the output below can get outdated, but you can always run the function WordList["Stopwords"]:

{"0","1","2","3","4","5","6","7","8","9","a","A","about","above","across","after","again","against","all","almost","alone","along","already","also","although","always","am","among","an","and","another","any","anyone","anything","anywhere","are","aren't","around","as","at","b","B","back","be","became","because","become","becomes","been","before","behind","being","below","between","both","but","by","c","C","can","cannot","can't","could","couldn't","d","D","did","didn't","do","does","doesn't","doing","done","don't","down","during","e","E","each","either","enough","even","ever","every","everyone","everything","everywhere","f","F","few","find","first","for","four","from","full","further","g","G","get","give","go","h","H","had","hadn't","has","hasn't","have","haven't","having","he","he'd","he'll","her","here","here's","hers","herself","he's","him","himself","his","how","however","how's","i","I","i'd","if","i'll","i'm","in","interest","into","is","isn't","it","it's","its","itself","i've","j","J","k","K","keep","l","L","last","least","less","let's","m","M","made","many","may","me","might","more","most","mostly","much","must","mustn't","my","myself","n","N","never","next","no","nobody","noone","nor","not","nothing","now","nowhere","o","O","of","off","often","on","once","one","only","or","other","others","ought","our","ours","ourselves","out","over","own","p","P","part","per","perhaps","put","q","Q","r","R","rather","s","S","same","see","seem","seemed","seeming","seems","several","shan't","she","she'd","she'll","she's","should","shouldn't","show","side","since","so","some","someone","something","somewhere","still","such","t","T","take","than","that","that's","the","their","theirs","them","themselves","then","there","therefore","there's","these","they","they'd","they'll","they're","they've","this","those","though","three","through","thus","to","together","too","toward","two","u","U","under","until","up","upon","us","v","V","very","w","W","was","wasn't","we","we'd","we'll","well","we're","were","weren't","we've","what","what's","when","when's","where","where's","whether","which","while","who","whole","whom","who's","whose","why","why's","will","with","within","without","won't","would","wouldn't","x","X","y","Y","yet","you","you'd","you'll","your","you're","yours","yourself","yourselves","you've","z","Z"}

  • 2
    Interestingly, the complement of this list and the list in my answer yields only a list of letters and numbers, so that list certainly captures all the words. However, applying DeleteStopwords[] seems to only remove "i" from the list. – J. M. is computer-less Sep 25 at 18:06
  • @J.M.issomewhatokay. a few inconsistencies are known, hopefully will be polished. Important thing is --- the docs is open about the subject, good thing to know :-) The docs is an attractor of stability in terms of known things of internal architecture. – Vitaliy Kaurov Sep 25 at 20:13

Here is a longer list using the following commands:

A = WordList["KnownWords"];
B = DeleteStopwords[A];
c = Complement[A, B]

{ a, A, about, above, across, A.D., add-in, add-on, A.E., after, again, against, a.k.a., all, all-around, all-or-nothing, all-out, almost, alone, along, al-Qur'an, already, also, although, always, a.m., A.M., Am, AM, among, an, AN, and, and/or, another, any, anyone, anything, anywhere, A-one, are, around, as, As, AS, at, At, back, back-to-back, balls-up, bang-up, be, Be, beaten-up, beat-up, because, become, beefed-up, before, behind, being, belly-up, below, between, bicycle-built-for-two, blown-up, booze-up, born-again, both, bottom-up, boxed-in, break-in, bride-to-be, broken-down, brush-off, built-in, built-up, bundled-up, burned-out, burned-over, burnt-out, bust-up, but, button-down, buttoned-down, buttoned-up, by, by-and-by, call-back, caller-out, caller-up, call-in, call-out, camp-made, can, can-do, cared-for, carry-over, cast-off, change-up, check-in, ch'i, Ch'in, chin-up, chock-full, choke-full, chucker-out, chuck-full, churned-up, climb-down, clip-on, coach-and-four, comb-out, come-on, cover-up, crack-up, cure-all, custom-made, cut-in, cut-up, D.A., dead-on, derring-do, do, DO, d.o.a., do-it-yourself, done, do-nothing, do-si-do, down, Down, down-and-out, drawn-out, dressed-up, dried-out, dried-up, drive-in, drop-off, during, each, eighty-four, eighty-one, eighty-three, eighty-two, either, end-all, enough, even, ever, every, everyone, everything, everywhere, eyes-only, face-off, factory-made, fare-thee-well, far-off, far-out, few, fifty-four, fifty-one, fifty-three, fifty-two, fill-in, find, first, F.I.S.C., flame-out, flare-up, fly-by, follow-on, follow-through, follow-up, for, force-out, fore-and-after, forget-me-not, form-only, forty-first, forty-four, forty-one, forty-three, forty-two, foul-up, four, frame-up, free-for-all, from, fucked-up, full, further, get, get-go, get-up-and-go, G.I., gill-less, give, give-and-go, give-and-take, go, go-around, go-between, going-over, good-by, good-for-nothing, goof-off, groom-to-be, half-seas-over, hand-down, handed-down, hand-me-down, hands-down, hands-off, hands-on, hanger-on, hang-up, hard-on, has-been, have, have-not, Hawai'i, he, He, head-on, heads-up, heart-whole, her, here, Here, hers, herself, higher-up, high-interest, high-up, him, himself, his, hold-down, hollow-back, hoped-for, hopped-up, how, how-do-you-do, how-d'ye-do, however, HSV-I, hundred-and-first, hushed-up, i, I, I.D., i.e., I.E.D., if, If, ill-being, in, In, IN, in-between, inside-out, interest, into, I.Q., it, IT, its, itself, I.W.W., jerk-off, Johnny-jump-up, jumped-up, keep, knock-down, knock-down-and-drag-out, knocked-out, know-all, know-how, know-it-all, ladder-back, laid-back, laid-off, lash-up, last, lay-by, lay-up, lead-in, lean-to, least, less, lie-in, lighting-up, lights-out, log-in, longed-for, looker-on, look-over, low-down, low-interest, lying-in, ma'am, machine-made, made, made-up, make-do, make-up, man-made, many, marked-up, match-up, matt-up, may, May, me, ME, mess-up, mid-May, mid-off, mid-on, might, might-have-been, mixed-up, mix-up, mock-up, more, More, most, mostly, much, must, my, myself, ne'er-do-well, never, never-never, Never-Never, new-made, next, next-to-last, ninety-four, ninety-one, ninety-three, ninety-two, no, No, no., nobody, no-go, no-one, nor, no-show, nosh-up, not, nothing, now, nowhere, odds-on, of, off, off-and-on, often, on, once, once-over, one, one-and-one, one-off, one-on-one, one-to-one, only, OR, other, our, ours, ourselves, out, out-and-out, over, own, p.a., P.A., paid-up, part, passer-by, pass-through, paste-up, pegged-down, pent-up, per, perhaps, phase-out, phone-in, pick-me-up, pick-off, pig-a-back, pin-up, piss-up, plug-in, pop-up, Post-It, press-up, pull-in, pull-off, pull-through, pull-up, pumped-up, punch-up, purpose-made, put, put-down, put-on, put-put, put-up, put-upon, rake-off, raree-show, rather, rave-up, read-out, ready-made, right-down, right-side-out, right-side-up, rip-off, roll-on, run-down, run-in, runner-up, run-on, run-through, run-up, same, Same, Sana'a, sauce-alone, save-all, sawed-off, sawn-off, say-so, schoolma'am, see, seem, seeming, see-through, self-interest, self-made, self-will, send-off, set-back, set-to, seventy-four, seventy-one, seventy-three, seventy-two, seven-up, several, shake-up, shape-up, share-out, she, shell-less, shoo-in, shoot-down, shoot-'em-up, show, show-off, shut-in, side, side-to-side, since, sit-down, sit-in, sit-up, sixty-four, sixty-one, sixty-three, sixty-two, slap-up, slip-on, slip-up, smash-up, snarl-up, so, so-and-so, sold-out, some, someone, something, somewhere, so-so, sought-after, spaced-out, spend-all, spin-off, spread-out, stand-alone, stand-down, stand-in, stand-up, start-off, step-down, step-in, step-up, stick-on, still, stock-still, stock-take, stopped-up, straight-out, stripped-down, strung-out, stuck-up, such, sum-up, sure-enough, tailor-made, take, take-in, take-up, tap-off, teach-in, than, that, the, their, theirs, them, themselves, then, there, therefore, these, they, thing-in-itself, thirty-first, thirty-four, thirty-one, thirty-something, thirty-three, thirty-two, this, those, though, three, through, throw-in, thus, tie-in, tie-on, tie-up, time-out, tip-off, tip-up, to, to-do, toe-in, together, too, top-down, top-up, toss-up, touch-and-go, touch-me-not, toward, trade-in, trade-last, trade-off, tricked-out, trip-up, trumped-up, try-on, tumble-down, tune-up, turn-on, twenty-first, twenty-four, twenty-one, twenty-three, twenty-two, two, two-by-four, two-part, uncalled-for, uncared-for, under, unheard-of, unhoped-for, unlooked-for, unthought-of, until, unwished-for, up, up-and-down, upon, upside-down, us, US, U.S.A., very, walk-in, walk-on, walk-through, walk-to, walk-up, warm-up, washed-out, washed-up, washing-up, wave-off, way-out, we, well, well-being, well-done, well-made, well-off, well-thought-of, well-to-do, what, when, where, where's, whether, which, while, whipper-in, white-out, who, WHO, whole, whom, whose, why, will, wished-for, with, within, with-it, without, work-in, worn-out, would-be, write-down, write-in, write-off, wrong-side-out, year-around, yearned-for, yet, you, you-all, your, yours, yourself }

  • Why uppercase A and B, but lowercase c? – Peter Mortensen Sep 25 at 17:09
  • What this longer list actually demonstrates is what could be called a peculiarity in DeleteStopwords[]: DeleteStopwords["ne'er-do-well wearing buttoned-down vests"] – J. M. is computer-less Sep 25 at 17:27

Another way

WordData[All, "Stopwords"] == WordList["Stopwords"]

True

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.