Mathematica Stack Exchange is a question and answer site for users of Mathematica. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to get a list of words from a string. Sounds like an easy task for Mathematica. I have the following code:

text = "Merçi d'avoir pris le temps.";
ToLowerCase[#] & /@ StringSplit[text, Except[WordCharacter] ..]

However, the output is

{"merçi", "d", "avoir", "pris", "le", "temps"}

and not

{"merçi", "d'avoir", "pris", "le", "temps"}

because the ' is not a word character. Hence, I'd like to ignore the ', just like the -. Any idea on how to do that?

share|improve this question
Can you give a short but complete example of the problem/difficulty you are seeing (i.e. code I can copy and and run directly), and mention what OS and what version of Mathematica you are using? I don't quite understand the question: what is the problem with é? WordCharacter does match it on my machine, as ToUpperCase/ToLowerCase work fine on it. – Szabolcs Mar 14 '14 at 20:49
@RunnyKine I'd rather have it omit all non-letter characters (except for ' and -), as there are some pretty weird ones in these articles. I'd rather not have to specify them all manually. – Tim Vermeulen Mar 14 '14 at 21:06
@Szabolcs You're right, é is matched by WordCharacter, I must've had some other error in my code. The other problem persists: I don't want ' and - taken out. Is there any way I can change Except[WordCharacter].. to something similar to Except[{WordCharacter,Characters["'-"]}]..? – Tim Vermeulen Mar 14 '14 at 21:11
@Szabolcs I've edited my question, hope it makes sense now. – Tim Vermeulen Mar 14 '14 at 21:14
@timvermeulen OK, sounds good. I thought those characters would not be matched on some other OSs. Could you simply use Except[WordCharacter | "'" | "-"]? – Szabolcs Mar 14 '14 at 21:19
up vote 7 down vote accepted
text = "Merçi d'avoir pris le temps."; 

ToLowerCase[#] & /@ StringSplit[text, Except[(LetterCharacter|"'"|"-")]..]


{"merçi", "d'avoir", "pris", "le", "temps"}

OR if some of the words contain digits then replace LetterCharacter with WordCharacter above.

share|improve this answer

Since you are trying to extract words I would suggest using StringCases rather than StringSplit. Also, there is no need to write ToLowerCase[#] & -- ToLowerCase /@ list would do. In fact because ToLowerCase is Listable it can be directly applied to the output list. Therefore I would write:

ToLowerCase @ StringCases[text, (WordCharacter | "'" | "-") ..]
{"merçi", "d'avoir", "pris", "le", "temps"}

StringCases is somewhat faster than the alternative:

big = "" <> Table[text, {50000}];

StringSplit[big, Except[(LetterCharacter | "'" | "-")] ..] // Timing // First

StringCases[big, (WordCharacter | "'" | "-") ..]           // Timing // First

share|improve this answer

Unless I'm missing some subtlety, this works:

    text = "Merçi d'avoir pris le temps.";
(* {Merçi, d'avoir, pris, le, temps.} *)
share|improve this answer
One still has to get rid of the terminal period. Ditto if the original text had commas, semicolons, dashes, etc. And then suppose one of the "words" were an abbreviation, e.g., "Dr."? – murray Mar 17 '14 at 3:48

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.