Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I'm trying to get a list of words from a string. Sounds like an easy task for Mathematica. I have the following code:

text = "Merçi d'avoir pris le temps.";
ToLowerCase[#] & /@ StringSplit[text, Except[WordCharacter] ..]

However, the output is

{"merçi", "d", "avoir", "pris", "le", "temps"}

and not

{"merçi", "d'avoir", "pris", "le", "temps"}

because the ' is not a word character. Hence, I'd like to ignore the ', just like the -. Any idea on how to do that?

share|improve this question
    
Can you give a short but complete example of the problem/difficulty you are seeing (i.e. code I can copy and and run directly), and mention what OS and what version of Mathematica you are using? I don't quite understand the question: what is the problem with é? WordCharacter does match it on my machine, as ToUpperCase/ToLowerCase work fine on it. –  Szabolcs Mar 14 at 20:49
    
@RunnyKine I'd rather have it omit all non-letter characters (except for ' and -), as there are some pretty weird ones in these articles. I'd rather not have to specify them all manually. –  timvermeulen Mar 14 at 21:06
    
@Szabolcs You're right, é is matched by WordCharacter, I must've had some other error in my code. The other problem persists: I don't want ' and - taken out. Is there any way I can change Except[WordCharacter].. to something similar to Except[{WordCharacter,Characters["'-"]}]..? –  timvermeulen Mar 14 at 21:11
    
@Szabolcs I've edited my question, hope it makes sense now. –  timvermeulen Mar 14 at 21:14
1  
@timvermeulen OK, sounds good. I thought those characters would not be matched on some other OSs. Could you simply use Except[WordCharacter | "'" | "-"]? –  Szabolcs Mar 14 at 21:19

3 Answers 3

up vote 5 down vote accepted
text = "Merçi d'avoir pris le temps."; 

ToLowerCase[#] & /@ StringSplit[text, Except[(LetterCharacter|"'"|"-")]..]

Gives:

{"merçi", "d'avoir", "pris", "le", "temps"}

OR if some of the words contain digits then replace LetterCharacter with WordCharacter above.

share|improve this answer

Since you are trying to extract words I would suggest using StringCases rather than StringSplit. Also, there is no need to write ToLowerCase[#] & -- ToLowerCase /@ list would do. In fact because ToLowerCase is Listable it can be directly applied to the output list. Therefore I would write:

ToLowerCase @ StringCases[text, (WordCharacter | "'" | "-") ..]
{"merçi", "d'avoir", "pris", "le", "temps"}

StringCases is somewhat faster than the alternative:

big = "" <> Table[text, {50000}];

StringSplit[big, Except[(LetterCharacter | "'" | "-")] ..] // Timing // First

StringCases[big, (WordCharacter | "'" | "-") ..]           // Timing // First
0.1342

0.1092
share|improve this answer

Unless I'm missing some subtlety, this works:

    text = "Merçi d'avoir pris le temps.";
    StringSplit[text]
(* {Merçi, d'avoir, pris, le, temps.} *)
share|improve this answer
1  
One still has to get rid of the terminal period. Ditto if the original text had commas, semicolons, dashes, etc. And then suppose one of the "words" were an abbreviation, e.g., "Dr."? –  murray Mar 17 at 3:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.