6
$\begingroup$

I'm trying to get a list of words from a string. Sounds like an easy task for Mathematica. I have the following code:

text = "Merçi d'avoir pris le temps.";
ToLowerCase[#] & /@ StringSplit[text, Except[WordCharacter] ..]

However, the output is

{"merçi", "d", "avoir", "pris", "le", "temps"}

and not

{"merçi", "d'avoir", "pris", "le", "temps"}

because the ' is not a word character. Hence, I'd like to ignore the ', just like the -. Any idea on how to do that?

Improve this question
$\endgroup$
8
  • $\begingroup$ Can you give a short but complete example of the problem/difficulty you are seeing (i.e. code I can copy and and run directly), and mention what OS and what version of Mathematica you are using? I don't quite understand the question: what is the problem with é? WordCharacter does match it on my machine, as ToUpperCase/ToLowerCase work fine on it. $\endgroup$
    – Szabolcs
    Commented Mar 14, 2014 at 20:49
  • $\begingroup$ @RunnyKine I'd rather have it omit all non-letter characters (except for ' and -), as there are some pretty weird ones in these articles. I'd rather not have to specify them all manually. $\endgroup$
    – Tim Vermeulen
    Commented Mar 14, 2014 at 21:06
  • $\begingroup$ @Szabolcs You're right, é is matched by WordCharacter, I must've had some other error in my code. The other problem persists: I don't want ' and - taken out. Is there any way I can change Except[WordCharacter].. to something similar to Except[{WordCharacter,Characters["'-"]}]..? $\endgroup$
    – Tim Vermeulen
    Commented Mar 14, 2014 at 21:11
  • $\begingroup$ @Szabolcs I've edited my question, hope it makes sense now. $\endgroup$
    – Tim Vermeulen
    Commented Mar 14, 2014 at 21:14
  • 1
    $\begingroup$ @timvermeulen OK, sounds good. I thought those characters would not be matched on some other OSs. Could you simply use Except[WordCharacter | "'" | "-"]? $\endgroup$
    – Szabolcs
    Commented Mar 14, 2014 at 21:19

4 Answers 4

Reset to default
8
$\begingroup$ 
text = "Merçi d'avoir pris le temps."; 

ToLowerCase[#] & /@ StringSplit[text, Except[(LetterCharacter|"'"|"-")]..]

Gives:

{"merçi", "d'avoir", "pris", "le", "temps"}

OR if some of the words contain digits then replace LetterCharacter with WordCharacter above.

Improve this answer
$\endgroup$
2
$\begingroup$

Since you are trying to extract words I would suggest using StringCases rather than StringSplit. Also, there is no need to write ToLowerCase[#] & -- ToLowerCase /@ list would do. In fact because ToLowerCase is Listable it can be directly applied to the output list. Therefore I would write:

ToLowerCase @ StringCases[text, (WordCharacter | "'" | "-") ..]

{"merçi", "d'avoir", "pris", "le", "temps"}

StringCases is somewhat faster than the alternative:

big = "" <> Table[text, {50000}];

StringSplit[big, Except[(LetterCharacter | "'" | "-")] ..] // Timing // First

StringCases[big, (WordCharacter | "'" | "-") ..]           // Timing // First

0.1342

0.1092
Improve this answer
$\endgroup$
1
$\begingroup$

It will be a discovery for some that:

DictionaryWordQ[#, Language -> "French"] & /@ {"merçi", "merci"}

{False, True}

Other than that the following does the job:

w1 = TextCases[text, "Word"]

{"Merçi", "d'avoir", "pris", "le", "temps"}

Another example:

TextCases["Répétez s'il vous plaît. Quand voulez-vous voyager?", \
"Word"]

{"Répétez", "s'il", "vous", "plaît", "Quand", "voulez-vous",
"voyager"}

Improve this answer
$\endgroup$
0
$\begingroup$

Unless I'm missing some subtlety, this works:

    text = "Merçi d'avoir pris le temps.";
    StringSplit[text]
(* {Merçi, d'avoir, pris, le, temps.} *)
Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ One still has to get rid of the terminal period. Ditto if the original text had commas, semicolons, dashes, etc. And then suppose one of the "words" were an abbreviation, e.g., "Dr."? $\endgroup$
    – murray
    Commented Mar 17, 2014 at 3:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.