Skip to main content
Mathematica

Return to Question

Tweeted twitter.com/#!/StackMma/status/444603518855700480
updated code, added output, clearer problem statement
Source Link
Tim Vermeulen
Tim Vermeulen
  • 407
  • 3
  • 11

I'm trying to get a list of words from an articlea string. Sounds like an easy task for Mathematica. I have the following code:

text = "Merçi d'avoir pris le temps.";
ToLowerCase[#] & /@ StringSplit[articleStringSplit[text, Except[WordCharacter] ..]

However, there is a problem with this: the articleoutput is in French. Many words use

{"merçi", "d", "avoir", "pris", "le", "temps"}

and not

{"merçi", "d'avoir", "pris", "le", "temps"}

because the characters ' and -is not a word character. Hence, and even more words use charactersI'd like é andto ignore the à' and, just like the î-. All those characters are omitted. Any idea on how to fix thisdo that?

I'm trying to get a list of words from an article. Sounds like an easy task for Mathematica. I have the following code:

ToLowerCase[#] & /@ StringSplit[article,Except[WordCharacter]..]

However, there is a problem with this: the article is in French. Many words use the characters ' and -, and even more words use characters like é and à and î. All those characters are omitted. Any idea how to fix this?

I'm trying to get a list of words from a string. Sounds like an easy task for Mathematica. I have the following code:

text = "Merçi d'avoir pris le temps.";
ToLowerCase[#] & /@ StringSplit[text, Except[WordCharacter] ..]

However, the output is

{"merçi", "d", "avoir", "pris", "le", "temps"}

and not

{"merçi", "d'avoir", "pris", "le", "temps"}

because the ' is not a word character. Hence, I'd like to ignore the ', just like the -. Any idea on how to do that?

Source Link
Tim Vermeulen
Tim Vermeulen
  • 407
  • 3
  • 11

Removing non-word characters from a string

I'm trying to get a list of words from an article. Sounds like an easy task for Mathematica. I have the following code:

ToLowerCase[#] & /@ StringSplit[article,Except[WordCharacter]..]

However, there is a problem with this: the article is in French. Many words use the characters ' and -, and even more words use characters like é and à and î. All those characters are omitted. Any idea how to fix this?