Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a Excelsheet with tweetsmessages. These tweets are all written in Dutch. I would like to analyse these tweets in relation to the timestamp.

tweets = Import["E:\\mathematica\\data\\tweets_gr2014_v5.xls", {"Data", 1}];

tweets[[2 ;;, {1, 2}]]

The result is : The result is: { {"19-3-2014 23:13", "##GR2014 - Een goeie in tv commentaar: #Wilders schreeuwt wel veel, maar hij wil helemaal niet in steden regeren. Hij doet bijna nergens mee"}, {"19-3-2014 21:49", "#020kiest eerste uitslag Amsterdam zo op AT5 #GR2014 #at5"}, {"19-3-2014 22:26", "#033kiest pvda verliest veel... was wel te verwachten nietwaar? #GR2014"}.......}

Then I use StringCases to split the individual words in the tweet messages.

tweets1 = Transpose@{tweets[[2 ;;, 1]],  StringCases[ToLowerCase[tweets[[2 ;;,
 2]]], RegularExpression["\\w(?<!\\d)[\\w'-]*"]]};

The next step is dat I would like tot delete all stop words. Therefor I have a listing with Dutch Stop words.

stopwoordenNL = { "aan", "al", "alles", "als", "altijd", "andere", "ben", "bij", "daar", "dan", "dat", "de", "der", "deze", "die", "dit", "doch", "doen", "door", "dus", "een", "eens", "en", "er", "ge", "geen", "geweest", "haar", "had", "heb", "hebben", "heeft", "hem", "het", "hier", "hij", "hoe", "hun", "iemand", "iets", "ik", "in", "is", "ja", "je", "kan", "kon", "kunnen", "maar", "me", "meer", "men", "met", "mij", "mijn", "moet", "na", "naar", "niet", "niets", "nog", "nu", "of", "om", "omdat", "onder", "ons", "ook", "op", "over", "reeds", "te", "tegen", "toch", "toen", "tot", "u", "uit", "uw", "van", "veel", "voor", "want", "waren", "was", "wat", "werd", "wezen", "wie", "wil", "worden", "wordt", "zal", "ze", "zelf", "zich", "zij", "zijn", "zo", "zonder", "zou", "ten", "per", "lid", "pagina", "toe", "indien", "alle", "den", "bent", "hebt", "nee", "ja", "wel", "mee", "ter", "zie", "aldus", "alsmede", "alsnog", "betreffende", "danwel", "derhalve", "zoals", "zowel", "hierbij", "hierboven", "hiermee", "hierna", "hiertoe", "hiervan", "hiervoor", "tenzij", "and", "for"};

Without using the timestamp I used the script:

DeleteCases[tweets[[All, 2]], w_ /; MemberQ[stopwoordenNL], ToLowerCase[w]]];

But it doesn't work when I analyse two columns. How can I delete the stop words?

share|improve this question
    
The problem is most likely that w_ matches the top level, i.e. your list of lists. You have to make the pattern more specific, like this: {___,{___,w_,___},___}, or you can add Level[w_] = ... where ... is the appropriate level to your condition. –  Pickett Mar 31 at 8:49
    
Michiel, your final step is to use DeleteCases on tweets[[All, 2]]. Do you not mean to do something with tweets1 instead? There is also a syntax error in your last line, you probably meant to place the square bracket that closes MemberQ after ToLowerCase[w]. I'm not sure how the timestamp is relevant to all this, let's try to focus on deleting the "stop woorden" first, right? –  Jacob Akkerboom Mar 31 at 10:26

1 Answer 1

up vote 2 down vote accepted

Let's make some data first

str1 = Import["http://nl.wikipedia.org/wiki/Kat_(dier)", 
   "Plaintext"];
str2 = Import["http://nl.wikipedia.org/wiki/Hond", "Plaintext"];

This is a shorter way to get the words

words1 = StringCases[str, WordCharacter ..];
words2 = StringCases[str2, WordCharacter ..];

We could now immediately delete the "stopwoorden"

katNonStop = DeleteCases[words1, Alternatives @@ stopwoordenNL];
hondNonStop = DeleteCases[words2, Alternatives @@ stopwoordenNL];

But let's try to remake the structure of tweets1

tweets1 = {{{"bla1"}, words1}, {{"bla2"}, words2}};

We could then do

DeleteCases[tweets1[[All, 2]], Alternatives @@ stopwoordenNL, Infinity]
share|improve this answer
    
Hi Jacob,Thanks for your answer. When I try the same logic for SringReplace I've got a error:'String or list of strings expected at position 1 in StringReplace'. Can you explain me why? tweets3 = StringReplace[tweets2[[All, 2]], Alternatives @@ Characters[punctuation] -> " ", Infinity] –  Michiel van Mens Apr 1 at 7:03
    
I forgot to write that punctuation = ",/.<>?;':\"()+-_!&"; –  Michiel van Mens Apr 1 at 7:10
    
@MichielvanMens You are welcome. Try StringReplace[#, Alternatives @@ Characters[punctuation] -> " "] & /@ tweets2[[All, 2]]. –  Jacob Akkerboom Apr 1 at 7:30
    
Great!! Thank you. –  Michiel van Mens Apr 1 at 8:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.