10
$\begingroup$

I saw that in new version of Mathematica there is an interesting function: DeleteStopwords that deletes stopwords from a list of words, but this function is only for the English language. But I'm Italian and my "stopwords" are different from English ones. So I'm wondering if there is a method to create a personal stopwords, and delete a text from it. This is my code 

Clear["Global`*"]
mytext = Import["https://gallery.mailchimp.com/66b77b8f219a30e5da628b18b/files/ps.txt"];
WordCloud[mytext]

enter image description here

As you can see there are a lot of word that are stopwords.

Another ideas is to cut all words with length less than three. Any ideas? Thanks.

Improve this question
$\endgroup$
3
  • $\begingroup$ Welcome to Mathematica.SE! I suggest the following: 1) As you receive help, try to give it too, by answering questions in your area of expertise. 2) Take the tour! 3) When you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge. Also, please remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign! $\endgroup$ – Michael E2 Oct 31 '15 at 15:23
  • $\begingroup$ You may have to roll your own. You can find lists of stop words for Italian on the web (e.g. code.google.com/p/stop-words and import them to Mathematica. Then you might use TextWords to get a list of words from your text, and Complement[yourwordlist, stopwordlist] to get a cleaned list to feed to WordCloud. Unfortunately I'm on a tablet at the moment and can't actually do it myself... $\endgroup$ – MarcoB Oct 31 '15 at 15:27
  • $\begingroup$ Here is a Github variant of a stopword collection, see github.com/stopwords-iso $\endgroup$ – FredrikD Oct 26 '20 at 13:45
6
$\begingroup$

I use:

RemoveStopWords[doc_, stopWords_] := 
  StringDelete[doc, 
   WordBoundary ~~ Alternatives @@ stopWords ~~ WordBoundary, 
   IgnoreCase -> True];

Where stopWords is a list of all stop words, I found a list of stop words for my language easily through an internet search.

In addition to this I also remove words that are too short. Another option is to use a method that puts less weight on common words, have a look at TF-IDF for example:

tf–idf can be successfully used for stop-words filtering in various subject fields including text summarization and classification.

You could essentially use it to figure out what words should be considered stop words. But you'd need more than one document for sure, so it depends on what data you're working with.

Improve this answer
$\endgroup$
3
$\begingroup$

Another ideas is to cut all words with length less than three. Any ideas?

StringReplace[mytext, 
 WordBoundary ~~ Repeated[WordCharacter, 2] ~~ WordBoundary -> ""]

But I suppose less or equal three would be better. Put there 3 then and:

enter image description here

Improve this answer
$\endgroup$
0
$\begingroup$

This is a very interesting question and very interesting answers. I tried to evaluate the answer of Kuba using the MarcoB suggestions. For instance:

Clear["Global`*"]
mytext = Import[
"https://gallery.mailchimp.com/66b77b8f219a30e5da628b18b/files/ps.\
txt"];
stopwordit = 
Import["https://sites.google.com/site/kevinbouge/stopwords-lists/\
stopwords_it.txt?attredirects=0&d=1", {"Text"}];
WordCloud[mytext]

enter image description here

Then I used the kuba function:

RemoveStopWords[doc_, stopWords_] := 
StringDelete[doc, 
WordBoundary ~~ Alternatives @@ stopWords ~~ WordBoundary, 
IgnoreCase -> True];
textwithoutSW = RemoveStopWords[mytext, stopwordit];
WordCloud[textwithoutSW]

enter image description here

but the WordClouds are the same, although the word "che" is on the StopWords in Italian language. What's wrong?

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ You don't import the stop words properly, try "List" or "Table" instead: stopwordit = Import["sites.google.com/site/kevinbouge/stopwords-lists\ stopwords_it.txt?attredirects=0&d=1", "List"]; (And that's not from Kuba's post!) $\endgroup$ – V.E. Oct 31 '15 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.