Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I've found that WordData has a database of English words and their possible parts of speech. But some words are ambiguous, e.g. result can be a verb or a noun:

In[7]:= WordData["result", "PartsOfSpeech"]
Out[7]= {"Noun", "Verb"}

Such cases can, however, be distinguished based on context.

So my question is: can one use Mathematica to perform parts-of-speech tagging of a text, either through built-in functions or as an extension? There are many known algorithms and implementations available, yet I have not been able to find a way to leverage them in Mathematica.

share|improve this question
3  
@IstvánZachar what about this well known applications nlp.stanford.edu/software/tagger.shtml and the nlp.stanford.edu/downloads/tagger.shtml . Can we port them to Mathematica easily...and build on other functionalities based on them. –  PlatoManiac Nov 30 '13 at 12:56
add comment

2 Answers 2

up vote 14 down vote accepted

After downloading and unpacking the stanford Part-Of-Speech tagger that @PlatoManiac mentioned it is easy to call it with JLink:

$POSTaggerPath = "/some/where/stanford-postagger-2013-11-12/";

<< JLink`
AddToClassPath[$POSTaggerPath];

tagString[str_String] := JavaBlock[
  Module[{tagger},
   LoadJavaClass["edu.stanford.nlp.tagger.maxent.MaxentTagger"];
   tagger = JavaNew[
     "edu.stanford.nlp.tagger.maxent.MaxentTagger", 
     FileNameJoin[{$POSTaggerPath, "models", "english-left3words-distsim.tagger"}]];
   tagger@tagString[str]
   ]]

tagString["$HistoryLength=0 may result in lower memory use by discarding old result"]

(* $_$ HistoryLength_JJ =_JJ 0_CD may_MD result_VB in_IN
   lower_JJR memory_NN use_NN by_IN discarding_VBG old_JJ result_NN  *)

To use the bidirectional tagger I had to increase the java memory:

ReinstallJava[JVMArguments -> "-Xmx500m"]
share|improve this answer
2  
That's a fast implementation..you guys are fluent in more than one language, I really envy that.+1 –  PlatoManiac Nov 30 '13 at 15:22
    
Thanks! I don't "talk" Java but that's a mighty powerful way to interface things to Mathematica! One question: if I am going to call this tagString many times in my code, is there some overhead incurred from doing the LoadJavaClass and JavaNew every time? Should I move them out of the tagString function, in that case? –  F'x Dec 1 '13 at 9:09
    
@F'x For sure, the JavaNew in particular. –  ssch Dec 1 '13 at 11:25
add comment

As long as we're talking about leveraging external resources, one could also use http://www.text-processing.com, which offers natural language processing functions (including POS tagging) as a web service.

According to their documentation, "The public API is for non-commercial purposes, and each method is throttled to 1000 calls per day per IP", but apparently you can pay for higher/commercial use.

Anyway, this is an example of how their POS tagging service could be consumed via Mathematica 9 onwards:

tagSentence[sent_String] :=
 Module[{fetched},
  fetched = 
   URLFetch["http://text-processing.com/api/tag", "Method" -> "POST", 
    "Parameters" -> {"text" -> sent}]; (* URLFetch is present in Mathematica 9+*)  
  StringCases[fetched, 
   word : Except[WhitespaceCharacter] .. ~~ "/" ~~ 
     tag : Except[WhitespaceCharacter] | ")" .. :> {word, tag}]
  ]

Note that the string parsing code above is almost certainly not robust.

tagSentence["He used to play the role of a used-car salesman in a play."]

yields

{{"He", "P"}, {"used", "V"}, {"to", "T"}, {"play", "V"}, {"the",
"D"}, {"role", "N"}, {"of", "I"}, {"a", "D"}, {"used", "V"}, {"-",
":"}, {"car", "N"}, {"salesman", "N"}, {"in", "I"}, {"a", "D"}, {"play", "N"}, {".", "."}}

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.