27
$\begingroup$

I've found that WordData has a database of English words and their possible parts of speech. But some words are ambiguous, e.g. result can be a verb or a noun:

In[7]:= WordData["result", "PartsOfSpeech"]
Out[7]= {"Noun", "Verb"}

Such cases can, however, be distinguished based on context.

So my question is: can one use Mathematica to perform parts-of-speech tagging of a text, either through built-in functions or as an extension? There are many known algorithms and implementations available, yet I have not been able to find a way to leverage them in Mathematica.

$\endgroup$
20
$\begingroup$

After downloading and unpacking the stanford Part-Of-Speech tagger that @PlatoManiac mentioned it is easy to call it with JLink:

$POSTaggerPath = "/some/where/stanford-postagger-2013-11-12/";

<< JLink`
AddToClassPath[$POSTaggerPath];

tagString[str_String] := JavaBlock[
  Module[{tagger},
   LoadJavaClass["edu.stanford.nlp.tagger.maxent.MaxentTagger"];
   tagger = JavaNew[
     "edu.stanford.nlp.tagger.maxent.MaxentTagger", 
     FileNameJoin[{$POSTaggerPath, "models", "english-left3words-distsim.tagger"}]];
   tagger@tagString[str]
   ]]

tagString["$HistoryLength=0 may result in lower memory use by discarding old result"]

(* $_$ HistoryLength_JJ =_JJ 0_CD may_MD result_VB in_IN
   lower_JJR memory_NN use_NN by_IN discarding_VBG old_JJ result_NN  *)

To use the bidirectional tagger I had to increase the java memory:

ReinstallJava[JVMArguments -> "-Xmx500m"]
$\endgroup$
  • 2
    $\begingroup$ That's a fast implementation..you guys are fluent in more than one language, I really envy that.+1 $\endgroup$ – PlatoManiac Nov 30 '13 at 15:22
  • $\begingroup$ Thanks! I don't "talk" Java but that's a mighty powerful way to interface things to Mathematica! One question: if I am going to call this tagString many times in my code, is there some overhead incurred from doing the LoadJavaClass and JavaNew every time? Should I move them out of the tagString function, in that case? $\endgroup$ – F'x Dec 1 '13 at 9:09
  • $\begingroup$ @F'x For sure, the JavaNew in particular. $\endgroup$ – ssch Dec 1 '13 at 11:25
7
$\begingroup$

As long as we're talking about leveraging external resources, one could also use http://www.text-processing.com, which offers natural language processing functions (including POS tagging) as a web service.

According to their documentation, "The public API is for non-commercial purposes, and each method is throttled to 1000 calls per day per IP", but apparently you can pay for higher/commercial use.

Anyway, this is an example of how their POS tagging service could be consumed via Mathematica 9 onwards:

tagSentence[sent_String] :=
 Module[{fetched},
  fetched = 
   URLFetch["http://text-processing.com/api/tag", "Method" -> "POST", 
    "Parameters" -> {"text" -> sent}]; (* URLFetch is present in Mathematica 9+*)  
  StringCases[fetched, 
   word : Except[WhitespaceCharacter] .. ~~ "/" ~~ 
     tag : Except[WhitespaceCharacter] | ")" .. :> {word, tag}]
  ]

Note that the string parsing code above is almost certainly not robust.

tagSentence["He used to play the role of a used-car salesman in a play."]

yields

{{"He", "P"}, {"used", "V"}, {"to", "T"}, {"play", "V"}, {"the",
"D"}, {"role", "N"}, {"of", "I"}, {"a", "D"}, {"used", "V"}, {"-",
":"}, {"car", "N"}, {"salesman", "N"}, {"in", "I"}, {"a", "D"}, {"play", "N"}, {".", "."}}

$\endgroup$
4
$\begingroup$

As of Mathematica 11, the function TextStructure seems to do pretty well. You can add "PartsOfSPeech" as an argument to specify that you just want to see the parts of speech of the sentence. The example the documentation gives is TextStructure["The cat sat on the mat.", "PartsOfSpeech"] which yields Parts of Speech

$\endgroup$
  • 1
    $\begingroup$ Additionally, there is also the function PartOfSpeech[] that can be used on isolated words. $\endgroup$ – J. M.'s technical difficulties Dec 30 '16 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.