# Can Mathematica determine part-of-speech for each word in a text?

I've found that WordData has a database of English words and their possible parts of speech. But some words are ambiguous, e.g. result can be a verb or a noun:

In[7]:= WordData["result", "PartsOfSpeech"]
Out[7]= {"Noun", "Verb"}

Such cases can, however, be distinguished based on context.

So my question is: can one use Mathematica to perform parts-of-speech tagging of a text, either through built-in functions or as an extension? There are many known algorithms and implementations available, yet I have not been able to find a way to leverage them in Mathematica.

-
@IstvánZachar what about this well known applications nlp.stanford.edu/software/tagger.shtml and the nlp.stanford.edu/downloads/tagger.shtml . Can we port them to Mathematica easily...and build on other functionalities based on them. – PlatoManiac Nov 30 '13 at 12:56

$POSTaggerPath = "/some/where/stanford-postagger-2013-11-12/"; << JLink` AddToClassPath[$POSTaggerPath];

tagString[str_String] := JavaBlock[
Module[{tagger},
tagger = JavaNew[
"edu.stanford.nlp.tagger.maxent.MaxentTagger",
FileNameJoin[{$POSTaggerPath, "models", "english-left3words-distsim.tagger"}]]; tagger@tagString[str] ]] tagString["$HistoryLength=0 may result in lower memory use by discarding old result"]

(* $_$ HistoryLength_JJ =_JJ 0_CD may_MD result_VB in_IN
lower_JJR memory_NN use_NN by_IN discarding_VBG old_JJ result_NN  *)

To use the bidirectional tagger I had to increase the java memory:

ReinstallJava[JVMArguments -> "-Xmx500m"]
-
That's a fast implementation..you guys are fluent in more than one language, I really envy that.+1 – PlatoManiac Nov 30 '13 at 15:22
Thanks! I don't "talk" Java but that's a mighty powerful way to interface things to Mathematica! One question: if I am going to call this tagString many times in my code, is there some overhead incurred from doing the LoadJavaClass and JavaNew every time? Should I move them out of the tagString function, in that case? – F'x Dec 1 '13 at 9:09
@F'x For sure, the JavaNew in particular. – ssch Dec 1 '13 at 11:25

As long as we're talking about leveraging external resources, one could also use http://www.text-processing.com, which offers natural language processing functions (including POS tagging) as a web service.

According to their documentation, "The public API is for non-commercial purposes, and each method is throttled to 1000 calls per day per IP", but apparently you can pay for higher/commercial use.

Anyway, this is an example of how their POS tagging service could be consumed via Mathematica 9 onwards:

tagSentence[sent_String] :=
Module[{fetched},
fetched =
URLFetch["http://text-processing.com/api/tag", "Method" -> "POST",
"Parameters" -> {"text" -> sent}]; (* URLFetch is present in Mathematica 9+*)
StringCases[fetched,
word : Except[WhitespaceCharacter] .. ~~ "/" ~~
tag : Except[WhitespaceCharacter] | ")" .. :> {word, tag}]
]

Note that the string parsing code above is almost certainly not robust.

tagSentence["He used to play the role of a used-car salesman in a play."]

yields

{{"He", "P"}, {"used", "V"}, {"to", "T"}, {"play", "V"}, {"the",
"D"}, {"role", "N"}, {"of", "I"}, {"a", "D"}, {"used", "V"}, {"-",
":"}, {"car", "N"}, {"salesman", "N"}, {"in", "I"}, {"a", "D"}, {"play", "N"}, {".", "."}}

-