2
$\begingroup$

I am playing with Natural language programming in Mathematica, and can't figure out how to extract the entity classes (i.e. male, female) for the following example text:

"This was a pleasant summer afternoon. The hilly road crisscrossing the fields was a breezy ride for William Henry Hampton the Third, a boy of 8 years of age, and his cousin, Elizabeth-Ann Randall, a girl of 10 years of age. The Victorian English countryside was the perfect summer vacation setting for the cousins taking time off from a busy school in London. Their grandmother, Henriette Blake, who recently turned 57 years old, has been living in the countryside with her husband her entire life after her great-great grandparents left the city of Birmingham to try their hand in farming. Henriette's husband passed on due to old age many years ago, and her days were enlightened with her grandkids visits during their summer vacations.";

Below is what I tried, with actual result and expected result noted inline; the goals to achieve on are outlined below the code:

novel = "This was a pleasant summer afternoon. The hilly road \
crisscrossing the fields was a breezy ride for William Henry Hampton \
the Third, a boy of 8 years of age, and his cousin, Elizabeth-Ann \
Randall, a girl of 10 years of age. The Victorian English countryside \
was the perfect summer vacation setting for the cousins taking time \
off from a busy school in London. Their grandmother, Henriette Blake, \
who recently turned 57 years old, has been living in the countryside \
with her husband her entire life after her great-great grandparents \
left the city of Birmingham to try their hand in farming. Henriette's \
husband passed on due to old age many years ago, and her days were \
enlightened with her grandkids visits during their summer vacations.";

EntityRegister[EntityStore["male" -> <|
     "Entities" -> <|"boy", "father", "grandfather", "son", "husband"|>
     |>]];
EntityRegister[EntityStore["female" -> <|
     "Entities" -> <|"girl", "mother", "grandmother", "daughter", "wife"|>
     |>]];

FindTextualAnswer[novel, "What are the names of the males?"]
(* Result: Elizabeth-Ann Randall, Expected: William Henry Hampton the Third *)
(* HOWEVER, is the above be better suited for TextCases? This is tested below. *)

FindTextualAnswer[novel, "What is the name of the boy?"]
(* Result: Elizabeth-Ann Randall, Expected: William Henry Hampton the Third *)

FindTextualAnswer[novel, "What is the name of the father?"]
(* Result: Elizabeth-Ann Randall, Expected: none *)

FindTextualAnswer[novel, "What is the name of the grandfather?"]
(* Result: Henriette Blake, Expected: not available *)

FindTextualAnswer[novel, "What are the names of the females?"]
(* Result: Elizabeth-Ann Randall, Expected: Elizabeth-Ann Randall, Henriette Blake *)
(* HOWEVER, is the above be better suited for TextCases? This is tested below. *)

FindTextualAnswer[novel, "What is the name of the girl?"]
(* Result: Elizabeth-Ann Randall, Expected: Elizabeth-Ann Randall *)

FindTextualAnswer[novel, "What is the name of the mother?"]
(* Result: Elizabeth-Ann Randall, Expected: none *)

FindTextualAnswer[novel, "What is the name of the grandmother?"]
(* Result: Elizabeth-Ann Randall, Expected: Henriette Blake *)

TextCases[novel, "male"]
(* Error result *)

TextCases[novel, "female"]
(* Error result *)

It seems, with further deep-dive needed, that spaCy combined with prodigy has a working approach to solve for these questions, however before taking the leap, I hope Mathematica would have one available too.

[Update in response to m_goldberg, with appreciation] I see this as a named entity recognition (NER) problem. The goal is to receive a recommendation on approach (or a code) to train a model (e.g. a neural network) to accept input (a category) and, accounting for context (i.e. novel), provide output, such as:

Input: "male", Context: novel, Output: "William Henry Hampton the Third", "grandfather, unnamed"

Input: "boy", Context: novel, Output: "William Henry Hampton the Third"

Input: "father", Context: novel, Output: "No such character"

Input: "grandfather", Context: novel, Output: "Unnamed character"

Input: "female", Context: novel, Output: "Elizabeth-Ann Randall", "Henriette Blake"

Input: "girl", Context: novel, Output: "Elizabeth-Ann Randall"

Input: "mother", Context: novel, Output: "No such character"

Input: "grandmother", Context: novel, Output: "Henriette Blake"

In the meantime, this Mathematica tutorial regretfully does not suggest a solution: https://reference.wolfram.com/language/tutorial/NeuralNetworksSequenceLearning.html The closest approach is under the tutorial section "Simple RNN Trained on the bAbI QA Dataset", where the neural net accepts Context and a question about Context, and returns a classifier. From here, I am unable to envision how to tweak the NN architecture to accept Context and a classifier to return an answer from the Context based on the classifier.

There are also examples, regretfully not relevant, under individual neural networks in Wolfram Neural Net Repository such as here: https://resources.wolframcloud.com/NeuralNetRepository/resources/BERT-Trained-on-BookCorpus-and-Wikipedia-Data (Note: for other models see Browse by Task Type->(Feature Extraction, Language Modeling))

$\endgroup$
  • $\begingroup$ It is not al all clear to me what your question is. What kind of answer do you hope to get from him post? $\endgroup$ – m_goldberg Oct 29 at 0:50
  • $\begingroup$ Thank you @m_goldberg, updated with section [Update in response to m_goldberg, with appreciation] $\endgroup$ – Matvei Kruglyak Oct 29 at 3:47
3
$\begingroup$

All functions attempted by OP are experimental. That said, the particular workflow of OP using EntityRegister / EntityStore probably should be expected to work. (Meaning maybe or maybe not...)

Text data

Here is the text from OP:

novel = "This was a pleasant summer afternoon. The hilly road \
crisscrossing the fields was a breezy ride for William Henry Hampton \
the Third, a boy of 8 years of age, and his cousin, Elizabeth-Ann \
Randall, a girl of 10 years of age. The Victorian English countryside \
was the perfect summer vacation setting for the cousins taking time \
off from a busy school in London. Their grandmother, Henriette Blake, \
who recently turned 57 years old, has been living in the countryside \
with her husband her entire life after her great-great grandparents \
left the city of Birmingham to try their hand in farming. Henriette's \
husband passed on due to old age many years ago, and her days were \
enlightened with her grandkids visits during their summer vacations.";

Different workflow

It seems that one of OP’s wants can be achieved with “proper” use of TextCases combined with one of Classify's pre-built classifiers:

(We use Named Entity Recognition to get person names and then use classification to get the corresponding genders.)

TextCases[novel, "Person", "AcceptanceThreshold" -> 0.2]

(*{"William Henry Hampton the Third", "Elizabeth-Ann Randall", "Victorian", "Henriette Blake", "Henriette"}*)
Association@Map[# -> Classify["NameGender", #] &, %]

(*<|"William Henry Hampton the Third" -> "Male", 
 "Elizabeth-Ann Randall" -> Indeterminate, 
 "Victorian" -> Indeterminate, 
 "Henriette Blake" -> "Female", 
 "Henriette" -> "Female"|>*)

Using better options and more comprehensive parameters

Better results are obtained using PerformanceGoal->”Quality” for FindTextualAnswer.

Additionally, using probabilities and an acceptance threshold produces results OP expects. (The association aQAs below is made using OP's posted comments of the results by FindTextualAnswer.)

opts = {PerformanceGoal -> "Quality"};
args = Sequence @@ {3, {"String", "Probability"}};
aQAs = <|
   "What are the names of the males?" -> "William Henry Hampton the Third",
   "What is the name of the boy?" -> "William Henry Hampton the Third",
   "What is the name of the father?" -> None,
   "What is the name of the grandfather?" -> None,
   "What are the names of the females?" -> "Elizabeth-Ann Randall,Henriette Blake",
   "What is the name of the girl?" -> "Elizabeth-Ann Randall",
   "What is the name of the mother?" -> None,
   "What is the name of the grandmother?" -> "Henriette Blake"
   |>;
threshold = 0.1;
res = 
  KeyValueMap[
    <|"Question" -> #1, 
      "Result" -> Select[Association[Rule @@@ FindTextualAnswer[novel, #1, args, opts]], # >= threshold &], 
      "Expected" -> #2|> &, 
    aQAs
  ];
ResourceFunction["GridTableForm"][Values /@ res, TableHeadings -> Keys[res[[1]]]]

enter image description here

For the question “What are the names of the females?” expected results are obtained if no threshold filtering is done:

FindTextualAnswer[novel, "What are the names of the females?", args, opts]

(*{
{"cousin, Elizabeth-Ann Randall, a girl",  1.32598*10^-6}, 
{"Henriette Blake", 6.72097*10^-8}, {"grandmother", 1.01825*10^-9}
}*)
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for refining the approach with FindTextualAnswer. The question now becomes whether, to paraphrase spaCy, the examples here are memorized by a model rather than a theory that can be generalized across other examples is developed. After all, we don’t just want the model to learn that this one instance of "William Henry Hampton the Third" right here is a boy and male – we want it to learn that "William Henry Hampton the Third" or another name, in contexts like this, is most likely a boy and male. spacy.io/usage/training#ner $\endgroup$ – Matvei Kruglyak Oct 29 at 20:22
  • $\begingroup$ To update, after I changed the names in novel to Thomas Hardy, Anastasia Smith, and Jane Eyre, the boy, girl, and grandmother result is coming with >95% confidence. I recall reading (possibly in Wolfram's blog) that FindTextualAnswer is built-on existing architecture (either BERT or GPT2) so the foundation is solid. Training, by the looks of it, can be done via KeyValueMap $\endgroup$ – Matvei Kruglyak Oct 31 at 0:02
  • $\begingroup$ Another interesting comment from Wolfram's blog is that Predict and Classify could be combined with a model from Neural Net Repository for refined control. $\endgroup$ – Matvei Kruglyak Oct 31 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.