1
$\begingroup$

I created a few functions that will retrieve some text from certain medical articles. However, not all articles are useful or even match the keywords I provide. Is there a way to "grade" each article to see which ones are a good match?

I'm not sure how to go about this. I created the function below that is returning {False, True}, so I'm thinking that Intersection or Tally is not what I should use. Should I just see how many sentences have my keywords out of all the sentences and score on that?

In this example, there are four sentences with two of them holding one of each keyword. What approach would you use?

 checkQuality[datatext_, searchtext_] := (
  sentences = TextSentences[datatext];
  tally = {};
  
  Map[Do[
     AppendTo[tally, 
      ToString[MemberQ[StringSplit[sentences[[s]]], #]]], {s, 1, 
      Length[sentences]}] &, StringSplit[DeleteStopwords[searchtext]]];
  
  Apply[Intersection, Partition[tally, Length[sentences]]]
  )

searchTerm = "carotenoids and hepatomegaly";

checkQuality["GGT can also be an early marker of oxidative stress \
since serum antioxidant carotenoids namely lycopene, \
\[Alpha]-carotene, \[Beta]-carotene, and \[Beta]-cryptoxanthin are \
inversely associated with alcohol-induced increase of serum GGT found \
in moderate and heavy drinkers [36]. GGT levels may be 2\[Dash]3 \
times greater than the upper reference value in more than 50% of the \
patients with nonalcoholic fatty liver disease [37]. There is a \
significant positive correlation between serum GGT and triglyceride \
levels in diabetes and the level decreases with treatment especially \
when treated with insulin. Whereas serum GGT does not correlate with \
hepatomegaly in diabetes mellitus [38].", searchTerm]
$\endgroup$

1 Answer 1

3
$\begingroup$

First I would clean up your function signature. You currently have the problem of cleaning up the search string with DeleteStopwords. I'd move that to a separate utility, and for checkQuality would assume that we're passed a list of actual search terms. Next, I'd probably start by using something simple like StringCount:

checkQuality[text_String, searchterm_String] := checkQuality[text, {searchterm}];
checkQuality[text_String, searchterms : {__String}] := 
  StringCount[text, #, IgnoreCase -> True] & /@ searchterms

So, on your sample text (which I've called testText this gives:

checkQuality[testText, {"carotenoids", "hepatomegaly"}]
(* {1,1} *)

From there, you might consider text that matched every single term to be better than one that matched the same number of times overall but missed some of the terms. So, for example, let's modify the function:

checkQuality[text_String, searchterms : {__String}] :=
  With[
    {counts = StringCount[text, #, IgnoreCase -> True] & /@ searchterms},
    {Total[counts], CountsBy[counts, Positive][True]/Length[counts]}]

and now:

checkQuality[testText, {"carotenoids", "hepatomegaly", "RABBIT"}]
(* {2, 2/3} *)

Maybe you could weight each of these scores to get a single score.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.