Skip to main content
Added a simpler matching expression
Source Link
MarcoB
  • 67.7k
  • 18
  • 96
  • 198

I suspect that your test doesn't work because, according to the docs for PatternTest, "In a form such as __?test, every element in the sequence matched by __ must yield True when test is applied."

Instead, a conditional pattern using /; will work as I think you intended with your definition of the words wordlist:

StringCases[ToLowerCase@string, word__ /; MemberQ[words, word]]

(* Out: {"what", "is", "thebes", "tap", "pro", "a", "c", "h", "to", "a", "problem", "like", 
         "this", "in", "math", "em", "at", "ic", "a"} *)

Nevertheless, I'd suggest a bit of cleanup of the word list::

words = DeleteDuplicates@
          Select[
            DeleteCases[
              DeleteMissing@words,
              string_ /; StringContainsQ[string, "'" | "-"]
            ],
            StringLength[#] > 1 &
         ];

StringCases[ToLowerCase@string, word__ /; MemberQ[words, word]]

(* Out: {"what", "is", "thebes", "tap", "pro", "to", "problem", "like", \
"this", "in", "math", "em", "at", "ic"} *)

This approach... and an easier way to look for all possible matches:

StringCases[ToLowerCase@string, words, Overlaps -> True]

(* Out:
 {"what", "ha", "hat", "at", "ti", "tis", "is", "the", "thebe", "thebes", "he", "be", 
  "best", "es", "ta", "tap", "appro", "approach", "pro", "roach", "to", "pro", "problem", 
  "rob", "roble", "em", "ml", "li", "like", "et", "this", "hi", "his", "is", "si", "sin", 
  "in", "nm", "ma", "mat", "math", "at", "the", "them", "thematic", "he", "hem", "hematic", 
  "em", "ma", "mat", "at", "ti", "tic", "ic"}
*)

These approaches will still run into trouble, though. String matching is greedy by default, which is not always good: for instance, instead of "the best", the underlying sequence is interpreted greedily as "thebes" + "tap".

I really don't know that one could simply switch between greedy and lazy matching as appropriate without writing a full-fledged natural language recognition engine. If you came up with anything of the sort, quite a few people would be very interested...

I suspect that your test doesn't work because, according to the docs for PatternTest, "In a form such as __?test, every element in the sequence matched by __ must yield True when test is applied."

Instead, a conditional pattern using /; will work as I think you intended with your definition of the words wordlist:

StringCases[ToLowerCase@string, word__ /; MemberQ[words, word]]

(* Out: {"what", "is", "thebes", "tap", "pro", "a", "c", "h", "to", "a", "problem", "like", 
         "this", "in", "math", "em", "at", "ic", "a"} *)

Nevertheless, I'd suggest a bit of cleanup of the word list:

words = DeleteDuplicates@
          Select[
            DeleteCases[
              DeleteMissing@words,
              string_ /; StringContainsQ[string, "'" | "-"]
            ],
            StringLength[#] > 1 &
         ];

StringCases[ToLowerCase@string, word__ /; MemberQ[words, word]]

(* Out: {"what", "is", "thebes", "tap", "pro", "to", "problem", "like", \
"this", "in", "math", "em", "at", "ic"} *)

This approach will still run into trouble, though. String matching is greedy by default, which is not always good: for instance, instead of "the best", the underlying sequence is interpreted greedily as "thebes" + "tap".

I really don't know that one could simply switch between greedy and lazy matching as appropriate without writing a full-fledged natural language recognition engine. If you came up with anything of the sort, quite a few people would be very interested...

I suspect that your test doesn't work because, according to the docs for PatternTest, "In a form such as __?test, every element in the sequence matched by __ must yield True when test is applied."

Instead, a conditional pattern using /; will work as I think you intended with your definition of the words wordlist:

StringCases[ToLowerCase@string, word__ /; MemberQ[words, word]]

(* Out: {"what", "is", "thebes", "tap", "pro", "a", "c", "h", "to", "a", "problem", "like", 
         "this", "in", "math", "em", "at", "ic", "a"} *)

Nevertheless, I'd suggest a bit of cleanup of the word list::

words = DeleteDuplicates@
          Select[
            DeleteCases[
              DeleteMissing@words,
              string_ /; StringContainsQ[string, "'" | "-"]
            ],
            StringLength[#] > 1 &
         ];

StringCases[ToLowerCase@string, word__ /; MemberQ[words, word]]

(* Out: {"what", "is", "thebes", "tap", "pro", "to", "problem", "like", \
"this", "in", "math", "em", "at", "ic"} *)

... and an easier way to look for all possible matches:

StringCases[ToLowerCase@string, words, Overlaps -> True]

(* Out:
 {"what", "ha", "hat", "at", "ti", "tis", "is", "the", "thebe", "thebes", "he", "be", 
  "best", "es", "ta", "tap", "appro", "approach", "pro", "roach", "to", "pro", "problem", 
  "rob", "roble", "em", "ml", "li", "like", "et", "this", "hi", "his", "is", "si", "sin", 
  "in", "nm", "ma", "mat", "math", "at", "the", "them", "thematic", "he", "hem", "hematic", 
  "em", "ma", "mat", "at", "ti", "tic", "ic"}
*)

These approaches will still run into trouble, though. String matching is greedy by default, which is not always good: for instance, instead of "the best", the underlying sequence is interpreted greedily as "thebes" + "tap".

I really don't know that one could simply switch between greedy and lazy matching as appropriate without writing a full-fledged natural language recognition engine. If you came up with anything of the sort, quite a few people would be very interested...

Source Link
MarcoB
  • 67.7k
  • 18
  • 96
  • 198

I suspect that your test doesn't work because, according to the docs for PatternTest, "In a form such as __?test, every element in the sequence matched by __ must yield True when test is applied."

Instead, a conditional pattern using /; will work as I think you intended with your definition of the words wordlist:

StringCases[ToLowerCase@string, word__ /; MemberQ[words, word]]

(* Out: {"what", "is", "thebes", "tap", "pro", "a", "c", "h", "to", "a", "problem", "like", 
         "this", "in", "math", "em", "at", "ic", "a"} *)

Nevertheless, I'd suggest a bit of cleanup of the word list:

words = DeleteDuplicates@
          Select[
            DeleteCases[
              DeleteMissing@words,
              string_ /; StringContainsQ[string, "'" | "-"]
            ],
            StringLength[#] > 1 &
         ];

StringCases[ToLowerCase@string, word__ /; MemberQ[words, word]]

(* Out: {"what", "is", "thebes", "tap", "pro", "to", "problem", "like", \
"this", "in", "math", "em", "at", "ic"} *)

This approach will still run into trouble, though. String matching is greedy by default, which is not always good: for instance, instead of "the best", the underlying sequence is interpreted greedily as "thebes" + "tap".

I really don't know that one could simply switch between greedy and lazy matching as appropriate without writing a full-fledged natural language recognition engine. If you came up with anything of the sort, quite a few people would be very interested...