Problem with WordBoundary in string patterns

Question

I am trying to extract from a string all words that end with "-man", but the problem is that WordBoundary does not seem to work as it returns whole string ignoring actual word boundaries.

Here is the code:

mystring = "I am a big fan of Superman, Spiderman and Batman.";
StringCases[mystring, WordBoundary ~~ __ ~~ "man"];

Answer 1

The problem does not lie with WordBoundary, it is due to the use of __.

The string pattern WordBoundary ~~ __ ~~ "man" will find a word boundary properly, but then will match a following sequence of characters up to "man" without restriction -- including characters that lie on word boundaries themselves. To exclude that possibility, we should restrict the characters between WordBoundary and "man" to be word characters:

StringCases[mystring, WordBoundary ~~ WordCharacter.. ~~ "man"]

(* {"Superman", "Spiderman", "Batman"} *)

Some Fiddly Details...

Consider the following:

mystring2 = "I like Superman, Spiderman and Batman.  But adamantly not 3man!";

StringCases[mystring2, WordBoundary ~~ WordCharacter.. ~~ "man"]

(* { "Superman", "Spiderman", "Batman", "adaman", "3man"} *)

"3man" was picked up properly, but we probably don't want to see "adaman" in the list. This is easily corrected:

StringCases[mystring2, WordBoundary ~~ WordCharacter.. ~~"man" ~~ WordBoundary]

(* { "Superman", "Spiderman", "Batman", "3man"} *)

... and a Bug

Now, consider this:

mystring3 = "Remember Underscore_man!";

StringCases[mystring3, WordBoundary ~~ WordCharacter.. ~~"man" ~~ WordBoundary]

(* {} *)

Strange... why didn't the legendary Underscore_man appear in the result? We find the answer in the documentation for WordCharacter: it represents a letter or digit character. Fair enough. But, wait...

StringCases["Dash-man & Underscore_man", WordBoundary ~~ Shortest@__ ~~ WordBoundary]

(* { "Dash", "-", "man", " & ", "Underscore_man" } *)

I thought that only letters and digits were word characters? But apparently an underscore does not create a word boundary even though it is not a word character. How can this be?

Let's take a look at the regular expressions generated for WordCharacter and WordBoundary:

StringPattern`PatternConvert[WordCharacter ~~ WordBoundary] // First

(* (?ms)[[:alnum:]]\b *)

We can see that [[:alnum:]] is being used for WordCharacter, in conformance to the statement in the documentation. However, we see that \b is being used for WordBoundary. The PCRE documentation tells us that \b represents adjacent word and non-word characters, and that word characters are letters, digits or underscore.

This means that in Mathematica, there is a mismatch between the definition of WordCharacter and WordBoundary. A small bug, to be sure, but beware.

Answer 2

There is a reason to use WordBoundary, but your example sentence doesn't bring it out. Consider

mystring = "I am a demanding fan of Superman, Spiderman and Batman.";

StringCases[mystring, LetterCharacter ... ~~ "man"]

{"deman", "Superman", "Spiderman", "Batman"}

so you might prefer

StringCases[mystring, LetterCharacter ... ~~ "man" ~~ WordBoundary]

{"Superman", "Spiderman", "Batman"}

Answer 3

In your example StringCases works as expected because by default it searches the Longest substring which matches the pattern __ (by default all repeated string patterns are greedy). Apparently the whole string without ending "." matches your pattern:

In[21]:= mystring = "I am a big fan of Superman, Spiderman and Batman.";
substr = StringCases[mystring, WordBoundary ~~ __ ~~ "man"]
StringMatchQ[substr, WordBoundary ~~ __ ~~ "man"]    

Out[22]= {"I am a big fan of Superman, Spiderman and Batman"}

Out[23]= {True}

One way to restrict the pattern for matching only single words ending with "man" is to use LetterCharacter instead of "any character" _:

In[24]:= StringCases[mystring, WordBoundary ~~ LetterCharacter .. ~~ "man" ~~ WordBoundary]

Out[24]= {"Superman", "Spiderman", "Batman"}

Note that switching to lazy version (Shortest) of the repeated pattern does not give the desired result:

In[25]:= StringCases[mystring, Shortest[WordBoundary ~~ __ ~~ "man" ~~ WordBoundary]]

Out[25]= {"I am a big fan of Superman", ", Spiderman", " and Batman"}