11
$\begingroup$

According to the documentation of WordCharacter:

WordCharacter matches any character for which either LetterQ or DigitQ yields True.

However, the results of the following code seem not compatible with the doc:

StringMatchQ[#, WordCharacter] & /@ {"a", "1", ".", " ", "中", "あ"}
LetterQ[#] || DigitQ[#] & /@ {"a", "1", ".", " ", "中", "あ"}

{True, True, False, False, True, True}

{True, True, False, False, False, False}

I suppose there is a special rule to handle CJK characters, so it's the doc which should be corrected, am I right?

(As the motivation of this "discovery", I was trying to match any characters not belong to CJK.)

$\endgroup$
  • 1
    $\begingroup$ Your example does not violate what the documentation states, it's just that the reverse isn't true (it would seem): WordCharacter does not not match any character which neither LetterQ nor DigitQ matches. $\endgroup$ – C. E. Jul 25 '15 at 3:14
  • $\begingroup$ @Pickett Your logic is right of course. But I can't agree with you, as it would make the doc sounding like playing with words... $\endgroup$ – Silvia Jul 25 '15 at 3:34
  • 5
    $\begingroup$ The documentation is certainly wrong. LetterQ is only usable for the Latin and Greek alphabets, and even then only the letters that exist in English and modern Greek. StringMatchQ relies on the PCRE library, so it considers WordCharacter whatever PCRE does. This extends at least to some other alphabets/character types. $\endgroup$ – Oleksandr R. Jul 26 '15 at 21:48
  • 1
    $\begingroup$ The result is difference with you here $\endgroup$ – yode Mar 21 '17 at 20:14
  • $\begingroup$ Thanks! @yode I just tried and my 11.0 agrees with your result. I think Alexey Popkov's answer addressed the change. $\endgroup$ – Silvia Mar 22 '17 at 18:43
4
$\begingroup$

Under the "Implementation Details" section of the official tutorial "Working with String Patterns" we read:

Because PCRE currently does not support preset character classes with characters beyond character code 255, the word and letter character classes (such as WordCharacter and LetterCharacter) only include character codes in the Unicode range 0–255. Thus LetterCharacter and _?LetterQ do not give equivalent results beyond character code 255.

The above-cited Documentation section wasn't updated since Mathematica version 5.2 (July-2005), and shouldn't be correct anymore because starting from PCRE version 8.32 (30-November-2012) the latter supports Unicode:

PCRE_UCP

This option changes the way PCRE processes \B, \b, \D, \d, \S, \s, \W, \w, and some of the POSIX character classes. By default, only ASCII characters are recognized, but if PCRE_UCP is set, Unicode properties are used instead to classify characters. More details are given in the section on generic character types in the pcrepattern page. If you set PCRE_UCP, matching one of the items it affects takes much longer. The option is available only if PCRE has been compiled with Unicode property support.

<...>

In UTF-8 (UTF-16, UTF-32) mode, characters with values greater than 255 (0xffff) can be included in a class as a literal string of data units, or by using the \x{ escaping mechanism.

Starting from Mathematica version 10 the two lines of code in the question produce identical results but in general these tests are different, and (as of Mathematica version 11.1.0) the results aren't the same for 705 characters from the Unicode table as one can see from the following:

unicode = FromCharacterCode /@ Range[0, 65276];
test1 = StringMatchQ[#, WordCharacter] & /@ unicode;
test2 = LetterQ[#] || DigitQ[#] & /@ unicode;
Pick[unicode, MapThread[Unequal, {test1, test2}]]

screenshot

$\endgroup$
  • $\begingroup$ Nice finding! Thanks! So maybe I should stick with a self-maintained Unicode block table. $\endgroup$ – Silvia Mar 22 '17 at 19:51
6
$\begingroup$

This is a pretty ugly hack, but maybe it will inspire you to something better.

leastCJK = ToCharacterCode["⺀"][[1]];
StringMatchQ[#, _?(ToCharacterCode[#][[1]] < leastCJK &)] & /@ 
  {"a", "1", ".", " ", "中", "あ"}
{True, True, True, True, False, False}

Note: "⺀" is unicode character U+2E80, CJK RADICAL REPEAT

This hack can be used with Repeat to ignore CJK strings

StringCases[
  "the quick fox...本当に日本語を勉強していますか...jumped", 
  Repeated[_?(ToCharacterCode[#][[1]] < leastCJK &)], 
  Overlaps -> False]
{"the quick fox...", "...jumped"}
$\endgroup$
  • $\begingroup$ Thanks @m_goldberg ! This is how I'm dealing the problem right now. I hope there be a CJKCharacter :) $\endgroup$ – Silvia Jul 26 '15 at 16:24
  • $\begingroup$ Hi @m_goldberg , I think Alexey's answer gives very insightful information. If it's fine with you, I would like to accept it. $\endgroup$ – Silvia Mar 22 '17 at 18:48
  • $\begingroup$ @Silvia. Please feel free to do so. $\endgroup$ – m_goldberg Mar 22 '17 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.