# Why is StringExpression faster than RegularExpression?

Edit: as noted by Albert Retey the performance difference is only seen when sub expression extraction is performed. If this test is used below the timings are similar:

First@Timing[r1 = StringCases[textBig, se];]
First@Timing[r2 = StringCases[textBig, re];]

According to the documentation:

Any symbolic string pattern is first translated to a regular expression. You can see this translation by using the internal StringPatternPatternConvert function.

StringPatternPatternConvert["a" | "" ~~ DigitCharacter ..] // InputForm
{"(?ms)a?\\d+", {}, {}, Hold[None]}

The first element returned is the regular expression, while the rest of the elements have to do with conditions, replacement rules, and named patterns.

The regular expression is then compiled by PCRE, and the compiled version is cached for future use when the same pattern appears again. The translation from symbolic string pattern to regular expression only happens once.

Based on this I would expect a StringExpression and the regular expression produced by PatternConvert to perform similarly, but they do not. Taking an example from this recent question please observe:

se = Shortest["(ICD-9-CM " ~~ code__ ~~ ")"];
re = First @ StringPattern`PatternConvert[se] // RegularExpression
RegularExpression["(?ms)$ICD-9-CM (.+?)$"]
text1 = "  A  Vitamin D Deficiency (ICD-9-CM 268.9) (ICD-9-CM 268.9) 09/11/2015  01 ";
textBig = StringJoin @ ConstantArray[text1, 1*^6];

First@Timing[r1 = StringCases[textBig, se :> code];]
First@Timing[r2 = StringCases[textBig, re :> "\$1"];]

r1 === r2
0.718
1.903
True
• Why is using the StringExpression more than twice as fast as the RegularExpression?
• Is there a way to make the RegularExpression matching run just as quickly?
• I think you have v7. Just mentioning that it's the same in 9 too. – Szabolcs May 22 '13 at 15:59
• @Szabolcs ?: means clustering but not capturing, so you can group regexes within (?:) but doesn't make backreferences as () does. – Stefan May 22 '13 at 16:26
• it seems like that StringExpression does something magical with RegularExpression. Presumably it does compile/cache the pattern – Stefan May 22 '13 at 17:06
• I have no time to investigate that more closely, but I think the difference could well come from how the resubstituting of the matches is implemented. If you try the same thing without such substitutions, then the runtime differences are marginal, e.g.: StringCases[textBig, RegularExpression["(?ms)$ICD-9-CM .+?$"]] vs. StringCases[textBig, Shortest["(ICD-9-CM " ~~ __ ~~ ")"]] seem to be equally fast (or probably slow when compared to pcregrep :-) – Albert Retey May 22 '13 at 19:56
• @Mr.Wizard after my first approach, which showed nearly similar performance behaviour and after Albert's further observation, I came up with the following: First@Timing[r3=StringCases[textBig,"(ICD-9-CM "~~x:RegularExpression[".+?"]~~")"->x];] Which shows similar performance behaviour. So we may conclude that you may choose to use RegularExpression in Mma, but you should avoid capturing/substitution if you want to be fast, plus capturing inside a StringExpression context does not work at all. – Stefan May 23 '13 at 7:53