Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Given this text:

text1 = "  A  Vitamin D Deficiency (ICD-9-CM 268.9) (ICD-9-CM 268.9) 09/11/2015  01 ";

Yes, the ICD code values are duplicates- possibly but not certainly in all the records.

What string expression will match only the code values - twice if duplicated? I've tried:

StringCases[text1, "(ICD-9-CM " ~~ code__ ~~ ")" .. :> code, 
 Overlaps -> False]

(* {"268.9) (ICD-9-CM 268.9"} *)

StringCases[text1, "(ICD-9-CM " ~~ code__ ~~ ")" .. :> code,   Overlaps -> True]

(* {"268.9) (ICD-9-CM 268.9", "268.9"} *)

StringCases[text1, "(ICD-9-CM " ~~ code__ ~~ ")" .. :> code, 
 Overlaps -> All]

(* {"268.9) (ICD-9-CM 268.9", "268.9", "268.9"} *)

Removing the .. repeat pattern doesn't alter the outputs. What's the explanation for this behavior?

share|improve this question
    
probably .. is stronger precedence than ~~ so you're asking for repeated ")" (if you FullForm you can see this). also look into Logest/Shortest –  amr May 21 '13 at 22:59
    
In addition to the answers below Union or DeleteDuplicates could be useful to check for non-identical matches. –  Yves Klett May 22 '13 at 10:39

5 Answers 5

up vote 6 down vote accepted

You could simply find the shortest match:

StringCases[text1, "(ICD-9-CM " ~~ Shortest[code__] ~~ ")" :> code]
{"268.9", "268.9"}

If it is possible that there is additional space or other characters a combination may be more robust:

text2 = "  A  Vitamin D Deficiency (ICD-9-CM   268.9) (ICD-9-CM: 268.9) 09/11/2015  01 ";

StringCases[text2, Shortest["(ICD-9-CM" ~~ __ ~~ code : NumberString ~~ ")"] :> code]
{"268.9", "268.9"}

Albert Retey suggested a method scanning for Except[")"] to restrict the pattern to a single field. I believe this is equivalent to my use of Shortest, but I was curious to find out how the two compared. We can view the Regular Expression that Mathematica internally converts each StringExpression to with StringPattern`PatternConvert. I will include two placements of Shortest.

StringPattern`PatternConvert /@ {
   "(ICD-9-CM " ~~ Shortest[code__] ~~ ")",
   Shortest["(ICD-9-CM " ~~ code__ ~~ ")"],
   "(ICD-9-CM " ~~ code : (Except[")"] ..) ~~ ")"
 } // Column
{"(?ms)\\(ICD-9-CM (.+?)\\)", {{Hold[code], 1}}, {}, Hold[None]}
{"(?ms)\\(ICD-9-CM (.+?)\\)", {{Hold[code], 1}}, {}, Hold[None]}
{"(?ms)\\(ICD-9-CM ((?:[^)])+)\\)", {{Hold[code], 1}}, {}, Hold[None]}

We see that both Shortest placements produce the same RE. We also see that the method using Except is more complex, and would appear to be performing additional testing that is ultimately unnecessary. I would expect it to be slower, and it is, but only slightly:

textBig = StringJoin @ ConstantArray[text1, 1*^6];

StringCases[textBig, Shortest["(ICD-9-CM " ~~ code__ ~~ ")"] :> code] // Timing // First

StringCases[textBig, "(ICD-9-CM " ~~ code : (Except[")"] ..) ~~ ")" :> code] // 
  Timing // First

0.718

0.812

Interestingly the hand-written RE is quite a bit slower:

StringCases[textBig, RegularExpression["\\(ICD-9-CM\\s*([^\\)]*)\\s*\\)"] :> "$1"] // 
  Timing // First

1.872

I started a new question about this here: Why is StringExpression faster than RegularExpression?

share|improve this answer

A couple of details:

  • Restricting code__ to NumberString will prevent it from being greedy (else it might stop only at the second ))
  • You need to wrap the entire pattern (which is what we want to repeat) in parentheses to respect the precedence of the .. operator.

The following pattern works:

StringCases[text1, ("ICD-9-CM " ~~ code : NumberString) .. :> code]
(* {"268.9", "268.9"} *)

Use ToExpression@code if you want them as numbers instead of a string.

share|improve this answer
    
Unfortunately ICD codes are not necessarily numbers, eg V58.11. –  alancalvitti May 22 '13 at 18:26
    
@alancalvitti Ok, but that should've been mentioned in the question. I don't know what "ICD codes" mean. –  rm -rf May 22 '13 at 18:29

With RegularExpression you could use:

text1 = "  A  Vitamin D Deficiency (ICD-9-CM 268.9) (ICD-9-CM 268.9) 09/11/2015  01 ";
StringCases[text1,RegularExpression@"\\(ICD-9-CM (\\d+(\\.)?(\\d+)?)\\)"-> "$1"]
share|improve this answer
    
Another way: StringCases[" A Vitamin D Deficiency (ICD-9-CM 268.9) (ICD-9-CM 268.9) 09/11/2015 01 ", RegularExpression["\\(ICD-9-CM ((\\d|\\.)+)\\)"] :> "$1"] –  J. M. May 22 '13 at 1:26

If you know that the code you are searching for is a number, then I think the suggestions to use patterns which recognize such number strings are your best bet. For completeness I wanted to show another way which I think is a useful standard approach for such tasks and will work even when you can't make such a premise. The idea is to simply only let code match everything except the next closing parenthesis:

StringCases[text1,"(ICD-9-CM " ~~ code : (Except[")"] ..) ~~ ")" :> code]

or, if you prefer with regular expressions (somewhat complicated with all those backslashes needed as escape character):

StringCases[text1, RegularExpression["\\(ICD-9-CM\\s*([^\\)]*)\\s*\\)"] :> "$1"]

EDIT: As Mr. Wizard has pointed out, this is somewhat slower and more complicated than the string expression shown above. That mainly is due to the additional complexity of allowing one or more whitespace characters (\\s*) before and after the matches. A slightly less complicated version of the regular expression which also is a more direct translation of the string expression is this:

StringCases[text1, RegularExpression["\\(ICD-9-CM ([^)]+)\\)"] :> "$1"]

which I would also expect to be slightly faster.

share|improve this answer
    
I don't think the Repeated (..) is necessary here. Also, I think your first method is conceptually equivalent to my use of Shortest but I don't know which is better. I need to explore this. –  Mr.Wizard May 22 '13 at 10:41
    
Your hand-written RE is not as fast as the one produced by Mathematica in my testing; do you understand why that is? (+1 btw) –  Mr.Wizard May 22 '13 at 10:54
    
@Mr.Wizard: yes, I also think that it is conceptually the same as your Shortest approach, which I think is more elegant, but I also don't have a well justified opinion about what is "better". I have only seen your suggestion after I finished my answer, but didn't want to delete it... –  Albert Retey May 22 '13 at 10:58
    
Please don't delete! This is an interesting comparison. At a lower level (PCRE) your "Except )" is probably how these patterns are checked anyway. I don't read RE comfortably so I'm not quite sure what you're doing in the hand-written RE, but I'd still like to know if you can figure out why it is slower than the rest. What does (?ms) at the beginning of the generated RE's do, for example? –  Mr.Wizard May 22 '13 at 11:02
    
@Mr.Wizard: ... and no, I don't understand why the regexp is slower, I guess it might be due to the \\s* which might be expensive and are not part of the Mathematica string expressions. I can ususally achieve regexps to do what I want but am not good enough to optimize them, so I well could be outperformed by auto-generated code. Honestly, I was quite surprised (and proud) that I got that monster working at first try :-) –  Albert Retey May 22 '13 at 11:18

Regular Expressions

This is typically solved with pattern matching. You specify a pattern by creating a regular expression (or short regex) using RegularExpression.

text1="  A  Vitamin D Deficiency (ICD-9-CM 268.9) (ICD-9-CM 268.9) 09/11/2015  01 ";

StringCases[text1,RegularExpression["\\(ICD-9-CM\\s*([0-9\\.]+)\\)"]:>"$1"]

yields: {268.9,268.9}

Explanation:

The regular expression language does have so called metacharacters:

\ | ( ) [ { ^ $ * + ? .

If you want to match these metacharacters you have to escape (prefix with \\) them:

(hello world)  <=> \\(hello world\\)

There are several character classes pre-defined in Mma's regex engine:

  • \\d = digit 0-9
  • \\w = word characters
  • \\s = space, new-line, tab, or other whitespace character

(you can find more of them in the Details section in the Documentation of RegularExpression.

Imagine you'd like to recognize ICD-9 codes with a regular expression.

An ICD-9 code looks like this:

  • two/three digit numeric code (may have leading zeroes to pad to three digits)
  • an optional dot
  • if that dot is present, there will be one or two following digits, depending on the preceding three digits. Which digits are allowed specifically is very variable.
  • Some codes are prefixed by an E or V.

Your example does not include the E or V prefix, so you're lucky with the solutions proposed so far. The best solution, imho, is to specify a grammatic of ICD-9 code using a regex:

Solution:

StringCases[text1,
    RegularExpression["ICD-9-CM\\s([EV]?\\d{2,3}(?:\\.\\d{1,2})?)"]:>"$1"]

This will match your ICD-9-CM codes and will verify on the fly if they are ill-formed (so to say, no match => someone produced a typo).

This regular expression is production safe and according the specification. So you're safe :)

share|improve this answer
    
Interestingly the RegularExpression patterns are again twice as slow as StringExpression patterns, despite the fact that they, according to the documentation, are converted to regular expressions internally. The problem is not poorly hand-written RE's because the generated RE's (see my post) used in RegularExpression are also slow. –  Mr.Wizard May 22 '13 at 15:00
    
@Mr.Wizard i just finished my post and then discovered yours. Yes, confusing isn't it? According to the documentation the patterns are compiled and cached. This seems not to be true if you explicitly define the regex on your own... –  Stefan May 22 '13 at 15:09
    
I would like to find a way to make the RegularExpression patterns as fast; it's troubling that they are not. Unless you would like to do so yourself I am going to ask why SE's are faster than RE's and if there is a way to improve the RE performance. Would you care to ask? –  Mr.Wizard May 22 '13 at 15:17
    
I'd like to let you take the lead, but please let me know or update your post, because this is really interesting in order to find out what's going on under the hood... –  Stefan May 22 '13 at 15:33
    
Posted: mathematica.stackexchange.com/q/25677/121 –  Mr.Wizard May 22 '13 at 15:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.