11
$\begingroup$

I would like to retrieve information associated with a given CAS registry number (Chemical Abstracts Service nr) from the NIST webbook web site in Mathematica, using the provided API (e.g. for geraniol, http://webbook.nist.gov/cgi/cbook.cgi?ID=106-24-1&Units=SI&Mask=2000#Gas-Chrom) (note I use the NCI CADD cactus server to convert InChiKeys to molecular structures if NIST doesn't list the molecular structure).

E.g. for a few example CAS chemical identifiers

    casnrs = {"106-24-1", "28115-92-6", "281-17-4", "28122-24-9", "28122-25-0", \
"28122-28-3", "281-23-2", "28123-26-4", "28123-68-4", "28123-70-8"};

I would like to download all associated information listed at the NIST webbook site. What is the best method to go about doing this in Mathematica?

Edit: with the help of some of the posts below I came to the following solution: (below is also an alternative method based on importing the website as XML)

Import molecular structures

getmolstrcactus[casnr_, threed_] := 
  Module[{str, notes, inchikey}, 
   If[threed == True, str = "?get3d=true"; notes = "Cactus/Corina MM",
     str = ""; notes = "Cactus"];
   Quiet[Check[{Import[
       "http://cactus.nci.nih.gov/chemical/structure/" <> casnr <> 
        "/sdf" <> str, "MOL"], "Cactus/Corina MM"}
     , inchikey = 
      StringReplace[
       ImportString[
         URLFetch["http://webbook.nist.gov/cgi/cbook.cgi", 
          "Parameters" -> {"ID" -> casnr, "UNITS" -> "SI", 
            "MASK" -> "2000"}], {"HTML", "Data"}][[2, 4]], 
       "IUPAC Standard InChIKey: " -> ""];
     Check[{Import[
        "http://cactus.nci.nih.gov/chemical/structure/" <> inchikey <>
          "/sdf" <> str, "MOL"], "Cactus/Corina MM"},
      {"NA", "NA"}]]]];
import2Dmolstr[casnr_] := 
  Module[{}, 
   raw = Import[
     "http://webbook.nist.gov/cgi/cbook.cgi?Str2File=C" <> casnr, 
     "String"];
   If[StringLength[raw] != 183,
    {Quiet[ImportString[raw, "MOL"]], "NIST"}, 
    getmolstrcactus[casnr, False]]];
AbsoluteTiming[import2Dmolstr /@ casnrs // TableForm]

2D molecular structure

import3Dmolstr[casnr_] := Module[{},
   Quiet[Check[{Import[
       "http://webbook.nist.gov/cgi/cbook.cgi?Str3File=C" <> casnr, 
       "MOL"], "NIST/MOPAC6/GAUSSIAN94 PM3"}, 
     getmolstrcactus[casnr, True]]]];
AbsoluteTiming[import3Dmolstr /@ casnrs // TableForm]

3D molecular structure

Import infrared and/or electron ionization mass spectra

importspectrum[casnr_, type_] := Quiet[Module[{}, If[type == "Mass",
     Check[
      ImportString[
        StringReplace[
         Import["http://webbook.nist.gov/cgi/cbook.cgi?JCAMP=C" <> 
           StringReplace[casnr, "-" -> ""] <> "&Index=0&Type=" <> 
           type, "String"], {x : NumberString ~~ " " ~~ 
            y : NumberString :> x <> "\n" <> y}], "JCAMPDX"][[1]], 
      "NA"], Check[
      Import["http://webbook.nist.gov/cgi/cbook.cgi?JCAMP=C" <> 
         StringReplace[casnr, "-" -> ""] <> "&Index=0&Type=" <> "IR", 
        "JCAMPDX"][[1]], "NA"]]]]; 
importmassspectrum[casnr_] := importspectrum[casnr, "Mass"];
importIRspectrum[casnr_] := importspectrum[casnr, "IR"];

plotIRspec[casnr_] := Module[{}, spec = importIRspectrum[casnr];
   Quiet[Check[
     ListPlot[spec, Joined -> True, PlotRange -> All, 
      PlotStyle -> PointSize[0]], "NA"]]];
plotmassspec[casnr_] := Module[{}, spec = importmassspectrum[casnr];
   Quiet[Check[
     ListPlot[spec, Filling -> Axis, FillingStyle -> Blue, 
      PlotRange -> All, PlotStyle -> PointSize[0]], "NA"]]];

plotIRspec["7647-01-0"]

IR spectrum

AbsoluteTiming[plotmassspec /@ casnrs // TableForm]

spectra

Import other info

removeHTML[str_] := 
  StringSplit[StringReplace[str, "<" ~~ Except[">"] .. ~~ ">" -> ""], 
   "\n"];
stringclean[str_] := 
  StringReplace[
   str, {"&raquo;" -> "", "&laquo;" -> "", ";" -> "", 
    "alpha" -> "\[Alpha]", "beta" -> "\[Beta]", "gamma" -> "\[Gamma]",
     "delta" -> "\[Delta]", "epsilon" -> "\[Epsilon]"}];
stringcheck[str_, part_] := 
  Quiet[Check[
    If[StringTake[str, StringLength[part]] == part, 
     StringReplace[str, part -> ""], "NA"], "NA"]];
CAScactusinfo[casnr_] := 
  Quiet[Module[{formula, mw, inchi, inchikey, name, synonyms, source},
    formula = 
     Check[Import[
       "http://cactus.nci.nih.gov/chemical/structure/" <> casnr <> 
        "/formula", "String"], "NA"];
    If[formula != "NA",
     mw = 
      Import["http://cactus.nci.nih.gov/chemical/structure/" <> 
        casnr <> "/mw", "String"];
     inchi = 
      StringSplit[
        Import["http://cactus.nci.nih.gov/chemical/structure/" <> 
          casnr <> "/stdinchi", "String"], "InChI="][[1]];
     inchikey = 
      StringSplit[
        Import["http://cactus.nci.nih.gov/chemical/structure/" <> 
          casnr <> "/stdinchikey", "String"], "InChIKey="][[1]];
     name = 
      Import["http://cactus.nci.nih.gov/chemical/structure/" <> 
        casnr <> "/iupac_name", "String"];
     synonyms = 
      StringJoin[
       Riffle[Import[
         "http://cactus.nci.nih.gov/chemical/structure/" <> casnr <> 
          "/names", "List"], "|"]];
     source = "NCI/CADD Cactus Chemical Identifier Resolver";,
     {formula, mw, inchi, inchikey, name, synonyms, source} = {"NA", 
        "NA", "NA", "NA", "NA", "NA", "NA"};];
    {formula, mw, inchi, inchikey, name, synonyms, source}]];
CASnistinfo[casnr_] := 
  Module[{raw, data, parts, formula, mw, inchi, inchikey, cas, name, 
    synonyms, stereoisomers, source, cactusinfo},
   raw = Check[
     ImportString[
      URLFetch["http://webbook.nist.gov/cgi/cbook.cgi", 
       "Parameters" -> {"ID" -> casnr, "UNITS" -> "SI", 
         "MASK" -> "2000"}], {"String"}], ""];
   data = ImportString[raw, {"HTML", "Data"}];
   If[And[data != {}, 
     data != {" Standard Reference Data Program", " Data Gateway", 
       "Chemistry WebBook"}],
    parts = {"Formula : ", "Molecular weight : ", "InChI=", 
      "IUPAC Standard InChIKey: ", "CAS Registry Number: ", 
      "Other names:", "Stereoisomers:"};
    formula = 
     StringReplace[stringcheck[data[[2, 1]], parts[[1]]], " " -> ""];
    mw = stringcheck[data[[2, 2]], parts[[2]]];
    cas = casnr;(* stringcheck[data[[2,5]],parts[[5]]]; *) 
    name = 
     stringclean[
      StringSplit[StringSplit[raw, "<title>"][[2]], "</title>"][[1]]];
    synonyms = 
     Quiet[
      Check[If[Length[StringPosition[raw, parts[[6]]]] != 0, 
        Quiet[stringclean[
          StringJoin[
           Riffle[removeHTML[
             StringSplit[
               StringSplit[raw, "strong>Other names:</strong>"][[2]], 
               "<strong>"][[1]]], "|"]]]], "NA"], "NA"]];
    stereoisomers = 
     Quiet[Check[
       If[Length[StringPosition[raw, parts[[7]]]] != 0, 
        Quiet[stringclean[
          StringJoin[
           Riffle[removeHTML[
             StringSplit[
               StringSplit[raw, "strong>Stereoisomers:</strong>"][[
                2]], "<strong>"][[1]]], "|"]]]], "NA"], "NA"]];
    inchi = Quiet[stringcheck[data[[2, 3, 1, 1]], parts[[3]]]];
    inchikey = stringcheck[data[[2, 4]], parts[[4]]];
    If[inchi == "NA", cactusinfo = CAScactusinfo[casnr];
     inchi = cactusinfo[[3]];
     inchikey = cactusinfo[[4]];
     source = 
      "NIST Webbook and NCI/CADD Cactus Chemical Identifier \
Resolver";, source = "NIST Webbook";];
    {casnr, formula, mw, inchi, inchikey, cas, name, synonyms, 
     stereoisomers, source},
    {formula, mw, inchi, inchikey, name, synonyms, source} = 
     CAScactusinfo[casnr];
    {casnr, formula, mw, inchi, inchikey, casnr, name, synonyms, "NA",
      source}]];
AbsoluteTiming[CASnistinfo /@ casnrs // TableForm]

chemical info

Import gas chromatography retention indices

(Note: here I partly parse the raw HTML, in the post below there is also a nice solution based on importing as XML)

CASnistRIs[casnr_] := 
  Quiet[Module[{raw, raw2, fun, RItypes, nRItypes, data, dat, pos, 
     RItypecol, RIdata, inchikey},
    Check[
     raw = URLFetch[
       "http://webbook.nist.gov/cgi/cbook.cgi?ID=" <> casnr <> 
        "&Units=SI&Mask=2000#Gas-Chrom"];
     raw2 = 
      Drop[StringSplit[
        StringSplit[
          StringSplit[raw, 
            "<h2><a id=\"Gas-Chrom\" name=\"Gas-Chrom\">Gas \
Chromatography</a></h2>"][[2]], 
          "<h2><a id=\"Refs\" name=\"Refs\">References"][[1]], 
        "<h3>"], 1];
     fun[s_] := StringSplit[StringSplit[s, "Type="][[2]], "#"][[1]];
     RItypes = Map[fun, raw2];
     nRItypes = Length[RItypes];
     data = ImportString[raw, {"HTML", "Data"}];
     pos = Position[data, "Column type"][[1, 1]];
     RItypecol = {};
     RIdata = ArrayFlatten[Table[dat = Drop[data[[pos + i - 1]], 1];
        If[Dimensions[dat][[2]] == 5, 
         dat = Transpose[
            Insert[Transpose[dat], 
             Table["NA", {Dimensions[dat][[1]]}], 3]];];
        RItypecol = 
         Append[RItypecol, 
          Table[RItypes[[i]], {Dimensions[dat][[1]]}]];
        dat, {i, 1, nRItypes}], 1];
     RItypecol = Flatten[RItypecol];
     RIdata = Transpose[Insert[Transpose[RIdata], RItypecol, 1]];
     RIdata = 
      Transpose[
       Insert[Transpose[RIdata], 
        Table[casnr, {Dimensions[RIdata][[1]]}], 1]];
     data = ImportString[raw, {"HTML", "Data"}];
     inchikey = stringcheck[data[[2, 4]], "IUPAC Standard InChIKey: "];
     If[inchikey == "NA", 
      inchikey = 
        Quiet[Check[
          StringSplit[
            Import["http://cactus.nci.nih.gov/chemical/structure/" <> 
              casnr <> "/stdinchikey", "String"], "InChIKey="][[1]], 
          "NA"]];];
     RIdata = 
      Transpose[
       Insert[Transpose[RIdata], 
        Table[inchikey, {Dimensions[RIdata][[1]]}], 2]];
     RIdata = 
      Insert[RIdata, {"CAS nr", "InChiKey", "RI type", "Column", 
        "Active phase", "Temperature (\[Degree]C)", "RI", "Reference",
         "Temperature program"}, 1], "NA"]
    ]];
AbsoluteTiming[CASnistRIs["106-24-1"]  // TableForm ]

RI info

$\endgroup$
3
  • 1
    $\begingroup$ Perhaps ImportString[raw, {"HTML", "Data"}] will be an easier starting point $\endgroup$
    – ssch
    Nov 15, 2013 at 15:26
  • $\begingroup$ You seem to use technical words, such as "Stereoisomers, Synonyms and Retention," which are not keywords in the raw data. Would you please clarify for the chemically illiterate? Thanks. :) $\endgroup$
    – Michael E2
    Nov 16, 2013 at 1:19
  • 1
    $\begingroup$ @MichaelE2 This sounds like a job for Chemistry.SE! In this view of an example entry of the NIST webook, you'll note that the OP wants the information following the bold bullets. Looks like this text gets stripped when importing certain elements or using ImportString. $\endgroup$ Nov 16, 2013 at 2:10

2 Answers 2

7
$\begingroup$

The question has changed so much since the last time I read it that what I had worked out hardly seems relevant anymore -- esp. to someone who doesn't understand chemistry jargon. Many website data servers can provide their data in JSON or XML format. These formats are easily parsed. The WebBook site seems not to, but I couldn't even find a description of the API that the OP mentions. However, the pages returned claim to be XHTML, and Mathematica can convert them to XML. The XML you get still needs some refining to separate the metal from slag. XML is nested like Mathematica expressions, so aside from the superabundance of XMLElement heads and other stuff that makes it hard for a human to read, it's relatively easy to deal with programmatically.

Getting the XMLObject is easy:

xmlGet[id_String] := 
 Import["http://webbook.nist.gov/cgi/cbook.cgi?ID=" <> id <> "&Units=SI&Mask=2000",
        {"HTML", "XMLObject"}]

Originally the problem seemed to be how to convert a web page like Caryophylladienol II to Mathematica data. In an XMLObject, the basic unit is an XMLElement, which has the form

XMLElement[tag, {attributes,...}, {data,...}]

The attributes are Rules and the data are usually strings or XMLElements.

To process the XMLObject, I usually use Cases (sometimes Position) to locate the desired elements and define a custom function to recursively process the data arguments as I wish. The function is defined by declarations like

fmt[XMLElement[tag, attr_, data_]] := ...

where tag might be a literal string or a pattern, and fmt is often called again on the elements of data.

You can collect the bold-faced data sections on the webpage with

Cases[xml, entry : {XMLElement["strong", _, _], __}, Infinity]

and you might take advantage of using RuleDelayed with the second argument to process the data. I would use it to create a list of rules of the form

sectionname -> data

You'll end up with extra entries because the Cases will match more sections of the webpage than is sought, but that is not a problem.

You can also add rules for the gas chromatology and references sections. They are between elements of the form

XMLElement["h2", {}, data_]

The argument data contains the title of the section, often embedded in another .

If you write a function cbookData along the lines of

cbookData[cas_String] := 
  Check[cbookData[cas] = webToRules[cas], cbookData[cas] =.; {}];
cbookData[cas_String, property_String] /; cbookData[cas] =!= {} := 
  property /. cbookData[cas];

where webToRules get and parses the XMLObject, the data will be imported from the web just once and stored in cbookData[cas] so that subsequent calls won't have to wait on the internet.

casno = "19431-79-9";
First /@ cbookData[casno]

(* {"Formula", "Molecular weight", "IUPAC Standard InChI",
    "IUPAC Standard InChIKey", "CAS Registry Number", "Chemical structure",
    "Species with the same structure", "Stereoisomers", "Other names",
    "Information on this page", "Options", "Data compiled by",
    "Gas Chromatography", "References", "XMLObject"}  *)

We caught a few extras ("Information on this page", etc.).

Examples

Row@cbookData[casno, "Formula"]

Mathematica graphics

ToExpression @ cbookData[casno, "Molecular weight"]
(* {220.351} *)

First@cbookData[casno, "Chemical structure"]

Mathematica graphics

The chromatography tables are stored as

"Gas Chromatography" -> {{attributes1, table1}, ...}

where table1 is the array of table entries and attributes1 contain various things such as the title. Here is the first one with a little formatting function I wrote called myGrid:

myGrid @ First @ cbookData[casno, "Gas Chromatography"]

Mathematica graphics

Here are the entries of the three tables under the heading I (the third column):

cbookData[casno, "Gas Chromatography"][[All, 2, 2 ;;, 3]]

(* {{"1624.", "1627.", "1623.", "1639.", "1622."},
    {"1637.", "1631.", "1627.", "1656.", "1615.", "1638.", "1628.", "1602.", "1611.", 
     "1635.", "1622.", "1622.", "1627."},
    {"1641.", "1641.", "1604.", "1641.", "1628.", "1623."}} *)

(Of course, these can be converted to numbers with, say, ToExpression.)

In summary, there's quite a bit of checking to do by hand to figure out patterns that will capture the data. But converting to an XMLObject does some of the parsing, like converting an HTML table to an array. If the data were available as an XML (or JSON) database, then it should be even easier. The Chemistry WebBook seems particularly troublesome -- at least I feel like I'm guessing at patterns for identifying data, and I can't be certain a guess will be accurate for the next molecule.

Code dump

How to format strings is something I played with a little, but it's not in a perfect state. A sentence often gets broken up at funny places, and when to strip or insert whitespace is something I stopped trying to figure out. (It probably needs to be completely rethought.) There are formatted tags from the HTML that don't have to be ignored. For instance, one might try either or both of the following:

fmt[XMLElement["em", _, {s_String}]] := Style[s, Italic];
fmt[XMLElement["em", _, ss_]] := Row[Style[#, Italic] & /@ ss];

One could process numeric strings thus:

fmt[s_String?(StringMatchQ[#, Whitespace ... ~~ NumberString ~~ Whitespace ...]&)] :=
  ToExpression[s];

But that might turn numeric labels into Mathematica numbers, which might or might not cause bugs.

It is worth noting that links may be formed by appending the "src" attribute to the base url:

fmt[XMLElement[tag, attr_, data_]] := ...cbook`urlBase <> ("src" /. attr)...

I changed the xmlGet function. The old one strangely wasn't working today, but it worked yesterday (in two different versions of Mathematica). It briefly did that yesterday, but worked after restarting Mathematica. Not today, though. Sigh.

  (*
   * formatting/parsing functions
   *)
Clear[fmt, refFmt];
fmt[XMLElement["tr", _, row_]] := fmt /@ row;
fmt[XMLElement["ul", _, list_]] := fmt /@ list;
fmt[XMLElement["sub", _, item_]] := Subscript["", fmt[item]];
fmt[XMLElement["img", attr_, _]] := (* load images *)
  Import[cbook`urlBase <> ("src" /. attr)];
fmt[XMLElement[_, _, item_]] := fmt[item];
  (* processing strings is problematic *)
fmt[{s_String}] := StringTrim[s, Whitespace | ":"];
fmt[_String?(StringMatchQ[#, Whitespace] &)] := Sequence[];
fmt[s_String] := StringTrim[s, Whitespace | ":"];
  (* default lists-- elements are processed. then the whole list as a
     2nd run indicated by the second argument (fmt[..., 2]) *)
fmt[{}] = fmt[{}, 2] = Sequence[];
fmt[{ss__String}, 2] := 
  StringJoin @@ ({ss} //. {{a___, 
        b_?(StringMatchQ[#, RegularExpression[".*[,.;:]"]] &), 
        c_?(StringMatchQ[#, WordCharacter ~~ ___] &), d___} :> {a, b, 
        " ", c, d},
      {a___, b_?(StringMatchQ[#, ___ ~~ WordCharacter] &), 
        c_?(StringMatchQ[#, WordCharacter ~~ ___] &), d___} :> {a, b, 
        " ", c, d}});
fmt[{ss__}, 2] := Row[{ss}];
fmt[{ss__}] := fmt[fmt /@ {ss}, 2];
  (* special formatting rules for post-processing *)
specialcases = {
   HoldPattern["Other names" -> {s_}] :>
     ("Other names" -> StringSplit[s, RegularExpression["[;\n]+"]]),
   {init___, s_, Subscript["", sub_], end___} :> {init, Subscript[s, sub], end},
   HoldPattern["Go To" -> _] :> Sequence[]};

  (* for formatting gas chromatography tables *)
myGrid[{attr_, tab_}] := 
  Style[Labeled[
    Grid[tab, Frame -> All, 
     ItemStyle -> {Automatic, {Bold, {Automatic}}}], 
    "summary" /. attr, Top, LabelStyle -> Bold], "Label"];

  (* for formatting references (redefine as desired) *)
refFmt[{ss__}] := refFmt /@ {ss};
refFmt[args__] := fmt[args];

  (*
   * input functions
   *)
Clear[xmlGet, cbook`casToRules, cbook`h2pos, cbook`h2entries];

cbook`urlBase = "http://webbook.nist.gov";

xmlGet[id_String] := 
  With[{url = 
     "http://webbook.nist.gov/cgi/cbook.cgi?ID=C" <> id <> 
      "&Units=SI&Mask=2000"}, 
   Print[url]; (* for debugging *)
   Quiet@ImportString[
     URLFetch[
      "http://webbook.nist.gov/cgi/cbook.cgi?ID=19431-79-9&Units=SI&\
Mask=2000"], "XML"]
   ];

  (* finds the "h2" headers *)
cbook`h2entries[xml : XMLObject[_][__], head_String, h2pos_] := 
  With[{pos = 
     Last@First@
       Position[xml, XMLElement["h2", {}, _?(! FreeQ[#, head] &)]]},
   1 + pos ;; h2pos[[2, 1 + Position[Last@h2pos, pos][[1, 1]]]] - 1];

  (* loads the NIST WebBook url as XML and converts to rules mapping keywords to data *)
cbook`casToRules[molCAS_String] := Module[{xml},
  Check[xml = xmlGet[molCAS]; If[xml === $Failed, Return[$Failed]], 
   Return[{}]];

    (* intialize h2pos
      {part for the html body (List), {indices of h2 headers}} *)
  cbook`h2pos[molCAS] = {Most@First@#, Last /@ #} &@
    Position[xml, XMLElement["h2", __]];

  Join[
    (* bold-faced sections *)
   Cases[xml, 
     entry : {XMLElement["strong", _, _], __} :>
       (fmt @ entry[[1, -1, 1]] -> 
         fmt /@ DeleteCases[Rest[entry], _String?(StringMatchQ[#, Whitespace] &)]), 
     Infinity] /. specialcases, (* splits "Other names", deletes some extra sections *)
    (* additional sections (between h2 headers) *)
   {"Gas Chromatography" -> 
      Cases[Extract[xml, 
         First@cbook`h2pos[molCAS]][[cbook`h2entries[xml, 
          "Gas Chromatography", cbook`h2pos[molCAS]]]], 
       XMLElement["table", attr : {___, "class" -> "data", ___}, data_] :>
          {attr, fmt /@ data /. specialcases}, Infinity],
    "References" -> Cases[
       Extract[xml, 
         First@cbook`h2pos[molCAS]][[cbook`h2entries[xml, "References",
           cbook`h2pos[molCAS]]]],
       XMLElement["p", {}, ref_] :> refFmt[ref]
       ],
    "XML" -> xml (* the xml document for further processing *)}
   ]
  ]

  (* Main user interface *)
Clear[cbookData];
cbookData::failed = "Failed to load CAS ``";
cbookData[cas_String] := 
  Check[If[(cbookData[cas] = cbook`casToRules[cas]) === $Failed, 
    Message[cbookData::failed, cas]]; cbookData[cas], 
   cbookData[cas] =.; {}];
cbookData[cas_String, property_String] /; cbookData[cas] =!= {} := 
  property /. cbookData[cas];
$\endgroup$
8
  • $\begingroup$ Hi @Michael E2 - yes sorry my question still changed a bit, as I was still trying to solve the problem myself - but this is really neat and useful - I'll try to see if I can rewrite what I have now along these lines, it does seem to be a lot cleaner! Thanks so much!!! $\endgroup$ Nov 17, 2013 at 12:57
  • $\begingroup$ @TomWenseleers I can post them. I got them to work in a sort of hit-and-miss fashion, and they're not particularly complete. But they probably illustrate the techniques and approach adequately. I'll have time to do it a little later on. $\endgroup$
    – Michael E2
    Nov 17, 2013 at 13:59
  • $\begingroup$ OK that's perfect - many thanks! Especially the part where you extract the retention index tables I would be very interested in, since my routine to do this doesn't seem to be robust (the rest seems kind of OK though based on some more extensive tests I did, though your solution clearly seems more elegant)... $\endgroup$ Nov 17, 2013 at 20:56
  • $\begingroup$ @TomWenseleers I don't know how elegant it will seem under the hood. :) The structure of the web pages are not so elegant, so a certain amount of drudgery is inevitable, I'm afraid. It seemed like bobthechemist was able to extract the RI numbers. For my own part, using XML is easier than parsing text. $\endgroup$
    – Michael E2
    Nov 17, 2013 at 22:19
  • $\begingroup$ Many thanks - that was really helpful and the RI import is working now - I'll post it back in my question when I tidied up my code... $\endgroup$ Nov 18, 2013 at 0:36
7
$\begingroup$

I've tried to do some scrubbing of web pages with chemical information and even when I think that a website has some consistent structure I find that there are always special cases or the web site provider chooses that particular time to revamp the website!

Nonetheless, here's a way to get at some of your information.

casno = "19431-79-9"
nistsearch[casstring_] := 
 Module[{data, formula, mw, inchi, inchikey, name},
  data = ImportString[
    URLFetch["http://webbook.nist.gov/cgi/cbook.cgi",
     "Parameters" ->
      {"ID" -> casstring,
       "UNITS" -> "SI",
       "MASK" -> "2000"}],
    {"HTML", "Data"}];
  formula = 
   First@Quiet@
     Cases[data[[2]], 
      x_ /; StringTake[x, 7] == "Formula" -> StringDrop[x, 10]];
  mw = First@
    Quiet@Cases[data[[2]], 
      x_ /; StringTake[x, 16] == "Molecular weight" -> 
       StringDrop[x, 19]];
  inchi = 
   First@Quiet@
     Cases[Flatten[data[[2]]], 
      x_ /; StringTake[x, 5] == "InChI" -> StringDrop[x, 6]];
  inchikey = 
   First@Quiet@
     Cases[data[[2]], 
      x_ /; StringTake[x, 23] == "IUPAC Standard InChIKey" -> 
       StringDrop[x, 25]];
  name = data[[2, 7]];
  {casstring, formula, mw, name, inchi, inchikey}
  ]

It works nicely for one of the two compounds you are searching for, but even with the two examples provided, the second one gives us trouble.

nistsearch /@ {"19431-79-9", "630035"} // TableForm

The output of the table is long because of the InChI keys so I won't reproduce it here. The name is the only string I search for with a list index to demonstrate the problem. It works fine for the first entry, but fails for the second. Because the data on the web page is not formatted consistently for each molecule, you'll have a very difficult time coming up with an automated method for scrubbing. A number of elements do have useful identifiers at the beginning of the strings, and the function demonstrates how you can get at this data.

Getting retention data

It looks like the webpage structure is reasonably consistent (I tried 4 molecules) in the region where the retention time data is reported. Therefore, we can use carefully designed searches to get the data you are interested in (although you lose the column information with this approach).

sstring = "19431-79-9";

retsearch[s_] := Module[{data, top, bottom, d1},
  (* Fetch the raw HTML data *)
  data = URLFetch[
    "http://webbook.nist.gov/cgi/cbook.cgi?ID=" <> s <> 
     "&Units=SI&Mask=2000"];

  (* Note that the GC data tables occur between the following text *)
  top = "<h2><a id=\"Gas-Chrom\" name=\"Gas-Chrom\">Gas \Chromatography</a></h2>";
  bottom = "<h2><a id=\"Refs\" name=\"Refs\">References</a></h2>";

  (* Grab the portion between these two strings *)
  d1 = StringCases[data, top ~~ x__ ~~ bottom -> x]; 

  (* StringPosition finds the starting and ending positions of the tables *)
  (* StringCases finds number strings between aligator brackets; retention entires*)
  (*    that match this pattern. *)

  MapThread[
   StringCases[
     First@StringTake[
       d1, {#1, #2}], ">" ~~ x : NumberString ~~ "<" -> x] &,
   {StringPosition[d1, "<table"][[1, All, 1]], 
    StringPosition[d1, "</table>"][[1, All, 2]]}]
  ]

This appears to work for several of the Caryophylladienol II isomers:

sstrings = { "19431-79-9", "R238914", "R288350", "R325538"};
retsearch /@ sstrings;
TableForm[%, TableHeadings -> {sstrings}]

Mathematica graphics

$\endgroup$
4
  • $\begingroup$ many thanks for these very useful pointers - I have now gotten almost there, but I am still left with some small issues in fetching the retention index information - would you know by any chance how to solve those? $\endgroup$ Nov 16, 2013 at 22:09
  • $\begingroup$ @TomWenseleers What more do you want in addition to my retsearch function? $\endgroup$ Nov 16, 2013 at 22:39
  • $\begingroup$ Well have a look in my edited post at the bottom, it shows how to get all the info from the full RI index tables, including column type used etc, but there is still a few small cosmetic issues left at the end that I am struggling with $\endgroup$ Nov 16, 2013 at 22:44
  • $\begingroup$ @TomWenseleers go ahead and delete the comment with your email address, i've got it and will be in touch. $\endgroup$ Nov 16, 2013 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.