I am using Mathematica to scrape information from webpages. To get file information, I am gathering plain text that is enclosed by <tr> and </tr> tags. I have made a list, of which each element is the entire data. I want to somehow scrape the plaintext readable element out of this.

By way of example, here are two elements from my list:

         <td class=\"td_data\" valign=\"top\">1.</td>
         <td class=\"td_data\" valign=\"top\">1841-1869 (Province of \
    Canada), number 195, 21 June 1845, page 15</a></td>
    <td class=\"td_data\" valign=\"top\"><a \
    f=g&PHPSESSID=kq2k6i3u6qodbjdp1ardk0ca96\">GIF</a> | <a \
    </tr>", "<tr>
         <td class=\"td_data\" valign=\"top\">2.</td>
         <td class=\"td_data\" valign=\"top\">1841-1869 (Province of \
    Canada), number 402, Extra, 16 May 1849, page 4</a></td>
    <td class=\"td_data\" valign=\"top\"><a \
    f=g&PHPSESSID=kq2k6i3u6qodbjdp1ardk0ca96\">GIF</a> | <a \

I've been playing around with Shorten[] and StringCases[] to try to find a way, but would there be a way to quickly map a function on this list that would result in the following:

"1841-1869 (Province of Canada), number 195, 21 June 1845, page 15",
"1841-1869 (Province of Canada), number 402, Extra, 16 May 1849, page 4."

The actual text changes, i.e. it doesn't always say 1841-1869 etc., but the general format of the <tr> portion remains consistent. I don't mind if the GIF | PDF part remains either. Would there be a quick way to just render the actually visible part of the HTML file in this list?

If somebody wants to reproduce the list I am using, here is my code (I include this only to make things simpler, but this is probably more of a theoretical question):

baseurl = 
pagelist = 
  Import[baseurl <> ToString[#], "Source"] & /@ Range[1, 1889, 199];
pagetext = Apply[StringJoin, pagelist];
trlist = StringCases[pagetext, Shortest["<tr>" ~~ ___ ~~ "</tr>"]];
  • $\begingroup$ Have you tried importing the html file as "Data"? It might be easier to extract your data from that. $\endgroup$ – Heike Feb 2 '12 at 0:26
  • 1
    $\begingroup$ HTML is hell of a complicated file format (Example: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"><><title//<p ltr<span></span</p></> is a valid document), can't you get your hands on some XML version of the page? (I didn't even consider what horrible code the average page has, catchword quirks mode) $\endgroup$ – David Feb 2 '12 at 1:10

For the two strings in your first example, this seems to work

ImportString[string, "HTML"]

For the baseurl as in the original post, Import[baseUrl, "Data"] gives something like

data = Import[baseUrl, "Data"]
data[[2, ;; 4]]

{{"Item", "View Options"}, {
  1., "1841-1869 (Province of Canada), number 195, 21 June 1845, page \
15", "GIF | PDF"}, {
  2., "1841-1869 (Province of Canada), number 402, Extra, 16 May \
1849, page 4", "GIF | PDF"}, {
  3., "1841-1869 (Province of Canada), number 405, 26 May 1849, page \
15", "GIF | PDF"}}

so it looks like data[[2, ;; ,2]] gives you the list you're after.

  • $\begingroup$ Not quite. It produces extra pieces. $\endgroup$ – Leonid Shifrin Feb 2 '12 at 0:33
  • $\begingroup$ @LeonidShifrin do you mean the indices and the "GIF | PDF" at the beginning and end? I guess importing as "Table" as in you solution gets rid of those, but it's probably easier to just import the file as "Data" in the first place in this case. $\endgroup$ – Heike Feb 2 '12 at 0:46
  • $\begingroup$ Yes, I meant that. When I was posting a comment, only the first part of your answer was available. Using "Data" is a good alternative, I agree. $\endgroup$ – Leonid Shifrin Feb 2 '12 at 0:48
  • $\begingroup$ Great, thank you. Let me play with this as soon as my internet connection works! $\endgroup$ – canadian_humanist Feb 2 '12 at 1:02

I don't know how robust this is, but this function seems to do what you want:

    ExportString[Delete[ImportString[#, "Table"], {{2}, {-2}}], "Table"], 
] &
  • $\begingroup$ Great, thank you. For some reason, my internet connection is having trouble with Mathematica so it may take me a bit of time to verify this! $\endgroup$ – canadian_humanist Feb 2 '12 at 1:02
  • $\begingroup$ @Ian No problem, was happy to help. $\endgroup$ – Leonid Shifrin Feb 2 '12 at 1:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.