How to scrape headlines from New York Times, Wall Street Journal main pages to create datasets similar to this service?

Importing HTML from nyt.com (HTML4) results in a String and the markup is not preserved. Is there a workaround? For wsj.com (XHTML) is either not valid (is there a W3C XHTML validator?) or a problem for XML`Parser. Any clues?

In[362]:= StringQ@Import["http://nyt.com","HTML"]
Out[362]= True

During evaluation of In[361]:= XML`Parser`XMLGet::nfprserr: Attribute 'property' is not declared for element 'meta' at Line: 11 Character: 71 in /tmp/m00009067531/wsj.
During evaluation of In[361]:= XML`Parser`XMLGet::prserr: Expected an attribute name at Line: 50 Character: 45 in /tmp/m00009067531/wsj.
During evaluation of In[361]:= Import::fmterr: Cannot import data as XML format. >>
Out[361]= $Failed
  • 2
    $\begingroup$ I don't think scraping the nyt site will be much fun. It's mostly javascript. $\endgroup$ Apr 20 '12 at 19:19
  • 2
    $\begingroup$ If you are after the headlines, feeds.nytimes.com/nyt/rss/HomePage is probably a better link to use. $\endgroup$
    – celtschk
    May 3 '12 at 15:50

You can always do Import["http://wsj.com","XMLObject"]. That has the side effect of producing some irregular XML whenever the underlying HTML doesn't quite map cleanly to XML, but it mostly produces an XMLObject[] expression tree that you can match over and extract data from, and I've never seen a web page for which it won't return something.


I agree wholeheartedly with the comment of celtschk to the OP. Both journals have RSS feeds (with pointers at the bottom of their main pages) that are designed exactly for the purpose that you describe. I doubt that either journal wants you to "scrape" their content; scraping is specifically forbidden by the WSJ Terms of Use.

I don't know how much easier the RSS feed could be:

NotebookPut[Import["http://blogs.wsj.com/digits/feed/", "RSS"]];

enter image description here

You can also import explicitly as XML, if you prefer to massage the result into some other form. Since RSS is ultimately expressed as genuine XML, you won't risk the dangers inherit in importing HTML as XML.

  • $\begingroup$ This answer is so obvious now that I think about it. Great job. $\endgroup$
    – Searke
    May 3 '12 at 17:07
  • $\begingroup$ @MarkMcClure Fantastic! I didn't know this Mathematica functionality! $\endgroup$
    – Rod
    Jun 8 '13 at 19:57
  • $\begingroup$ @RodLm Yes, that is pretty awesome, isn't it. Mathematica's import/export is generally outstanding in two ways - the broad range of filetypes it works with and the way it consistently represents things in it's own language. You might also be interested in this question, which deals with importing HTML as XML. $\endgroup$ Jun 8 '13 at 21:13
  • $\begingroup$ @MarkMcClure Yes, I've seen this question before... I actually developed a similar code to count "unanswered" questions at Yahoo! Answers according to given keywords... It's also amazing! $\endgroup$
    – Rod
    Jun 8 '13 at 21:19

Just import the source of the page instead of its rendered content:

Import["http://nyt.com", "Source"]
  • $\begingroup$ Import using "Source" works but "XMLObject yields more structured results. It comes down to being able to match XML vs Text expressions. Either way Most of the work is in modeling the schema. $\endgroup$ Apr 21 '12 at 2:20
  • 1
    $\begingroup$ Yes… also, as we all know, you can’t parse HTML/XML with regexps $\endgroup$
    – F'x
    Apr 21 '12 at 6:07
  • $\begingroup$ maybe you can't parse general HTML (why not XML?) but that doesn't imply you can't parse NYT or WSJ, modulo legal restrictions as pointed out by McClure. $\endgroup$ May 4 '12 at 3:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.