I've been working on scraping an article from CNBC. The link is here

I want to scrape the main body, and the code I have so far is this:

firstCaseContent = Cases[largeXml, XMLElement["p", {}, {___}], Infinity];
secondCaseContent = Cases[#, XMLElement["p", {}, {content_}] -> content, 1] &@firstCaseContent
contents = StringJoin@secondCaseContent

This returns the following:

"The company unveiled Messenger Platform 2.1 on Thursday, which will \ make it simpler for business to use its bots and conduct sales and \ customer service tasks. More than 1.2 billion people use Messenger \ each month, and users send more than 2 billion messages to businesses \ monthly, according to Facebook. A Nielsen and Facebook study showed \ 63 percent of survey takers increased messaging with companies over \ the last two years in 14 different markets. The new features make it \ easier for businesses to set up payments and checkout services \ through the chat function. It will also let them change the typical \ \"Send Message\" button on its Facebook page to a more specific task, \ like \"Shop Now,\" \"Get Support\" or \"Play Now.\""

This works somewhat well, but the code does not return "p" (paragraphs) that have an "a" or URL embedded in the text. Is there any way I can get my scraper to also scrape the text with URLs embedded?

  • $\begingroup$ Use content__ instead of content_. The single Blank is just matching raw strings. Easiest thing to do is to capture all "p" tags at that level and just export them using ImportString[ExportString[#,"XML"],"HTML"]. It's slow, but it's easy and robust. Of course if you need the URLs, as Hyperlink that won't work. $\endgroup$
    – b3m2a1
    Jul 31, 2017 at 8:39
  • $\begingroup$ Another way: html = Import[url, "HTML"]; StringCases[html, "<p>" ~~ Shortest[str__] ~~ "</p>"] $\endgroup$
    – C. E.
    Jul 31, 2017 at 10:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.