I've been working on scraping an article from CNBC. The link is here
I want to scrape the main body, and the code I have so far is this:
firstCaseContent = Cases[largeXml, XMLElement["p", {}, {___}], Infinity];
secondCaseContent = Cases[#, XMLElement["p", {}, {content_}] -> content, 1] &@firstCaseContent
contents = StringJoin@secondCaseContent
This returns the following:
"The company unveiled Messenger Platform 2.1 on Thursday, which will \ make it simpler for business to use its bots and conduct sales and \ customer service tasks. More than 1.2 billion people use Messenger \ each month, and users send more than 2 billion messages to businesses \ monthly, according to Facebook. A Nielsen and Facebook study showed \ 63 percent of survey takers increased messaging with companies over \ the last two years in 14 different markets. The new features make it \ easier for businesses to set up payments and checkout services \ through the chat function. It will also let them change the typical \ \"Send Message\" button on its Facebook page to a more specific task, \ like \"Shop Now,\" \"Get Support\" or \"Play Now.\""
This works somewhat well, but the code does not return "p" (paragraphs) that have an "a" or URL embedded in the text. Is there any way I can get my scraper to also scrape the text with URLs embedded?
content__instead ofcontent_. The singleBlankis just matching raw strings. Easiest thing to do is to capture all"p"tags at that level and just export them usingImportString[ExportString[#,"XML"],"HTML"]. It's slow, but it's easy and robust. Of course if you need the URLs, asHyperlinkthat won't work. $\endgroup$html = Import[url, "HTML"]; StringCases[html, "<p>" ~~ Shortest[str__] ~~ "</p>"]$\endgroup$