Short version: How can I convert an XMLElement that represents part of a HTML document to plain text?

Long version: The more general problem is extracting information from webpages. We can import the webpage as an XMLObject and extract the relevant part. But this may still be a complex expression of many nested XMLElements (several paragraphs, links, emphasis, etc.), while I'm typically only interested in the text.

Let's take a random example: extracting the text from this article. Using the developer tools of any modern browser it's easy to find out that the relevant part is in a div with id="article-body-blocks". So we do

page = Import[

body = Cases[page, 
   XMLElement["div", {"id" -> "article-body-blocks"}, ___], Infinity];

The body is still a compound expression. Is there a built-in, direct way to extract the text?

My workaround is

ImportString[ExportString[First@body, "XML"], "HTML"]

but this is a hack (i.e. fragile, likely to break in future versions or with an input I didn't anticipate). Is there anything specifically meant for this purpose?

| improve this question | | | | |
  • $\begingroup$ Related question: How to obtain tabular data nested somewhere inside HTML. I had a look at this page earlier today. What i ended up doing was hackish indeed. Would like to see a clean approach. $\endgroup$ – Daniel Lichtblau Nov 14 '12 at 21:50
  • $\begingroup$ @Daniel I've been using a small palette for that for a long time now. Please search for "tabular" here (and I apologise for all the other terribly outdated stuff on that page). Copying works best in Firefox: it always copies as tab-delimited table, and it's possible to select an arbitrary rectangular region by holding down Control and dragging (not all browsers have this). So, select the relevant part of the table in Firefox, copy, and paste using the Table or TSV buttons in the palette. A bit messy but fast and convenient. $\endgroup$ – Szabolcs Nov 14 '12 at 22:15
  • $\begingroup$ Took only a few seconds to bring that data into Mathematica with this method (I have the palette installed). $\endgroup$ – Szabolcs Nov 14 '12 at 22:16
  • 1
    $\begingroup$ Related question: stackoverflow.com/q/8809812/884752 $\endgroup$ – faysou Nov 14 '12 at 22:26
  • $\begingroup$ @Faysal Aberkane & Szabolcs, Thank you both for your informative comments. $\endgroup$ – Daniel Lichtblau Nov 14 '12 at 22:30

Here's an easy way:

body //. XMLElement[_, _, t_] :> t // Flatten // StringJoin

| improve this answer | | | | |
  • $\begingroup$ This works, but it'd require some extra code to know where to put line breaks. Take for example this piece of HTML, html = "<p>one two <em>three<em> four <p>next paragraph"; ImportString[html, {"HTML", "XMLObject"}]. A <p> creates a line break, so does a <div>. An <em> (or a <span>) should not. Of course I don't need a full HTML renderer, but line breaks/paragraphs and &nbsp;-like things should be handled. $\endgroup$ – Szabolcs Nov 14 '12 at 20:56
  • 3
    $\begingroup$ @Szabolcs Yes, but it should be trivial to handle. Just add a line break at the end to t in XMLElement["p",_,t_] and so on. I'll update it later when I have some time. Of course, if you want a full blown and robust parser, it'll take a hell lot more work and testing, which is beyond the scope here... $\endgroup$ – rm -rf Nov 14 '12 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.