9
$\begingroup$

Short version: How can I convert an XMLElement that represents part of a HTML document to plain text?

Long version: The more general problem is extracting information from webpages. We can import the webpage as an XMLObject and extract the relevant part. But this may still be a complex expression of many nested XMLElements (several paragraphs, links, emphasis, etc.), while I'm typically only interested in the text.

Let's take a random example: extracting the text from this article. Using the developer tools of any modern browser it's easy to find out that the relevant part is in a div with id="article-body-blocks". So we do

page = Import[
   "http://www.guardian.co.uk/science/blog/2012/nov/13/science-enforced-humility",
   "XMLObject"];

body = Cases[page, 
   XMLElement["div", {"id" -> "article-body-blocks"}, ___], Infinity];

The body is still a compound expression. Is there a built-in, direct way to extract the text?

My workaround is 

ImportString[ExportString[First@body, "XML"], "HTML"]

but this is a hack (i.e. fragile, likely to break in future versions or with an input I didn't anticipate). Is there anything specifically meant for this purpose?

Improve this question
$\endgroup$
5
  • $\begingroup$ Related question: How to obtain tabular data nested somewhere inside HTML. I had a look at this page earlier today. What i ended up doing was hackish indeed. Would like to see a clean approach. $\endgroup$
    – Daniel Lichtblau
    Nov 14, 2012 at 21:50
  • $\begingroup$ @Daniel I've been using a small palette for that for a long time now. Please search for "tabular" here (and I apologise for all the other terribly outdated stuff on that page). Copying works best in Firefox: it always copies as tab-delimited table, and it's possible to select an arbitrary rectangular region by holding down Control and dragging (not all browsers have this). So, select the relevant part of the table in Firefox, copy, and paste using the Table or TSV buttons in the palette. A bit messy but fast and convenient. $\endgroup$
    – Szabolcs
    Nov 14, 2012 at 22:15
  • $\begingroup$ Took only a few seconds to bring that data into Mathematica with this method (I have the palette installed). $\endgroup$
    – Szabolcs
    Nov 14, 2012 at 22:16
  • 1
    $\begingroup$ Related question: stackoverflow.com/q/8809812/884752 $\endgroup$
    – faysou
    Nov 14, 2012 at 22:26
  • $\begingroup$ @Faysal Aberkane & Szabolcs, Thank you both for your informative comments. $\endgroup$
    – Daniel Lichtblau
    Nov 14, 2012 at 22:30

1 Answer 1

Reset to default
6
$\begingroup$

Here's an easy way:

body //. XMLElement[_, _, t_] :> t // Flatten // StringJoin

Improve this answer
$\endgroup$
2
  • $\begingroup$ This works, but it'd require some extra code to know where to put line breaks. Take for example this piece of HTML, html = "<p>one two <em>three<em> four <p>next paragraph"; ImportString[html, {"HTML", "XMLObject"}]. A <p> creates a line break, so does a <div>. An <em> (or a <span>) should not. Of course I don't need a full HTML renderer, but line breaks/paragraphs and &nbsp;-like things should be handled. $\endgroup$
    – Szabolcs
    Nov 14, 2012 at 20:56
  • 3
    $\begingroup$ @Szabolcs Yes, but it should be trivial to handle. Just add a line break at the end to t in XMLElement["p",_,t_] and so on. I'll update it later when I have some time. Of course, if you want a full blown and robust parser, it'll take a hell lot more work and testing, which is beyond the scope here... $\endgroup$
    – rm -rf ♦
    Nov 14, 2012 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.