What is the Mathematica string expression to extract the innertext of an HTML element?

Question

Consider the following example of text extraction.

lst = {
"foo bar <span>feyenoord - ADO</span>",
"foo bar foo <span>Chelsea - Leicester</span>",
"foo bar <span id=\"puin\">feyenoord - ADO Den Haag</span>"};

The goal is to take the text in the span element.

Map[StringTake[StringCases[#, ">" ~~ __ ~~ "<"], {2, -2}] &, lst]

The result is as expected.

{
 {"feyenoord - ADO"}, 
 {"Chelsea - Leicester"},
 {"feyenoord - ADO Den Haag"}
}

Then I received a somewhat different input text. Note the ">" following "foo" in the third element.

lst1 = {
"foo bar <span>feyenoord - ADO</span>",
"foo bar foo <span>Chelsea - Leicester</span>",
"foo> bar <span id=\"puin\">feyenoord - ADO Den Haag</span>"};

The goal is still to take the text in the span element.

Map[StringTake[StringCases[#, ">" ~~ __ ~~ "<"], {2, -2}] &, lst1]

But now, the result is not as expected.

{
 {"feyenoord - ADO"}, 
 {"Chelsea - Leicester"}, 
 {" bar <span id=\"puin\">feyenoord - ADO Den Haag"}
}

Whatever texts are received, i.e.:

 "foo bar <<< foo <span id="id" attr="name">payload</span> foo>span>>>"

it is always "payload", the innertext of the span element that needs to be extracted.

I thought about removing id="__" but other attributes may come in at other times.

So I am basically looking ( and not wanting to reinvent the wheel ) for the regular expression or Mathematica string expression to extract the innertext of an HTML element ( span in this case ).

Note that the strings are not valid HTML.

Answer 1

jSoupLink can be used if the markup looks like HTML. So even though you contend that your input is not HTML, the following still works:

<< jSoupLink`
getSpan[str_] := First[ImportString[str, "HTMLDOM"]["Select", "span"]]["OwnText"]

getSpan["foo bar <span id=\"puin\">feyenoord - ADO Den Haag</span>"]

"feyenoord - ADO Den Haag"

getSpan["foo> bar <span id=\"puin\">feyenoord - ADO Den Haag</span>"]

"feyenoord - ADO Den Haag"

getSpan["foo bar <<< foo <span id=\"id\" attr=\"name\">payload</span> foo>span>>>"]

"payload"

Answer 2

Clear["Global`*"]

lst = {"foo bar <span>feyenoord - ADO</span>", 
   "foo bar foo <span>Chelsea - Leicester</span>", 

   "foo bar <span id=\"puin\">feyenoord - ADO Den Haag</span>"};

h1 = "foo bar <<< foo <span id=\"id\" attr=\"name\">payload</span>foo>span>>>"

---

Start with the simpler case:

StringCases[#, "<span>" ~~ s : __ ~~ "</span>" -> s] & /@ lst

{{"feyenoord - ADO"}, {"Chelsea - Leicester"}, {}}

Let's keep building the pattern up to handle the missed case where the span tag has more attributes.

StringCases[#, 
   "<span" ~~ ___ ~~ ">" ~~ s : ___ ~~ "</span>" -> s] & /@ lst

{{"feyenoord - ADO"}, {"Chelsea - Leicester"}, {"feyenoord - ADO Den
Haag"}}

For the other example, the same solution extracts innertext:

StringCases[#, 
   "<span" ~~ ___ ~~ ">" ~~ s : ___ ~~ "</span>" -> s] &@h1

{"payload"}

---

EDIT-1

With a text file containing many span tags, the above requires a tweak using Shortest, otherwise the last span block is grabbed. For example:

h2 = "foo bar <span>first inner - ABC</span>foo bar foo <span>Second \
one</span>foo bar <span id=\"ghi\">Bit Next one</span>foo bar \
<span>Not the Last one</span>foo bar <<< foo <span id=\"id\" \
attr=\"name\">payload</span> foo>span>>>"


StringCases[#,  
   "<span" ~~ Shortest[___] ~~ ">" ~~ s : Shortest[___] ~~ 
     "</span>" :> s] &@h2

{"first inner - ABC", "Second one", "Bit Next one", "Not the Last
one", "payload"}