4
$\begingroup$

Consider the following example of text extraction.

lst = {
"foo bar <span>feyenoord - ADO</span>",
"foo bar foo <span>Chelsea - Leicester</span>",
"foo bar <span id=\"puin\">feyenoord - ADO Den Haag</span>"};

The goal is to take the text in the span element.

Map[StringTake[StringCases[#, ">" ~~ __ ~~ "<"], {2, -2}] &, lst]

The result is as expected.

{
 {"feyenoord - ADO"}, 
 {"Chelsea - Leicester"},
 {"feyenoord - ADO Den Haag"}
}

Then I received a somewhat different input text. Note the ">" following "foo" in the third element.

lst1 = {
"foo bar <span>feyenoord - ADO</span>",
"foo bar foo <span>Chelsea - Leicester</span>",
"foo> bar <span id=\"puin\">feyenoord - ADO Den Haag</span>"};

The goal is still to take the text in the span element.

Map[StringTake[StringCases[#, ">" ~~ __ ~~ "<"], {2, -2}] &, lst1]

But now, the result is not as expected.

{
 {"feyenoord - ADO"}, 
 {"Chelsea - Leicester"}, 
 {" bar <span id=\"puin\">feyenoord - ADO Den Haag"}
}

Whatever texts are received, i.e.:

 "foo bar <<< foo <span id="id" attr="name">payload</span> foo>span>>>"

it is always "payload", the innertext of the span element that needs to be extracted.

I thought about removing id="__" but other attributes may come in at other times.

So I am basically looking ( and not wanting to reinvent the wheel ) for the regular expression or Mathematica string expression to extract the innertext of an HTML element ( span in this case ).

Note that the strings are not valid HTML.

$\endgroup$
  • 1
    $\begingroup$ Using regexp to parse HTML is not a great idea in general. If you have a specific pattern, involving span or other HTML tags, you can probably write a more specialized regexp for it, but it certainly has to be more specific than the one you are using. This has nothing to do with Mathematica per se - you would have to construct it for whatever language you'd use for extraction. Or you could use one of the proper HTML parsers, and then postprocess the result symbolically. $\endgroup$ – Leonid Shifrin Feb 19 '17 at 18:32
  • $\begingroup$ Actually I am in the post-post-processing, and it is not HTML. $\endgroup$ – nilo de roock Feb 19 '17 at 18:57
  • 3
    $\begingroup$ Is this something useful: StringCases["foo> bar <span id=\"puin\">feyenoord - ADO Den Haag</span>", "<span" ~~ Shortest[__] ~~ ">" ~~ x : Shortest[__] ~~ "</span" :> x] ? $\endgroup$ – Fred Simons Feb 19 '17 at 19:03
  • $\begingroup$ @LeonidShifrin The Mathematica Tutorial for Advanced String Patterns says: "String patterns are useful for taking raw HTML and extracting information from it" $\endgroup$ – nilo de roock Feb 20 '17 at 11:14
  • $\begingroup$ String patterns are surely useful for this, I don't dispute that. They just stop working well when the complexity of a piece you want to parse increases. In your particular case, something like what @FredSimons suggested should work pretty well, I guess. $\endgroup$ – Leonid Shifrin Feb 20 '17 at 11:17
3
$\begingroup$

jSoupLink can be used if the markup looks like HTML. So even though you contend that your input is not HTML, the following still works:

<< jSoupLink`
getSpan[str_] := First[ImportString[str, "HTMLDOM"]["Select", "span"]]["OwnText"]

getSpan["foo bar <span id=\"puin\">feyenoord - ADO Den Haag</span>"]

"feyenoord - ADO Den Haag"

getSpan["foo> bar <span id=\"puin\">feyenoord - ADO Den Haag</span>"]

"feyenoord - ADO Den Haag"

getSpan["foo bar <<< foo <span id=\"id\" attr=\"name\">payload</span> foo>span>>>"]

"payload"

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I am using a stack of Conductor / Selenium / WebDriver / PhantomJS at the moment to query websites using CSS selectors. - I have looked at jSoup in the design phase but I'll definitely have another look. - Your reply is highly appreciated, thank you. $\endgroup$ – nilo de roock Feb 20 '17 at 8:58
  • $\begingroup$ @C.E. After PacletInstalling the latest build, the << doesn't work for me... $\endgroup$ – M.R. Apr 28 '19 at 23:05
  • $\begingroup$ @M.R. Thank you for telling me, I will look into it when I have the time. $\endgroup$ – C. E. Apr 28 '19 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.