7
$\begingroup$

I'm trying to scrape this Wikipedia page ('Deaths in 2019- July', I'm writing this on the 14th.). I am failing miserably and I'm starting to think that it is due to some misconception on my part on the use of patterns.

Namely, the use of __ vs .. and PatternSequence.

Please help.

I use Import[<URL>, "XML"] to import the page. I use pattn0 (see below) to identify the positions pos0 = Position[<XML>, pattn0] and pos1 = MapAt[# + 1 &, pos0, {All, -1}].

pos and pos1 are used to identify the positions of the desired entries.

I'm using src = Apply[Part[<XML>, ##] &, Riffle[pos0, pos1], 1] in all pattern matching attempts as the source over which I try to extract (or match against) parts I want.

(I provide a link to the src used, at the end of the question)

Snippet of desired entries

I will report below a random pair of entries I am interested in:

XMLElement["h3", {}, {XMLElement["span", {"class" -> "mw-headline", "id" -> "14"}, {"14"}]}],

XMLElement["ul",{},{
  XMLElement["li",{},{
    XMLElement["a",{"href"->"/wiki/Hussain_Muhammad_Ershad","title"->"Hussain Muhammad Ershad"},{"Hussain Muhammad Ershad"}],", 89, Bangladeshi military officer and politician,",
    XMLElement["a",{"href"->"/wiki/Chief_of_Army_Staff_(Bangladesh)","title"->"Chief of Army Staff (Bangladesh)"},{"Chief of Army Staff"}],"(1978\[Dash]1986) and",
    XMLElement["a",{"href"->"/wiki/President_of_Bangladesh","title"->"President of Bangladesh"},{"President"}],"(1983\[Dash]1990), myelodysplastic syndrome.",
    XMLElement["sup",{"id"->"cite_ref-1","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-1"},{"[1]"}]}]
   }
  ],
XMLElement["li",{},{
    XMLElement["a",{"href"->"/wiki/Nereo_Laroni","title"->"Nereo Laroni"},{"Nereo Laroni"}],", 76, Italian politician,",
    XMLElement["a",{"href"->"/wiki/Mayor_of_Venice","title"->"Mayor of Venice"},{"Mayor of Venice"}],"(1985\[Dash]1987) and",
    XMLElement["a",{"href"->"/wiki/Member_of_the_European_Parliament","title"->"Member of the European Parliament"},{"MEP"}],"(1989\[Dash]1994), complications from heart surgery.",
    XMLElement["sup",{"id"->"cite_ref-2","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-2"},{"[2]"}]}]
   }
  ],
XMLElement["li",{},{
    XMLElement["a",{"href"->"/wiki/Lavenia_Padarath","title"->"Lavenia Padarath"},{"Lavenia Padarath"}],", 74, Fijian politician,",
    XMLElement["a",{"href"->"/wiki/Parliament_of_Fiji","title"->"Parliament of Fiji"},{"MP"}],"(1999\[Dash]2001, 2006) and President of the",
    XMLElement["a",{"href"->"/wiki/Fiji_Labour_Party","title"->"Fiji Labour Party"},{"Labour Party"}],"(since 2015).",
    XMLElement["sup",{"id"->"cite_ref-3","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-3"},{"[3]"}]}]
   }
  ]
 }
]

Patterns I-am trying to-use

I am using the following patterns to match the entries I am interested in:

pattn0 = XMLElement["h3", _, {
   XMLElement["span", {"class" -> "mw-headline", "id" -> id_}, {idnm_}]
  }
 ]

(pattn0 matches against the day numbers reported in the header for each day)

and

pattn1 = XMLElement["a", {"href" -> href_, Longest[othr___], "title" -> tit_, last___}, {nm_}];
pattn2 = PatternSequence[pattn1, Longest[cmnt___]];
  • pattn2 is used to match against entries like XMLElement["a", {__}, {_}] which are the data entries I'm (mostly) interested in.
  • In pattn2 I am using Longest[othr___] because there are some entries where, between "href" and "title", there interjects some other minor entry (some XMLElement[__]).
  • Also, last___ is there to match against any remaining entries I might have missed.
  • Lastly, Longest[cmnts___] is used to match against entries (usually, as far as I can tell, a single one) after XMLElement["a", __].

pattn3 and pattn4 match against reference entries:

pattn3 = XMLElement["a", {"href" -> href2_}, {refn_}];
pattn4 = XMLElement["sup", {"id" -> id_, "class" -> "reference", {pattn3}];

(pattn4 is used to match against XMLElement["sup", {__, {__}] which are the entries related to references and their links)

pattn5 combines all the patterns into one (unfortunately non-performing) pattern:

pattn5 = PatternSequence[pattn0, x : XMLElement["ul", _, {XMLElement["li", _, {pattn2 .., pattn4}] ..}]];

Things that work

  • Retrieving day numbers Evaluating Cases[src, pattn0 :> idnm] works! and returns the day numbers

    {14,13,12,11,10,9,8,7,6,5,4,3,2,1}
    

    as expected.

  • Retrieving footnotes/references

    Both pattn3 and pattn4 work. Here, evaluating Cases[src, pattn4, Infinity] works (first five results):

    { 
        XMLElement[sup,{id->cite_ref-1,class->reference},{XMLElement[a,{href->#cite_note-1},{[1]}]}],
        XMLElement[sup,{id->cite_ref-2,class->reference},{XMLElement[a,{href->#cite_note-2},{[2]}]}],
        XMLElement[sup,{id->cite_ref-3,class->reference},{XMLElement[a,{href->#cite_note-3},{[3]}]}],
        XMLElement[sup,{id->cite_ref-4,class->reference},{XMLElement[a,{href->#cite_note-4},{[4]}]}],
        XMLElement[sup,{id->cite_ref-5,class->reference},{XMLElement[a,{href->#cite_note-5},{[5]}]}],...}
    

Things that partially work

  • pattn2

    In using pattn2 I get partially matched results; namely, the part related to Longest[cmnt___] does not get matched, ever!. Essentially what works seems to be the part related to pattn1.

    Cases[src, pattn2, Infinity]
    

    evaluates to the partially correct result (only the first five entries shown here to avoid clutter)

    {
    XMLElement[a,{href->/wiki/Hussain_Muhammad_Ershad,title->Hussain Muhammad Ershad},{Hussain Muhammad Ershad}],
    XMLElement[a,{href->/wiki/Chief_of_Army_Staff_(Bangladesh),title->Chief of Army Staff (Bangladesh)},{Chief of Army Staff}],
    XMLElement[a,{href->/wiki/President_of_Bangladesh,title->President of Bangladesh},{President}],
    XMLElement[a,{href->/wiki/Nereo_Laroni,title->Nereo Laroni},{Nereo Laroni}],
    XMLElement[a,{href->/wiki/Mayor_of_Venice,title->Mayor of Venice},{Mayor of Venice}], ...}
    

Things that should have(?) worked but don't!

  • Combined pattern pattn5

    Evaluating Cases[src, pattn5, Infinity] evaluates to {}. Nothing gets matched!

Suggestions, Nudges, Tips on how to get a decent match

Are all welcome.

Thanks!


Appendix

I provide the src used throughout the question (pastebin). Run Uncompress on the string to retrieve the XML elements.

$\endgroup$
1
  • $\begingroup$ related to pastebin: 1. click on the link 2. when you are redirected, copy the text (place your cursor in the body of the text, select all the text and copy selection) 3. in a notebook type Uncompress[" then paste in the contents of the clipboard and continue typing without moving the cursor "] 4. evaluate $\endgroup$
    – user42582
    Jul 15, 2019 at 22:03

1 Answer 1

2
$\begingroup$

This answer will be about the section entitled

Things that partially work

I got the input from the web (I was unable to use pastebin successfully).

xml = Import["https://en.wikipedia.org/wiki/Deaths_in_2019#July", "XML"];

pattn0 = XMLElement[
  "h3", _, {XMLElement["span",
     {"class" -> "mw-headline", "id" -> id_}, {idnm_}]}];

pos0 = Position[xml, pattn0];

pos1 = MapAt[# + 1 &, pos0, {All, -1}];

src = Apply[Part[xml, ##] &, Riffle[pos0, pos1], 1];

src changes with time so mine may not be identical to the data in pastebin.

Below are some example parts of src that serve as your data entries.

Column[src[[8, 3]][[1 ;; 2]], Dividers -> All]

Mathematica graphics

Note carefully that it looks like you have multiple elements following the XMLElement. However if you check the Head you will find that it is a single string.

src[[8, 3, 3, 3, 2]]
(* ", 91, Vietnamese mathematician." *)

Head[src[[8, 3, 3, 3, 2]]]
(* String *)

Patterns that match XML

You had set pattn1 to

pattn1 = XMLElement["a", {"href" -> href_, Longest[othr___],
  "title" -> tit_, last___}, {nm_}]

When applied to src I got 576 entries

pattn1List = Cases[src, pattn1, Infinity];

Below is a sample of the result

Column[pattn1List[[1 ;; 3]], Dividers -> All]

Mathematica graphics

Patterns that match XML and comments

You had wanted to match both the XML and comments that might follow. You tried:

pattn2 = PatternSequence[pattn1, Longest[cmnt___]]

But when it is applied it to src you didn't get any comments. In fact, it was identical to only using pattn1.

pattn2List = Cases[src, pattn2, Infinity];

SameQ[pattn1List, pattn2List]
(* True *)

PatternSequence

The documentation indicates that PatternSequence should not be used as a standalone pattern. pattn2 violates that principle.

I will show two attempts that produce a result, however neither is the desired result.

Failure 1

pattn2 = {___, PatternSequence[pattn1, cmnt_String], ___}
pattn2List = Cases[src, pattn2, Infinity];

This produces a result but rather than individual entries it returns the entire list with interspersed XMLElements and comments. Below is an example.

pattn2List[[4]]

  (* {XMLElement[
  "a", {"href" -> "/wiki/Giuseppe_Merlo", 
   "title" -> "Giuseppe Merlo"}, {"Giuseppe Merlo"}],
   ", 91, Italian tennis player.", 
 XMLElement[
  "sup", {"id" -> "cite_ref-4", 
   "class" -> "reference"}, {XMLElement[
    "a", {"href" -> "#cite_note-4"}, {"[4]"}]}]} *)

Failure 2

In order to extract both the XMLElement and its associated comment I tried.

pattn2 = {___, group : PatternSequence[pattn1, cmnt_String], ___}
pattn2List = Cases[src, pattn2 :> group, Infinity];

This succeeded in the sense that an XMLElement and its associated comment were returned. However only the first element from a multi-element list was returned, the others were missing.

Below is a sample

Column[pattn2List[[57 ;; 62]], Dividers -> All]

Mathematica graphics

XMElement and comment

Hopefully this answer has been helpful in achieving a better understanding of PatternSequence.

I could stop here but would also like to present a method for retrieving XMLElement that match pattn1 along its associated comment. Note, not all XMLElements that match pattn1 are followed by comments.

The strategy is to make a list of positions (posPattn1)that match pattn1.

Also make a list of positions (posCmnt) that are strings.

Iterate over posPattn1, increment the position by one and determine if it is a member of posCmnt (i.e., a string).

If it is, output both, otherwise only output the XMLElement.

posPattn1 = Position[src, pattn1];
posCmnt = Position[src, _String];

pattn2List  = Map[
   Module[
     {
      pos1 = #,
      pos2 = #,
      pat1,
      str2
      },
     pat1 = Extract[src, pos1];
     pos2[[-1]] = pos2[[-1]] + 1;
     str2 = If[
       MemberQ[posCmnt, pos2],
       Extract[src, pos2],
       Nothing
       ];
     {pat1, str2}
     ] &,
   posPattn1
   ];
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.